git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump
- From
- Dipl. Ing. Sergey Brester <serg.brester@sebres.de>
- Date
- Oct 14, 2020, 09:22 UTC
- Message-ID
- <1eeb49305cb7c712e141dcae2c434d96@sebres.de>
Steps to reproduce the issue:
1. export from fossil and import the dump to the git: ``` fossil export --git --import-marks .git/.fossil2git-fssl --export-marks .git/.fossil2git-fssl.tmp ^ | git fast-import --import-marks=.git/.fossil2git-git --export-marks=.git/.fossil2git-git.tmp ``` during the import git-fast-import.exe is growing on memory (more than I have physically, e.g. noticed over 20GB), SO FINALLY IT IS CRASHING WITH: ``` fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes) fast-import: dumping crash report to .git/fast_import_crash_1800 ``` the crash report contains: ``` fast-import crash report: fast-import process: 1800 parent process : 1 at 2020-10-13 18:55:19 +0000
fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes)
Most Recent Commands Before Crash ---------------------------------
Active Branch LRU ----------------- active_branches = 0 cur, 5 max
pos clock name ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inactive Branches -----------------
Marks ----- exported to .git/.fossil2git-git.tmp
------------------- END OF CRASH REPORT ```
2. if I do the export firstly (redirect to file), fossil creates 22MB large export dump-file, if I import it via git hereafter, it crashes in the same way. ``` fossil export --git --import-marks .git/.fossil2git-fssl --export-marks .git/.fossil2git-fssl.tmp > tmp-dump-out-of-mem.txt
dir tmp-dump-out-of-mem.txt 13.10.2020 20:37 22.916.280 tmp-dump-out-of-mem.txt
type tmp-dump-out-of-mem.txt | git fast-import --import-marks=.git/.fossil2git-git --export-marks=.git/.fossil2git-git.tmp
fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes) fast-import: dumping crash report to .git/fast_import_crash_1800 ```
I did not see any issues with (even much larger) imports, before I upgraded git to 2.28.0 (from 2.25.1, I guess).
[System Info] git version: git version 2.28.0.windows.1 cpu: x86_64 built from commit: 77982caf269b7ee713a76da2bcf260c34d3bf7a7 sizeof-long: 4 sizeof-size_t: 8 shell-path: /bin/sh uname: Windows 10.0 18363 compiler info: gnuc: 10.2 libc info: no libc information available $SHELL (typically, interactive shell): <unset>
An attempt to repeat this with 2.27.0 (portable) changes nothing (crashed with the same issue).
BUT TRYING THAT WITH 2.14.4 (MINGIT-2.14.4.WINDOWS.7-64-BIT) WORKS WELL: ``` C:SoftDevGit-2.14mingw64bingit-fast-import.exe statistics: --------------------------------------------------------------------- Alloc'd objects: 130000 Total objects: 591 ( 80 duplicates ) blobs : 224 ( 0 duplicates 151 deltas of 224 attempts) trees : 260 ( 80 duplicates 191 deltas of 260 attempts) commits: 107 ( 0 duplicates 0 deltas of 0 attempts) tags : 0 ( 0 duplicates 0 deltas of 0 attempts) Total branches: 201 ( 46 loads ) marks: 1048576 ( 129437 unique ) atoms: 1503 Memory total: 10439 KiB pools: 4346 KiB objects: 6093 KiB --------------------------------------------------------------------- pack_report: getpagesize() = 65536 pack_report: core.packedGitWindowSize = 1073741824 pack_report: core.packedGitLimit = 35184372088832 pack_report: pack_used_ctr = 131659 pack_report: pack_mmap_calls = 119 pack_report: pack_open_windows = 66 / 66 pack_report: pack_mapped = 488903419 / 488903419 --------------------------------------------------------------------- ```
The crash with newer versions is pretty well reproducible, so I have frozen this state to be able to test it later (or check whether it gets fixed). Don't hesitate to ping me if you need some data or tests.
-- Regards, Sergey Brester