git/list[1] front-page[2] threads[3] people[4] search[5] about
 

git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump

From
DBDipl. Ing. Sergey Brester <serg.brester@sebres.de>
Date
Oct 14, 2020, 09:22 UTC
Message-ID
<1eeb49305cb7c712e141dcae2c434d96@sebres.de>
Steps to reproduce the issue:
  1. export from fossil and import the dump to the git:
```
fossil export --git --import-marks .git/.fossil2git-fssl --export-marks 
.git/.fossil2git-fssl.tmp ^
  | git fast-import --import-marks=.git/.fossil2git-git 
--export-marks=.git/.fossil2git-git.tmp
```
  during the import git-fast-import.exe is growing on memory (more than I 
have physically, e.g. noticed over 20GB),
  SO FINALLY IT IS CRASHING WITH:
```
fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes)
fast-import: dumping crash report to .git/fast_import_crash_1800
```
  the crash report contains:
```
fast-import crash report:
  fast-import process: 1800
  parent process : 1
  at 2020-10-13 18:55:19 +0000
fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes)

Most Recent Commands Before Crash ---------------------------------

Active Branch LRU
-----------------
  active_branches = 0 cur, 5 max
  pos clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Inactive Branches -----------------

Marks
-----
  exported to .git/.fossil2git-git.tmp

------------------- END OF CRASH REPORT ```

  2. if I do the export firstly (redirect to file), fossil creates 22MB 
large export dump-file,
  if I import it via git hereafter, it crashes in the same way.
```
fossil export --git --import-marks .git/.fossil2git-fssl --export-marks 
.git/.fossil2git-fssl.tmp > tmp-dump-out-of-mem.txt

dir tmp-dump-out-of-mem.txt 13.10.2020 20:37 22.916.280 tmp-dump-out-of-mem.txt

type tmp-dump-out-of-mem.txt | git fast-import --import-marks=.git/.fossil2git-git --export-marks=.git/.fossil2git-git.tmp

fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes) fast-import: dumping crash report to .git/fast_import_crash_1800 ```

I did not see any issues with (even much larger) imports, before I upgraded git to 2.28.0 (from 2.25.1, I guess).

[System Info] git version: git version 2.28.0.windows.1 cpu: x86_64 built from commit: 77982caf269b7ee713a76da2bcf260c34d3bf7a7 sizeof-long: 4 sizeof-size_t: 8 shell-path: /bin/sh uname: Windows 10.0 18363 compiler info: gnuc: 10.2 libc info: no libc information available $SHELL (typically, interactive shell): <unset>

An attempt to repeat this with 2.27.0 (portable) changes nothing (crashed with the same issue).

BUT TRYING THAT WITH 2.14.4 (MINGIT-2.14.4.WINDOWS.7-64-BIT) WORKS WELL:
```
C:SoftDevGit-2.14mingw64bingit-fast-import.exe statistics:
---------------------------------------------------------------------
Alloc'd objects: 130000
Total objects: 591 ( 80 duplicates )
  blobs : 224 ( 0 duplicates 151 deltas of 224 attempts)
  trees : 260 ( 80 duplicates 191 deltas of 260 attempts)
  commits: 107 ( 0 duplicates 0 deltas of 0 attempts)
  tags : 0 ( 0 duplicates 0 deltas of 0 attempts)
Total branches: 201 ( 46 loads )
  marks: 1048576 ( 129437 unique )
  atoms: 1503
Memory total: 10439 KiB
  pools: 4346 KiB
  objects: 6093 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 65536
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit = 35184372088832
pack_report: pack_used_ctr = 131659
pack_report: pack_mmap_calls = 119
pack_report: pack_open_windows = 66 / 66
pack_report: pack_mapped = 488903419 / 488903419
---------------------------------------------------------------------
```

The crash with newer versions is pretty well reproducible, so I have frozen this state to be able to test it later (or check whether it gets fixed). Don't hesitate to ping me if you need some data or tests.

-- 
Regards,
Sergey Brester
Next: Jeff King
Message 1 of 20 in “git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump”
  1. Dipl. Ing. Sergey BresterOct 14, 2020
  2. Jeff KingOct 15, 2020
  3. Dipl. Ing. Sergey BresterOct 15, 2020
  4. fast-import: fix over-allocation of marks storageJeff King, Oct 15, 2020
  5. Junio C HamanoOct 15, 2020
  6. Junio C HamanoOct 15, 2020
  7. Dipl. Ing. Sergey BresterOct 15, 2020
  8. Junio C HamanoOct 15, 2020
  9. Jeff KingOct 15, 2020
  10. Junio C HamanoOct 15, 2020
  11. brian m. carlsonOct 16, 2020
  12. Jeff KingOct 15, 2020
  13. Jeff KingOct 15, 2020
  14. brian m. carlsonOct 16, 2020
  15. Jeff KingOct 16, 2020
  16. Dipl. Ing. Sergey BresterOct 15, 2020
  17. Junio C HamanoOct 15, 2020
  18. René ScharfeOct 15, 2020
  19. René ScharfeOct 15, 2020
  20. Dipl. Ing. Sergey BresterOct 15, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.