git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump

From
René Scharfe <l.s.r@web.de>
Date
Oct 15, 2020, 15:52 UTC
Message-ID
<79cbeb4c-5840-d5b7-a2b9-d72cf47968df@web.de>
In-Reply-To
<72a4d4d8dff95351122bd192976dd6b1@sebres.de>
Am 15.10.20 um 13:50 schrieb Dipl. Ing. Sergey Brester:
Show 10 quoted lines
> ```
> # clone or copy git repository (we'll use it for export and import):
> git clone https://github.com/git/git.git
> cd git
>
> # make 1st fast-export in order to generate marks (don't need the dump, only marks are needed):
> git fast-export --reencode=yes --export-marks=.tmp.exp-1.tmp e83c5163316f89bfbde7d9ab23ca2e25604af290..dc04167d378fb29d30e1647ff6ff51dd182bc9a3 > /dev/null
>
> # make 2nd fast-export in order to generate partial export dump (file ".tmp.dump" will be ca. 87MB):
> git fast-export --reencode=yes --import-marks=.tmp.exp-1.tmp --export-marks=.tmp.exp-2.tmp 61addb841f2a6d74a1737a01e03df1f773e04944..master > .tmp.dump

In case someone else is wondering about the meaning of those hashes, here are their reference strings:

e83c516331 (Initial revision of "git", the information manager from hell, 2005-04-07) dc04167d37 (Fourth batch, 2020-08-04) 61addb841f (Merge branch 'jk/strvec' into next, 2020-08-04)

> # now try to import this dump, using first marks as import marks (we have all revs in git-repo):
> git fast-import --import-marks=.tmp.exp-1.tmp --export-marks=.tmp.imp.tmp < .tmp.dump
>
> ```

So you use the marks generated by the first export to import the second export. I wonder if that's relevant to trigger the memory allocation issue.

Show 6 quoted lines
> And see how git-fast-import eating your whole memory and enjoy the crash :)
>
> ```
> fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes)
> fast-import: dumping crash report to .git/fast_import_crash_6684
> ```

I can reproduce this on Debian -- but don't get a crash report, just a notice from the OOM killer. It bisects to ddddf8d7e2 (fast-import: permit reading multiple marks files, 2020-02-22).

René
Previous: René ScharfeNext: Dipl. Ing. Sergey Brester
Message 19 of 20 in “git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump”
  1. Dipl. Ing. Sergey BresterOct 14, 2020
  2. Jeff KingOct 15, 2020
  3. Dipl. Ing. Sergey BresterOct 15, 2020
  4. fast-import: fix over-allocation of marks storageJeff King, Oct 15, 2020
  5. Junio C HamanoOct 15, 2020
  6. Junio C HamanoOct 15, 2020
  7. Dipl. Ing. Sergey BresterOct 15, 2020
  8. Junio C HamanoOct 15, 2020
  9. Jeff KingOct 15, 2020
  10. Junio C HamanoOct 15, 2020
  11. brian m. carlsonOct 16, 2020
  12. Jeff KingOct 15, 2020
  13. Jeff KingOct 15, 2020
  14. brian m. carlsonOct 16, 2020
  15. Jeff KingOct 16, 2020
  16. Dipl. Ing. Sergey BresterOct 15, 2020
  17. Junio C HamanoOct 15, 2020
  18. René ScharfeOct 15, 2020
  19. René ScharfeOct 15, 2020
  20. Dipl. Ing. Sergey BresterOct 15, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.