git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump

From
DBDipl. Ing. Sergey Brester <serg.brester@sebres.de>
Date
Oct 15, 2020, 11:50 UTC
Message-ID
<72a4d4d8dff95351122bd192976dd6b1@sebres.de>
In-Reply-To
<20201015012636.GA387901@coredump.intra.peff.net>
Hi Jeff,
well, I don't know how you were trying to reproduce it.

My first attempt with a git-repository (cloned from https://github.com/git/git.git) showed that immediately to me. Here you go (I used git bash here):

``` # clone or copy git repository (we'll use it for export and import): git clone https://github.com/git/git.git cd git

# make 1st fast-export in order to generate marks (don't need the dump, 
only marks are needed):
git fast-export --reencode=yes --export-marks=.tmp.exp-1.tmp 
e83c5163316f89bfbde7d9ab23ca2e25604af290..dc04167d378fb29d30e1647ff6ff51dd182bc9a3 
 > /dev/null

# make 2nd fast-export in order to generate partial export dump (file ".tmp.dump" will be ca. 87MB): git fast-export --reencode=yes --import-marks=.tmp.exp-1.tmp --export-marks=.tmp.exp-2.tmp 61addb841f2a6d74a1737a01e03df1f773e04944..master > .tmp.dump

# now try to import this dump, using first marks as import marks (we have all revs in git-repo): git fast-import --import-marks=.tmp.exp-1.tmp --export-marks=.tmp.imp.tmp < .tmp.dump

```

And see how git-fast-import eating your whole memory and enjoy the crash :)

``` fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes) fast-import: dumping crash report to .git/fast_import_crash_6684 ```

Regards, Sergey

15.10.2020 03:26, Jeff King wrote:
Show 20 quoted lines
> On Wed, Oct 14, 2020 at 11:22:03AM +0200, Dipl. Ing. Sergey Brester 
> wrote:
> 
>> I did not see any issues with (even much larger) imports, before I 
>> upgraded git to 2.28.0 (from 2.25.1, I guess).
> 
> I wasn't able to reproduce, but it sounds like there's something 
> special
> about this repo (since the dump is not that big, and you said much
> larger dumps have succeeded).
> 
> Is it possible to make either the fossil repo, or the fast-import input
> file available?
> 
> If not, since you said it works with 2.14.4, is it possible to bisect 
> in
> Git to see when it stopped working? You'll need to be able to build Git
> from source.
> 
> -Peff
Previous: Jeff KingNext: Jeff King
Message 3 of 20 in “git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump”
  1. Dipl. Ing. Sergey BresterOct 14, 2020
  2. Jeff KingOct 15, 2020
  3. Dipl. Ing. Sergey BresterOct 15, 2020
  4. fast-import: fix over-allocation of marks storageJeff King, Oct 15, 2020
  5. Junio C HamanoOct 15, 2020
  6. Junio C HamanoOct 15, 2020
  7. Dipl. Ing. Sergey BresterOct 15, 2020
  8. Junio C HamanoOct 15, 2020
  9. Jeff KingOct 15, 2020
  10. Junio C HamanoOct 15, 2020
  11. brian m. carlsonOct 16, 2020
  12. Jeff KingOct 15, 2020
  13. Jeff KingOct 15, 2020
  14. brian m. carlsonOct 16, 2020
  15. Jeff KingOct 16, 2020
  16. Dipl. Ing. Sergey BresterOct 15, 2020
  17. Junio C HamanoOct 15, 2020
  18. René ScharfeOct 15, 2020
  19. René ScharfeOct 15, 2020
  20. Dipl. Ing. Sergey BresterOct 15, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.