Re: git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump

From: Dipl. Ing. Sergey Brester <serg.brester@sebres.de>
Date: Oct 15, 2020, 16:19 UTC
Message-ID: <181f35ca5dff05d8d2288f020354a860@sebres.de>
In-Reply-To: <79cbeb4c-5840-d5b7-a2b9-d72cf47968df@web.de>

15.10.2020 17:52, René Scharfe wrote:

Show 7 quoted lines

> In case someone else is wondering about the meaning of those hashes,
> here are their reference strings:
> e83c516331 (Initial revision of "git", the information manager from 
> hell, 2005-04-07)
> dc04167d37 (Fourth batch, 2020-08-04)
> 61addb841f (Merge branch 'jk/strvec' into next, 2020-08-04)
>

These are just revisions of master branch, and the process was splittet into two parts to generate a large marks file with a small partial dump, that can be imported over first marks. So quasi to simulate normal situation (large repo, small partial export dump), for which import/export with marks is basically used. It doesn't matter here, just helpful to export/import single branch only (to bypass signed tags, avoid installing gpg keys, etc).

> So you use the marks generated by the first export to import the second
> export.

It also doesn't matter, because you could import whole first dump (several gigabytes, I guess) in order to generate new import marks... Which will then be the same after all :) just because it is the same repository used for export (and all the revisions are already there, moreover having the same hash values, let alone internal marks index and the export sequence).

> I wonder if that's relevant to trigger the memory allocation issue.

No, it is not relevant. Also note my initial report where it is affected by real situation (over import marks generated previosly with git fast-import).

> I can reproduce this on Debian -- but don't get a crash report, just a
> notice from the OOM killer. It bisects to ddddf8d7e2 (fast-import:
> permit reading multiple marks files, 2020-02-22).

Sure, OOM manager kills this process (before it is able to crash e. g. create a dump on failed malloc)... If you'll disable OOMK (or try to run it on windows), you'll see the crash and dump is there. :)

Regards, Sergey.