From: Dipl. Ing. Sergey Brester <serg.brester@sebres.de>
Date: Thu, 15 Oct 2020 11:50:30 GMT
Subject: Re: git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump
Message-ID: <72a4d4d8dff95351122bd192976dd6b1@sebres.de>
In-Reply-To: <20201015012636.GA387901@coredump.intra.peff.net>

Hi Jeff,

well, I don't know how you were trying to reproduce it.

My first attempt with a git-repository (cloned from 
https://github.com/git/git.git) showed that immediately to me.
Here you go (I used git bash here):

```
# clone or copy git repository (we'll use it for export and import):
git clone https://github.com/git/git.git
cd git

# make 1st fast-export in order to generate marks (don't need the dump, 
only marks are needed):
git fast-export --reencode=yes --export-marks=.tmp.exp-1.tmp 
e83c5163316f89bfbde7d9ab23ca2e25604af290..dc04167d378fb29d30e1647ff6ff51dd182bc9a3 
 > /dev/null

# make 2nd fast-export in order to generate partial export dump (file 
".tmp.dump" will be ca. 87MB):
git fast-export --reencode=yes --import-marks=.tmp.exp-1.tmp 
--export-marks=.tmp.exp-2.tmp 
61addb841f2a6d74a1737a01e03df1f773e04944..master > .tmp.dump

# now try to import this dump, using first marks as import marks (we 
have all revs in git-repo):
git fast-import --import-marks=.tmp.exp-1.tmp 
--export-marks=.tmp.imp.tmp < .tmp.dump

```

And see how git-fast-import eating your whole memory and enjoy the crash 
:)

```
fatal: Out of memory, malloc failed (tried to allocate 2097152 bytes)
fast-import: dumping crash report to .git/fast_import_crash_6684
```

Regards,
Sergey

15.10.2020 03:26, Jeff King wrote:

> On Wed, Oct 14, 2020 at 11:22:03AM +0200, Dipl. Ing. Sergey Brester 
> wrote:
> 
>> I did not see any issues with (even much larger) imports, before I 
>> upgraded git to 2.28.0 (from 2.25.1, I guess).
> 
> I wasn't able to reproduce, but it sounds like there's something 
> special
> about this repo (since the dump is not that big, and you said much
> larger dumps have succeeded).
> 
> Is it possible to make either the fossil repo, or the fast-import input
> file available?
> 
> If not, since you said it works with 2.14.4, is it possible to bisect 
> in
> Git to see when it stopped working? You'll need to be able to build Git
> from source.
> 
> -Peff