Re: git-fast-import out of memory

From: brian m. carlson <sandals@crustytoothpaste.net>
Date: Jun 5, 2020, 22:43 UTC
Message-ID: <20200605224342.GF6569@camp.crustytoothpaste.net>
In-Reply-To: <c53bb69b-682d-3b47-4ed0-5f4559e69e37@gmx.com>

On 2020-06-05 at 05:15:04, Billes Tibor wrote:

Show 5 quoted lines

> Can you help me fix this issue? I hope the information I gathered is
> enough to
> help you find the cause of this behavior. I'd be happy to provide more
> information if needed or test patches.  Unfortunately the source code I was
> fetching is proprietary, I cannot post it.

Hey, thanks for this report. I can confirm this with git.git and the following commands:

  git fast-export --all --reencode=yes --signed-tags=verbatim --tag-of-filtered-object=drop >$TMP/git.fe
  cd $TMP && rm -fr test-repo && git init test-repo &&
    (cd test-repo && /usr/bin/time git fast-import --export-marks=$TMP/git.marks < $TMP/git.fe &&
    /usr/bin/time git fast-import --import-marks=$TMP/git.marks < $TMP/git.fe)

The latter fast-import command gets killed with SIGKILL due to using 16 GiB of memory, which I would agree is unreasonable.

It looks like with Junio's patch at <xmqqeeqto4x1.fsf@gitster.c.googlers.com>, the memory usage is down to 1.4 GiB, which considering that a pack must be written, is much more reasonable.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204