Re: Surprising use of memory and time when repacking mozilla's gecko repository

From: Mike Hommey <mh@glandium.org>
Date: Jul 5, 2019, 04:45 UTC
Message-ID: <20190705044530.mehpvei5lysy3czl@glandium.org>
In-Reply-To: <20190704100530.smn4rpiekwtfylhz@glandium.org>

On Thu, Jul 04, 2019 at 07:05:30PM +0900, Mike Hommey wrote:

Show 6 quoted lines

> My guess is all those stalls are happening when processing the files I
> already had problems with in the past[3], except there are more of them
> now (thankfully, they were removed, so there won't be more, but that
> doesn't make the existing ones go away).
>
> 3. https://public-inbox.org/git/20180703223823.qedmoy2imp4dcvkp@glandium.org/T/

I've more or less confirmed that's the cause of the long stalls during the compression phase. It can be reproduced to some extent with:

$ git clone https://github.com/mozilla/gecko $ cd gecko $ git rev-list --all -- testing/web-platform/meta/MANIFEST.json | xargs -I '{}' git ls-tree '{}' testing/web-platform/meta/MANIFEST.json | sort -u | awk '{print $3}' | git cat-file --batch-check | sort -n -k 3 | awk '{print $1}' | git pack-objects --window=250 --depth=50 --no-reuse-delta my_pack

There might be some other file than testing/web-platform/meta/MANIFEST.json involved, because this uses "only" 40GB RAM, but that file is the one I know of.

This however doesn't reproduce the "writing takes forever" part of the problem.

Mike