Re: Performance issue: initial git clone causes massive repack

From: Jakub Narebski <jnareb@gmail.com>
Date: Apr 7, 2009, 13:37 UTC
Message-ID: <200904071537.04225.jnareb@gmail.com>
In-Reply-To: <alpine.LFD.2.00.0904070903020.6741@xanadu.home>

On Tue, 7 Apr 2009, Nicolas Pitre wrote:

> On Tue, 7 Apr 2009, Jakub Narebski wrote:
>> Björn Steinbrink <B.Steinbrink@gmx.de> writes:

[...]

Show 21 quoted lines

>>> It's not primarily pack-objects, I think. It's the rev-list that's run
>>> by upload-pack.  Running "git rev-list --objects --all" on that repo
>>> eats about 2G RSS, easily killing the system's cache on a small box,
>>> leading to swapping and a painful time reading the packfile contents
>>> afterwards to send them to the client.
>> 
>> Than I think that "packfile caching" GSoC project (which is IIRC
>> "object enumeration caching", or at least includes it) should help
>> here.
> 
> NO!
> 
> Please people stop being so creative with all sort of ways to simply 
> avoid the real issue and focussing on a real fix.  Git has not become 
> what it is today by the accumulation of workarounds and ignorance of 
> fundamental issues.
> 
> Having git-rev-list consume about 2G RSS for the enumeration of 4M 
> objects is simply inacceptable, period.  This is the equivalent of 500 
> bytes per object pinned in memory on average, just for listing object, 
> which is completely silly. We ought to do better than that.

I have thought that the large amount of memory consumed by git-rev-list was caused by not-so-sequential access to very large packfile (1.5GB+ if I remember correctly), which I thought causes the whole packfile to be mmapped and not only window, plus large amount of objects in 300MB+ mem range or something; those both would account for around 2GB.

Besides even if git-rev-list wouldn't take so much memory, object enumeration caching would still help with CPU load... admittedly less.

-- 
Jakub Narebski
Poland