Re: Surprising use of memory and time when repacking mozilla's gecko repository

From: Jakub Narebski <jnareb@gmail.com>
Date: Jul 5, 2019, 11:29 UTC
Message-ID: <86bly81zyn.fsf@gmail.com>
In-Reply-To: <20190705054736.6llkhnopt2fcta2k@glandium.org>

Mike Hommey <mh@glandium.org> writes:

> On Fri, Jul 05, 2019 at 01:14:13AM -0400, Jeff King wrote:
>> On Thu, Jul 04, 2019 at 10:13:20PM +0900, Mike Hommey wrote:

[...]

Show 13 quoted lines

>> I think I explained all of the memory-usage questions in my earlier
>> response, but just for reference: if you have access to it, valgrind's
>> "massif" tool is really good for this kind of profiling. Something like:
>> 
>>   valgrind --tool=massif git pack-objects ...
>>   ms_print massif.out.*
>> 
>> which shows heap usage at various times, points out the snapshot with
>> peak usage, and shows a backtrace of the main culprits at a few
>> snapshots.
>
> At the expense of time ;) A run would likely last an entire day under
> massif (by which I mean a full 24 hours, not a 9-5 day).

Valgrind, as I understand it, runs the program under emulation. I wonder if perf / OProfile based solution could help here (gathering memory-based events and metrics).

There is also trace2 built-in into Git, but I don't know if it could be used for this purpose or not.

Best, -- Jakub Narębski