Re: Something is broken in repack

From: Jon Smirl <jonsmirl@gmail.com>
Date: Dec 11, 2007, 02:25 UTC
Message-ID: <9e4733910712101825l33cdc2c0mca2ddbfd5afdb298@mail.gmail.com>
In-Reply-To: <9e4733910712071505y6834f040k37261d65a2d445c4@mail.gmail.com>

New run using same configuration. With the addition of the more efficient load balancing patches and delta cache accounting.

Seconds are wall clock time. They are lower since the patch made threading better at using all four cores. I am stuck at 380-390% CPU utilization for the git process.

complete seconds RAM 10% 60 900M (includes counting) 20% 15 900M 30% 15 900M 40% 50 1.2G 50% 80 1.3G 60% 70 1.7G 70% 140 1.8G 80% 180 2.0G 90% 280 2.2G 95% 530 2.8G - 1,420 total to here, previous was 1,983 100% 1390 2.85G During the writing phase RAM fell to 1.6G What is being freed in the writing phase??

I have no explanation for the change in RAM usage. Two guesses come to mind. Memory fragmentation. Or the change in the way the work was split up altered RAM usage.

Total CPU time was 195 minutes in 70 minutes clock time. About 70% efficient. During the compress phase all four cores were active until the last 90 seconds. Writing the objects took over 23 minutes CPU bound on one core.

New pack file is: 270,594,853 Old one was: 344,543,752 It still has 828,660 objects

On 12/7/07, Jon Smirl <jonsmirl@gmail.com> wrote:

Show 33 quoted lines

> Using this config:
> [pack]
>         threads = 4
>         deltacachesize = 256M
>         deltacachelimit = 0
>
> And the 330MB gcc pack for input
>  git repack -a -d -f  --depth=250 --window=250
>
> complete seconds RAM
> 10%  47 1GB
> 20%  29 1Gb
> 30%  24 1Gb
> 40%  18 1GB
> 50%  110 1.2GB
> 60%  85 1.4GB
> 70%  195 1.5GB
> 80%  186 2.5GB
> 90%  489 3.8GB
> 95%  800 4.8GB
> I killed it because it started swapping
>
> The mmaps are only about 400MB in this case.
> At the end the git process had 4.4GB of physical RAM allocated.
>
> Starting from a highly compressed pack greatly aggravates the problem.
> Starting with a 2GB pack of the same data my process size only grew to
> 3GB with 2GB of mmaps.
>
> --
> Jon Smirl
> jonsmirl@gmail.com
>

-- 
Jon Smirl
jonsmirl@gmail.com