Re: [PATCH 2/2] pack-objects: fix threaded load balancing

From: Jon Smirl <jonsmirl@gmail.com>
Date: Dec 10, 2007, 06:06 UTC
Message-ID: <9e4733910712092206o40e0c748s3796b95f637bf2b3@mail.gmail.com>
In-Reply-To: <9e4733910712092159s24cf5a7cx4610f797f61b1de5@mail.gmail.com>

On 12/10/07, Jon Smirl <jonsmirl@gmail.com> wrote:

Show 16 quoted lines

> I just deleted the section looking for identical hashes.
>
> +                       while (sub_size && list[0]->hash &&
> +                              list[0]->hash == list[-1]->hash) {
> +                               list++;
> +                               sub_size--;
> +                       }
>
> Doing that allows the long chains to be split over the cores.
>
> My last 5% of objects is taking over 50% of the total CPU time in the
> repack. I think these objects are the ones from that 103,817 entry
> chain. It is also causing the explosion in RAM consumption.
>
> At the end I can only do 20 objects per clock second on four cores. It
> takes 30 clock minutes (120 CPU minutes) to do the last 3% of objects.

It's all in create_delta...

samples % symbol name 10344074 98.5961 create_delta 138010 1.3155 create_delta_index 4380 0.0417 find_deltas 2526 0.0241 patch_delta 776 0.0074 unpack_entry

Show 9 quoted lines

>
> Can the chains be limited to not grow over some reasonable number, say
> 5,000? It will make the pack a little bigger but it will help a lot
> with performance.
>
> --
> Jon Smirl
> jonsmirl@gmail.com
>

-- 
Jon Smirl
jonsmirl@gmail.com