From: Jon Smirl <jonsmirl@gmail.com>
Date: Mon, 10 Dec 2007 05:59:38 GMT
Subject: Re: [PATCH 2/2] pack-objects: fix threaded load balancing
Message-ID: <9e4733910712092159s24cf5a7cx4610f797f61b1de5@mail.gmail.com>
In-Reply-To: <9e4733910712092123r6987c3edua74a15890ae45a3d@mail.gmail.com>

I just deleted the section looking for identical hashes.

+                       while (sub_size && list[0]->hash &&
+                              list[0]->hash == list[-1]->hash) {
+                               list++;
+                               sub_size--;
+                       }

Doing that allows the long chains to be split over the cores.

My last 5% of objects is taking over 50% of the total CPU time in the
repack. I think these objects are the ones from that 103,817 entry
chain. It is also causing the explosion in RAM consumption.

At the end I can only do 20 objects per clock second on four cores. It
takes 30 clock minutes (120 CPU minutes) to do the last 3% of objects.

Can the chains be limited to not grow over some reasonable number, say
5,000? It will make the pack a little bigger but it will help a lot
with performance.

-- 
Jon Smirl
jonsmirl@gmail.com