Re: [PATCH 2/2] pack-objects: fix threaded load balancing

I just deleted the section looking for identical hashes.

+                       while (sub_size && list[0]->hash &&
+                              list[0]->hash == list[-1]->hash) {
+                               list++;
+                               sub_size--;
+                       }

Doing that allows the long chains to be split over the cores.

My last 5% of objects is taking over 50% of the total CPU time in the repack. I think these objects are the ones from that 103,817 entry chain. It is also causing the explosion in RAM consumption.

At the end I can only do 20 objects per clock second on four cores. It takes 30 clock minutes (120 CPU minutes) to do the last 3% of objects.

Can the chains be limited to not grow over some reasonable number, say 5,000? It will make the pack a little bigger but it will help a lot with performance.