Re: Git push performance problems with ~100K refs

From: Jeff King <peff@peff.net>
Date: Mar 30, 2012, 09:40 UTC
Message-ID: <20120330094052.GB12298@sigill.intra.peff.net>
In-Reply-To: <20120330093207.GA12298@sigill.intra.peff.net>

On Fri, Mar 30, 2012 at 05:32:08AM -0400, Jeff King wrote:

Show 16 quoted lines

> But that's just a suspicion. I don't have time tonight to work out a
> test case. Is it possible for you to run something like:
> 
>   # make a new commit on top of HEAD, but not yet referenced
>   sha1=`git commit-tree HEAD^{tree} -p HEAD </dev/null`
> 
>   # now do the same "connected" test that receive-pack would do
>   git rev-list --objects $sha1 --not --all
> 
> That should replicate the slow behavior you are seeing. If that works,
> try running the latter command under "perf"; my guess is that you will
> see commit_list_insert_by_date as a hot-spot.
> 
> Even doing this simple test on a moderate repository (my git.git has
> ~1100 refs), commit_list_insert_by_date accounts for 10% of the CPU
> according to perf.

Actually, I did have time for a simple test. Doing:

  git rev-list HEAD |
  while read sha1; do
    echo $sha1 refs/heads/$sha1
  done >>packed-refs
  git pack-refs

in git.git slows down the test above considerably, and perf reports 90% of the time spent in commit_list_insert_by_date. So I think that is indeed the problem.

At one point, I looked at replacing the commit_list implementation with a heap-based priority queue, but unfortunately many parts of the code depend on the list-like nature and would need to be rewritten. We might be able to hack around it by at least adding all of the initial items to an unordered list, then sorting it into its final form.

-Peff