Re: Git is not scalable with too many refs/*

From: Martin Fick <mfick@codeaurora.org>
Date: Sep 27, 2011, 00:12 UTC
Message-ID: <201109261812.31738.mfick@codeaurora.org>
In-Reply-To: <ece30e6a1b74bcddde5634003408f61f@quantumfyre.co.uk>

On Monday, September 26, 2011 05:26:55 pm Julian Phillips wrote:

> On Mon, 26 Sep 2011 15:52:04 -0600, Martin Fick wrote:
> > On Monday, September 26, 2011 03:39:33 pm Martin Fick

wrote:

Show 28 quoted lines

> >> On Monday, September 26, 2011 02:28:53 pm Julian
> >> Phillips
> >> In my case it was called 18785 times!  Any other tests
> >> I should run?
> > 
> > Gerrit stores the changes in directories under
> > refs/changes named after the last 2 digits of the
> > change.  Then under each change it stores each
> > patchset.  So it looks like this:
> > refs/changes/dd/change_num/ps_num
> > 
> > I noticed that:
> >  ls refs/changes/* | wc -l
> >  -> 18876
> > 
> > somewhat close, but not super close to 18785,  I am not
> > sure if that is a clue.  It's almost like each change
> > is causing a re-sort,
> 
> basically, it is ...
> 
> Back when I made that change, I failed to notice that
> get_ref_dir was recursive for subdirectories ... sorry
> ...
> 
> Hopefully this should speed things up.  My test repo went
> from ~17m user time, to ~2.5s.
> Packing still make things much faster of course.

Excellent! This works (almost, in my refs.c it is called sort_ref_list, not sort_refs_list). So, on the non garbage collected repo, git branch now takes ~.5s, and in the garbage collected one it takes only ~.05s!

Thanks way much!!!

-Martin

-- 
Employee of Qualcomm Innovation Center, Inc. which is a 
member of Code Aurora Forum