Re: Git is not scalable with too many refs/*

> On Mon, 26 Sep 2011 14:01:38 -0600, Martin Fick wrote:
> -- snip --
> 
> > So, maybe you are correct, maybe my repo is the corner
> > case? Is a repo which needs to be gced considered a
> > corner case? Should git be able to detect that the
> > repo is so in desperate need of gcing?  Is it normal
> > for git to need to gc right after a clone and then
> > fetching ~100K refs?
> 
> Were you 100k refs packed before the gc?  If not, perhaps
> your refs are causing a lot of trouble for the merge
> sort?  They will be written out sorted to the
> packed-refs file, so the merge sort won't have to do any
> real work when loading them after that...

> > I am not sure what is right here, if this patch makes a
> > repo which needs gcing degrade 5 to 10 times worse
> > than the benefit of this patch, it still seems
> > questionable to me.
> 
> Well - it does this _for your repo_, that doesn't
> automatically mean that it does generally, or
> frequently.  

> For instance, none of my normal repos that
> have a lot of refs are Gerrit ones, and I wouldn't be
> surprised if they benefitted from the merge sort
> (assuming that I am right that the merge sort is taking
> a long time on your gerrit refs).
> 
> Besides, you would be better off running gc, and thus
> getting the benefit too.

> >> Random thought.  What happens to the with compression
> >> case if you leave the commit in, but add a sleep(15)
> >> to the end of sort_refs_list?
> > 
> > Why, what are you thinking?  Hmm, I am trying this on
> > the non gced repo and it doesn't seem to be completing
> > (no cpu usage)!  It appears that perhaps it is being
> > called many times (the sleeping would explain no cpu
> > usage)?!?  This could be a real problem, this should
> > only get called once right?
> 
> I was just wondering if the time taken to get the refs
> was changing the interaction with something else.  Not
> very likely, but ...
> 
> I added a print statement, and it was called four times
> when I had unpacked refs, and once with packed.  So,
> maybe you are hitting some nasty case with unpacked
> refs.  If you use a print statement instead of a sleep,
> how many times does sort_refs_lists get called in your
> unpacked case?  It may well also be worth calculating
> the time taken to do the sort.