From: Julian Phillips Date: Thu, 29 Sep 2011 00:54:00 GMT Subject: Re: Git is not scalable with too many refs/* Message-ID: In-Reply-To: <201109281610.49322.mfick@codeaurora.org> On Wed, 28 Sep 2011 16:10:48 -0600, Martin Fick wrote: > On Wednesday, September 28, 2011 01:38:04 pm Martin Fick > wrote: -- snip -- >> So with that bug fixed, the thing taking the most time >> now for a git checkout with ~100K refs seems to be the >> orphan check as Thomas predicted. The strange part with >> this, is that the orphan check seems to take only about >> ~20s in the repo where the refs aren't packed. However, >> in the repo where they are packed, this check takes at >> least 5min! This seems a bit unusual, doesn't it? Is >> the filesystem that much better at indexing refs than >> git's pack mechanism? Seems unlikely, the unpacked refs >> take 312M in the FS, the packed ones only take about >> 4.3M. I suspect their is something else unexpected >> going on here in the packed ref case. >> >> Any thoughts? I will dig deeper... > > I think the problem is that resolve_ref() walks a linked > list of searching for the packed ref. Does this mean that > packed refs are not indexed at all? Are you sure that it is walking the linked list that is the problem? I've created a test repo with ~100k refs/changes/... style refs, and ~40000 refs/heads/... style refs, and checkout can walk the list of ~140k refs seven times in 85ms user time including doing whatever other processing is needed for checkout. The real time is only 114ms - but then my test repo has no real data in. If resolve_ref() walking the linked list of refs was the problem, then I would expect my test repo to show the same problem. It doesn't, a pre ref-packing checkout took minutes (~0.5s user time), whereas a ref-packed checkout takes ~0.1s. So, I would suggest that the problem lies elsewhere. Have you tried running a checkout whilst profiling? -- Julian