Re: another packed-refs race

From: Michael Haggerty <mhagger@alum.mit.edu>
Date: May 6, 2013, 12:12 UTC
Message-ID: <51879E2D.8060506@alum.mit.edu>
In-Reply-To: <20130503172853.GB21715@sigill.intra.peff.net>

On 05/03/2013 07:28 PM, Jeff King wrote:

Show 15 quoted lines

> On Fri, May 03, 2013 at 11:26:11AM +0200, Johan Herland wrote:
> 
>> You don't really need to be sure that packed-refs is up-to-date. You
>> only need to make sure that don't rely on lazily loading loose refs
>> _after_ you have loaded packed-refs.
> 
> True. As long as you load them both together, and always make sure you
> do loose first, you'd be fine. But I think there will be corner cases
> where you have loaded _part_ of the loose ref namespace. I think part of
> the point of Michael's ref work is that if you call "for_each_tag_ref",
> we would not waste time loading refs/remotes/ at all. If you then follow
> that with a call to "for_each_ref", you would want to re-use the cached
> work from traversing refs/tags/, and then traverse refs/remotes/. You
> know that your cached packed-refs is good with respect to refs/tags/,
> but you don't with respect to refs/remotes.

Correct.

[I wonder if there really are a lot of iterations over overlapping parts of the reference namespace within a single git process...]

Show 18 quoted lines

>> The following solution might work in both the resolve-a-single-ref and
>> enumerating-refs case:
>>
>> 0. Look for ref already cached in memory. If found, OK.
>>
>> 1. Look for loose ref. If found, OK.
>>
>> 2. If not found, load all loose refs and packed-refs from disk (in
>> that order), and store in memory for remainder of this process. Never
>> reload packed-refs from disk (unless you also reload all loose refs
>> first).
> 
> I think that would be correct (modulo that step 1 cannot happen for
> enumeration). But we would like to avoid loading all loose refs if we
> can. Especially on a cold cache, it can be quite slow, and you may not
> even care about those refs for the current operation (I do not recall
> the exact original motivation for the lazy loading, but it was something
> along those lines).

Lazy loading was first inspired by the observation that effectively every git invocation was loading *all* loose references to do an iteration over refs/replace/ (which I've never even used!) This was absolutely killing the performance of filter-branch, which creates a lot of loose references and invokes git many times--even though the cache was warm.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/