Re: Slow fetches of tags

From: Junio C Hamano <junkio@cox.net>
Date: May 24, 2006, 18:08 UTC
Message-ID: <7v64jv8fdx.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <Pine.LNX.4.64.0605240947580.5623@g5.osdl.org>

Linus Torvalds <torvalds@osdl.org> writes:

Show 5 quoted lines

> So the problem may be that we basically send a totally unnecessary list of 
> all the objects we have, when the other end really only cares about the 
> fact that we have the objects that the tags point to. Which we know we do, 
> but we didn't say so, because "git-fetch" didn't really mark them that 
> way.

I think this speculation is correct. We should be able to do better.

Show 7 quoted lines

> I almost suspect that we need to have a syntax where-by the local 
> fetch-list ends up doing
>
> 	"$tagname:$tagname:$sha1wehave"
>
> as the argument to fetch-pack, and then fetch-pack would be modified to 
> send those "$sha1wehave" objects early as "have" objects.

But this logic has to be a bit more involved.

A "have" object is not just has_sha1_file(), but it needs to be reachable from one of our tips we have already verified as complete, so either the caller of fetch-pack does the verification and give a verified $sha1wehave, or fetch-pack takes $sha1weseemtohave and does its own verification and then send it as one of the "have" objects (the issue is the same as the one in my previous message to Eric W. Biederman -- we trust only refs not just having a single object).

It might be useful to have a helper script you can give N object names and M refs (and/or --all flag to mean "all of the refs"), which returns the ones that are reachable from the given refs. It would be even more useful if it were a helper function, but given that the computation would involve walking the ancestry chain, I suspect it would have a bad interaction with any user of such a helper function that wants to do its own ancestry walking, because many of them seem to assume an object that has already been parsed are the ones they parsed for their own purpose.