Re: GTP/0.1 terminology 101: commit reels and references

From: Sam Vilain <sam@vilain.net>
Date: Jul 28, 2008, 10:03 UTC
Message-ID: <1217239419.6750.73.camel@maia.lan>
In-Reply-To: <7vk5f6tqsj.fsf@gitster.siamese.dyndns.org>

On Mon, 2008-07-28 at 00:24 -0700, Junio C Hamano wrote:

Show 12 quoted lines

> >>   Commit reels can also, and generally do, include the objects required
> >>  for a specific commit.
> >
> > Yes.  The only times where they wouldn't contain all the objects
> > required for the commits within the reel, is when those objects happened
> > to be contained by a previous reel.
> 
> What do you mean by "previous" reel?  It is not quite defined in your
> message but perhaps defined elsewhere?
> 
> How is this different from a bundle?  Does a reel, unlike a bundle,
> contain the full tree for the bottom commits?

They are almost identical, both being defined by a set of starting and ending refs. And now that you mention it, I feel slightly embarrassed for not spotting the connection before. I only really compared reels to packs, which is what the original specification tried to chop up bitwise and distribute chunk by chunk.

The differences are:

  - the reel has a defined object order (which as I hoped to demonstrate
    in the test cases, is just a refinement of rev-list --date-order)

  - deltas always point in one direction, to objects "earlier" on
    the reel, so that slices of the reel sent on the network can be made
    thin without resulting in unresolvable deltas (which should be
    possible to do on commit boundaries using rev-list --objects-edge)

  - the behaviour at the beginning of the reel is precisely defined
    (although as I said, I think that the decision might be worth
    revisiting - perhaps getting just the latest reel is a useful
    'shallow clone')

Show 7 quoted lines

> > This is one of the design decisions which I think may be a mistake; a
> > less expensive to calculate definition of a reel would be *all* objects
> > between the starting and ending Reference objects.
> 
> Do you mean all such objects and nothing else?  That would imply that a
> reel is quite similar to a bundle (but neither rev-list --objects-edge
> nor bundle guarantees that the result is minimal).

It's the lack of guarantees which is the issue, really. In order to take the download work of the entire pack and distribute it over multiple peers, you need a way to carve the bundle up. This has to happen in such a way that the fragments that you get back will actually fit together at the end, and also in such a way that you don't lose the benefits of delta compression.

The way I thought would be best to do that would be to line up all the objects in an exactly defined way - hence, the "reel" concept - and then chop that up.

If a pack is already arranged to line up with the commit reel's structure, then it's possible that the amount of work required to answer a "play" request is as little as looking up in the reel index the local on-disk location within the local pack, and copying that to the network.

I've certainly wondered how much baggage could be removed from this whole thing, like replacing the "tracker" with a simple git-daemon message that holds a register of mirrors/peers, possibly layering things over git:// instead of the bittorrent-like protocol, dividing up blocks by the commit graph and not all objects, etc. But I think that it would be best to defer that kind of design change until the conclusion of this prototype experiment.

That being said, anything which does shortcut the distance to the finish line and can be agreed on wouldn't go amiss.

Sam