Re: [PATCH v2 4/4] bundle v3: the beginning

From: Christian Couder <christian.couder@gmail.com>
Date: May 31, 2016, 13:18 UTC
Message-ID: <CAP8UFD3jPQFk2deSk5JXC3PTz5yWcvXJ4=Qjam5Qw6P9SrLzFQ@mail.gmail.com>
In-Reply-To: <CACsJy8Dr_Z886Jb-O8gbAv_vzBLicNH6bPPpKwb9HWZTKQ9muw@mail.gmail.com>

On Tue, May 31, 2016 at 2:43 PM, Duy Nguyen <pclouds@gmail.com> wrote:

Show 11 quoted lines

> On Fri, May 20, 2016 at 7:39 PM, Christian Couder
> <christian.couder@gmail.com> wrote:
>> I am responding to this 2+ month old email because I am investigating
>> adding an alternate object store at the same level as loose and packed
>> objects. This alternate object store could be used for large files. I
>> am working on this for GitLab. (Yeah, I am working, as a freelance,
>> for both Booking.com and GitLab these days.)
>
> I'm also interested in this from a different angle, narrow clone that
> potentially allows to skip download some large blobs (likely old ones
> from the past that nobody will bother).

Interesting!

[...]

Show 16 quoted lines

>> I wonder if this mechanism could also be used or extended to clone and
>> fetch an alternate object database.
>>
>> In [1], [2] and [3], and this was also discussed during the
>> Contributor Summit last month, Peff says that he started working on
>> alternate object database support a long time ago, and that the hard
>> part is a protocol extension to tell remotes that you can access some
>> objects in a different way.
>>
>> If a Git client would download a "$name.bndl" v3 bundle file that
>> would have a "data: $URL/alt-odb-$name.odb" extended header, the Git
>> client would just need to download "$URL/alt-odb-$name.odb" and use
>> the alternate object database support on this file.
>
> What does this file contain exactly? A list of SHA-1 that can be
> retrieved from this remote/alternate odb?

It would depend on the external odb. Git could support different external odb that have different trade-offs.

> I wonder if we could just
> git-replace for this marking. The replaced content could contain the
> uri pointing to the alt odb.

Yeah, interesting! That's indeed another possibility that might not need the transfer of any external odb.

But in this case it might be cleaner to just have a separate ref hierarchy like:

refs/external-odbs/my-ext-odb/<sha1>

instead of using the replace one.

Or maybe:

refs/replace/external-odbs/my-ext-odb/<sha1>

if we really want to use the replace hierarchy.

> We could optionally contact alt odb to
> retrieve real content, or just show the replaced/fake data when alt
> odb is out of reach.

Yeah, I wonder if that really needs the replace mechanism.

> Transferring git-replace is basically ref
> exchange, which may be fine if you don't have a lot of objects in this
> alt odb.

Yeah sure, great idea!

By the way this makes me wonder if we could implement resumable clone using some kind of replace ref.

The client while cloning nearly as usual would download one or more special replace refs that would points to objects with links to download bundles using standard protocols. Just after the clone, the client would read these objects and download the bundles from these objects. And then it would clone from these bundles.

Show 8 quoted lines

> If you do, well, we need to deal with lots of refs anyway.
> This may benefit from it too.
>
>> [3] http://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020
>
> This points to  https://github.com/peff/git/commits/jk/external-odb
> which is dead. Jeff, do you still have it somewhere, or is it not
> worth looking at anymore?

I have rebased, fixed and improved it a bit. I added write support for blobs. But the result is not very clean right now. I was going to send a RFC patch series after cleaning the result, but as you ask, here are some links to some branches:

- https://github.com/chriscool/git/commits/gl-external-odb3 (the
updated patches from Peff, plus 2 small patches from me)
- https://github.com/chriscool/git/commits/gl-external-odb7 (the same
as above, plus a number of WIP patches to add blob write support)

Thanks, Christian.