Re: [RFC/PATCH v3 00/16] Add initial experimental external ODB support

From: Lars Schneider <larsxschneider@gmail.com>
Date: Dec 18, 2016, 13:13 UTC
Message-ID: <3819FEA8-BF58-47D3-B60D-2840062022B7@gmail.com>
In-Reply-To: <CAP8UFD2uyq3Uf1co_BUKJX_eogdCDJ30KJZmQ1BQXNQ1dw=w3A@mail.gmail.com>

Show 24 quoted lines

> On 13 Dec 2016, at 18:20, Christian Couder <christian.couder@gmail.com> wrote:
> 
> On Sat, Dec 3, 2016 at 7:47 PM, Lars Schneider <larsxschneider@gmail.com> wrote:
>> 
>>> On 30 Nov 2016, at 22:04, Christian Couder <christian.couder@gmail.com> wrote:
>>> 
>>> Goal
>>> ~~~~
>>> 
>>> Git can store its objects only in the form of loose objects in
>>> separate files or packed objects in a pack file.
>>> 
>>> To be able to better handle some kind of objects, for example big
>>> blobs, it would be nice if Git could store its objects in other object
>>> databases (ODB).
>> ...
>> 
>> Minor nit: I feel the term "other" could be more expressive. Plus
>> "database" might confuse people. What do you think about
>> "External Object Storage" or something?
> 
> In the current Git code, "DB" is already used a lot. For example in
> cache.h there is:
> ...

I am not worried about Git core developers as I don't think they would be confused by the term "DB". I wonder if it would make sense to have a clearer "external name" for the average Git user (== non Git devs) or if this would create just more confusion.

Show 25 quoted lines

>>> * Transfer
>>> 
>>> To tranfer information about the blobs stored in external ODB, some
>>> special refs, called "odb ref", similar as replace refs, are used.
>>> 
>>> For now there should be one odb ref per blob. Each ref name should be
>>> refs/odbs/<odbname>/<sha1> where <sha1> is the sha1 of the blob stored
>>> in the external odb named <odbname>.
>>> 
>>> These odb refs should all point to a blob that should be stored in the
>>> Git repository and contain information about the blob stored in the
>>> external odb. This information can be specific to the external odb.
>>> The repos can then share this information using commands like:
>>> 
>>> `git fetch origin "refs/odbs/<odbname>/*:refs/odbs/<odbname>/*"`
>> 
>> The "odbref" would point to a blob and the blob could contain anything,
>> right? E.g. it could contain an existing GitLFS pointer, right?
>> 
>> version https://git-lfs.github.com/spec/v1
>> oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
>> size 12345
> 
> Yes, but I think that the sha1 should be added. So yes, it could
> easily be made compatible with git LFS.

What do you mean with "the sha1 should be added"? Do you suggest to add sha1 to GitLFS?

Show 17 quoted lines

>>> Future work
>>> ~~~~~~~~~~~
>>> 
>>> I think that the odb refs don't prevent a regular fetch or push from
>>> wanting to send the objects that are managed by an external odb. So I
>>> am interested in suggestions about this problem. I will take a look at
>>> previous discussions and how other mechanisms (shallow clone, bundle
>>> v3, ...) handle this.
>> 
>> If the ODB configuration is stored in the Git repo similar to
>> .gitmodules then every client that clones ODB references would be able
>> to resolve them, right?
> 
> Yeah, but I am not sure that being able to resolve the odb refs will
> prevent the big blobs from being sent.
> With Git LFS, git doesn't know about the big blobs, only about the
> substituted files, but that is not the case in what I am doing.

I think the biggest problem in Git are huge blobs that are not in the head revision. In the great majority of cases you don't need these blobs but you always have to transfer them during clone. That's what GitLFS is solving today and what I hope your protocol could solve better in the future!

Cheers, Lars