Re: [RFC/PATCH 0/8] Add initial experimental external ODB support

From: Duy Nguyen <pclouds@gmail.com>
Date: Jun 13, 2016, 10:10 UTC
Message-ID: <CACsJy8ARuqKX1z4zze6Xj47X14tJ1eApCWivDwwuQh2gqRBYMw@mail.gmail.com>
In-Reply-To: <20160613085546.11784-1-chriscool@tuxfamily.org>

On Mon, Jun 13, 2016 at 3:55 PM, Christian Couder <christian.couder@gmail.com> wrote:

Show 12 quoted lines

> Future work
> ~~~~~~~~~~~
>
> From the discussions it appear that using the bundle v3 mechanism to
> tranfer external ODB data could work, but only if the server has access
> to its own external ODB.
>
> Another possible mechanism to transfer external ODB data would be some
> kind of replace refs. This would be slower but the mechanism for the
> transfer already fully exists.
>
> So I think I am going to experiment with some kind of replace refs.

Or we can go "fault-in" all the way.

1) Extend rev-list to support large blob exclusion, e.g. blobs whose
size above a certain limit are excluded, or blobs in certain paths are
excluded (maybe except those at the tip)

2) Add a new command, similar to "shallow", in the git protocol for
the client to tell the server that it does not want the server to send
certain large blobs (and the server probably should acknowledge with
"ok you can get them from this url...")

3) upload-pack passes the relevant rev-list blob filtering arguments
to pack-objects, no big blobs are transferred

4) External odb setting must be done (temporarily) before index-pack
is run (as part of the clone/fetch process). index-pack will notice
that certain blobs are missing from the received pack, but it will
also notice that it already has them via standard has_sha1_file() (we
just need to make sure it does not read blob content back, i think
we're ok here). I'm assuming here that EODB in fact contains _all_
blobs, so no extra setup is required at the server side.

5) If index-pack says "ok" then you can make this new external odb
permanent, else remove it.

Subsequent fetches need to send the same "do not include these blobs" instructions, of course. Not sure how well it interacts with pack-bitmap, but I guess when you use this, you probably already use shallow clone (which disables pack bitmap).

> One interesting thing also would be to use the streaming api when
> reading from or writing to the external ODB.

If EODB is about large blobs only, you probably want to stream them directly to a pack as you get them and not keep in loose object form. Infrastructure is already in place (I think it's used by git-add), but hooking it at sha1_file.c layer might be tricky. This is just cherry on top, not strictly needed of course.

-- 
Duy