Re: Implementing reftable in Git

From: Stefan Beller <sbeller@google.com>
Date: May 9, 2018, 17:42 UTC
Message-ID: <CAGZ79kZx=wHKc=2WLz-8pQWv1VhRq+pKVV9=Shq3gEMdkX-Q=A@mail.gmail.com>
In-Reply-To: <CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com>

Hi Christian,

On Wed, May 9, 2018 at 7:33 AM, Christian Couder <christian.couder@gmail.com> wrote:

> Hi,
>
> I might start working on implementing reftable in Git soon.

Cool! Everyone is waiting for it as they dream about the performance and correctness benefits this brings.

Benefits that I know of:
* performance in repos with many refs
* no capitalization issues on case insensitive FS
* replay-ability of the last fetch ("show the last reflog
  of any ref under refs/remote/origin") is easier to do
  in a correct way. (This is one of my motivations to desire reftables)
* We *might* be able to use reftables in negotiation later
  ("client: Last I fetched, you said your latest transaction
  number was '5' with the hash over all refs to be <sha1>;
  server: ok, here are the refs and the pack, you're welcome").

Why are you (or rather booking.com) interested in this?

> During the last Git Merge conference last March Stefan talked about
> reftable. In Alex Vandiver's notes [1] it is asked that people
> announce it on the list when they start working on it,

Mostly because many parties want to see it implemnented and were not sure when they could start implementing it.

> and it appears
> that there is a reference implementation in JGit.

The reference implementation can be used in tests to see if we can interact with them, using the JGIT pre-requisite.

Show 11 quoted lines

> Looking it up, there is indeed some documentation [2], code [3], tests
> [4] and other related stuff [5] in the JGit repo. It looks like the
> JGit repo and the reftable code there are licensed under the Eclipse
> Distribution License - v 1.0 [7] which is very similar to the 3-Clause
> BSD License also called Modified BSD License which is GPL compatible
> according to gnu.org [9]. So from a quick look it appears that I
> should be able to port the JGit to Git if I just keep the copyright
> and license header comments in all the related files.
>
> So I think the most straightforward and compatible way to do it would
> be to port the JGit implementation.

I would think you can go by the spec and then test if it is compatible with JGit; that way the spec will be ironed out in corner cases.

> Thanks in advance for any suggestion or comment about this.

I volunteer for reviewing.

(Advanced:) The spec allows for some tune-able parameters and JGits use is heavily optimized for the server side. I think git-core may need to have slightly different tweaks in different situations, e.g. block sizes and how many restarts are put into the block. On the FS we may want to have faster access at the cost of more disk space, whereas in the future when using reftables on the wire as well for ref advertisement we may want to opt for smallest tables. (largest blocks, no restarts)

With that said, please implement it in a way that it can not just be used as a refs backend, but can easily be re-used to write ref advertisements onto the wire?

Thanks, Stefan