Re: Experimental TDB support for GIT REFS

From: Ronnie Sahlberg <sahlberg@google.com>
Date: Jul 2, 2014, 17:11 UTC
Message-ID: <CAL=YDW=pHB4sjpxjYvO_w=RadK4Rhq6Zp+pc+NAqWT3hWjkWXg@mail.gmail.com>
In-Reply-To: <CAJo=hJu3QD09JccSvUrpnNSpFE5ppDHEzozkKyZZAyutvCQTGg@mail.gmail.com>

On Fri, Jun 27, 2014 at 5:37 PM, Shawn Pearce <spearce@spearce.org> wrote:

Show 20 quoted lines

> On Fri, Jun 27, 2014 at 2:30 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:
>> List,
>>
>> One of my ref transaction aims is to make define a stable public API
>> for accessing refs.
>> Once this is done I want to make it possible to replace the current
>> .git/refs/* model with a
>> different type of backend.
>> In my case I want to add support to store all refs and reflogs in a TDB database
>> but once we have a pluggable backend framework for refs, if someone
>> wants to store the refs
>> in a SQL database, that should be fairly trivial too.
>>
>> There are a few series queued before this becomes possible, but is
>> anyone wants to test or play with my "git can use TDB database" you
>> can find an implementation of this at
>>
>> https://github.com/rsahlberg/git/tree/backend-struct-tdb
>
> Interesting! But the link 404s :(

Yeah, sorry but there were issues :-( Issues solved now though and refactoring done.

Please see https://github.com/rsahlberg/git/tree/backend-struct-db

This branch adds a refs backend that communicates via a domain socket to a refs daemon. The refs daemon in turn interfaces with the actual database.

My branch contains one hack refs daemon that uses a TDB database for the refs storage. This daemon is only about 600 lines of code, most of which is marshalling and socket handling and only a small amount of TDB specific code. This small daemon should make it easy for folks to add arbitrary database interfaces easily. Taking refsd-tdb.c and modifying it to instead attach to a SQL database should only take a few hundred lines of changes or so.

To build the daemon and start it :

$ gcc refsd-tdb.c -o refsd-tdb -ltdb <create /var/lib/git and /var/log/git> $ ./refsd-tdb /var/lib/git/refs.socket /var/lib/git /var/log/git/refsd.log

Then you can inspect the database with tdbdump /var/lib/git/refs.tdb

You can then create a repository that uses this database : $ git init --db-repo-name=ROCC --db-socket=/var/lib/git/refs.socket .

Then further commands will operate on the refs tdb.

See this as a prototype to experiment with and see the direction of the refs transaction and pluggable backend support. There is a lot additional work and cleanup that needs to be done before this will become production code.

(yes I know we should not do hand marshalling and unmarshalling for the PDUs on the domain socket. We should use some lightweight encoding library like XDR and rpcgen or similar.)

Please test, comment and have fun.

regards ronnie sahlberg