Experimental TDB support for GIT REFS

List,

One of my ref transaction aims is to make define a stable public API for accessing refs. Once this is done I want to make it possible to replace the current .git/refs/* model with a different type of backend. In my case I want to add support to store all refs and reflogs in a TDB database but once we have a pluggable backend framework for refs, if someone wants to store the refs in a SQL database, that should be fairly trivial too.

There are a few series queued before this becomes possible, but is anyone wants to test or play with my "git can use TDB database" you can find an implementation of this at

https://github.com/rsahlberg/git/tree/backend-struct-tdb

Most everything works (except submodules) but the code needs more work. It is also VERY LIKELY that the database schema WILL CHANGE so this is only for testing and you will need to blow the repo away and start all over again at some stage. I.e. no production use.

To activate TDB support a new flag is added to the init-db and clone commands:

git init --tdb-repo-name=RONNIE .

This creates a new git repo and sets core.tdb_repo_name in the config. This config variable tells git to dlopen and use libtdb instead of using the files under .git/refs

repo-name=RONNIE is a string that identifies the repository and all keys in the database is prefixed with repo-name + '\0'

By default this will create the two database refs.tdb and logs.tdb in the repositories .git directory but it is also possible to store the databases somehwere else by adding --tdb-repo-dir=/var/lib/git/ when creating the repo.

Since the keys in the database are prefixed with the repo name it is even possible to store all the refs for multiple independent repos in the same database :

- First repository
  $ cd <somewhere>
  $ git init-db --tdb-repo-name=MyFirstRepo --tdb-repo-dir=/var/lib/git

- Second repository
  $ cd <somewhere else>
  $ git init-db --tdb-repo-name=MySecondRepo --tdb-repo-dir=/var/lib/git

This can also be a solutions for platforms lacking case sensitive filesystems where today two refs that only differ in case can not be used.

The current prototype will still apply the naming collision rules that the existing files backend has. For example if you have a ref 'm' you can not create another ref 'm/foo'. I left those checks as is in order to not break compatibility between the TDB backend and the current Files backend.

Please feel free to test this out. And comment.

regards ronnie sahlberg

On Fri, Jun 27, 2014 at 2:30 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:

Interesting! But the link 404s :(

On Fri, Jun 27, 2014 at 5:37 PM, Shawn Pearce <spearce@spearce.org> wrote:

Yeah, sorry but there were issues :-( Issues solved now though and refactoring done.

Please see https://github.com/rsahlberg/git/tree/backend-struct-db

This branch adds a refs backend that communicates via a domain socket to a refs daemon. The refs daemon in turn interfaces with the actual database.

My branch contains one hack refs daemon that uses a TDB database for the refs storage. This daemon is only about 600 lines of code, most of which is marshalling and socket handling and only a small amount of TDB specific code. This small daemon should make it easy for folks to add arbitrary database interfaces easily. Taking refsd-tdb.c and modifying it to instead attach to a SQL database should only take a few hundred lines of changes or so.

To build the daemon and start it :

$ gcc refsd-tdb.c -o refsd-tdb -ltdb <create /var/lib/git and /var/log/git> $ ./refsd-tdb /var/lib/git/refs.socket /var/lib/git /var/log/git/refsd.log

Then you can inspect the database with tdbdump /var/lib/git/refs.tdb

You can then create a repository that uses this database : $ git init --db-repo-name=ROCC --db-socket=/var/lib/git/refs.socket .

Then further commands will operate on the refs tdb.

See this as a prototype to experiment with and see the direction of the refs transaction and pluggable backend support. There is a lot additional work and cleanup that needs to be done before this will become production code.

(yes I know we should not do hand marshalling and unmarshalling for the PDUs on the domain socket. We should use some lightweight encoding library like XDR and rpcgen or similar.)

Please test, comment and have fun.

regards ronnie sahlberg

On Wed, Jul 2, 2014 at 10:11 AM, Ronnie Sahlberg <sahlberg@google.com> wrote:

One enhancement to this could be to make it easier to use for trivial users that do not want to share one database across multiple repos.

We could have something like $ git init --db=tdb .

core.db=tdb could then mean that instead of connecting to a domain socket for a shared daemon instead it would just popen("refsd-<core.db> ...", "rw") and automaticallt set the asguments ro refsd-tdb to point to the local .git directory.

This would make the setup simpler for the trivial usecase since it would avoid having to manuallt initializing a dedicated daemon before the git commands will start working.