{"thread":{"id":"37013","subject":"Experimental TDB support for GIT REFS","startedAt":"2014-06-27T21:30:07Z","lastAt":"2014-07-02T22:34:08Z","messageCount":4,"participants":["Ronnie Sahlberg","Shawn Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"245100","messageId":"CAL=YDWn-CGZGr5bXNTiZmzr4-w_8CERx3r2bmLWbczqJ0Sn7dg@mail.gmail.com","threadId":"37013","inReplyTo":null,"subject":"Experimental TDB support for GIT REFS","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-06-27T21:30:07Z","receivedAt":"2014-06-27T21:30:07Z","isPatch":false,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"List,\n\nOne of my ref transaction aims is to make define a stable public API\nfor accessing refs.\nOnce this is done I want to make it possible to replace the current\n.git/refs/* model with a\ndifferent type of backend.\nIn my case I want to add support to store all refs and reflogs in a TDB database\nbut once we have a pluggable backend framework for refs, if someone\nwants to store the refs\nin a SQL database, that should be fairly trivial too.\n\nThere are a few series queued before this becomes possible, but is\nanyone wants to test or play with my \"git can use TDB database\" you\ncan find an implementation of this at\n\nhttps://github.com/rsahlberg/git/tree/backend-struct-tdb\n\nMost everything works (except submodules) but the code needs more work.\nIt is also VERY LIKELY that the database schema WILL CHANGE so this is\nonly for testing\nand you will need to blow the repo away and start all over again at some stage.\nI.e. no production use.\n\n\nTo activate TDB support a new flag is added to the init-db and clone commands:\n\ngit init --tdb-repo-name=RONNIE .\n\nThis creates a new git repo and sets core.tdb_repo_name in the config.\nThis config variable tells git to dlopen and use libtdb instead of\nusing the files under .git/refs\n\nrepo-name=RONNIE is a string that identifies the repository and all\nkeys in the database is prefixed\nwith repo-name + '\\0'\n\nBy default this will create the two database refs.tdb and logs.tdb in\nthe repositories .git directory\nbut it is also possible to store the databases somehwere else by\nadding --tdb-repo-dir=/var/lib/git/\nwhen creating the repo.\n\nSince the keys in the database are prefixed with the repo name it is\neven possible to store all the\nrefs for multiple independent repos in the same database :\n\n- First repository\n  $ cd <somewhere>\n  $ git init-db --tdb-repo-name=MyFirstRepo --tdb-repo-dir=/var/lib/git\n\n- Second repository\n  $ cd <somewhere else>\n  $ git init-db --tdb-repo-name=MySecondRepo --tdb-repo-dir=/var/lib/git\n\n\nThis can also be a solutions for platforms lacking case sensitive\nfilesystems where today two refs that only differ in case can not be\nused.\n\n\n\nThe current prototype will still apply the naming collision rules that\nthe existing files backend has.\nFor example if you have a ref 'm'  you can not create another ref 'm/foo'.\nI left those checks as is in order to not break compatibility between\nthe TDB backend and the current Files backend.\n\n\nPlease feel free to test this out. And comment.\n\n\nregards\nronnie sahlberg\n"},{"id":"245108","messageId":"CAJo=hJu3QD09JccSvUrpnNSpFE5ppDHEzozkKyZZAyutvCQTGg@mail.gmail.com","threadId":"37013","inReplyTo":"CAL=YDWn-CGZGr5bXNTiZmzr4-w_8CERx3r2bmLWbczqJ0Sn7dg@mail.gmail.com","subject":"Re: Experimental TDB support for GIT REFS","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2014-06-28T00:37:52Z","receivedAt":"2014-06-28T00:37:52Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Fri, Jun 27, 2014 at 2:30 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:\n> List,\n>\n> One of my ref transaction aims is to make define a stable public API\n> for accessing refs.\n> Once this is done I want to make it possible to replace the current\n> .git/refs/* model with a\n> different type of backend.\n> In my case I want to add support to store all refs and reflogs in a TDB database\n> but once we have a pluggable backend framework for refs, if someone\n> wants to store the refs\n> in a SQL database, that should be fairly trivial too.\n>\n> There are a few series queued before this becomes possible, but is\n> anyone wants to test or play with my \"git can use TDB database\" you\n> can find an implementation of this at\n>\n> https://github.com/rsahlberg/git/tree/backend-struct-tdb\n\nInteresting! But the link 404s :(\n"},{"id":"245314","messageId":"CAL=YDW=pHB4sjpxjYvO_w=RadK4Rhq6Zp+pc+NAqWT3hWjkWXg@mail.gmail.com","threadId":"37013","inReplyTo":"CAJo=hJu3QD09JccSvUrpnNSpFE5ppDHEzozkKyZZAyutvCQTGg@mail.gmail.com","subject":"Re: Experimental TDB support for GIT REFS","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-07-02T17:11:10Z","receivedAt":"2014-07-02T17:11:10Z","isPatch":false,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Fri, Jun 27, 2014 at 5:37 PM, Shawn Pearce <spearce@spearce.org> wrote:\n> On Fri, Jun 27, 2014 at 2:30 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:\n>> List,\n>>\n>> One of my ref transaction aims is to make define a stable public API\n>> for accessing refs.\n>> Once this is done I want to make it possible to replace the current\n>> .git/refs/* model with a\n>> different type of backend.\n>> In my case I want to add support to store all refs and reflogs in a TDB database\n>> but once we have a pluggable backend framework for refs, if someone\n>> wants to store the refs\n>> in a SQL database, that should be fairly trivial too.\n>>\n>> There are a few series queued before this becomes possible, but is\n>> anyone wants to test or play with my \"git can use TDB database\" you\n>> can find an implementation of this at\n>>\n>> https://github.com/rsahlberg/git/tree/backend-struct-tdb\n>\n> Interesting! But the link 404s :(\n\n\nYeah, sorry but there were issues :-(   Issues solved now though and\nrefactoring done.\n\nPlease see\nhttps://github.com/rsahlberg/git/tree/backend-struct-db\n\nThis branch adds a refs backend that communicates via a domain socket\nto a refs daemon.\nThe refs daemon in turn interfaces with the actual database.\n\nMy branch contains one hack refs daemon that uses a TDB database for\nthe refs storage.\nThis daemon is only about 600 lines of code, most of which is\nmarshalling and socket handling and only a small amount of\nTDB specific code.\nThis small daemon should make it easy for folks to add arbitrary\ndatabase interfaces easily.\nTaking refsd-tdb.c  and modifying it to instead attach to a SQL\ndatabase should only take a few hundred lines of changes or so.\n\n\nTo build the daemon and start it :\n\n$ gcc refsd-tdb.c -o refsd-tdb -ltdb\n<create /var/lib/git and /var/log/git>\n$ ./refsd-tdb /var/lib/git/refs.socket /var/lib/git /var/log/git/refsd.log\n\nThen you can inspect the database with\ntdbdump /var/lib/git/refs.tdb\n\n\nYou can then create a repository that uses this database :\n$ git init --db-repo-name=ROCC --db-socket=/var/lib/git/refs.socket .\n\nThen further commands will operate on the refs tdb.\n\nSee this as a prototype to experiment with and see the direction of\nthe refs transaction and pluggable backend support.\nThere is a lot additional work and cleanup that needs to be done\nbefore this will become production code.\n\n(yes I know we should not do hand marshalling and unmarshalling for\nthe PDUs on the domain socket. We should use some lightweight encoding\nlibrary like XDR and rpcgen or similar.)\n\n\nPlease test, comment and have fun.\n\nregards\nronnie sahlberg\n"},{"id":"245363","messageId":"CAL=YDWmBqOvRyPk3=LKCM_k2=2XRW8zUwZJv-GzBLPo9w_BYfw@mail.gmail.com","threadId":"37013","inReplyTo":"CAL=YDW=pHB4sjpxjYvO_w=RadK4Rhq6Zp+pc+NAqWT3hWjkWXg@mail.gmail.com","subject":"Re: Experimental TDB support for GIT REFS","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-07-02T22:34:08Z","receivedAt":"2014-07-02T22:34:08Z","isPatch":false,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Wed, Jul 2, 2014 at 10:11 AM, Ronnie Sahlberg <sahlberg@google.com> wrote:\n> On Fri, Jun 27, 2014 at 5:37 PM, Shawn Pearce <spearce@spearce.org> wrote:\n>> On Fri, Jun 27, 2014 at 2:30 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:\n>>> List,\n>>>\n>>> One of my ref transaction aims is to make define a stable public API\n>>> for accessing refs.\n>>> Once this is done I want to make it possible to replace the current\n>>> .git/refs/* model with a\n>>> different type of backend.\n>>> In my case I want to add support to store all refs and reflogs in a TDB database\n>>> but once we have a pluggable backend framework for refs, if someone\n>>> wants to store the refs\n>>> in a SQL database, that should be fairly trivial too.\n>>>\n>>> There are a few series queued before this becomes possible, but is\n>>> anyone wants to test or play with my \"git can use TDB database\" you\n>>> can find an implementation of this at\n>>>\n>>> https://github.com/rsahlberg/git/tree/backend-struct-tdb\n>>\n>> Interesting! But the link 404s :(\n>\n>\n> Yeah, sorry but there were issues :-(   Issues solved now though and\n> refactoring done.\n>\n> Please see\n> https://github.com/rsahlberg/git/tree/backend-struct-db\n>\n> This branch adds a refs backend that communicates via a domain socket\n> to a refs daemon.\n> The refs daemon in turn interfaces with the actual database.\n>\n> My branch contains one hack refs daemon that uses a TDB database for\n> the refs storage.\n> This daemon is only about 600 lines of code, most of which is\n> marshalling and socket handling and only a small amount of\n> TDB specific code.\n> This small daemon should make it easy for folks to add arbitrary\n> database interfaces easily.\n> Taking refsd-tdb.c  and modifying it to instead attach to a SQL\n> database should only take a few hundred lines of changes or so.\n>\n>\n> To build the daemon and start it :\n>\n> $ gcc refsd-tdb.c -o refsd-tdb -ltdb\n> <create /var/lib/git and /var/log/git>\n> $ ./refsd-tdb /var/lib/git/refs.socket /var/lib/git /var/log/git/refsd.log\n>\n> Then you can inspect the database with\n> tdbdump /var/lib/git/refs.tdb\n>\n>\n> You can then create a repository that uses this database :\n> $ git init --db-repo-name=ROCC --db-socket=/var/lib/git/refs.socket .\n>\n> Then further commands will operate on the refs tdb.\n>\n> See this as a prototype to experiment with and see the direction of\n> the refs transaction and pluggable backend support.\n> There is a lot additional work and cleanup that needs to be done\n> before this will become production code.\n>\n> (yes I know we should not do hand marshalling and unmarshalling for\n> the PDUs on the domain socket. We should use some lightweight encoding\n> library like XDR and rpcgen or similar.)\n>\n>\n> Please test, comment and have fun.\n\nOne enhancement to this could be to make it easier to use for trivial\nusers that do not want to share one database across multiple repos.\n\nWe could have something like\n$ git init --db=tdb .\n\ncore.db=tdb could then mean that instead of connecting to a domain\nsocket for a shared daemon instead it would just\npopen(\"refsd-<core.db> ...\", \"rw\")\nand automaticallt set the asguments ro refsd-tdb to point to the local\n.git directory.\n\nThis would make the setup simpler for the trivial usecase since it\nwould avoid having to manuallt initializing a dedicated daemon before\nthe git commands will start working.\n"}]}