{"thread":{"id":"37294","subject":"Pluggable backends for refs,wip","startedAt":"2014-08-05T21:40:50Z","lastAt":"2014-08-08T15:53:51Z","messageCount":6,"participants":["Ronnie Sahlberg","Nico Williams","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"247294","messageId":"CAL=YDWk5C+bQS76_+jx1BSYSxecCxaS95RG181ecbJBLCNWLkQ@mail.gmail.com","threadId":"37294","inReplyTo":null,"subject":"Pluggable backends for refs,wip","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-08-05T21:40:50Z","receivedAt":"2014-08-05T21:40:50Z","isPatch":false,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"List, Michael,\n\nPlease see\nhttps://github.com/rsahlberg/git/tree/backend-struct-db-2\nfor an example of a pluggable backend for refs storage.\n\nThis series contain changes to make it possible to add new backends\nfor handling/storage of refs and implements one new backend :\nrefs-be-be.c .\n\nThis new backend offloads the actual refs handling to a small database\ndaemon with which ita talks via a very simple rpc protocol. That\ndaemon in turn then connects to the datastore and read/writes the\nvalues to it.\n\nBy having an always running database daemon it will allow faster\nstartup of the git commands since they will now only need to connect\nto a domain socket instead of having to traverse a potentially very\nlarge number of files during the \"build ref cache\" phase.\nAnother nice feature is that it can allow running one single database\ndaemon and use it to host the refs for multiple independent git\nrepositoris (by using the new repository name config to distinguish\nbetween them).\n\nIt can not yet apply to origin/* since it is based on some small\nseries that have yet not arrived there\nand is still a wip. But if you want to test/look at what we could be\ndoing one day, please feel free to clone this repo.\n\n\nFAQ:\nQ:\nThis sound cool. How do I test this?\n\nA:\n1, Clone https://github.com/rsahlberg/git/tree/backend-struct-db-2 and\nbuild git.\n\n2, gcc refsd-tdb.c -o refsd-tdb -l tdb\n3, ./refsd-tdb /tmp/refsd.socket /tmp /tmp/refsd.log\n\n4, git clone --db-repo-name=ROCKet --db-socket=/tmp/refsd.socket <some-repo> foo\n\n./foo should now contain a git repository that store its refs in a\nseparate database.\n(teh databases are store under /tmp  so don't use this for anything\nimportant because bad things happens to things stored under /tmp)\n\n\nregards\nronnie sahlberg\n"},{"id":"247295","messageId":"CAK3OfOiu39H8oZMKjra97ZNLHNatNYtFwr8Do_QwoxGXzg4SGw@mail.gmail.com","threadId":"37294","inReplyTo":"CAL=YDWk5C+bQS76_+jx1BSYSxecCxaS95RG181ecbJBLCNWLkQ@mail.gmail.com","subject":"Re: Pluggable backends for refs,wip","fromName":"Nico Williams","fromEmail":"nico@cryptonector.com","sentAt":"2014-08-05T21:56:42Z","receivedAt":"2014-08-05T21:56:42Z","isPatch":false,"sender":{"key":"nico@cryptonector.com","avatar":null},"body":"Personally (a user of, not a maintainer of, git) I really want some\nalternative backends.  In particular I'm after something like Fossil's\nuse of SQLite3; I want a SQLite3 backend for several reasons, not the\nleast of which is the power of SQL for looking at history.\n\nI'm not sure that I necessarily want a daemon/background process.  I\nget the appeal (add inotify and bingo, very fast git status, always),\nbut it seems likely to add obnoxious failure modes.\n\nAs to a SQLite3-type backend, I am of two minds: either add it as a\nbolt-on to the builtin backend, or add it as a first-class backend\nthat replaces the builtin one.  The former is nice because the SQLite3\nDB becomes more of a cache/index and query engine than a store, and\ncan be used without migrating any repos, but the latter is also nice\nbecause SQLite3 provides strong ACID transactional semantics on local\nfilesystems.\n\nNico\n--\n"},{"id":"247301","messageId":"CAL=YDWnYQJVTkCW0GYE6qOK9a+VBiO6ozUOPWkGNEr8H0-4a4w@mail.gmail.com","threadId":"37294","inReplyTo":"CAK3OfOiu39H8oZMKjra97ZNLHNatNYtFwr8Do_QwoxGXzg4SGw@mail.gmail.com","subject":"Re: Pluggable backends for refs,wip","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-08-05T22:23:55Z","receivedAt":"2014-08-05T22:23:55Z","isPatch":false,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Tue, Aug 5, 2014 at 2:56 PM, Nico Williams <nico@cryptonector.com> wrote:\n> Personally (a user of, not a maintainer of, git) I really want some\n> alternative backends.  In particular I'm after something like Fossil's\n> use of SQLite3; I want a SQLite3 backend for several reasons, not the\n> least of which is the power of SQL for looking at history.\n>\n> I'm not sure that I necessarily want a daemon/background process.  I\n> get the appeal (add inotify and bingo, very fast git status, always),\n> but it seems likely to add obnoxious failure modes.\n>\n> As to a SQLite3-type backend, I am of two minds: either add it as a\n> bolt-on to the builtin backend, or add it as a first-class backend\n> that replaces the builtin one.  The former is nice because the SQLite3\n> DB becomes more of a cache/index and query engine than a store, and\n> can be used without migrating any repos, but the latter is also nice\n> because SQLite3 provides strong ACID transactional semantics on local\n> filesystems.\n\n\nThis will allow you to do either or both, depending on what you want.\n\nI am adding one new first-class backend to talk to a separate daemon :\n  refs-be-db.c\nwhich then talks to a separate daemon   refsd-tdb.c\n\nrefsd-tdb.c is 7 RPCs and ~500 lines of code for a naive\nimplementation for a standalone separate daemon implementation.\n\n\nIf you rather want want a new first-class backend builtin to git\nitself instead of as a separate daemon, then that will be possible\ntoo.\nIt just means that you will have to base the work on refs-be-db.c\nwhich is a much larger and complex code base than refsd-tdb.c.\n\nBut yeah, once this work is finished, you will be able to build new\nfirst-class ref backends if you so wish.\nPlease see refs-be-db.c  that is the file and the methods you will\nneed to implement in order to have a first-class SQL* backend.\n\n\nregards\nronnie sahlberg\n"},{"id":"247302","messageId":"CAK3OfOhZ1ni+u2beM3yRRhy3jC9WRTFPhJ9ArzVxv56jew04iA@mail.gmail.com","threadId":"37294","inReplyTo":"CAL=YDWnYQJVTkCW0GYE6qOK9a+VBiO6ozUOPWkGNEr8H0-4a4w@mail.gmail.com","subject":"Re: Pluggable backends for refs,wip","fromName":"Nico Williams","fromEmail":"nico@cryptonector.com","sentAt":"2014-08-05T22:28:30Z","receivedAt":"2014-08-05T22:28:30Z","isPatch":false,"sender":{"key":"nico@cryptonector.com","avatar":null},"body":"Excellent.  Thanks!\n"},{"id":"247422","messageId":"53E377AD.606@alum.mit.edu","threadId":"37294","inReplyTo":"CAL=YDWk5C+bQS76_+jx1BSYSxecCxaS95RG181ecbJBLCNWLkQ@mail.gmail.com","subject":"Re: Pluggable backends for refs,wip","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2014-08-07T12:57:17Z","receivedAt":"2014-08-07T12:57:17Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 08/05/2014 02:40 PM, Ronnie Sahlberg wrote:\n> Please see\n> https://github.com/rsahlberg/git/tree/backend-struct-db-2\n> for an example of a pluggable backend for refs storage.\n> \n> This series contain changes to make it possible to add new backends\n> for handling/storage of refs and implements one new backend :\n> refs-be-be.c .\n> \n> This new backend offloads the actual refs handling to a small database\n> daemon with which ita talks via a very simple rpc protocol. That\n> daemon in turn then connects to the datastore and read/writes the\n> values to it.\n> [...]\n\nRonnie,\n\nThis is awesome!  Congratulations on your progress.\n\nI'm still on vacation and haven't yet looked at the code.  I will be\nback next week and hope to find time to check it out, and also to do\nsome more review of the code that you have already submitted to git core.\n\n\nHave you thought about how to test alternate reference backends?  This\nwill be very important to getting one or more of them accepted into git\ncore (not to mention giving people confidence to actually *use* them!)\n\nIt seems to me that a few steps are needed:\n\n* Each backend would need a suite of backend-aware tests that verify\nproper operation *within* the backend.  These tests would mostly use\nlow-level plumbing commands like update-refs to create/modify/delete\nreferences, and would be allowed to grub around in the filesystem, talk\ndirectly with the database, etc. to make sure that the commands have the\ncorrect effects.  For example, for the traditional filesystem backend,\nthese tests would be the ones to check that creating a reference causes\na file to spring into existence under $GIT_DIR/refs.\n\nThe tests for pack-refs, and all tests that care about the distinction\nbetween packed and loose refs, would become part of the backend-aware\ntests for the filesystem backend.\n\nAll of the backend-aware tests should be run every time the test suite\nis run (provided, of course, that the correct prerequisites are\navailable, and subject to being turned off manually).\n\n* The rest of the test suite has to be made backend-agnostic.  For\nexample, such tests should *not* be allowed to look under $GIT_DIR for\nthe existence/absence of loose reference files [1] but would rather have\nto inquire about references via git commands.\n\n* It should be possible for the developer to choose easily which\nreference backend to use when running the agnostic part of the test\nsuite.  The chosen backend should be used to run *all* backend-agnostic\ntests.\n\nA database-backed backend might even want to be testable in two modes:\none with the DB daemon running constantly, and one where the daemon is\nstopped and started between each pair of Git commands.\n\nSo after the changes, a single run of the test suite should run the\nbackend-aware tests for *all* known backends followed by the\nbackend-agnostic tests for a single selected backend.\n\nMichael\n\n[1] When I was working on my quagga-reference spike [2] I found that a\nlot of the test suite uses knowledge about how references and reflogs\nare stored by the filesystem backend and just grabs at the files rather\nthan accessing the references using git commands.  It will take some\nwork to clean this up.\n\n[2] http://thread.gmane.org/gmane.comp.version-control.git/243726\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"247479","messageId":"CAL=YDWm5DAnjg_7Y+xUW3YVo5o83GFxdhMYTf4tAFPbRNEUX3w@mail.gmail.com","threadId":"37294","inReplyTo":"53E377AD.606@alum.mit.edu","subject":"Re: Pluggable backends for refs,wip","fromName":"Ronnie Sahlberg","fromEmail":"sahlberg@google.com","sentAt":"2014-08-08T15:53:51Z","receivedAt":"2014-08-08T15:53:51Z","isPatch":false,"sender":{"key":"sahlberg@google.com","avatar":"https://avatars.githubusercontent.com/u/7320636?v=4"},"body":"On Thu, Aug 7, 2014 at 5:57 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:\n> On 08/05/2014 02:40 PM, Ronnie Sahlberg wrote:\n>> Please see\n>> https://github.com/rsahlberg/git/tree/backend-struct-db-2\n>> for an example of a pluggable backend for refs storage.\n>>\n>> This series contain changes to make it possible to add new backends\n>> for handling/storage of refs and implements one new backend :\n>> refs-be-be.c .\n>>\n>> This new backend offloads the actual refs handling to a small database\n>> daemon with which ita talks via a very simple rpc protocol. That\n>> daemon in turn then connects to the datastore and read/writes the\n>> values to it.\n>> [...]\n>\n> Ronnie,\n>\n> This is awesome!  Congratulations on your progress.\n>\n> I'm still on vacation and haven't yet looked at the code.  I will be\n> back next week and hope to find time to check it out, and also to do\n> some more review of the code that you have already submitted to git core.\n\nThanks!\n\n>\n>\n> Have you thought about how to test alternate reference backends?  This\n> will be very important to getting one or more of them accepted into git\n> core (not to mention giving people confidence to actually *use* them!)\n\nI have thought about it and also done some experiments.\nFor the initial git support, I think we first should try to get the\npluggable backend support\ninto git, and also the work to change the current files backend into a\nbuilt-in pluggable backend.\n\nI.e. get everything in the\nhttps://github.com/rsahlberg/git/tree/backend-struct-db-2\nbranch except the last three patches.\nThat brings us to a stage where we have pluggable backend support and\nwe have one backend, the files backend, that works just like today.\n\nThe last three patches in that series are then just confirmation that\nthe pluggable backend approach works and we can add that a little\nlater once we finish tests and other things.\n\n\n\nFor tests there are the issues with \"git-clone\" and \"git-init\"\nrequiring two additional arguments in order to set up and initialize a\nrepository to use the \"database daemon backend\".\nOther future backends I would imagine would have similar needs.\nThe way I handle in the experiments I did was to use two new\nenvironment variables GIT_INIT and GIT_CLONE that would default to\n\"git-clone\" and \"git-init\" respectively\nand then just override them with GIT_INIT=\"git-init\n--db-repo-name=ROCKy --db-socket=/tmp/refsd.socket\" when I wanted the\ntests to initialize a \"database backend\" repository.\nThis required some updates to test-lib.sh and test-lib-functions.sh as\nwell as the tests themself to use ${GIT_INIT} instead of git-init\ndirectly.\n\nI am not sure what is the best approach here is and would love if you\ncould help out with this once we get the basic pluggable backend stuff\nin.\n\n\n\n>\n> It seems to me that a few steps are needed:\n>\n> * Each backend would need a suite of backend-aware tests that verify\n> proper operation *within* the backend.  These tests would mostly use\n> low-level plumbing commands like update-refs to create/modify/delete\n> references, and would be allowed to grub around in the filesystem, talk\n> directly with the database, etc. to make sure that the commands have the\n> correct effects.  For example, for the traditional filesystem backend,\n> these tests would be the ones to check that creating a reference causes\n> a file to spring into existence under $GIT_DIR/refs.\n\nYes.\nQuite a few tests do muck around with the files directly. Some for\ngood reasons but I think there are a lot of cases where the tests do\nit just out of convenience.\n\nFor this we will need to convert the tests that don't strictly need to\nmuck around with the files to use a backend agnostic method to do the\nsame checks.\nFor the tests that are truly testing the backend itself, such as a\nhypothetical test to check that a symbolic link to a ref behaves as it\nshould, we will need a mechanism where we can conditionalize the tests\nbased on what is the current backend.\nSo lots of \"if backend == database then skip this test\"\n\n\n>\n> The tests for pack-refs, and all tests that care about the distinction\n> between packed and loose refs, would become part of the backend-aware\n> tests for the filesystem backend.\n>\n> All of the backend-aware tests should be run every time the test suite\n> is run (provided, of course, that the correct prerequisites are\n> available, and subject to being turned off manually).\n>\n> * The rest of the test suite has to be made backend-agnostic.  For\n> example, such tests should *not* be allowed to look under $GIT_DIR for\n> the existence/absence of loose reference files [1] but would rather have\n> to inquire about references via git commands.\n>\n> * It should be possible for the developer to choose easily which\n> reference backend to use when running the agnostic part of the test\n> suite.  The chosen backend should be used to run *all* backend-agnostic\n> tests.\n>\n\nAgree.\nIt would be great if we could work on this together.\n\n\n> A database-backed backend might even want to be testable in two modes:\n> one with the DB daemon running constantly, and one where the daemon is\n> stopped and started between each pair of Git commands.\n>\n> So after the changes, a single run of the test suite should run the\n> backend-aware tests for *all* known backends followed by the\n> backend-agnostic tests for a single selected backend.\n\nACK.\n\n>\n> Michael\n>\n> [1] When I was working on my quagga-reference spike [2] I found that a\n> lot of the test suite uses knowledge about how references and reflogs\n> are stored by the filesystem backend and just grabs at the files rather\n> than accessing the references using git commands.  It will take some\n> work to clean this up.\n>\n> [2] http://thread.gmane.org/gmane.comp.version-control.git/243726\n>\n> --\n> Michael Haggerty\n> mhagger@alum.mit.edu\n>\n"}]}