{"thread":{"id":"48441","subject":"Implementing reftable in Git","startedAt":"2018-05-09T14:33:22Z","lastAt":"2018-05-11T22:21:40Z","messageCount":13,"participants":["Christian Couder","Derrick Stolee","Duy Nguyen","Jonathan Nieder","Stefan Beller","Carlos Martín Nieto","Ævar Arnfjörð Bjarmason","Michael Haggerty","David Turner"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"347081","messageId":"CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com","threadId":"48441","inReplyTo":null,"subject":"Implementing reftable in Git","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-05-09T14:33:17Z","receivedAt":"2018-05-09T14:33:22Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nI might start working on implementing reftable in Git soon.\n\nDuring the last Git Merge conference last March Stefan talked about\nreftable. In Alex Vandiver's notes [1] it is asked that people\nannounce it on the list when they start working on it, and it appears\nthat there is a reference implementation in JGit.\n\nLooking it up, there is indeed some documentation [2], code [3], tests\n[4] and other related stuff [5] in the JGit repo. It looks like the\nJGit repo and the reftable code there are licensed under the Eclipse\nDistribution License - v 1.0 [7] which is very similar to the 3-Clause\nBSD License also called Modified BSD License which is GPL compatible\naccording to gnu.org [9]. So from a quick look it appears that I\nshould be able to port the JGit to Git if I just keep the copyright\nand license header comments in all the related files.\n\nSo I think the most straightforward and compatible way to do it would\nbe to port the JGit implementation.\n\nThanks in advance for any suggestion or comment about this.\n\nReftable was first described by Shawn and then discussed last July on\nthe list [6].\n\nMy work on this would be sponsored by Booking.com.\n\nThanks,\nChristian.\n\n[1] https://public-inbox.org/git/alpine.DEB.2.20.1803091557510.23109@alexmv-linux/\n\n[2] https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md\n\n[3] https://github.com/eclipse/jgit/tree/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/reftable\n\n[4] https://github.com/eclipse/jgit/tree/master/org.eclipse.jgit.test/tst/org/eclipse/jgit/internal/storage/reftable\n\n[5] https://github.com/eclipse/jgit/tree/master/org.eclipse.jgit.pgm/src/org/eclipse/jgit/pgm/debug\n\n[6] https://public-inbox.org/git/CAJo=hJtyof=HRy=2sLP0ng0uZ4=S-DpZ5dR1aF+VHVETKG20OQ@mail.gmail.com/\n\n[7] http://www.eclipse.org/org/documents/edl-v10.php\n\n[8] https://opensource.org/licenses/BSD-3-Clause\n\n[9] https://www.gnu.org/licenses/license-list.en.html#ModifiedBSD\n"},{"id":"347085","messageId":"059b8daf-c990-aad5-90a4-5ec38c42b7b3@gmail.com","threadId":"48441","inReplyTo":"CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com","subject":"Re: Implementing reftable in Git","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-09T14:52:33Z","receivedAt":"2018-05-09T14:52:39Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/9/2018 10:33 AM, Christian Couder wrote:\n> Hi,\n>\n> I might start working on implementing reftable in Git soon.\n>\n> During the last Git Merge conference last March Stefan talked about\n> reftable. In Alex Vandiver's notes [1] it is asked that people\n> announce it on the list when they start working on it, and it appears\n> that there is a reference implementation in JGit.\n\nThanks for starting on this! In addition to the performance gains, this \nwill help a lot of users with case-insensitive file systems from getting \ncase-errors on refnames.\n\n> Looking it up, there is indeed some documentation [2], code [3], tests\n> [4] and other related stuff [5] in the JGit repo. It looks like the\n> JGit repo and the reftable code there are licensed under the Eclipse\n> Distribution License - v 1.0 [7] which is very similar to the 3-Clause\n> BSD License also called Modified BSD License which is GPL compatible\n> according to gnu.org [9]. So from a quick look it appears that I\n> should be able to port the JGit to Git if I just keep the copyright\n> and license header comments in all the related files.\n>\n> So I think the most straightforward and compatible way to do it would\n> be to port the JGit implementation.\n>\n> Thanks in advance for any suggestion or comment about this.\n>\n> Reftable was first described by Shawn and then discussed last July on\n> the list [6].\n\nThe hope is that such a direct port should be possible, but someone else \nshould comment on the porting process.\n\nThis is also something that could be created independently based on the \ndocumentation you mention. I was planning to attempt that during a \nhackathon in July, but I'm happy you are able to start earlier (and that \nyou are announcing your intentions). I would be happy to review your \npatch series, so please keep me posted.\n\nThanks,\n-Stolee\n"},{"id":"347093","messageId":"CACsJy8De1U6FbdMi4yF_AF2OYGrhF8qLO1ZAJ1PK37p8yv0m0g@mail.gmail.com","threadId":"48441","inReplyTo":"CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com","subject":"Re: Implementing reftable in Git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-09T16:07:38Z","receivedAt":"2018-05-09T16:08:12Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, May 9, 2018 at 4:33 PM, Christian Couder\n wrote:\n> Hi,\n>\n> I might start working on implementing reftable in Git soon.\n\nAdding Michael Haggerty who did lots of work on ref stuff. He probably\ncan give a few suggestions.\n\nYou probably should also look at the last attempt to add lmdb as a new\nref backend. I'm not sure why it's still not in, maybe it wasn't the\nright time (e.g. infrastructure was not ready).\n\n> During the last Git Merge conference last March Stefan talked about\n> reftable. In Alex Vandiver's notes [1] it is asked that people\n> announce it on the list when they start working on it, and it appears\n> that there is a reference implementation in JGit.\n>\n> Looking it up, there is indeed some documentation [2], code [3], tests\n> [4] and other related stuff [5] in the JGit repo. It looks like the\n> JGit repo and the reftable code there are licensed under the Eclipse\n> Distribution License - v 1.0 [7] which is very similar to the 3-Clause\n> BSD License also called Modified BSD License which is GPL compatible\n> according to gnu.org [9]. So from a quick look it appears that I\n> should be able to port the JGit to Git if I just keep the copyright\n> and license header comments in all the related files.\n>\n> So I think the most straightforward and compatible way to do it would\n> be to port the JGit implementation.\n>\n> Thanks in advance for any suggestion or comment about this.\n>\n> Reftable was first described by Shawn and then discussed last July on\n> the list [6].\n>\n> My work on this would be sponsored by Booking.com.\n>\n> Thanks,\n> Christian.\n>\n> [1] https://public-inbox.org/git/alpine.DEB.2.20.1803091557510.23109@alexmv-linux/\n>\n> [2] https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md\n>\n> [3] https://github.com/eclipse/jgit/tree/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/reftable\n>\n> [4] https://github.com/eclipse/jgit/tree/master/org.eclipse.jgit.test/tst/org/eclipse/jgit/internal/storage/reftable\n>\n> [5] https://github.com/eclipse/jgit/tree/master/org.eclipse.jgit.pgm/src/org/eclipse/jgit/pgm/debug\n>\n> [6] https://public-inbox.org/git/CAJo=hJtyof=HRy=2sLP0ng0uZ4=S-DpZ5dR1aF+VHVETKG20OQ@mail.gmail.com/\n>\n> [7] http://www.eclipse.org/org/documents/edl-v10.php\n>\n> [8] https://opensource.org/licenses/BSD-3-Clause\n>\n> [9] https://www.gnu.org/licenses/license-list.en.html#ModifiedBSD\n-- \nDuy\n"},{"id":"347098","messageId":"20180509164807.GI10348@aiede.svl.corp.google.com","threadId":"48441","inReplyTo":"CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com","subject":"Re: Implementing reftable in Git","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-05-09T16:48:07Z","receivedAt":"2018-05-09T16:48:14Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nChristian Couder wrote:\n\n> I might start working on implementing reftable in Git soon.\n\nYay!\n\n[...]\n> So I think the most straightforward and compatible way to do it would\n> be to port the JGit implementation.\n\nI suspect following the spec[1] would be even more compatible, since it\nwould force us to tighten the spec where it is unclear.\n\n> It looks like the\n> JGit repo and the reftable code there are licensed under the Eclipse\n> Distribution License - v 1.0 [7] which is very similar to the 3-Clause\n> BSD License also called Modified BSD License\n\nIf you would like the patches at https://git.eclipse.org/r/q/topic:reftable\nrelicensed for Git's use so that you don't need to include that\nlicense header, let me know. Separate from any legal concerns, if\nyou're doing a straight port, a one-line comment crediting the JGit\nproject would still be appreciated, of course.\n\nThat said, I would not be surprised if going straight from the spec is\neasier than porting the code.\n\nThanks,\nJonathan\n\n[1] https://eclipse.googlesource.com/jgit/jgit/+/master/Documentation/technical/reftable.md\n"},{"id":"347111","messageId":"CAGZ79kZx=wHKc=2WLz-8pQWv1VhRq+pKVV9=Shq3gEMdkX-Q=A@mail.gmail.com","threadId":"48441","inReplyTo":"CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com","subject":"Re: Implementing reftable in Git","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-09T17:42:49Z","receivedAt":"2018-05-09T17:42:53Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Hi Christian,\n\nOn Wed, May 9, 2018 at 7:33 AM, Christian Couder\n wrote:\n> Hi,\n>\n> I might start working on implementing reftable in Git soon.\n\nCool! Everyone is waiting for it as they dream about the\nperformance and correctness benefits this brings.\n\nBenefits that I know of:\n* performance in repos with many refs\n* no capitalization issues on case insensitive FS\n* replay-ability of the last fetch (\"show the last reflog\n of any ref under refs/remote/origin\") is easier to do\n in a correct way. (This is one of my motivations to desire reftables)\n* We *might* be able to use reftables in negotiation later\n (\"client: Last I fetched, you said your latest transaction\n number was '5' with the hash over all refs to be ;\n server: ok, here are the refs and the pack, you're welcome\").\n\nWhy are you (or rather booking.com) interested in this?\n\n> During the last Git Merge conference last March Stefan talked about\n> reftable. In Alex Vandiver's notes [1] it is asked that people\n> announce it on the list when they start working on it,\n\nMostly because many parties want to see it implemnented\nand were not sure when they could start implementing it.\n\n> and it appears\n> that there is a reference implementation in JGit.\n\nThe reference implementation can be used in tests\nto see if we can interact with them, using the JGIT pre-requisite.\n\n> Looking it up, there is indeed some documentation [2], code [3], tests\n> [4] and other related stuff [5] in the JGit repo. It looks like the\n> JGit repo and the reftable code there are licensed under the Eclipse\n> Distribution License - v 1.0 [7] which is very similar to the 3-Clause\n> BSD License also called Modified BSD License which is GPL compatible\n> according to gnu.org [9]. So from a quick look it appears that I\n> should be able to port the JGit to Git if I just keep the copyright\n> and license header comments in all the related files.\n>\n> So I think the most straightforward and compatible way to do it would\n> be to port the JGit implementation.\n\nI would think you can go by the spec and then test if it is compatible with\nJGit; that way the spec will be ironed out in corner cases.\n\n> Thanks in advance for any suggestion or comment about this.\n\nI volunteer for reviewing.\n\n(Advanced:) The spec allows for some tune-able parameters and JGits use\nis heavily optimized for the server side. I think git-core may need to have\nslightly different tweaks in different situations, e.g. block sizes and how\nmany restarts are put into the block.\nOn the FS we may want to have faster access at the cost of more disk space,\nwhereas in the future when using reftables on the wire as well for ref\nadvertisement we may want to opt for smallest tables. (largest blocks,\nno restarts)\n\nWith that said, please implement it in a way that it can not just be used as\na refs backend, but can easily be re-used to write ref advertisements\nonto the wire?\n\nThanks,\nStefan\n"},{"id":"347112","messageId":"20180509174830.GJ10348@aiede.svl.corp.google.com","threadId":"48441","inReplyTo":"CAGZ79kZx=wHKc=2WLz-8pQWv1VhRq+pKVV9=Shq3gEMdkX-Q=A@mail.gmail.com","subject":"Re: Implementing reftable in Git","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-05-09T17:48:30Z","receivedAt":"2018-05-09T17:48:37Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Stefan Beller wrote:\n\n> * We *might* be able to use reftables in negotiation later\n> (\"client: Last I fetched, you said your latest transaction\n> number was '5' with the hash over all refs to be ;\n> server: ok, here are the refs and the pack, you're welcome\").\n\nDo you mean that reftable's reflog layout makes this easier?\n\nIt's not clear to me why this wouldn't work with the current\nreflogs.\n\n[...]\n> On Wed, May 9, 2018 at 7:33 AM, Christian Couder\n> wrote:\n\n>> During the last Git Merge conference last March Stefan talked about\n>> reftable. In Alex Vandiver's notes [1] it is asked that people\n>> announce it on the list when they start working on it,\n>\n> Mostly because many parties want to see it implemnented\n> and were not sure when they could start implementing it.\n\nAnd to coordinate / help each other!\n\n[...]\n> I volunteer for reviewing.\n\n\\o/\n\n[...]\n> With that said, please implement it in a way that it can not just be used as\n> a refs backend, but can easily be re-used to write ref advertisements\n> onto the wire?\n\nCan you spell this out a little more for me? At first glance it's not\nobvious to me how knowing about this potential use would affect the\ninitial code.\n\nThanks,\nJonathan\n"},{"id":"347114","messageId":"67fd1816c4da0e54fb88dc29a44b897d41a36602.camel@dwim.me","threadId":"48441","inReplyTo":"20180509164807.GI10348@aiede.svl.corp.google.com","subject":"Re: Implementing reftable in Git","fromName":"Carlos Martín Nieto","fromEmail":"cmn@dwim.me","sentAt":"2018-05-09T17:51:01Z","receivedAt":"2018-05-09T17:51:11Z","isPatch":false,"sender":{"key":"cmn@dwim.me","avatar":"https://avatars.githubusercontent.com/u/335443?v=4"},"body":"Hi all,\n\nOn Wed, 2018-05-09 at 09:48 -0700, Jonathan Nieder wrote:\n> Hi,\n> \n> Christian Couder wrote:\n> \n> > I might start working on implementing reftable in Git soon.\n> \n> Yay!\n> \n> [...]\n> > So I think the most straightforward and compatible way to do it would\n> > be to port the JGit implementation.\n> \n> I suspect following the spec[1] would be even more compatible, since it\n> would force us to tighten the spec where it is unclear.\n> \n> > It looks like the\n> > JGit repo and the reftable code there are licensed under the Eclipse\n> > Distribution License - v 1.0 [7] which is very similar to the 3-Clause\n> > BSD License also called Modified BSD License\n> \n> If you would like the patches at https://git.eclipse.org/r/q/topic:reftable\n> relicensed for Git's use so that you don't need to include that\n> license header, let me know. Separate from any legal concerns, if\n> you're doing a straight port, a one-line comment crediting the JGit\n> project would still be appreciated, of course.\n> \n> That said, I would not be surprised if going straight from the spec is\n> easier than porting the code.\n\nWould you expect that this port would keep the Eclipse Distribution\nLicense or would it get relicensed to GPLv2?\n\nWe would also want to have reftable functionality in the libgit2\nproject, but it has a slightly different license from git (GPLv2 with\nlinking exception) which requires explicit consent from the authors for\nus to port over the code from git with its GPLv2 license.\n\nThe libgit2 project does have permission from Shawn to relicense his\ngit code, but this would presumably not cover this kind of porting. I\ndon't believe we would have issues if the code remained this BSD-like\nlicense.\n\nSorry for being difficult, but fewer distinct reimplementations is\nprobably a good thing overall.\n\ncc the core libgit2 team\n\nCheers,\n cmn\n\n"},{"id":"347115","messageId":"20180509175445.GK10348@aiede.svl.corp.google.com","threadId":"48441","inReplyTo":"67fd1816c4da0e54fb88dc29a44b897d41a36602.camel@dwim.me","subject":"Re: Implementing reftable in Git","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-05-09T17:54:45Z","receivedAt":"2018-05-09T17:54:52Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Carlos Martín Nieto wrote:\n> On Wed, 2018-05-09 at 09:48 -0700, Jonathan Nieder wrote:\n\n>> If you would like the patches at https://git.eclipse.org/r/q/topic:reftable\n>> relicensed for Git's use so that you don't need to include that\n>> license header, let me know. Separate from any legal concerns, if\n>> you're doing a straight port, a one-line comment crediting the JGit\n>> project would still be appreciated, of course.\n[...]\n> Would you expect that this port would keep the Eclipse Distribution\n> License or would it get relicensed to GPLv2?\n\nI think you're way overcomplicating things.\n\nThe patches are copyright Google. We can handle issues as they come.\n\nJonathan\n"},{"id":"347117","messageId":"CAGZ79kYwdTriaoev5EYvoSVA+ZummdKm3rjY261KucptjhytUQ@mail.gmail.com","threadId":"48441","inReplyTo":"20180509174830.GJ10348@aiede.svl.corp.google.com","subject":"Re: Implementing reftable in Git","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-09T17:55:11Z","receivedAt":"2018-05-09T17:55:15Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 9, 2018 at 10:48 AM, Jonathan Nieder wrote:\n> Stefan Beller wrote:\n>\n>> * We *might* be able to use reftables in negotiation later\n>> (\"client: Last I fetched, you said your latest transaction\n>> number was '5' with the hash over all refs to be ;\n>> server: ok, here are the refs and the pack, you're welcome\").\n>\n> Do you mean that reftable's reflog layout makes this easier?\n>\n> It's not clear to me why this wouldn't work with the current\n> reflogs.\n\nBecause of D/F conflicts we may not know all remote refs\n(and their ref logs), such that \"the hash over all refs\" on the remote\nis error prone to compute. Without transaction numbers it is also\ncumbersome for the server to remember the state.\nWe could try it based on the current refs, but I'd think\nit is not easy to do, whereas reftables bring some subtle\nadvantages that allow for such easier negotiation.\n\n>\n> [...]\n>> On Wed, May 9, 2018 at 7:33 AM, Christian Couder\n>> wrote:\n>\n>>> During the last Git Merge conference last March Stefan talked about\n>>> reftable. In Alex Vandiver's notes [1] it is asked that people\n>>> announce it on the list when they start working on it,\n>>\n>> Mostly because many parties want to see it implemnented\n>> and were not sure when they could start implementing it.\n>\n> And to coordinate / help each other!\n\nYes. Usually open source contributions are so sparse, that\njust doing it and then sending it to the mailing list does not\nproduce contention or conflict (double work), but this seemed\nlike a race condition waiting to happen. ;)\n\n>> With that said, please implement it in a way that it can not just be used as\n>> a refs backend, but can easily be re-used to write ref advertisements\n>> onto the wire?\n>\n> Can you spell this out a little more for me? At first glance it's not\n> obvious to me how knowing about this potential use would affect the\n> initial code.\n\nYeah me neither. I just want to make Christian aware of the potential\nuse cases, that come afterwards, so it can influence his design decisions\nfor the implementation.\n"},{"id":"347119","messageId":"e6f23463b1bd577d1c70848b11a0fd217b78f8a8.camel@dwim.me","threadId":"48441","inReplyTo":"20180509175445.GK10348@aiede.svl.corp.google.com","subject":"Re: Implementing reftable in Git","fromName":"Carlos Martín Nieto","fromEmail":"cmn@dwim.me","sentAt":"2018-05-09T18:05:41Z","receivedAt":"2018-05-09T18:05:47Z","isPatch":false,"sender":{"key":"cmn@dwim.me","avatar":"https://avatars.githubusercontent.com/u/335443?v=4"},"body":"On Wed, 2018-05-09 at 10:54 -0700, Jonathan Nieder wrote:\n> Carlos Martín Nieto wrote:\n> > On Wed, 2018-05-09 at 09:48 -0700, Jonathan Nieder wrote:\n> > > If you would like the patches at https://git.eclipse.org/r/q/topi\n> > > c:reftable\n> > > relicensed for Git's use so that you don't need to include that\n> > > license header, let me know. Separate from any legal concerns,\n> > > if\n> > > you're doing a straight port, a one-line comment crediting the\n> > > JGit\n> > > project would still be appreciated, of course.\n> \n> [...]\n> > Would you expect that this port would keep the Eclipse Distribution\n> > License or would it get relicensed to GPLv2?\n> \n> I think you're way overcomplicating things.\n> \n> The patches are copyright Google. We can handle issues as they come.\n\nFair enough. I just wanted to avoid coming back to this in a few months\nand realising we can't use it at all.\n\nCheers,\n cmn\n\n"},{"id":"347124","messageId":"874ljgy83h.fsf@evledraar.gmail.com","threadId":"48441","inReplyTo":"CAGZ79kZx=wHKc=2WLz-8pQWv1VhRq+pKVV9=Shq3gEMdkX-Q=A@mail.gmail.com","subject":"Re: Implementing reftable in Git","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-05-09T18:52:50Z","receivedAt":"2018-05-09T18:52:56Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 09 2018, Stefan Beller wrote:\n\n> Hi Christian,\n>\n> On Wed, May 9, 2018 at 7:33 AM, Christian Couder\n> wrote:\n>> Hi,\n>>\n>> I might start working on implementing reftable in Git soon.\n>\n> Cool! Everyone is waiting for it as they dream about the\n> performance and correctness benefits this brings.\n>\n> Benefits that I know of:\n> * performance in repos with many refs\n> * no capitalization issues on case insensitive FS\n> * replay-ability of the last fetch (\"show the last reflog\n> of any ref under refs/remote/origin\") is easier to do\n> in a correct way. (This is one of my motivations to desire reftables)\n> * We *might* be able to use reftables in negotiation later\n> (\"client: Last I fetched, you said your latest transaction\n> number was '5' with the hash over all refs to be ;\n> server: ok, here are the refs and the pack, you're welcome\").\n>\n> Why are you (or rather booking.com) interested in this?\n\nWe have a lot of refs, which is a longer-term scalability issue (which\nI've implemented hacks around (ref archiving)), and we also run into the\ncapitalization issues you mentioned.\n"},{"id":"347349","messageId":"CAMy9T_H=_+9Z=CpX85Ma4gCyUuvNAPR7fSBHi2J=4nC1XzF2sg@mail.gmail.com","threadId":"48441","inReplyTo":"CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com","subject":"Re: Implementing reftable in Git","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2018-05-11T09:31:57Z","receivedAt":"2018-05-11T09:32:05Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On Wed, May 9, 2018 at 4:33 PM, Christian Couder\n wrote:\n> I might start working on implementing reftable in Git soon.\n> [...]\n\nNice. It'll be great to have a reftable implementation in git core\n(and ideally libgit2, as well). It seems to me that it could someday\nbecome the new default reference storage method. The file format is\nconsiderably more complicated than the current loose/packed scheme,\nwhich is definitely a disadvantage (for example, for other Git\nimplementations). But implementing it *with good performance and\nwithout races* might be no more complicated than the current scheme.\n\nTesting will be important. There are already many tests specifically\nabout testing loose/packed reference storage. These will always have\nto run against repositories that are forced to use that reference\nscheme. And there will need to be new tests specifically about the\nreftable scheme. Both classes of tests should be run every time. That\nmuch is pretty obvious.\n\nBut currently, there are a lot of tests that assume the loose/packed\nreference format on disk even though the tests are not really related\nto references at all. ISTM that these should be converted to work at a\nhigher level, for example using `for-each-ref`, `rev-parse`, etc. to\nexamine references rather than reading reference files directly. That\nway the tests should run correctly regardless of which scheme is in\nuse.\n\nAnd since it's too expensive to run the whole test suite with both\nreference storage schemes, it seems to me that the reference storage\nscheme that is used while running the scheme-neutral tests should be\neasy to choose at runtime.\n\nDavid Turner did some analogous work for wiring up and testing his\nproposed LMDB ref storage backend that might be useful [1]. I'm CCing\nhim, since he might have thoughts on this topic.\n\nRegarding the reftable spec itself:\n\nI recently gave a little internal talk about it, and while preparing\nthe talk I noticed a couple of things that should maybe be tweaked:\n\n* The spec proposes to change `$GIT_DIR/refs`, which is currently a\ndirectory that holds the loose refs, into a file that holds the table\nof contents of reftable files comprising the full set of references.\nThis was my suggestion. I was thinking that this would prevent old\nrefs code from being used accidentally on a reftable-enabled\nrepository, while still enabling old versions of Git recognize this as\na git directory [2]. I think that the latter is important to make\nthings like `git rev-parse --git-dir` work correctly, even if the\ninstalled version of git can't actually *read* the repository.\n\n The problem is that `is_git_directory()` checks not only whether\n`$GIT_DIR/refs` exists, but also whether it is executable (i.e., since\nit is normally a directory, that it is searchable). It would be silly\nto make the reftable table of contents executable, so this doesn't\nseem like a good approach after all.\n\n So probably `$GIT_DIR/refs` should continue to be a directory. If\nit's there, it would probably make sense to place the reftable files\nand maybe the ToC inside of it. We would have to rely on older Git\nversions refusing to work in the directory because its `config` file\nhas an unrecognized `core.repositoryFormatVersion`, but that should be\nOK I think.\n\n* The scheme for naming reftable files [3] is, I believe, just a\nsuggestion as far as the spec is concerned (except for the use of\n`.ref`/`.log` file extensions). It might be more less unwieldy to use\n`%d` rather than `%08d`, and more convenient to name compacted files\nto `${min_update_index}-${max_update_index}_${n}.{ref,log}` to make it\nclearer to see by inspection what each file contains. That would also\nmake it unnecessary, in most cases, to insert a `_${n}` to make the\nfilename unique.\n\nMichael\n\n[1] https://github.com/dturner-tw/git/tree/dturner/pluggable-backends\n[2] https://github.com/git/git/blob/ccdcbd54c4475c2238b310f7113ab3075b5abc9c/setup.c#L309-L347\n[3] https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md#layout\n https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md#compaction\n[4] https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md#footer\n"},{"id":"347395","messageId":"1526077294.16035.33.camel@novalis.org","threadId":"48441","inReplyTo":"CAMy9T_H=_+9Z=CpX85Ma4gCyUuvNAPR7fSBHi2J=4nC1XzF2sg@mail.gmail.com","subject":"Re: Implementing reftable in Git","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2018-05-11T22:21:34Z","receivedAt":"2018-05-11T22:21:40Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2018-05-11 at 11:31 +0200, Michael Haggerty wrote:\n> On Wed, May 9, 2018 at 4:33 PM, Christian Couder\n> wrote:\n> > I might start working on implementing reftable in Git soon.\n> > [...]\n> \n> Nice. It'll be great to have a reftable implementation in git core\n> (and ideally libgit2, as well). It seems to me that it could someday\n> become the new default reference storage method. The file format is\n> considerably more complicated than the current loose/packed scheme,\n> which is definitely a disadvantage (for example, for other Git\n> implementations). But implementing it *with good performance and\n> without races* might be no more complicated than the current scheme.\n\nI am somewhat concerned about perf, because as I recall, we have a\nbunch of code which effectively load all refs, which will be more\nexpensive with reftable than packed-refs (though maybe cheaper than\nloose refs). But maybe we have eliminated this code or can work around\nit.\n\n> Testing will be important. There are already many tests specifically\n> about testing loose/packed reference storage. These will always have\n> to run against repositories that are forced to use that reference\n> scheme. And there will need to be new tests specifically about the\n> reftable scheme. Both classes of tests should be run every time. That\n> much is pretty obvious.\n> \n> But currently, there are a lot of tests that assume the loose/packed\n> reference format on disk even though the tests are not really related\n> to references at all. ISTM that these should be converted to work at\n> a\n> higher level, for example using `for-each-ref`, `rev-parse`, etc. to\n> examine references rather than reading reference files directly. That\n> way the tests should run correctly regardless of which scheme is in\n> use.\n\nI agree with that, and I think some of my patches from years ago\nattempted to do that. I probably should have broken those out into a\nseparate series so that they could have been applied separately.\n\n> And since it's too expensive to run the whole test suite with both\n> reference storage schemes, it seems to me that the reference storage\n> scheme that is used while running the scheme-neutral tests should be\n> easy to choose at runtime.\n\nI ran the whole suite with both schemes during my testing, and I think\nit was quite valuable in flushing out bugs.\n\n> David Turner did some analogous work for wiring up and testing his\n> proposed LMDB ref storage backend that might be useful [1]. I'm CCing\n> him, since he might have thoughts on this topic.\n\nInline, above.\n"}]}