{"thread":{"id":"61259","subject":"reftable & jgit compatibility","startedAt":"2024-04-03T10:36:15Z","lastAt":"2024-04-04T08:36:28Z","messageCount":11,"participants":["Han-Wen Nienhuys","Patrick Steinhardt","Luca Milanesio","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"492151","messageId":"CAOw_e7Z_10b73n91ihsaao_S-XPkNqvY7gTcHvqUODKD-SwPSA@mail.gmail.com","threadId":"61259","inReplyTo":null,"subject":"reftable & jgit compatibility","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2024-04-03T10:36:04Z","receivedAt":"2024-04-03T10:36:15Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"Thanks again for taking up this work.\n\nAs I'm browsing over your patches (and realizing how much of the\narcana of the format I've forgotten), I hope that I did not make any\nerrors in implementing the spec (and/or that Shawn didn't deviate his\nimplementation from the spec). It would be extremely unfortunate if an\nincompatibility between CGit and JGit were discovered after it is\nreleased.\n\nSo far I have always been able to read JGit reftables using the C / Go\ncode, but it would be good to systematically test this, ie. generate a\n bunch of tables using JGit and check that passing them through the C\ncode (read & write) leaves them unchanged. Or perhaps check in some\ntables as golden reference data.\n\nJosh can probably connect you to the right folks to help with this on\nthe JGit side.\n\n-- \nHan-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen\n"},{"id":"492152","messageId":"Zg0zs2_QLpXv2PwT@tanuki","threadId":"61259","inReplyTo":"CAOw_e7Z_10b73n91ihsaao_S-XPkNqvY7gTcHvqUODKD-SwPSA@mail.gmail.com","subject":"Re: reftable & jgit compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-03T10:47:15Z","receivedAt":"2024-04-03T10:47:20Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 03, 2024 at 12:36:04PM +0200, Han-Wen Nienhuys wrote:\n> Thanks again for taking up this work.\n> \n> As I'm browsing over your patches (and realizing how much of the\n> arcana of the format I've forgotten), I hope that I did not make any\n> errors in implementing the spec (and/or that Shawn didn't deviate his\n> implementation from the spec). It would be extremely unfortunate if an\n> incompatibility between CGit and JGit were discovered after it is\n> released.\n> \n> So far I have always been able to read JGit reftables using the C / Go\n> code, but it would be good to systematically test this, ie. generate a\n>  bunch of tables using JGit and check that passing them through the C\n> code (read & write) leaves them unchanged. Or perhaps check in some\n> tables as golden reference data.\n> \n> Josh can probably connect you to the right folks to help with this on\n> the JGit side.\n\nI very much agree, this thought has crossed my mind multiple times while\nworking on the whole reftable saga. Ideally, we would have integration\ntests that write reftables with one of the implementations and then read\nthem with the respective other implementation. I wouldn't really know\nwhere to put those though. CGit is very unlikely to pull in JGit as a\ntest dependency. Does JGit have any tests that already use CGit?\n\nAdding a bunch of reftables pre-generated by JGit might be an okayish\ntradeoff, I guess. I also don't really expect the format to evolve\nsignificantly, so these should be reasonably static over the long term.\n\nPatrick\n"},{"id":"492166","messageId":"64970945-5E37-4792-9F37-790CFD82A1BF@gmail.com","threadId":"61259","inReplyTo":"CAOw_e7Z_10b73n91ihsaao_S-XPkNqvY7gTcHvqUODKD-SwPSA@mail.gmail.com","subject":"Re: reftable & jgit compatibility","fromName":"Luca Milanesio","fromEmail":"luca.milanesio@gmail.com","sentAt":"2024-04-03T15:51:04Z","receivedAt":"2024-04-03T15:51:17Z","isPatch":false,"sender":{"key":"luca.milanesio@gmail.com","avatar":"https://gravatar.com/avatar/64e45570bb8baaba9a566592150e6c7341a300e314501a944cb95b0bc1380f49?d=mp&s=160"},"body":"Hi Han-Wen,\nThanks for completing the ref-table on JGit and kicking off the work on CGit.\n\n> On 3 Apr 2024, at 11:36, Han-Wen Nienhuys <hanwenn@gmail.com> wrote:\n> \n> Thanks again for taking up this work.\n> \n> As I'm browsing over your patches (and realizing how much of the\n> arcana of the format I've forgotten), I hope that I did not make any\n> errors in implementing the spec (and/or that Shawn didn't deviate his\n> implementation from the spec). It would be extremely unfortunate if an\n> incompatibility between CGit and JGit were discovered after it is\n> released.\n> \n> So far I have always been able to read JGit reftables using the C / Go\n> code, but it would be good to systematically test this, ie. generate a\n> bunch of tables using JGit and check that passing them through the C\n> code (read & write) leaves them unchanged. Or perhaps check in some\n> tables as golden reference data.\n> Josh can probably connect you to the right folks to help with this on\n> the JGit side.\n\nI am happy to experiment the support on GerritHub.io, we have over 40k repositories !\n\nLuca.\n\n\n"},{"id":"492167","messageId":"CAOw_e7Y_MwgrrJzuHk7tzBR9a2kDfTnwCzC-7_rgj8UJPKqp9g@mail.gmail.com","threadId":"61259","inReplyTo":"Zg0zs2_QLpXv2PwT@tanuki","subject":"Re: reftable & jgit compatibility","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2024-04-03T15:57:19Z","receivedAt":"2024-04-03T15:57:31Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"On Wed, Apr 3, 2024 at 4:41 PM Patrick Steinhardt <ps@pks.im> wrote:\n>\n> On Wed, Apr 03, 2024 at 12:36:04PM +0200, Han-Wen Nienhuys wrote:\n> > Thanks again for taking up this work.\n> >\n> > As I'm browsing over your patches (and realizing how much of the\n> > arcana of the format I've forgotten), I hope that I did not make any\n> > errors in implementing the spec (and/or that Shawn didn't deviate his\n> > implementation from the spec). It would be extremely unfortunate if an\n> > incompatibility between CGit and JGit were discovered after it is\n> > released.\n> >\n> > So far I have always been able to read JGit reftables using the C / Go\n> > code, but it would be good to systematically test this, ie. generate a\n> >  bunch of tables using JGit and check that passing them through the C\n> > code (read & write) leaves them unchanged. Or perhaps check in some\n> > tables as golden reference data.\n> >\n> > Josh can probably connect you to the right folks to help with this on\n> > the JGit side.\n>\n> I very much agree, this thought has crossed my mind multiple times while\n> working on the whole reftable saga. Ideally, we would have integration\n> tests that write reftables with one of the implementations and then read\n> them with the respective other implementation. I wouldn't really know\n> where to put those though. CGit is very unlikely to pull in JGit as a\n> test dependency. Does JGit have any tests that already use CGit?\n\nYes, but not many (eg. CGitIgnoreTest.java).\n\nI think the easiest way to make this happen is if CGit would ship a\ncommand to dump a raw reftable in a release soonish. Then JGit could\nuse that command to cross-check that a JGit-written reftable can be\nread correctly by the CGit code.  By shipping just the dumper you\navoid having to wait for proper reftable support to land in git.\n\nProbably the dumper should be extended to also support seeks, so you\ncan also exercise the indexing/searching code.\n\n-- \nHan-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen\n"},{"id":"492173","messageId":"xmqqjzle8kj4.fsf@gitster.g","threadId":"61259","inReplyTo":"64970945-5E37-4792-9F37-790CFD82A1BF@gmail.com","subject":"Re: reftable & jgit compatibility","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-03T16:42:55Z","receivedAt":"2024-04-03T16:42:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Luca Milanesio <luca.milanesio@gmail.com> writes:\n\n> Hi Han-Wen,\n> Thanks for completing the ref-table on JGit and kicking off the work on CGit.\n> ...\n>> So far I have always been able to read JGit reftables using the C / Go\n>> code, but it would be good to systematically test this, ie. generate a\n>> bunch of tables using JGit and check that passing them through the C\n>> code (read & write) leaves them unchanged. Or perhaps check in some\n>> tables as golden reference data.\n> ...\n> I am happy to experiment the support on GerritHub.io, we have over 40k repositories !\n\nThanks.\n"},{"id":"492189","messageId":"20240403205451.GD1949464@coredump.intra.peff.net","threadId":"61259","inReplyTo":"Zg0zs2_QLpXv2PwT@tanuki","subject":"Re: reftable & jgit compatibility","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-03T20:54:51Z","receivedAt":"2024-04-03T20:54:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 03, 2024 at 12:47:15PM +0200, Patrick Steinhardt wrote:\n\n> I very much agree, this thought has crossed my mind multiple times while\n> working on the whole reftable saga. Ideally, we would have integration\n> tests that write reftables with one of the implementations and then read\n> them with the respective other implementation. I wouldn't really know\n> where to put those though. CGit is very unlikely to pull in JGit as a\n> test dependency. Does JGit have any tests that already use CGit?\n\nWe do have some tests that use jgit to check bitmap interoperability.\nBut obviously they're optional, and I suspect they are not run very\noften (I do have jgit in my path these days, so I run them, but I assume\nmost people don't). It probably wouldn't be too hard to include it in\none of the CI runs, though. You can grep for the JGIT prereq in t/.\n\nWe had another test that used jgit to check for some protocol\ninteroperability. But it was broken with sha256 and nobody noticed. ;)\nThere I replaced it with a hard-coded input. See 13e67aa39b (v0\nprotocol: fix sha1/sha256 confusion for capabilities^{}, 2023-04-14) for\nsome discussion.\n\nI think using actual jgit (versus a hard-coded input) is a good basic\nsmoke test: it tells us if the two can interoperate generally. But for\ntesting specific inputs like the case in 13e67aa39b, we are depending on\njgit producing that specific behavior (which in this case, it probably\nwasn't any more). And there we are better off just with a manual test\nvector.\n\n-Peff\n"},{"id":"492216","messageId":"Zg5HXZrL_4BsyzfG@tanuki","threadId":"61259","inReplyTo":"CAOw_e7Y_MwgrrJzuHk7tzBR9a2kDfTnwCzC-7_rgj8UJPKqp9g@mail.gmail.com","subject":"Re: reftable & jgit compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T06:23:25Z","receivedAt":"2024-04-04T06:23:30Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 03, 2024 at 05:57:19PM +0200, Han-Wen Nienhuys wrote:\n> On Wed, Apr 3, 2024 at 4:41 PM Patrick Steinhardt <ps@pks.im> wrote:\n> >\n> > On Wed, Apr 03, 2024 at 12:36:04PM +0200, Han-Wen Nienhuys wrote:\n> > > Thanks again for taking up this work.\n> > >\n> > > As I'm browsing over your patches (and realizing how much of the\n> > > arcana of the format I've forgotten), I hope that I did not make any\n> > > errors in implementing the spec (and/or that Shawn didn't deviate his\n> > > implementation from the spec). It would be extremely unfortunate if an\n> > > incompatibility between CGit and JGit were discovered after it is\n> > > released.\n> > >\n> > > So far I have always been able to read JGit reftables using the C / Go\n> > > code, but it would be good to systematically test this, ie. generate a\n> > >  bunch of tables using JGit and check that passing them through the C\n> > > code (read & write) leaves them unchanged. Or perhaps check in some\n> > > tables as golden reference data.\n> > >\n> > > Josh can probably connect you to the right folks to help with this on\n> > > the JGit side.\n> >\n> > I very much agree, this thought has crossed my mind multiple times while\n> > working on the whole reftable saga. Ideally, we would have integration\n> > tests that write reftables with one of the implementations and then read\n> > them with the respective other implementation. I wouldn't really know\n> > where to put those though. CGit is very unlikely to pull in JGit as a\n> > test dependency. Does JGit have any tests that already use CGit?\n> \n> Yes, but not many (eg. CGitIgnoreTest.java).\n> \n> I think the easiest way to make this happen is if CGit would ship a\n> command to dump a raw reftable in a release soonish. Then JGit could\n> use that command to cross-check that a JGit-written reftable can be\n> read correctly by the CGit code.  By shipping just the dumper you\n> avoid having to wait for proper reftable support to land in git.\n\nYou do realize that \"proper reftable support\" has already landed, right?\nSo you can just use Git to create a reftable-enabled repository, write\ncommits and then use JGit to access the whole repository instead of only\nchecking a single table.\n\nMight be I'm missing your point though, not sure.\n\nPatrick\n\n> Probably the dumper should be extended to also support seeks, so you\n> can also exercise the indexing/searching code.\n\n\n"},{"id":"492217","messageId":"Zg5IrZIg-DOuf5nr@tanuki","threadId":"61259","inReplyTo":"20240403205451.GD1949464@coredump.intra.peff.net","subject":"Re: reftable & jgit compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T06:29:01Z","receivedAt":"2024-04-04T06:29:06Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 03, 2024 at 04:54:51PM -0400, Jeff King wrote:\n> On Wed, Apr 03, 2024 at 12:47:15PM +0200, Patrick Steinhardt wrote:\n> \n> > I very much agree, this thought has crossed my mind multiple times while\n> > working on the whole reftable saga. Ideally, we would have integration\n> > tests that write reftables with one of the implementations and then read\n> > them with the respective other implementation. I wouldn't really know\n> > where to put those though. CGit is very unlikely to pull in JGit as a\n> > test dependency. Does JGit have any tests that already use CGit?\n> \n> We do have some tests that use jgit to check bitmap interoperability.\n> But obviously they're optional, and I suspect they are not run very\n> often (I do have jgit in my path these days, so I run them, but I assume\n> most people don't). It probably wouldn't be too hard to include it in\n> one of the CI runs, though. You can grep for the JGIT prereq in t/.\n\nOh, that's great, I didn't know about that! I will take a look at\nupdating our CI systems to include JGit...\n\n> We had another test that used jgit to check for some protocol\n> interoperability. But it was broken with sha256 and nobody noticed. ;)\n> There I replaced it with a hard-coded input. See 13e67aa39b (v0\n> protocol: fix sha1/sha256 confusion for capabilities^{}, 2023-04-14) for\n> some discussion.\n\n... also to avoid rotting tests like this.\n\n> I think using actual jgit (versus a hard-coded input) is a good basic\n> smoke test: it tells us if the two can interoperate generally. But for\n> testing specific inputs like the case in 13e67aa39b, we are depending on\n> jgit producing that specific behavior (which in this case, it probably\n> wasn't any more). And there we are better off just with a manual test\n> vector.\n\nAgreed. I will add some basic interop tests that ensure that JGit and\nCGit can read their respective formats. I don't want it to be too fancy\ninitially, but it's good to have a baseline which we can iterate from in\nthe future.\n\nPatrick\n"},{"id":"492218","messageId":"CAOw_e7Zzc2uLX0FtkJ3fB+wuNJt5piMmoYVes+ayApe8BEN3+g@mail.gmail.com","threadId":"61259","inReplyTo":"Zg5HXZrL_4BsyzfG@tanuki","subject":"Re: reftable & jgit compatibility","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2024-04-04T06:44:44Z","receivedAt":"2024-04-04T06:44:56Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"On Thu, Apr 4, 2024 at 8:23 AM Patrick Steinhardt <ps@pks.im> wrote:\n\n> > I think the easiest way to make this happen is if CGit would ship a\n> > command to dump a raw reftable in a release soonish. Then JGit could\n> > use that command to cross-check that a JGit-written reftable can be\n> > read correctly by the CGit code.  By shipping just the dumper you\n> > avoid having to wait for proper reftable support to land in git.\n>\n> You do realize that \"proper reftable support\" has already landed, right?\n\nI had not realized this, and that's great news!\n\n> So you can just use Git to create a reftable-enabled repository, write\n> commits and then use JGit to access the whole repository instead of only\n> checking a single table.\n\nFor testing, it's probably easier if you can work in terms of\nindividual tables (because that is where the complexity lies:\ndifferent blocksizes, restart frequencies, with index, without index,\nwith reflog, without reflog etc.), but one can create controlled\nindividual tables by creating a whole repo and then compacting it.\nOTOH, this would necessitate exposing all writer options to the git\nCLI, which is maybe a bit much.\n\n-- \nHan-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen\n"},{"id":"492222","messageId":"Zg5Up_kSvOmQODO3@tanuki","threadId":"61259","inReplyTo":"CAOw_e7Zzc2uLX0FtkJ3fB+wuNJt5piMmoYVes+ayApe8BEN3+g@mail.gmail.com","subject":"Re: reftable & jgit compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T07:20:07Z","receivedAt":"2024-04-04T07:20:14Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 04, 2024 at 08:44:44AM +0200, Han-Wen Nienhuys wrote:\n> On Thu, Apr 4, 2024 at 8:23 AM Patrick Steinhardt <ps@pks.im> wrote:\n> \n> > > I think the easiest way to make this happen is if CGit would ship a\n> > > command to dump a raw reftable in a release soonish. Then JGit could\n> > > use that command to cross-check that a JGit-written reftable can be\n> > > read correctly by the CGit code.  By shipping just the dumper you\n> > > avoid having to wait for proper reftable support to land in git.\n> >\n> > You do realize that \"proper reftable support\" has already landed, right?\n> \n> I had not realized this, and that's great news!\n> \n> > So you can just use Git to create a reftable-enabled repository, write\n> > commits and then use JGit to access the whole repository instead of only\n> > checking a single table.\n> \n> For testing, it's probably easier if you can work in terms of\n> individual tables (because that is where the complexity lies:\n> different blocksizes, restart frequencies, with index, without index,\n> with reflog, without reflog etc.), but one can create controlled\n> individual tables by creating a whole repo and then compacting it.\n> OTOH, this would necessitate exposing all writer options to the git\n> CLI, which is maybe a bit much.\n\nPotentially, yeah. But as you say, it's likely quite some complexity to\nexpose this via the CLI directly. So for now, I'm going to focus on some\nbasic interoperability tests in Git that act on the repository level. We\ncan build on that and expand them as required when the need arises.\n\nDifferent blocksizes is definitely a bit of a sore spot right now. I do\nplan to expose write options via Git config options in the future, e.g.\nsomething like \"reftable.blockSize\" or \"reftable.restartCount\". But for\nall I know the CGit reftable library doesn't yet play nice with block\nsizes other than 4k.\n\nI didn't yet want to introduce configs which are specific to reftables\nin the first release of Git with the reftable backend, so I pushed this\nissue further down. I do plan to work on that in the next release cycle\nthough.\n\nPatrick\n"},{"id":"492226","messageId":"CAOw_e7b-zTZVGW_u6_ZY-CboHGOSWHbW7mxnqmxh+uuQa-VsTw@mail.gmail.com","threadId":"61259","inReplyTo":"Zg5Up_kSvOmQODO3@tanuki","subject":"Re: reftable & jgit compatibility","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2024-04-04T08:36:16Z","receivedAt":"2024-04-04T08:36:28Z","isPatch":false,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"On Thu, Apr 4, 2024 at 9:20 AM Patrick Steinhardt <ps@pks.im> wrote:\n> Different blocksizes is definitely a bit of a sore spot right now. I do\n> plan to expose write options via Git config options in the future, e.g.\n> something like \"reftable.blockSize\" or \"reftable.restartCount\". But for\n> all I know the CGit reftable library doesn't yet play nice with block\n> sizes other than 4k.\n\nIt shouldn't be too bad. Many unittests use blocksizes other than 4k\nsimply because populating a multi-block table takes less space at\nsmaller blocksizes.\n\n-- \nHan-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen\n"}]}