{"thread":{"id":"55040","subject":"ls-refs protocol extension","startedAt":"2021-01-24T12:41:14Z","lastAt":"2021-01-26T04:59:38Z","messageCount":3,"participants":["Christian Schilling","Taylor Blau"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"415156","messageId":"CAJsFKUCuvO7Y7-MJZ-GG9Jv+4-ei=-tvx4PF1PbeU61m+CV9RA@mail.gmail.com","threadId":"55040","inReplyTo":null,"subject":"ls-refs protocol extension","fromName":"Christian Schilling","fromEmail":"christian.schilling.de@gmail.com","sentAt":"2021-01-24T12:40:08Z","receivedAt":"2021-01-24T12:41:14Z","isPatch":false,"sender":{"key":"christian.schilling.de@gmail.com","avatar":null},"body":"Hi list,\nI have been working on a custom git server where the sha values\nrequired to respond to a ls-refs command can occasionally be expensive\nto compute.\nIt would be a great improvement of user experience if it was possible\nto show progress to the user while this is happening.\nI was hoping that this could be solved by using the already existing\nsideband-all capability, but unfortunately it seems that even with\nsideband-all the sideband is not available during ls-refs.\nSo my plan would be to start working on a patch to add sideband\nsupport to ls-refs, unless there is\nsome other protocol feature that I have overlooked and that would\nenable my use case.\n\nAny pointers to either alternative solutions and/or hints to potential\npitfalls in the implementation\nof my suggestion would be greatly appreciated.\n\nBest regards,\nChristian Schilling\n"},{"id":"415187","messageId":"YA4ZOOccXK0YEOWu@nand.local","threadId":"55040","inReplyTo":"CAJsFKUCuvO7Y7-MJZ-GG9Jv+4-ei=-tvx4PF1PbeU61m+CV9RA@mail.gmail.com","subject":"Re: ls-refs protocol extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T01:04:56Z","receivedAt":"2021-01-25T01:07:08Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jan 24, 2021 at 01:40:08PM +0100, Christian Schilling wrote:\n> Hi list,\n> I have been working on a custom git server where the sha values\n> required to respond to a ls-refs command can occasionally be expensive\n> to compute.\n\nAlmost certainly the expensive part of ls-refs is actually traversing\nthe references, not parsing the objects they point at and determining\ntheir hash.\n\nIncidentally, we had a discussion recently [1] that resulted in some\npatches that make it so that ls-refs often only has to read through part\nof the refs in your repository, not all of them.\n\n> It would be a great improvement of user experience if it was possible\n> to show progress to the user while this is happening.\n\nIt's possible that that might help, but honestly I'd be surprised if\nthere was a real use-case that needed it (especially after the patches\nthat I mentioned which should make it fast enough that you don't have to\ncare :-)).\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/20210119144251.27924-1-jacob@gitlab.com/\n"},{"id":"415264","messageId":"CAJsFKUBKoD-PvroY7F7=PtF6+KYszTgUD1mndH8vZmh-fSC90g@mail.gmail.com","threadId":"55040","inReplyTo":"YA4ZOOccXK0YEOWu@nand.local","subject":"Re: ls-refs protocol extension","fromName":"Christian Schilling","fromEmail":"christian.schilling.de@gmail.com","sentAt":"2021-01-25T08:44:15Z","receivedAt":"2021-01-26T04:59:38Z","isPatch":false,"sender":{"key":"christian.schilling.de@gmail.com","avatar":null},"body":"On Mon 25. Jan 2021 at 02:04, Taylor Blau <me@ttaylorr.com> wrote:\n>\n> On Sun, Jan 24, 2021 at 01:40:08PM +0100, Christian Schilling wrote:\n> > Hi list,\n> > I have been working on a custom git server where the sha values\n> > required to respond to a ls-refs command can occasionally be expensive\n> > to compute.\n>\n> Almost certainly the expensive part of ls-refs is actually traversing\n> the references, not parsing the objects they point at and determining\n> their hash.\n\nMy use case is very different. I’m working on a custom git server that\ncreates refs as well as the objects they are referring to on demand.\n\nMore specifically the references are derived from the repo url to\nimplement partial cloning.\nSo you can do:\ngit clone http://repository-url.git:/any/subdir/you/want.git\n\nThe server will do something similar to git-filter-branch and\ndeliver the result.\nDue to the way my own branch filtering implementation works this is\nusually fast enough for the users not to notice what is going on,\nbut the fact remains that the shas are only known after the filtering\nis complete.\nAlso my server acts as a proxy with the canonical backing repository\nhosted on a different machine.\nSo to start filtering the proxy has to first ask the upstream for the\ncurrent state of the repo and download\nany new changes. I have done a lot of optimizations to make\nall of this fast and in most cases it is, but for the occasional case,\nfor example if a completely new repo is cloned for the first time,\nit leaves the users looking at a seemingly “frozen” command prompt\nnot knowing if they made a mistake or are encountering some bug or\nnetwork problem.\n\nAt first I also thought about suggesting a protocol extension that allows\nto start transferring pack data before all refs are known to reduce the\ntotal time needed, but the slow cases are infrequent enough so that this\nprobably not worth it.\n\n> Incidentally, we had a discussion recently [1] that resulted in some\n> patches that make it so that ls-refs often only has to read through part\n> of the refs in your repository, not all of them.\n\nI had my own struggles with the “to many refs” problem, but this is\nunrelated :-)\n\n>\n> > It would be a great improvement of user experience if it was possible\n> > to show progress to the user while this is happening.\n>\n> It's possible that that might help, but honestly I'd be surprised if\n> there was a real use-case that needed it (especially after the patches\n> that I mentioned which should make it fast enough that you don't have to\n> care :-)).\n\nSorry for not being clearer about what I am trying to accomplish the first time,\nI hope I convinced you that this is a very different, but real, use case.\n\nThanks,\nChristian\n\n>\n> Thanks,\n> Taylor\n>\n> [1]: https://lore.kernel.org/git/20210119144251.27924-1-jacob@gitlab.com/\n"}]}