{"thread":{"id":"56395","subject":"Exec upload-pack on remote with what parameters to get direntries.","startedAt":"2021-08-28T12:56:30Z","lastAt":"2021-08-31T16:23:15Z","messageCount":12,"participants":["Stef Bon","Jeff King","Junio C Hamano","Ævar Arnfjörð Bjarmason","Bruno Albuquerque"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"434018","messageId":"CANXojcyWnFY60bXG6MDS9WAYkcFQHf+Oef0VREBkvgsuX9e=Kg@mail.gmail.com","threadId":"56395","inReplyTo":null,"subject":"Exec upload-pack on remote with what parameters to get direntries.","fromName":"Stef Bon","fromEmail":"stefbon@gmail.com","sentAt":"2021-08-28T12:56:17Z","receivedAt":"2021-08-28T12:56:30Z","isPatch":false,"sender":{"key":"stefbon@gmail.com","avatar":null},"body":"Hi,\n\nI've got a custom ssh library which I use to make a connection to a\ngit server like www.github.com, user stefbon.\n\nNow I want to get the direntries of a remote repo, and I know I have\nto use upload-pack for that, but with what parameters?\n\nI want to use the outcome to make a fuse fs, user can browse the\nfiles. Possibly the user can also view the contents.\n\nStef\n"},{"id":"434147","messageId":"YS0tNoAa/0VQe1OW@coredump.intra.peff.net","threadId":"56395","inReplyTo":"CANXojcyWnFY60bXG6MDS9WAYkcFQHf+Oef0VREBkvgsuX9e=Kg@mail.gmail.com","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-30T19:10:46Z","receivedAt":"2021-08-30T19:10:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 28, 2021 at 02:56:17PM +0200, Stef Bon wrote:\n\n> I've got a custom ssh library which I use to make a connection to a\n> git server like www.github.com, user stefbon.\n> \n> Now I want to get the direntries of a remote repo, and I know I have\n> to use upload-pack for that, but with what parameters?\n> \n> I want to use the outcome to make a fuse fs, user can browse the\n> files. Possibly the user can also view the contents.\n\nThe protocol used by upload-pack is described in\nDocumentation/technical/pack-protocol.txt, but in short: I don't think\nit will do what you want.\n\nThere is no operation to list the tree contents, for example, nor really\neven a good way to fetch a single object. The protocol is geared around\nefficiently transferring slices of history, so it is looking at sets of\nreachable objects (what the client is asking for, and what it claims to\nhave).\n\nYou might be able to cobble something together with shallow and partial\nfetches. E.g., something like:\n\n  git clone --depth 1 --filter=blob:none --single-branch -b $branch\n\nis basically asking to send only a single commit, plus all of its trees,\nbut no blobs. From there you could parse the tree objects to assemble a\ndirectory listing. Possibly with a tree:depth filter you could even do\nit iteratively.\n\nSome hosts offer a separate API that would give you a much nicer\ninterface. E.g., GitHub has:\n\n  https://docs.github.com/en/rest/reference/git#trees\n\nBut of course that won't work with GitLab, etc, and you'd have to\nimplement against the API for each hosting provider.\n\n-Peff\n"},{"id":"434149","messageId":"xmqq35qq4t1x.fsf@gitster.g","threadId":"56395","inReplyTo":"YS0tNoAa/0VQe1OW@coredump.intra.peff.net","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-30T19:43:38Z","receivedAt":"2021-08-30T19:43:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> There is no operation to list the tree contents, for example, nor really\n> even a good way to fetch a single object. The protocol is geared around\n> efficiently transferring slices of history, so it is looking at sets of\n> reachable objects (what the client is asking for, and what it claims to\n> have).\n>\n> You might be able to cobble something together with shallow and partial\n> fetches. E.g., something like:\n>\n>   git clone --depth 1 --filter=blob:none --single-branch -b $branch\n\nI was hoping that our support for fetching a single object (not\nnecessarily a commit) at the protocol level was good enough, so that\nStef's fuse/nfs daemon can fetch the tree object it is interested\nin.\n\nThere also is an effort, slowly moving to add verbs like object-info\nto the protocol to help the vfs usecase, but primitives at too low a\nlevel would be killed by latency, so it is somewhat unknown how\neffective it would be.\n\n\n"},{"id":"434162","messageId":"YS1DnGTDo5ZG8Iw4@coredump.intra.peff.net","threadId":"56395","inReplyTo":"xmqq35qq4t1x.fsf@gitster.g","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-30T20:46:20Z","receivedAt":"2021-08-30T20:46:23Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 30, 2021 at 12:43:38PM -0700, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> > There is no operation to list the tree contents, for example, nor really\n> > even a good way to fetch a single object. The protocol is geared around\n> > efficiently transferring slices of history, so it is looking at sets of\n> > reachable objects (what the client is asking for, and what it claims to\n> > have).\n> >\n> > You might be able to cobble something together with shallow and partial\n> > fetches. E.g., something like:\n> >\n> >   git clone --depth 1 --filter=blob:none --single-branch -b $branch\n> \n> I was hoping that our support for fetching a single object (not\n> necessarily a commit) at the protocol level was good enough, so that\n> Stef's fuse/nfs daemon can fetch the tree object it is interested\n> in.\n\nI don't think there's a clean way to ask for a single object. But\nthinking on it more, I suspect you could do something _really_ hacky\nusing the new object-type filters:\n\n  git fetch --filter=object:type=commit --filter=object:type=blob\n\nBecause we AND the filters together, no object can satisfy both. But\nbecause we also send any objects which were _explicitly_ requested by\nthe client, you can now fetch whatever single objects you want.\n\nAnd as long as you tell the other side you don't have any objects, it\nwon't send any deltas.\n\n> There also is an effort, slowly moving to add verbs like object-info\n> to the protocol to help the vfs usecase, but primitives at too low a\n> level would be killed by latency, so it is somewhat unknown how\n> effective it would be.\n\nYes. At GitHub we actually have a custom endpoint which hooks up\n\"cat-file --batch\" with a format of the client's choosing. That's what\n(indirectly) feeds things like raw.github.com.\n\nI've been tempted to send it upstream, but it's pretty ugly, and does\ngive the client a lot of power (for now, the placeholders you can use\nwith cat-file are not that powerful, but if we start to unify with\nref-filter, etc, then we run into situations like we had with\n%(describe) recently). Likewise, the v2 object-info endpoint _could_\naccept arbitrary format strings (it's the same idea, just with\n--batch-check instead of --batch).\n\n-Peff\n"},{"id":"434178","messageId":"xmqq4kb639xt.fsf@gitster.g","threadId":"56395","inReplyTo":"YS1DnGTDo5ZG8Iw4@coredump.intra.peff.net","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-30T21:21:50Z","receivedAt":"2021-08-30T21:21:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Yes. At GitHub we actually have a custom endpoint which hooks up\n> \"cat-file --batch\" with a format of the client's choosing. That's what\n> (indirectly) feeds things like raw.github.com.\n>\n> I've been tempted to send it upstream, but it's pretty ugly, and does\n> give the client a lot of power (for now, the placeholders you can use\n> with cat-file are not that powerful, but if we start to unify with\n> ref-filter, etc, then we run into situations like we had with\n> %(describe) recently). Likewise, the v2 object-info endpoint _could_\n> accept arbitrary format strings (it's the same idea, just with\n> --batch-check instead of --batch).\n\nYeah, the object-info actually was from folks who are interested in\ndoing something similar, and it would be nice if we can share the\nprotocol endpoint that is more suitable for interactive tree and\nhistory traversal to help those who want to do virtual filesystem.\n\nThanks.\n\n"},{"id":"434241","messageId":"CANXojczR1hMrzz7t0P6AkqL3kjdk+NzBKyCQnm-9cWFbULifow@mail.gmail.com","threadId":"56395","inReplyTo":"YS0tNoAa/0VQe1OW@coredump.intra.peff.net","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Stef Bon","fromEmail":"stefbon@gmail.com","sentAt":"2021-08-31T06:38:39Z","receivedAt":"2021-08-31T06:38:52Z","isPatch":false,"sender":{"key":"stefbon@gmail.com","avatar":null},"body":"Hi,\n\nthank you for the answer.\n\nI understand that the core of git is to make people work together when\nwriting code.\nTo get a tree of the source files is not directly part of that, but\npure informational. That is also the intent of my fuse fs: provide the\nuser information about the source files.\n\nNow I have a working ssh connection to the server, and open a channel\nfor running the upload-pack on the server using the exec channel\nrequest:\n\nhttps://datatracker.ietf.org/doc/html/rfc4254#section-6.5\n\nSo in my program I do not have to do something like:\n\nssh -x git@server \"git-upload-pack 'simplegit-progit.git'\"\n\nIt is only the sending of an exec message with the right command.\nVia the SSH_MSG_CHANNEL_DATA message the server will return the\noutput. In my program I have to write a parser to get the\ntree/direntries.\n\nNow you suggest the git clone --depth 1 --filter=blob:none\n--single-branch -b $branch\ncommand. How does that look when writing it in lowlevel git messages\nas described in\n\nhttps://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols\n\n?\nI'm programming at this low level, so I have to write the messages to\nsend to the server myself.\n\nAnd you mention the api github has for a git tree object. But git2 has\nalready the git_tree object?\n\nStef\n\nMy project  by the way is: https://github.com/stefbon/OSNS\n"},{"id":"434244","messageId":"YS3VLh8SFvpDZy84@coredump.intra.peff.net","threadId":"56395","inReplyTo":"CANXojczR1hMrzz7t0P6AkqL3kjdk+NzBKyCQnm-9cWFbULifow@mail.gmail.com","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-31T07:07:26Z","receivedAt":"2021-08-31T07:07:30Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 31, 2021 at 08:38:39AM +0200, Stef Bon wrote:\n\n> So in my program I do not have to do something like:\n> \n> ssh -x git@server \"git-upload-pack 'simplegit-progit.git'\"\n> \n> It is only the sending of an exec message with the right command.\n> Via the SSH_MSG_CHANNEL_DATA message the server will return the\n> output. In my program I have to write a parser to get the\n> tree/direntries.\n> \n> Now you suggest the git clone --depth 1 --filter=blob:none\n> --single-branch -b $branch\n> command. How does that look when writing it in lowlevel git messages\n> as described in\n> \n> https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols\n> \n> ?\n> I'm programming at this low level, so I have to write the messages to\n> send to the server myself.\n\nYou'll have to read the documentation I pointed to earlier:\n\n  https://github.com/git/git/blob/master/Documentation/technical/pack-protocol.txt\n\nIn short: the server tells you which refs it has and what they point to,\nthen the client says which objects it wants and which objects it has,\nand then the server sends a packfile. The flow of the protocol and the\nformat of the messages is laid out there.\n\nYou might also set GIT_TRACE_PACKET=1 in your environment and try\nrunning some Git commands. They will show you what's being said on the\nwire, up until the packfile is sent (decoding the packfile itself is a\nwhole other story).\n\n> And you mention the api github has for a git tree object. But git2 has\n> already the git_tree object?\n\nIf you mean libgit2, then yes, it has a git_tree struct. Just like we\nhave internally within regular Git. But those are for accessing _local_\nobjects, that have already been fetched.\n\nYou could build a fuse filesystem around a local Git repository pretty\neasily, either by using libgit2 or around tools like \"git ls-tree\" and\n\"git cat-file\". But if your purpose is to access a remote one without\ndownloading all of the objects first, then no, Git does not expose any\nof the endpoints you'd need remotely (but provider-specific APIs like\nGitHub's do).\n\n-Peff\n"},{"id":"434256","messageId":"CANXojcxF8V2RR=xMLrwcpwa=R8fvhsn2Wj=pnthXNnvxX7YLxQ@mail.gmail.com","threadId":"56395","inReplyTo":"YS3VLh8SFvpDZy84@coredump.intra.peff.net","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Stef Bon","fromEmail":"stefbon@gmail.com","sentAt":"2021-08-31T09:44:43Z","receivedAt":"2021-08-31T09:44:56Z","isPatch":false,"sender":{"key":"stefbon@gmail.com","avatar":null},"body":"Op di 31 aug. 2021 om 09:07 schreef Jeff King <peff@peff.net>:\n>\n> On Tue, Aug 31, 2021 at 08:38:39AM +0200, Stef Bon wrote:\n>\n\n> You might also set GIT_TRACE_PACKET=1 in your environment and try\n> running some Git commands. They will show you what's being said on the\n> wire, up until the packfile is sent (decoding the packfile itself is a\n> whole other story).\n>\n\nYes that will give me the insight I need.\nI will come back when it comes to decoding the packfile.\n\nThanks,\nStef\n"},{"id":"434283","messageId":"87v93lwv7m.fsf@evledraar.gmail.com","threadId":"56395","inReplyTo":"CANXojcxF8V2RR=xMLrwcpwa=R8fvhsn2Wj=pnthXNnvxX7YLxQ@mail.gmail.com","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T14:01:58Z","receivedAt":"2021-08-31T14:21:53Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 31 2021, Stef Bon wrote:\n\n> Op di 31 aug. 2021 om 09:07 schreef Jeff King <peff@peff.net>:\n>>\n>> On Tue, Aug 31, 2021 at 08:38:39AM +0200, Stef Bon wrote:\n>>\n>\n>> You might also set GIT_TRACE_PACKET=1 in your environment and try\n>> running some Git commands. They will show you what's being said on the\n>> wire, up until the packfile is sent (decoding the packfile itself is a\n>> whole other story).\n>>\n>\n> Yes that will give me the insight I need.\n> I will come back when it comes to decoding the packfile.\n\nAside from the \"here's how you can do it\", you haven't said why you'd\nlike to do such \"online\" browsing of the repository.\n\nI'd think that even for something that e.g. implements a file browser\nwith magic git-remote support (think GNOME VFS-like), what you'd want to\ndo in the background would be to do a \"clone\", although a clone with\nsome combination of --single-branch, --no-tags, and perhaps --depth and\nthe filters discussed upthread.\n\nIt will take the same time to get the pack, but once you do you can use\nlibgit2, git's plumbing etc. to do really fast browsing/wildcarding\netc. of the entries locally.\n\nSo is there a real performance or other use-case for wanting to do this,\nor does it just come down a lack of nice a \"one-shot\" API for \"list\nremote files?\".\n\nIn any case, on the topic of clever things you can (ab)use to do this,\nsome remotes support running \"git archive\" for you. Notably GitHub\ndoesn't, but GitLab does. Please don't take this as an endorsement to\nrun this command \"in production\"\n\n    $ time (git archive --format=tar  --remote=git@gitlab.com:git-vcs/git.git --prefix=t/t4018/ HEAD:t/t4018 | tar -tf- | head -n 3)\n    t/t4018/\n    t/t4018/README\n    t/t4018/bash-arithmetic-function\n\n    real    0m1.545s\n\nI idly wonder if there's a want/need for a file listing API whether\ndoing so via the tar/zip format wouldn't be a more viable & widely\nsupported thing than expecting everyone to come up with their own git\npackfile decoders. I.e. if we just supported some option to create\nall-empty dummy files via \"git archive\" this could be even better as a\ndummy file listing API. Right now this (ab)use of it requires\ne.g. sending ~10MB of t/'s content just to list everything in the t/\ndirectory.\n\n"},{"id":"434285","messageId":"87sfypwuwx.fsf@evledraar.gmail.com","threadId":"56395","inReplyTo":"xmqq4kb639xt.fsf@gitster.g","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-31T14:23:34Z","receivedAt":"2021-08-31T14:28:26Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 30 2021, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n>\n>> Yes. At GitHub we actually have a custom endpoint which hooks up\n>> \"cat-file --batch\" with a format of the client's choosing. That's what\n>> (indirectly) feeds things like raw.github.com.\n>>\n>> I've been tempted to send it upstream, but it's pretty ugly, and does\n>> give the client a lot of power (for now, the placeholders you can use\n>> with cat-file are not that powerful, but if we start to unify with\n>> ref-filter, etc, then we run into situations like we had with\n>> %(describe) recently). Likewise, the v2 object-info endpoint _could_\n>> accept arbitrary format strings (it's the same idea, just with\n>> --batch-check instead of --batch).\n>\n> Yeah, the object-info actually was from folks who are interested in\n> doing something similar, and it would be nice if we can share the\n> protocol endpoint that is more suitable for interactive tree and\n> history traversal to help those who want to do virtual filesystem.\n\nWhile this is all clever, I think this discussion really suggests that\nthe first thing we should do is make the relatively recent \"object-info\"\nprotocol verb not a default part of the supported v2 protocol we ship in\ngit.git.\n\nI.e. someone setting up a git server probably isn't going to suspect\nthat one day their server load is going to go up by some big % because\nsome developer somewhere is using a local IDE whose every file click on\na directory is a new remote server request (i.e. the case where\n\"object-info\"'s functionality is expanded like this).\n\nI found myself wondering this when reading serve.c the other day,\ni.e. why we have \"always_advertise\" for object-info, but it seemed\ninnocuous enough given how it's described in a2ba162cda2 (object-info:\nsupport for retrieving object info, 2021-04-20).\n\nBut just as a general thing, while I'm very much in favor of git growing\n*optional* support for more server<->client cooperation and CPU\noffloading, even things like \"git grep\" or \"git log\" optimistically\nrunning server-side, I think those sorts of features should definitely\nbe off by default for the reasons noted above.\n"},{"id":"434292","messageId":"CAPeR6H7_V+ypzyN39e27+NCRqG-nA17sgq=qtefzFF3Dg50bnA@mail.gmail.com","threadId":"56395","inReplyTo":"87sfypwuwx.fsf@evledraar.gmail.com","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Bruno Albuquerque","fromEmail":"bga@google.com","sentAt":"2021-08-31T15:35:22Z","receivedAt":"2021-08-31T15:35:45Z","isPatch":false,"sender":{"key":"bga@google.com","avatar":"https://avatars.githubusercontent.com/u/80971974?v=4"},"body":"On Tue, Aug 31, 2021 at 7:28 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n\n[Replying again as I used HTML mail by mistake. Sorry.]\n\n> I.e. someone setting up a git server probably isn't going to suspect\n> that one day their server load is going to go up by some big % because\n> some developer somewhere is using a local IDE whose every file click on\n> a directory is a new remote server request (i.e. the case where\n> \"object-info\"'s functionality is expanded like this).\n\nDo you mean by someone directly sending object-info requests? I am\nworking on wiring object-info to some of the existing tools\n(cat-file/ls-tree) so this is the general idea about how I see this\nbeing used:\n\n- object-info would be used when it made sense but only if the actual\nobject being queried is not already fetched locally. If you think of a\nvirtual filesystem that is backed by, say, partial clones, this mostly\nmeans retrieving metadata information to be displayed to the user.\n- Still in the context of a virtual filesystem, metadata is usually\ncached locally independently of Git itself, further reducing the need\nto call object-info (but, of course, this is a brittle assumption as\nit is not controlled by Git).\n- git cat-file, for example, would be changed to support real batching\nand then send a single request instead of the multiple requests it\ndoes currently.\n\nMy point is that I understand where your worry is coming from and as\nlong as someone can send arbitrary requests then it is possible your\nscenario of a heavier server load can potentially happen but as far as\nthe expected canonical usage, I do not think this would be a problem\nand, in fact, under some usage patterns it might make things better\n(mostly due to batching support in object-info).\n\nWith all that being said, I don' t think making it optional would be\nan issue so I have no strong feelings about this. I am fine with\nwhatever is agreed to be the best approach.\n\n> I found myself wondering this when reading serve.c the other day,\n> i.e. why we have \"always_advertise\" for object-info, but it seemed\n> innocuous enough given how it's described in a2ba162cda2 (object-info:\n> support for retrieving object info, 2021-04-20).\nFor what it is worth, The same change is now being reviewed in JGit\nand there the feature is conditionally enabled. But that was a\nside-effect of needing to deploy it to multiple servers before making\nthe feature available to clients.\n\n--\n\nBruno Albuquerque | Software Engineer | bga@google.com | +1 650-395-8242\n"},{"id":"434298","messageId":"xmqqv93lwplf.fsf@gitster.g","threadId":"56395","inReplyTo":"CAPeR6H7_V+ypzyN39e27+NCRqG-nA17sgq=qtefzFF3Dg50bnA@mail.gmail.com","subject":"Re: Exec upload-pack on remote with what parameters to get direntries.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-31T16:23:08Z","receivedAt":"2021-08-31T16:23:15Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Bruno Albuquerque <bga@google.com> writes:\n\n> With all that being said, I don' t think making it optional would be\n> an issue so I have no strong feelings about this. I am fine with\n> whatever is agreed to be the best approach.\n>\n>> I found myself wondering this when reading serve.c the other day,\n>> i.e. why we have \"always_advertise\" for object-info, but it seemed\n>> innocuous enough given how it's described in a2ba162cda2 (object-info:\n>> support for retrieving object info, 2021-04-20).\n> For what it is worth, The same change is now being reviewed in JGit\n> and there the feature is conditionally enabled. But that was a\n> side-effect of needing to deploy it to multiple servers before making\n> the feature available to clients.\n\nFWIW, I do not mind, and probably prefer if I think about it a bit\nlonger, to make it an opt-in feature, like all other capabilities\ndefined in the serve.c file.\n\nThanks.\n\n"}]}