Exec upload-pack on remote with what parameters to get direntries.

Hi,

I've got a custom ssh library which I use to make a connection to a git server like www.github.com, user stefbon.

Now I want to get the direntries of a remote repo, and I know I have to use upload-pack for that, but with what parameters?

I want to use the outcome to make a fuse fs, user can browse the files. Possibly the user can also view the contents.

Stef

On Sat, Aug 28, 2021 at 02:56:17PM +0200, Stef Bon wrote:

The protocol used by upload-pack is described in Documentation/technical/pack-protocol.txt, but in short: I don't think it will do what you want.

There is no operation to list the tree contents, for example, nor really even a good way to fetch a single object. The protocol is geared around efficiently transferring slices of history, so it is looking at sets of reachable objects (what the client is asking for, and what it claims to have).

You might be able to cobble something together with shallow and partial fetches. E.g., something like:

  git clone --depth 1 --filter=blob:none --single-branch -b $branch

is basically asking to send only a single commit, plus all of its trees, but no blobs. From there you could parse the tree objects to assemble a directory listing. Possibly with a tree:depth filter you could even do it iteratively.

Some hosts offer a separate API that would give you a much nicer interface. E.g., GitHub has:

  https://docs.github.com/en/rest/reference/git#trees

But of course that won't work with GitLab, etc, and you'd have to implement against the API for each hosting provider.

-Peff

Jeff King <peff@peff.net> writes:

I was hoping that our support for fetching a single object (not necessarily a commit) at the protocol level was good enough, so that Stef's fuse/nfs daemon can fetch the tree object it is interested in.

There also is an effort, slowly moving to add verbs like object-info to the protocol to help the vfs usecase, but primitives at too low a level would be killed by latency, so it is somewhat unknown how effective it would be.

On Mon, Aug 30, 2021 at 12:43:38PM -0700, Junio C Hamano wrote:

I don't think there's a clean way to ask for a single object. But thinking on it more, I suspect you could do something _really_ hacky using the new object-type filters:

  git fetch --filter=object:type=commit --filter=object:type=blob

Because we AND the filters together, no object can satisfy both. But because we also send any objects which were _explicitly_ requested by the client, you can now fetch whatever single objects you want.

And as long as you tell the other side you don't have any objects, it won't send any deltas.

Yes. At GitHub we actually have a custom endpoint which hooks up "cat-file --batch" with a format of the client's choosing. That's what (indirectly) feeds things like raw.github.com.

I've been tempted to send it upstream, but it's pretty ugly, and does give the client a lot of power (for now, the placeholders you can use with cat-file are not that powerful, but if we start to unify with ref-filter, etc, then we run into situations like we had with %(describe) recently). Likewise, the v2 object-info endpoint _could_ accept arbitrary format strings (it's the same idea, just with --batch-check instead of --batch).

-Peff

Jeff King <peff@peff.net> writes:

Yeah, the object-info actually was from folks who are interested in doing something similar, and it would be nice if we can share the protocol endpoint that is more suitable for interactive tree and history traversal to help those who want to do virtual filesystem.

Thanks.

On Mon, Aug 30 2021, Junio C Hamano wrote:

While this is all clever, I think this discussion really suggests that the first thing we should do is make the relatively recent "object-info" protocol verb not a default part of the supported v2 protocol we ship in git.git.

I.e. someone setting up a git server probably isn't going to suspect that one day their server load is going to go up by some big % because some developer somewhere is using a local IDE whose every file click on a directory is a new remote server request (i.e. the case where "object-info"'s functionality is expanded like this).

I found myself wondering this when reading serve.c the other day, i.e. why we have "always_advertise" for object-info, but it seemed innocuous enough given how it's described in a2ba162cda2 (object-info: support for retrieving object info, 2021-04-20).

But just as a general thing, while I'm very much in favor of git growing *optional* support for more server<->client cooperation and CPU offloading, even things like "git grep" or "git log" optimistically running server-side, I think those sorts of features should definitely be off by default for the reasons noted above.

On Tue, Aug 31, 2021 at 7:28 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

[Replying again as I used HTML mail by mistake. Sorry.]

Do you mean by someone directly sending object-info requests? I am working on wiring object-info to some of the existing tools (cat-file/ls-tree) so this is the general idea about how I see this being used:

- object-info would be used when it made sense but only if the actual
object being queried is not already fetched locally. If you think of a
virtual filesystem that is backed by, say, partial clones, this mostly
means retrieving metadata information to be displayed to the user.
- Still in the context of a virtual filesystem, metadata is usually
cached locally independently of Git itself, further reducing the need
to call object-info (but, of course, this is a brittle assumption as
it is not controlled by Git).
- git cat-file, for example, would be changed to support real batching
and then send a single request instead of the multiple requests it
does currently.

My point is that I understand where your worry is coming from and as long as someone can send arbitrary requests then it is possible your scenario of a heavier server load can potentially happen but as far as the expected canonical usage, I do not think this would be a problem and, in fact, under some usage patterns it might make things better (mostly due to batching support in object-info).

With all that being said, I don' t think making it optional would be an issue so I have no strong feelings about this. I am fine with whatever is agreed to be the best approach.

For what it is worth, The same change is now being reviewed in JGit and there the feature is conditionally enabled. But that was a side-effect of needing to deploy it to multiple servers before making the feature available to clients.

--

Bruno Albuquerque | Software Engineer | bga@google.com | +1 650-395-8242

Bruno Albuquerque <bga@google.com> writes:

FWIW, I do not mind, and probably prefer if I think about it a bit longer, to make it an opt-in feature, like all other capabilities defined in the serve.c file.

Thanks.

Hi,

thank you for the answer.

I understand that the core of git is to make people work together when writing code. To get a tree of the source files is not directly part of that, but pure informational. That is also the intent of my fuse fs: provide the user information about the source files.

Now I have a working ssh connection to the server, and open a channel for running the upload-pack on the server using the exec channel request:

https://datatracker.ietf.org/doc/html/rfc4254#section-6.5

So in my program I do not have to do something like:

ssh -x git@server "git-upload-pack 'simplegit-progit.git'"

It is only the sending of an exec message with the right command. Via the SSH_MSG_CHANNEL_DATA message the server will return the output. In my program I have to write a parser to get the tree/direntries.

Now you suggest the git clone --depth 1 --filter=blob:none --single-branch -b $branch command. How does that look when writing it in lowlevel git messages as described in

https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols

? I'm programming at this low level, so I have to write the messages to send to the server myself.

And you mention the api github has for a git tree object. But git2 has already the git_tree object?

Stef

My project  by the way is: https://github.com/stefbon/OSNS

On Tue, Aug 31, 2021 at 08:38:39AM +0200, Stef Bon wrote:

You'll have to read the documentation I pointed to earlier:

  https://github.com/git/git/blob/master/Documentation/technical/pack-protocol.txt

In short: the server tells you which refs it has and what they point to, then the client says which objects it wants and which objects it has, and then the server sends a packfile. The flow of the protocol and the format of the messages is laid out there.

You might also set GIT_TRACE_PACKET=1 in your environment and try running some Git commands. They will show you what's being said on the wire, up until the packfile is sent (decoding the packfile itself is a whole other story).

If you mean libgit2, then yes, it has a git_tree struct. Just like we have internally within regular Git. But those are for accessing _local_ objects, that have already been fetched.

You could build a fuse filesystem around a local Git repository pretty easily, either by using libgit2 or around tools like "git ls-tree" and "git cat-file". But if your purpose is to access a remote one without downloading all of the objects first, then no, Git does not expose any of the endpoints you'd need remotely (but provider-specific APIs like GitHub's do).

-Peff

Op di 31 aug. 2021 om 09:07 schreef Jeff King <peff@peff.net>:

Yes that will give me the insight I need. I will come back when it comes to decoding the packfile.

Thanks, Stef

On Tue, Aug 31 2021, Stef Bon wrote:

Aside from the "here's how you can do it", you haven't said why you'd like to do such "online" browsing of the repository.

I'd think that even for something that e.g. implements a file browser with magic git-remote support (think GNOME VFS-like), what you'd want to do in the background would be to do a "clone", although a clone with some combination of --single-branch, --no-tags, and perhaps --depth and the filters discussed upthread.

It will take the same time to get the pack, but once you do you can use libgit2, git's plumbing etc. to do really fast browsing/wildcarding etc. of the entries locally.

So is there a real performance or other use-case for wanting to do this, or does it just come down a lack of nice a "one-shot" API for "list remote files?".

In any case, on the topic of clever things you can (ab)use to do this, some remotes support running "git archive" for you. Notably GitHub doesn't, but GitLab does. Please don't take this as an endorsement to run this command "in production"

    $ time (git archive --format=tar  --remote=git@gitlab.com:git-vcs/git.git --prefix=t/t4018/ HEAD:t/t4018 | tar -tf- | head -n 3)
    t/t4018/
    t/t4018/README
    t/t4018/bash-arithmetic-function

    real    0m1.545s

I idly wonder if there's a want/need for a file listing API whether doing so via the tar/zip format wouldn't be a more viable & widely supported thing than expecting everyone to come up with their own git packfile decoders. I.e. if we just supported some option to create all-empty dummy files via "git archive" this could be even better as a dummy file listing API. Right now this (ab)use of it requires e.g. sending ~10MB of t/'s content just to list everything in the t/ directory.