Git dumb HTTP protocol should work without update-server-info

this works:

git ls-remote /path/to/repo2/ git ls-remote /path/to/repo2/.git/ git ls-remote file:///path/to/repo2/ git ls-remote file:///path/to/repo2/.git/

this fails:

python -m http.server -d /path/to/repo2/ & git ls-remote http://localhost:8000/ git ls-remote http://localhost:8000/.git/

workaround:

pushd /path/to/repo2/.git/ git --bare update-server-info mv hooks/post-update.sample hooks/post-update popd git ls-remote http://localhost:8000/

expected: dumb http remotes should behave like file remotes

that command creates the file /path/to/repo2/.git/info/refs but that is just an optimization for http servers with high latency

my "dumb" http server is smart enough to handle http range requests so there is no need to download all the files from .git/

but also without http range requests this should "just work" and the user should be responsible for optimizations

as another workaround i tried to mount the .git/ directory with httpdirfs but httpdirfs fails to mount git repos, see https://github.com/fangfufu/httpdirfs/issues/183

related:

https://stackoverflow.com/questions/2085402/what-does-git-update-server-info-do

https://stackoverflow.com/questions/2278888/private-git-repository-over-http

On 2025-09-07 at 11:24:13, Milan Hauth wrote:

In general, that's not possible, because HTTP doesn't support native atomic operations. An HTTP push locks the remote with DAV by preventing other changes to `info/refs`, which makes the operation atomic. If that file isn't there, then there's no way to guarantee that the ref update isn't competing with others, which might cause data loss.

In addition, HTTP also doesn't support native machine-readable directory listings except with DAV. However, we don't require DAV for fetches, so we need a list of the refs and the packs in order to be able to download objects correctly.

As I said above, I don't think that's the concern here.

I will also note that the dumb HTTP protocol doesn't work with reftable and there was some suggestion of removing it for Git 3.0. It certainly will not work out of the box with Git 3.0, since the default is reftable.

im only talking about "read" operations:

git ls-remote git fetch git pull

On 2025-09-07 at 17:23:28, Milan Hauth wrote:

Yes, as I said, reading directories is only possible with WebDAV since HTTP doesn't offer native directory listing. However, we don't use WebDAV for fetches and other read operations and not all web servers support it. We get better web server support in many cases by requiring that the server side do the work of updating the lists of packs and refs.

Without some way to list directories, you cannot in the general case iterate over the refs and packs in the file system without a manifest, so creating a manifest is what update-server-info does.

This is also why most tools which provide HTTP access in the file system require WebDAV, since it isn't very useful to have a file system where you can't list directories. (We abandoned directory-less file systems in the early DOS and Macintosh days.)

On Sun, Sep 07, 2025 at 03:07:11PM +0000, brian m. carlson wrote:

Yes, indeed. In theory though reftables could also be the solution to the underlying issue: the client can be tought to read the "tables.list" file and then fetch all tables listed therein. The result would be fully consistent, unless any of the tables gets garbage collected. The client would notice and abort the operation, after which it could restart the operation.

In that case there would be no need for git-update-server-info(1) anymore. The "tables.list" file sits in a well-known location, identifies all other tables we have to download, and there are no atomicity issues anymore.

The catch of course is that somebody would have to implement this :)

Patrick

Patrick Steinhardt <ps@pks.im> writes:

Does tables.list list what pack files there are in the repository? I somehow doubt it.

The dumb HTTP transport was meant to be able to operate with a truly dumb HTTP server, that does not even have to support WebDAV at all, so there needs some tables at known name that lists _all_ the files the cloners are expected to be able to download from. We still need the output from update-server-info [*] to tell what packs are there even if tables.list is stored at the known path.

[Footnote]

* ... or its equivalent generated offline and uploaded manually to
  the repository, which was what I did before I got an account at
  kernel.org.  There was a small web space at local ISP provided for
  its subscribers, so my "push" was to ftp upload the loose objects,
  packs, refs, and the info/refs + objects/info/packs files there X-<.

On Mon, Sep 08, 2025 at 07:43:20AM -0700, Junio C Hamano wrote:

Oh, you're right. I only remembered that we need it for refs, but of course we also need it for packs.

Patrick

On Sun, Sep 07, 2025 at 01:24:13PM +0200, Milan Hauth wrote:

File remotes are running a local git-upload-pack to act as the server. A dumb http remote can't run anything on the server side. So we are stuck pretending that the http endpoint is a filesystem. Besides performing terribly, as brian noted it is not even portable to do, since there is no readdir() equivalent (and things like httpdirfs have to resort to scraping auto-generated directory listings).

If you want to go that route, I think you're better off using a fuse wrapper and just letting Git work against the mounted filesystem, as you tried here:

I think this does work. The instructions you gave there won't do it, though, because python's http.server module doesn't support range requests.

Something like:

  sudo apt install python3-rangehttpserver
  (cd /path/to/repo && python -m RangeHTTPServer)
  mkdir mnt
  sudo httpdirfs http://localhost:8000/.git mnt

got me a mount that worked with:

  # you could probably avoid sudo here with better mount perms above
  sudo git clone mnt foo

It's painfully slow, though.

Possibly dumb-http could learn to do the same scraping that httpdirfs does to get the refs and pack listings (though this might be quite slow for unpacked refs, if the ref tree is deep). But I doubt you will find anybody that enthused about working on or reviewing dumb-http patches these days. The code is not very well maintained, IMO.

-Peff

Jeff King <peff@peff.net> writes:

Please don't. Once you go that route, that is no longer "dumb http" at all.

On 2025-09-08 at 00:05:43, Jeff King wrote:

That kind of scraping is really not a good idea.

It's the equivalent of trying to parse the FTP LIST output, which is customarily the equivalent of `ls -l`, but doesn't have to be, and often isn't on Windows. I can tell you from experience how painful doing that is and how many bug reports come in when you try to use that (because the FTP server also doesn't support either variant of the machine-readable format that solves this problem).

It's also prone to breaking things because some HTTP servers have weird redirect loops due to sorting entries in the index pages that you have to be careful not to trigger. And, of course, that depends on the server having index pages turned on, which many do not.

And then you have to do all of this in C with pointer arithmetic, with all of the terrifying security properties that has, especially because you can't actually use an XML parser to do it (unlike in DAV) since many servers have index pages that are (possibly invalid) autogenerated HTML 3.2 or something.

I've seen exactly one piece of software (lftp) that hasn't done a terribly awful implementation of this and that happens to have worked every time I've tried it (which is extremely infrequently, so I probably just haven't found a server which breaks it yet). Every other piece of software I've used that's done this kind of thing has just been broken and I anticipate Git would be no different.

I do want to be very clear that this is a bad idea and I hope never to see such patches come into Git.

On Mon, Sep 08, 2025 at 09:27:23PM +0000, brian m. carlson wrote:

I hear you could do it in rust these days. ;) But more seriously, yes, I don't think it's a good idea. Maybe I should have been more emphatic in my original message about that. But there is nothing stopping somebody who wanted to implement such a thing for themselves, which is probably more productive than asking somebody on this list to work on it.

-Peff