{"thread":{"id":"64102","subject":"Git dumb HTTP protocol should work without update-server-info","startedAt":"2025-09-07T11:24:25Z","lastAt":"2025-09-09T05:26:26Z","messageCount":11,"participants":["Milan Hauth","brian m. carlson","Jeff King","Junio C Hamano","Patrick Steinhardt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"525706","messageId":"CAGiEHCtP29bQRsEyLabNrLuiP96P-o7EEGi88B7pJbP0tfprxw@mail.gmail.com","threadId":"64102","inReplyTo":null,"subject":"Git dumb HTTP protocol should work without update-server-info","fromName":"Milan Hauth","fromEmail":"milahu@gmail.com","sentAt":"2025-09-07T11:24:13Z","receivedAt":"2025-09-07T11:24:25Z","isPatch":false,"sender":{"key":"milahu@gmail.com","avatar":"https://gravatar.com/avatar/68beefdc29b5666a84773ff8094481ddd56ff3d2f997948c05dedb586574afa9?d=mp&s=160"},"body":"this works:\n\ngit ls-remote /path/to/repo2/\ngit ls-remote /path/to/repo2/.git/\ngit ls-remote file:///path/to/repo2/\ngit ls-remote file:///path/to/repo2/.git/\n\nthis fails:\n\npython -m http.server -d /path/to/repo2/ &\ngit ls-remote http://localhost:8000/\ngit ls-remote http://localhost:8000/.git/\n\nworkaround:\n\npushd /path/to/repo2/.git/\ngit --bare update-server-info\nmv hooks/post-update.sample hooks/post-update\npopd\ngit ls-remote http://localhost:8000/\n\nexpected:\ndumb http remotes should behave like file remotes\n\n\n\n> git --bare update-server-info\n\nthat command creates the file\n/path/to/repo2/.git/info/refs\nbut that is just an optimization\nfor http servers with high latency\n\nmy \"dumb\" http server\nis smart enough to handle http range requests\nso there is no need\nto download all the files from .git/\n\nbut also without http range requests\nthis should \"just work\"\nand the user should be responsible for optimizations\n\n\n\nas another workaround\ni tried to mount the .git/ directory with httpdirfs\nbut httpdirfs fails to mount git repos, see\nhttps://github.com/fangfufu/httpdirfs/issues/183\n\n\n\nrelated:\n\nhttps://stackoverflow.com/questions/2085402/what-does-git-update-server-info-do\n\nhttps://stackoverflow.com/questions/2278888/private-git-repository-over-http\n"},{"id":"525708","messageId":"aL2fn4j5uhrVWrDp@fruit.crustytoothpaste.net","threadId":"64102","inReplyTo":"CAGiEHCtP29bQRsEyLabNrLuiP96P-o7EEGi88B7pJbP0tfprxw@mail.gmail.com","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-09-07T15:07:11Z","receivedAt":"2025-09-07T15:07:14Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-09-07 at 11:24:13, Milan Hauth wrote:\n> this works:\n> \n> git ls-remote /path/to/repo2/\n> git ls-remote /path/to/repo2/.git/\n> git ls-remote file:///path/to/repo2/\n> git ls-remote file:///path/to/repo2/.git/\n> \n> this fails:\n> \n> python -m http.server -d /path/to/repo2/ &\n> git ls-remote http://localhost:8000/\n> git ls-remote http://localhost:8000/.git/\n> \n> workaround:\n> \n> pushd /path/to/repo2/.git/\n> git --bare update-server-info\n> mv hooks/post-update.sample hooks/post-update\n> popd\n> git ls-remote http://localhost:8000/\n> \n> expected:\n> dumb http remotes should behave like file remotes\n\nIn general, that's not possible, because HTTP doesn't support native\natomic operations.  An HTTP push locks the remote with DAV by preventing\nother changes to `info/refs`, which makes the operation atomic.  If\nthat file isn't there, then there's no way to guarantee that the ref\nupdate isn't competing with others, which might cause data loss.\n\nIn addition, HTTP also doesn't support native machine-readable directory\nlistings except with DAV.  However, we don't require DAV for fetches, so\nwe need a list of the refs and the packs in order to be able to download\nobjects correctly.\n\n> > git --bare update-server-info\n> \n> that command creates the file\n> /path/to/repo2/.git/info/refs\n> but that is just an optimization\n> for http servers with high latency\n> \n> my \"dumb\" http server\n> is smart enough to handle http range requests\n> so there is no need\n> to download all the files from .git/\n> \n> but also without http range requests\n> this should \"just work\"\n> and the user should be responsible for optimizations\n\nAs I said above, I don't think that's the concern here.\n\nI will also note that the dumb HTTP protocol doesn't work with reftable\nand there was some suggestion of removing it for Git 3.0.  It certainly\nwill not work out of the box with Git 3.0, since the default is\nreftable.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"525722","messageId":"CAGiEHCvFnhZRXwNVRyzQ4ABi3p0Opdo8+draXoDu5p1R6HZn1Q@mail.gmail.com","threadId":"64102","inReplyTo":"aL2fn4j5uhrVWrDp@fruit.crustytoothpaste.net","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"Milan Hauth","fromEmail":"milahu@gmail.com","sentAt":"2025-09-07T17:23:28Z","receivedAt":"2025-09-07T17:23:41Z","isPatch":false,"sender":{"key":"milahu@gmail.com","avatar":"https://gravatar.com/avatar/68beefdc29b5666a84773ff8094481ddd56ff3d2f997948c05dedb586574afa9?d=mp&s=160"},"body":"> HTTP push\n\nim only talking about \"read\" operations:\n\ngit ls-remote\ngit fetch\ngit pull\n"},{"id":"525723","messageId":"aL3EE7dLMuj6ONZQ@fruit.crustytoothpaste.net","threadId":"64102","inReplyTo":"CAGiEHCvFnhZRXwNVRyzQ4ABi3p0Opdo8+draXoDu5p1R6HZn1Q@mail.gmail.com","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-09-07T17:42:43Z","receivedAt":"2025-09-07T17:42:50Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-09-07 at 17:23:28, Milan Hauth wrote:\n> > HTTP push\n> \n> im only talking about \"read\" operations:\n> \n> git ls-remote\n> git fetch\n> git pull\n\nYes, as I said, reading directories is only possible with WebDAV since\nHTTP doesn't offer native directory listing.  However, we don't use\nWebDAV for fetches and other read operations and not all web servers\nsupport it.  We get better web server support in many cases by requiring\nthat the server side do the work of updating the lists of packs and\nrefs.\n\nWithout some way to list directories, you cannot in the general case\niterate over the refs and packs in the file system without a manifest,\nso creating a manifest is what update-server-info does.\n\nThis is also why most tools which provide HTTP access in the file system\nrequire WebDAV, since it isn't very useful to have a file system where\nyou can't list directories.  (We abandoned directory-less file systems\nin the early DOS and Macintosh days.)\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"525745","messageId":"20250908000543.GB1281511@coredump.intra.peff.net","threadId":"64102","inReplyTo":"CAGiEHCtP29bQRsEyLabNrLuiP96P-o7EEGi88B7pJbP0tfprxw@mail.gmail.com","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-09-08T00:05:43Z","receivedAt":"2025-09-08T00:05:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Sep 07, 2025 at 01:24:13PM +0200, Milan Hauth wrote:\n\n> expected:\n> dumb http remotes should behave like file remotes\n\nFile remotes are running a local git-upload-pack to act as the server.\nA dumb http remote can't run anything on the server side. So we are\nstuck pretending that the http endpoint is a filesystem. Besides\nperforming terribly, as brian noted it is not even portable to do,\nsince there is no readdir() equivalent (and things like httpdirfs have\nto resort to scraping auto-generated directory listings).\n\nIf you want to go that route, I think you're better off using a fuse\nwrapper and just letting Git work against the mounted filesystem, as you\ntried here:\n\n> as another workaround\n> i tried to mount the .git/ directory with httpdirfs\n> but httpdirfs fails to mount git repos, see\n> https://github.com/fangfufu/httpdirfs/issues/183\n\nI think this does work. The instructions you gave there won't do it,\nthough, because python's http.server module doesn't support range\nrequests.\n\nSomething like:\n\n  sudo apt install python3-rangehttpserver\n  (cd /path/to/repo && python -m RangeHTTPServer)\n  mkdir mnt\n  sudo httpdirfs http://localhost:8000/.git mnt\n\ngot me a mount that worked with:\n\n  # you could probably avoid sudo here with better mount perms above\n  sudo git clone mnt foo\n\nIt's painfully slow, though.\n\nPossibly dumb-http could learn to do the same scraping that httpdirfs\ndoes to get the refs and pack listings (though this might be quite slow\nfor unpacked refs, if the ref tree is deep). But I doubt you will find\nanybody that enthused about working on or reviewing dumb-http patches\nthese days. The code is not very well maintained, IMO.\n\n-Peff\n"},{"id":"525748","messageId":"xmqqo6rlzil3.fsf@gitster.g","threadId":"64102","inReplyTo":"20250908000543.GB1281511@coredump.intra.peff.net","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-08T04:14:32Z","receivedAt":"2025-09-08T04:14:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Possibly dumb-http could learn to do the same scraping that httpdirfs\n> does to get the refs and pack listings (though this might be quite slow\n> for unpacked refs, if the ref tree is deep).\n\nPlease don't.  Once you go that route, that is no longer \"dumb http\"\nat all.\n\n"},{"id":"525786","messageId":"aL6kevExmhesoEWN@pks.im","threadId":"64102","inReplyTo":"aL2fn4j5uhrVWrDp@fruit.crustytoothpaste.net","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-08T09:40:10Z","receivedAt":"2025-09-08T09:40:22Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Sep 07, 2025 at 03:07:11PM +0000, brian m. carlson wrote:\n> I will also note that the dumb HTTP protocol doesn't work with reftable\n> and there was some suggestion of removing it for Git 3.0.  It certainly\n> will not work out of the box with Git 3.0, since the default is\n> reftable.\n\nYes, indeed. In theory though reftables could also be the solution to\nthe underlying issue: the client can be tought to read the \"tables.list\"\nfile and then fetch all tables listed therein. The result would be fully\nconsistent, unless any of the tables gets garbage collected. The client\nwould notice and abort the operation, after which it could restart the\noperation.\n\nIn that case there would be no need for git-update-server-info(1)\nanymore. The \"tables.list\" file sits in a well-known location,\nidentifies all other tables we have to download, and there are no\natomicity issues anymore.\n\nThe catch of course is that somebody would have to implement this :)\n\nPatrick\n"},{"id":"525828","messageId":"xmqqy0qpxawn.fsf@gitster.g","threadId":"64102","inReplyTo":"aL6kevExmhesoEWN@pks.im","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-08T14:43:20Z","receivedAt":"2025-09-08T14:43:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> On Sun, Sep 07, 2025 at 03:07:11PM +0000, brian m. carlson wrote:\n>> I will also note that the dumb HTTP protocol doesn't work with reftable\n>> and there was some suggestion of removing it for Git 3.0.  It certainly\n>> will not work out of the box with Git 3.0, since the default is\n>> reftable.\n>\n> Yes, indeed. In theory though reftables could also be the solution to\n> the underlying issue: the client can be tought to read the \"tables.list\"\n> file and then fetch all tables listed therein. The result would be fully\n> consistent, unless any of the tables gets garbage collected. The client\n> would notice and abort the operation, after which it could restart the\n> operation.\n>\n> In that case there would be no need for git-update-server-info(1)\n> anymore. The \"tables.list\" file sits in a well-known location,\n> identifies all other tables we have to download, and there are no\n> atomicity issues anymore.\n\nDoes tables.list list what pack files there are in the repository?  \nI somehow doubt it.\n\nThe dumb HTTP transport was meant to be able to operate with a truly\ndumb HTTP server, that does not even have to support WebDAV at all,\nso there needs some tables at known name that lists _all_ the files\nthe cloners are expected to be able to download from.  We still need\nthe output from update-server-info [*] to tell what packs are there\neven if tables.list is stored at the known path.\n\n\n[Footnote]\n\n* ... or its equivalent generated offline and uploaded manually to\n  the repository, which was what I did before I got an account at\n  kernel.org.  There was a small web space at local ISP provided for\n  its subscribers, so my \"push\" was to ftp upload the loose objects,\n  packs, refs, and the info/refs + objects/info/packs files there X-<.\n"},{"id":"525883","messageId":"aL9KOyR0eb7zAg2R@fruit.crustytoothpaste.net","threadId":"64102","inReplyTo":"20250908000543.GB1281511@coredump.intra.peff.net","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-09-08T21:27:23Z","receivedAt":"2025-09-08T21:27:31Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-09-08 at 00:05:43, Jeff King wrote:\n> Possibly dumb-http could learn to do the same scraping that httpdirfs\n> does to get the refs and pack listings (though this might be quite slow\n> for unpacked refs, if the ref tree is deep). But I doubt you will find\n> anybody that enthused about working on or reviewing dumb-http patches\n> these days. The code is not very well maintained, IMO.\n\nThat kind of scraping is really not a good idea.\n\nIt's the equivalent of trying to parse the FTP LIST output, which is\ncustomarily the equivalent of `ls -l`, but doesn't have to be, and often\nisn't on Windows.  I can tell you from experience how painful doing that\nis and how many bug reports come in when you try to use that (because\nthe FTP server also doesn't support either variant of the\nmachine-readable format that solves this problem).\n\nIt's also prone to breaking things because some HTTP servers have weird\nredirect loops due to sorting entries in the index pages that you have\nto be careful not to trigger.  And, of course, that depends on the\nserver having index pages turned on, which many do not.\n\nAnd then you have to do all of this in C with pointer arithmetic, with\nall of the terrifying security properties that has, especially because\nyou can't actually use an XML parser to do it (unlike in DAV) since many\nservers have index pages that are (possibly invalid) autogenerated HTML\n3.2 or something.\n\nI've seen exactly one piece of software (lftp) that hasn't done a\nterribly awful implementation of this and that happens to have worked\nevery time I've tried it (which is extremely infrequently, so I probably\njust haven't found a server which breaks it yet).  Every other piece of\nsoftware I've used that's done this kind of thing has just been broken\nand I anticipate Git would be no different.\n\nI do want to be very clear that this is a bad idea and I hope never to\nsee such patches come into Git.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"525891","messageId":"20250909013523.GA1333147@coredump.intra.peff.net","threadId":"64102","inReplyTo":"aL9KOyR0eb7zAg2R@fruit.crustytoothpaste.net","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-09-09T01:35:23Z","receivedAt":"2025-09-09T01:35:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 08, 2025 at 09:27:23PM +0000, brian m. carlson wrote:\n\n> And then you have to do all of this in C with pointer arithmetic, with\n> all of the terrifying security properties that has, especially because\n> you can't actually use an XML parser to do it (unlike in DAV) since many\n> servers have index pages that are (possibly invalid) autogenerated HTML\n> 3.2 or something.\n\nI hear you could do it in rust these days. ;) But more seriously, yes, I\ndon't think it's a good idea. Maybe I should have been more emphatic in\nmy original message about that. But there is nothing stopping somebody\nwho wanted to implement such a thing for themselves, which is probably\nmore productive than asking somebody on this list to work on it.\n\n-Peff\n"},{"id":"525893","messageId":"aL-6fPzaOhGpUafK@pks.im","threadId":"64102","inReplyTo":"xmqqy0qpxawn.fsf@gitster.g","subject":"Re: Git dumb HTTP protocol should work without update-server-info","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T05:26:20Z","receivedAt":"2025-09-09T05:26:26Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Sep 08, 2025 at 07:43:20AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > On Sun, Sep 07, 2025 at 03:07:11PM +0000, brian m. carlson wrote:\n> >> I will also note that the dumb HTTP protocol doesn't work with reftable\n> >> and there was some suggestion of removing it for Git 3.0.  It certainly\n> >> will not work out of the box with Git 3.0, since the default is\n> >> reftable.\n> >\n> > Yes, indeed. In theory though reftables could also be the solution to\n> > the underlying issue: the client can be tought to read the \"tables.list\"\n> > file and then fetch all tables listed therein. The result would be fully\n> > consistent, unless any of the tables gets garbage collected. The client\n> > would notice and abort the operation, after which it could restart the\n> > operation.\n> >\n> > In that case there would be no need for git-update-server-info(1)\n> > anymore. The \"tables.list\" file sits in a well-known location,\n> > identifies all other tables we have to download, and there are no\n> > atomicity issues anymore.\n> \n> Does tables.list list what pack files there are in the repository?  \n> I somehow doubt it.\n> \n> The dumb HTTP transport was meant to be able to operate with a truly\n> dumb HTTP server, that does not even have to support WebDAV at all,\n> so there needs some tables at known name that lists _all_ the files\n> the cloners are expected to be able to download from.  We still need\n> the output from update-server-info [*] to tell what packs are there\n> even if tables.list is stored at the known path.\n\nOh, you're right. I only remembered that we need it for refs, but of\ncourse we also need it for packs.\n\nPatrick\n"}]}