{"thread":{"id":"56262","subject":"Bug with Git shallow clones and submodules","startedAt":"2021-08-11T14:59:18Z","lastAt":"2021-09-04T14:51:27Z","messageCount":5,"participants":["Kevin Phillips","Philippe Blain","Jeff King","Konstantin Ryabitsev"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"432487","messageId":"CANNNLMC_XEqjYPOig+pbqrS30dPCTmcDSev6yg+5EEBkJCaaig@mail.gmail.com","threadId":"56262","inReplyTo":null,"subject":"Bug with Git shallow clones and submodules","fromName":"Kevin Phillips","fromEmail":"thefriendlycoder@gmail.com","sentAt":"2021-08-11T14:59:02Z","receivedAt":"2021-08-11T14:59:18Z","isPatch":false,"sender":{"key":"thefriendlycoder@gmail.com","avatar":null},"body":"We recently came across a bug in one of our Git projects which make\nuse of submodules. The basic problem is we have a Git repo that has 1\nsubmodule within it which is set up to check out a specific branch,\nand when we attempt to perform a shallow clone on the parent repo the\ncheckout fails with a \"Server does not allow request for unadvertised\nobject\" error. After doing a lot of testing and reading online I found\nthis StackOverflow thread (https://stackoverflow.com/a/61492323) that\nexplains our problem in great detail, which indicates to me that this\nis likely not a problem specific to us, so I thought I'd report it to\nsee if it could be fixed.\n\nIn our specific use case, we found the problem specifically when\nrunning a build of our project using Gitlab CI. This tool\nautomatically performs a shallow checkout of projects being built by\nusing a \"--depth=50\" parameter. This caused odd Git checkout errors to\nappear on our CI builds which weren't directly reproducible by our\ndevelopers local builds. Upon closer examination we were able to\nreproduce outside of the CI environment by performing a checkout as\nfollows:\n\n    git clone --recurse-submodules --depth=50  git@<path_to_repo>.git .\n\nBased on my initial research, which was helped greatly by the\nStackOverflow thread I linked to above, I was able to deduce the\nfollowing contributing factors:\n\nThe submodule that was causing us grief was the dpdk library located\nhere: http://git.dpdk.org/dpdk-stable/\n\nThis repository has a default branch named \"default\" which has only 2\ncommits on it: one for initializing the repo and one additional commit\nAll releases of this library are created from branches named by the\nrelease version, and they all seem disjoint from one another, and they\nare all disjoint from the default branch (ie: changes from the release\nbranches are never merged into the default branch)\n\nour parent repo, which is an internal / non-public repository that can\nnot be shared, has a single submodule defined in it with the following\n.gitmodule definition:\n\n\n[submodule \"dpdk-stable\"]\npath = dpdk-stable\nurl = https://git@dpdk.org/git/dpdk-stable\nbranch = 19.11\n\nthe specific revision we are checking out has the OID\n78bc27846101e481438a98c68cac47e4476085c0 which corresponds to the\n19.11.8 tag\nthe following git clone operation succeeds:\n\ngit clone --recurse-submodules git@<path_to_repo>.git .\n\nthe following git clone operation fails:\n\ngit clone --recurse-submodules --depth=50  git@<path_to_repo>.git .\n\nwe can work around the problem by doing a shallow clone with the\nno-single-branch parameter, as in:\n\ngit clone --recurse-submodules --depth=50 --no-single-branch\ngit@<path_to_repo>.git .\n\nBased on my current level of understanding, Git is automatically\nsetting the --single-branch flag when cloning submodules but only when\ndoing a shallow clone of the parent. This seems wrong to me. In\naddition, even if Git was going to enforce the single branch option on\nsubmodules it stands to reason that it should be respecting the branch\ndefined in the .gitmodules file (ie: 19.11 in our case) instead of\nusing the default branch (ie: \"default\" in this case). My guess is, if\nit were respecting the branch name then we wouldn't be getting the\n\"unadvertised object\" error because the referenced OID would exist in\nthe submodule checkout because it would have the correct branch\nchecked out.\n\nFor reference, this problem appears to be reproducible on several\ndifferent platforms (ie: several different Linux distros and Mac OSX\nat least) and several different Git versions up to v2.32.0 which is\nthe latest version at the moment. Hopefully this problem can be fixed\nin a future release of the command line tools. In the meantime we've\nhad to hack around the problem in our Gitlab CI builds by disabling\nthe shallow checkout features, which is less than ideal, so again I am\nhopeful that this can be fixed directly in Git itself.\n\nKevin Phillips\nLead Developer, Configuration Management Team, IBM\n"},{"id":"432761","messageId":"1a98c659-e7db-50a6-faf3-b3b4c15df679@gmail.com","threadId":"56262","inReplyTo":"CANNNLMC_XEqjYPOig+pbqrS30dPCTmcDSev6yg+5EEBkJCaaig@mail.gmail.com","subject":"Re: Bug with Git shallow clones and submodules","fromName":"Philippe Blain","fromEmail":"levraiphilippeblain@gmail.com","sentAt":"2021-08-15T14:31:46Z","receivedAt":"2021-08-15T14:31:54Z","isPatch":false,"sender":{"key":"levraiphilippeblain@gmail.com","avatar":"https://avatars.githubusercontent.com/u/44212482?v=4"},"body":"Hi Kevin,\n\nLe 2021-08-11 à 10:59, Kevin Phillips a écrit :\n> We recently came across a bug in one of our Git projects which make\n> use of submodules. The basic problem is we have a Git repo that has 1\n> submodule within it which is set up to check out a specific branch,\n\nJust to make sure we are on the same page: you can't have a submodule\n\"set up to check out a specific branch\", at least\nnot in the sense that the submodule is always \"checked out\" on a branch,\nand the superproject knows that the submodule should always be \"on that branch\";\nthat's not how submodules work. As specified in [1], the config `submodule.<name>.branch`\nwhich your repo has in '.gitmodules' is only ever used by\n`git submodule update --remote`. The superproject itself only records at which *commit* the\nsubmodule is at.\n\n> and when we attempt to perform a shallow clone on the parent repo the\n> checkout fails with a \"Server does not allow request for unadvertised\n> object\" error. After doing a lot of testing and reading online I found\n> this StackOverflow thread (https://stackoverflow.com/a/61492323) that\n> explains our problem in great detail, which indicates to me that this\n> is likely not a problem specific to us, so I thought I'd report it to\n> see if it could be fixed.\n\nI think your situation is related, but not exactly the same as the SO\npost. But torek's answer sure explains a lot of important points that\nare relevant in your situation.\n\n> \n> In our specific use case, we found the problem specifically when\n> running a build of our project using Gitlab CI. This tool\n> automatically performs a shallow checkout of projects being built by\n> using a \"--depth=50\" parameter. This caused odd Git checkout errors to\n> appear on our CI builds which weren't directly reproducible by our\n> developers local builds. Upon closer examination we were able to\n> reproduce outside of the CI environment by performing a checkout as\n> follows:\n> \n>      git clone --recurse-submodules --depth=50  git@<path_to_repo>.git .\n> \n> Based on my initial research, which was helped greatly by the\n> StackOverflow thread I linked to above, I was able to deduce the\n> following contributing factors:\n> \n> The submodule that was causing us grief was the dpdk library located\n> here: http://git.dpdk.org/dpdk-stable/\n> \n> This repository has a default branch named \"default\" which has only 2\n> commits on it: one for initializing the repo and one additional commit\n> All releases of this library are created from branches named by the\n> release version, and they all seem disjoint from one another, and they\n> are all disjoint from the default branch (ie: changes from the release\n> branches are never merged into the default branch)\n> \n> our parent repo, which is an internal / non-public repository that can\n> not be shared, has a single submodule defined in it with the following\n> .gitmodule definition:\n> \n> \n> [submodule \"dpdk-stable\"]\n> path = dpdk-stable\n> url = https://git@dpdk.org/git/dpdk-stable\n> branch = 19.11\n> \n> the specific revision we are checking out has the OID\n> 78bc27846101e481438a98c68cac47e4476085c0 which corresponds to the\n> 19.11.8 tag\n> the following git clone operation succeeds:\n> \n> git clone --recurse-submodules git@<path_to_repo>.git .\n> \n> the following git clone operation fails:\n> \n> git clone --recurse-submodules --depth=50  git@<path_to_repo>.git .\n\nThanks for the details here. I came up with this short reproducer script,\nwhich I think shows the behaviour you are seeing:\n\n~~~bash\n#!/bin/bash\n\nset -e\n\nrm -rf test && mkdir test && cd test\nrm -rf project clone\n\n# Create project\ngit init project && cd project\necho \"content\">file\ngit add file\ngit commit -m \"initial in proj\"\ngit submodule add -b 19.11 https://dpdk.org/git/dpdk-stable\ngit -C dpdk-stable -q checkout 78bc27846101e481438a98c68cac47e4476085c0\ngit add dpdk-stable\ngit commit -m \"add lib to project\"\ncd ../\n\n# Clone project\nGIT_TRACE2=3 git -c protocol.version=2 clone --no-local --recurse-submodules --depth=50 project clone 3> >(grep -E 'git clone|git fetch|git submodule|git rev-list' )\n~~~\n\nNote that in the \"Create project\" phase, I use the HTTPS URL\nhttps://dpdk.org/git/dpdk-stable, which is the recommended HTTPS URL at [2].\n\n> \n> we can work around the problem by doing a shallow clone with the\n> no-single-branch parameter, as in:\n> \n> git clone --recurse-submodules --depth=50 --no-single-branch\n> git@<path_to_repo>.git .\n> \n> Based on my current level of understanding, Git is automatically\n> setting the --single-branch flag when cloning submodules but only when\n> doing a shallow clone of the parent. This seems wrong to me. \n\nIn fact, the logic in the code is twofold:\n\n1. In 'git clone', '--depth' implies '--single-branch' [3]\n2. In 'git clone', '--single-branch' is used to clone submodules if\n    '--recurse-submodules' is used. This is the case since 132f600b06\n    (clone: pass --single-branch during --recurse-submodules, 2020-02-20)\n\nI'm not sure why you think this is wrong (in theory)...\n\n> In\n> addition, even if Git was going to enforce the single branch option on\n> submodules it stands to reason that it should be respecting the branch\n> defined in the .gitmodules file (ie: 19.11 in our case) instead of\n> using the default branch (ie: \"default\" in this case). \n\n... in practice, here you are right: it is indeed unfortunate that the\nbranch specified in '.gitmodules' is not used for the initial clone of\nthe submodule if they are cloned with '--single-branch'.\n\n> My guess is, if\n> it were respecting the branch name then we wouldn't be getting the\n> \"unadvertised object\" error because the referenced OID would exist in\n> the submodule checkout because it would have the correct branch\n> checked out.\n\nI think your are right, since the following command does work:\n\n     git clone --single-branch --branch 19.11 https://dpdk.org/git/dpdk-stable &&\n     git -C dpdk-stable checkout 78bc27846101e481438a98c68cac47e4476085c0\n\n> \n> For reference, this problem appears to be reproducible on several\n> different platforms (ie: several different Linux distros and Mac OSX\n> at least) and several different Git versions up to v2.32.0 which is\n> the latest version at the moment. Hopefully this problem can be fixed\n> in a future release of the command line tools. In the meantime we've\n> had to hack around the problem in our Gitlab CI builds by disabling\n> the shallow checkout features, which is less than ideal\n\nIf you run my reproducer script above, you should see the steps that the code\ndoes to try to retrieve the commit you need, 78bc27846101e481438a98c68cac47e4476085c0:\n\n1. It invokes 'git clone' for the submodule:\n\n     child_start[0] git clone --no-checkout --progress --separate-git-dir $PWD/test/clone/.git/modules/dpdk-stable --single-branch -- https://dpdk.org/git/dpdk-stable $PWD/test/clone/dpdk-stable\n\n2. It checks if the clone downloaded the needed object:\n\n     start git rev-list -n 1 78bc27846101e481438a98c68cac47e4476085c0 --not --all\n\n    We know that it did not, because as noted above, the branch from '.gitmodules'\n    is not used in the single-branch clone, so only object from the 'default' branch\n    are available.\n3. It realizes the object is not there, so it runs \"git fetch\" in the submodule\n4. It checks again with the same 'git rev-list command' if the fetch downloaded\n    the needed object. Of course, it did not, because since the branch in '.gitmodules'\n    is not used for the initial single-branch clone, at that point the fetch refspec\n    in the submodule is '+refs/heads/default:refs/remotes/origin/default', and so a\n    simple 'git fetch' does not bring any new object.\n5. It falls back to fetching the needed commit by its hash:\n\n     start git fetch origin 78bc27846101e481438a98c68cac47e4476085c0\n\n6. The server at https://dpdk.org/git/dpdk-stable refuses this request and you get\n\n     error: Server does not allow request for unadvertised object 78bc27846101e481438a98c68cac47e4476085c0\n\nOK so why does the server refuses the request ? If you take a look at [4], you will\nunderstand that it's probably because the configs 'uploadpack.allowReachableSHA1InWant'\nand 'uploadpack.allowAnySHA1InWant' are both false (i.e. their default value) on the\nhttps://dpdk.org/git/dpdk-stable server. So the behaviour here is not a bug, it's\nworking as it should (but leads to a bad experience for you).\n\nWhat is not unfortunately not mentioned in the documentation at [4]\nis that the two configs that I mention above only apply if version 0 of the Git transfer\nprotocol is in effect. So this would mean that even though we specify '-c protocol.version=2'\nin the 'git clone' command in my script, the server uses protocol v0. Protocol v2 works\nstarting in Git 2.18, so let's check what version is running at dpdk.org:\n\n$ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://dpdk.org/git/dpdk-stable 2>&1 | grep agent\npacket:          git< 6ee0521feb765d9105241a3f6693762c471655cf HEAD\\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow deepen-since deepen-not deepen-relative no-progress include-tag multi_ack_detailed no-done symref=HEAD:refs/heads/default agent=git/2.20.1\n\nOK, it's running Git 2.20.1. And the list of capabilities above implies that it's responding\nwith protocol v0, even if my local Git version if 2.29, which defaults to protocol v2. If\nit were responding with protocol v2, we would see:\n\n$ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://github.com/git/git 2>&1 | grep agent\npacket:          git< agent=git/github-g78b452413e8b  # server version\npacket:          git< agent=git/github-g78b452413e8b\npacket:          git> agent=git/2.29.2  # client version\npacket:          git< agent=git/2.29.2\n\nAnd grepping for \"version\" would confirm it:\n\n$ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://github.com/git/git 2>&1 | grep version\npacket:          git< version 2\npacket:          git< version 2\n\nI can't explain why the server refuses to talk with protocol v2. Maybe Jonathan (CC-ed),\nwho worked on the implementation of protocol v2, would be able to explain that.\n\nAnd it gets weirder. If you instead use the Git URL (instead of the HTTPS URL)\nfor the dpdk-stable submodule [2], then the server responds with protocol v2!\n\n$ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote git://dpdk.org/dpdk-stable 2>&1 | grep agent\npacket:          git< agent=git/2.20.1\npacket:          git> agent=git/2.29.2\n$ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote git://dpdk.org/dpdk-stable 2>&1 | grep version\npacket:          git> git-upload-pack /dpdk-stable\\0host=dpdk.org\\0\\0version=2\\0\npacket:          git< version 2\n\nAnd indeed, if you use the 'git://' URL in the 'git submodule add' command in my reproducer\nscript, it succeeds !\n\nIt don't understand this behaviour either.\n\nSo a workaround for you would be to use the 'git://' URL in your '.gitmodules',\nif the Git port (9418) is not blocked in your infrastructure.\n\nI hope this helps,\n\nPhilippe.\n\nP.S. for Jonathan I noticed some additional weirdness while debugging this:\nSome servers respond with a different Git version depending on the protocol:\n\n$ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote  git://gcc.gnu.org/git/gcc.git 2>&1 | grep -E 'agent|version '\npacket:          git< version 2\npacket:          git< agent=git/2.26.2\npacket:          git> agent=git/2.29.2\n$ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote  https://gcc.gnu.org/git/gcc.git 2>&1 | grep agent -E 'agent|version '\npacket:          git< 882f1d58bfa56737ff2de84c3cd1e0acfc318b86 HEAD\\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow deepen-since deepen-not deepen-relative no-progress include-tag multi_ack_detailed no-done symref=HEAD:refs/heads/master agent=git/2.27.0\n\nHere again, with HTTPS we fall back to protocol v0. The same behaviour happens\nwith git://sourceware.org/git/binutils-gdb.git...\n\n1. https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt--bltbranchgt\n2. https://core.dpdk.org/contribute/#clone\n3. https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt\n4. https://git-scm.com/docs/git-config#Documentation/git-config.txt-uploadpackallowReachableSHA1InWant\n"},{"id":"434561","messageId":"542a68a5-d439-921e-2691-bb8890584f27@gmail.com","threadId":"56262","inReplyTo":"1a98c659-e7db-50a6-faf3-b3b4c15df679@gmail.com","subject":"Weird interaction between Git protocol versions and git:// and https:// URLs (was: Re: Bug with Git shallow clones and submodules)","fromName":"Philippe Blain","fromEmail":"levraiphilippeblain@gmail.com","sentAt":"2021-09-02T13:49:38Z","receivedAt":"2021-09-02T13:49:34Z","isPatch":false,"sender":{"key":"levraiphilippeblain@gmail.com","avatar":"https://avatars.githubusercontent.com/u/44212482?v=4"},"body":"Hi everyone,\n\nLe 2021-08-15 à 10:31, Philippe Blain a écrit :\n\n> 5. It falls back to fetching the needed commit by its hash:\n> \n>      start git fetch origin 78bc27846101e481438a98c68cac47e4476085c0\n> \n> 6. The server at https://dpdk.org/git/dpdk-stable refuses this request and you get\n> \n>      error: Server does not allow request for unadvertised object 78bc27846101e481438a98c68cac47e4476085c0\n> \n> OK so why does the server refuses the request ? If you take a look at [4], you will\n> understand that it's probably because the configs 'uploadpack.allowReachableSHA1InWant'\n> and 'uploadpack.allowAnySHA1InWant' are both false (i.e. their default value) on the\n> https://dpdk.org/git/dpdk-stable server. So the behaviour here is not a bug, it's\n> working as it should (but leads to a bad experience for you).\n> \n> What is not unfortunately not mentioned in the documentation at [4]\n> is that the two configs that I mention above only apply if version 0 of the Git transfer\n> protocol is in effect. So this would mean that even though we specify '-c protocol.version=2'\n> in the 'git clone' command in my script, the server uses protocol v0. Protocol v2 works\n> starting in Git 2.18, so let's check what version is running at dpdk.org:\n> \n> $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://dpdk.org/git/dpdk-stable 2>&1 | grep agent\n> packet:          git< 6ee0521feb765d9105241a3f6693762c471655cf HEAD\\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow deepen-since deepen-not deepen-relative no-progress include-tag multi_ack_detailed no-done symref=HEAD:refs/heads/default agent=git/2.20.1\n> \n> OK, it's running Git 2.20.1. And the list of capabilities above implies that it's responding\n> with protocol v0, even if my local Git version if 2.29, which defaults to protocol v2. If\n> it were responding with protocol v2, we would see:\n> \n> $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://github.com/git/git 2>&1 | grep agent\n> packet:          git< agent=git/github-g78b452413e8b  # server version\n> packet:          git< agent=git/github-g78b452413e8b\n> packet:          git> agent=git/2.29.2  # client version\n> packet:          git< agent=git/2.29.2\n> \n> And grepping for \"version\" would confirm it:\n> \n> $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://github.com/git/git 2>&1 | grep version\n> packet:          git< version 2\n> packet:          git< version 2\n> \n> I can't explain why the server refuses to talk with protocol v2. Maybe Jonathan (CC-ed),\n> who worked on the implementation of protocol v2, would be able to explain that.\n\nI'm sorry to bump this, but I would really like to understand what is going on here.\nI'm not familiar at all with the protocol code, but my understanding was that after\nGit 2.18, if a client requested protocol v2, the server would answer with v2 (at least\nfor git:// and http[s]://, and also for ssh:// if AcceptEnv is correctly configured).\nHere, Git 2.20.1 at https://dpdk.org/git/dpdk-stable answers with v0 for https://,\nbut with v2 for git:// !\n\n> \n> And it gets weirder. If you instead use the Git URL (instead of the HTTPS URL)\n> for the dpdk-stable submodule [2], then the server responds with protocol v2!\n> \n> $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote git://dpdk.org/dpdk-stable 2>&1 | grep agent\n> packet:          git< agent=git/2.20.1\n> packet:          git> agent=git/2.29.2\n> $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote git://dpdk.org/dpdk-stable 2>&1 | grep version\n> packet:          git> git-upload-pack /dpdk-stable\\0host=dpdk.org\\0\\0version=2\\0\n> packet:          git< version 2\n> \n> And indeed, if you use the 'git://' URL in the 'git submodule add' command in my reproducer\n> script, it succeeds !\n> \n> It don't understand this behaviour either.\n> \n--- snip ---\n> \n> P.S. for Jonathan I noticed some additional weirdness while debugging this:\n> Some servers respond with a different Git version depending on the protocol:\n> \n> $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote  git://gcc.gnu.org/git/gcc.git 2>&1 | grep -E 'agent|version '\n> packet:          git< version 2\n> packet:          git< agent=git/2.26.2\n> packet:          git> agent=git/2.29.2\n> $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote  https://gcc.gnu.org/git/gcc.git 2>&1 | grep agent -E 'agent|version '\n> packet:          git< 882f1d58bfa56737ff2de84c3cd1e0acfc318b86 HEAD\\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow deepen-since deepen-not deepen-relative no-progress include-tag multi_ack_detailed no-done symref=HEAD:refs/heads/master agent=git/2.27.0\n> \n> Here again, with HTTPS we fall back to protocol v0. The same behaviour happens\n> with git://sourceware.org/git/binutils-gdb.git...\n\nThis is also puzzling to me...\n\nThanks,\n\nPhilippe.\n"},{"id":"434694","messageId":"YTNtVJy6sCfQ7T3L@coredump.intra.peff.net","threadId":"56262","inReplyTo":"542a68a5-d439-921e-2691-bb8890584f27@gmail.com","subject":"Re: Weird interaction between Git protocol versions and git:// and https:// URLs (was: Re: Bug with Git shallow clones and submodules)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-04T12:57:56Z","receivedAt":"2021-09-04T12:57:59Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Sep 02, 2021 at 09:49:38AM -0400, Philippe Blain wrote:\n\n> > OK, it's running Git 2.20.1. And the list of capabilities above implies that it's responding\n> > with protocol v0, even if my local Git version if 2.29, which defaults to protocol v2. If\n> > it were responding with protocol v2, we would see:\n> > \n> > $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://github.com/git/git 2>&1 | grep agent\n> > packet:          git< agent=git/github-g78b452413e8b  # server version\n> > packet:          git< agent=git/github-g78b452413e8b\n> > packet:          git> agent=git/2.29.2  # client version\n> > packet:          git< agent=git/2.29.2\n> > \n> > And grepping for \"version\" would confirm it:\n> > \n> > $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://github.com/git/git 2>&1 | grep version\n> > packet:          git< version 2\n> > packet:          git< version 2\n> > \n> > I can't explain why the server refuses to talk with protocol v2. Maybe Jonathan (CC-ed),\n> > who worked on the implementation of protocol v2, would be able to explain that.\n> \n> I'm sorry to bump this, but I would really like to understand what is going on here.\n> I'm not familiar at all with the protocol code, but my understanding was that after\n> Git 2.18, if a client requested protocol v2, the server would answer with v2 (at least\n> for git:// and http[s]://, and also for ssh:// if AcceptEnv is correctly configured).\n> Here, Git 2.20.1 at https://dpdk.org/git/dpdk-stable answers with v0 for https://,\n> but with v2 for git:// !\n\nThe server has to be configured correctly at the HTTP level, too. The v2\nprobe for HTTP is a \"Git-Protocol: version=2\" header in the HTTP\nrequest. The webserver has to be configured to communicate that to the\nrest of Git somehow. In our test setup for Apache, we do:\n\n  SetEnvIf Git-Protocol \".*\" GIT_PROTOCOL=$0\n\nIf that doesn't happen, then the server doesn't see the header at all,\nand just speaks v0 as usual.\n\nI suspect this could be better documented. I have to admit that I have\nnot actually run a regular HTTP server with Git in a long time (at\nGitHub, we have our own custom layer that terminates the HTTP\nconnections).\n\n-Peff\n"},{"id":"434696","messageId":"20210904145122.4glsdicbx56cs3ji@meerkat.local","threadId":"56262","inReplyTo":"YTNtVJy6sCfQ7T3L@coredump.intra.peff.net","subject":"Re: Weird interaction between Git protocol versions and git:// and https:// URLs (was: Re: Bug with Git shallow clones and submodules)","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2021-09-04T14:51:22Z","receivedAt":"2021-09-04T14:51:27Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Sat, Sep 04, 2021 at 08:57:56AM -0400, Jeff King wrote:\n> The server has to be configured correctly at the HTTP level, too. The v2\n> probe for HTTP is a \"Git-Protocol: version=2\" header in the HTTP\n> request. The webserver has to be configured to communicate that to the\n> rest of Git somehow. In our test setup for Apache, we do:\n> \n>   SetEnvIf Git-Protocol \".*\" GIT_PROTOCOL=$0\n> \n> If that doesn't happen, then the server doesn't see the header at all,\n> and just speaks v0 as usual.\n> \n> I suspect this could be better documented.\n\nIndeed, I was surprised to find out that all this time we were serving v0 for\nhttp requests. /o\\\n\nFor the record, the magic nginx+uwsgi incantation is:\n\n    uwsgi_param GIT_PROTOCOL $http_git_protocol;\n\nthat gets us what we want:\n\n    $ GIT_TRACE_PACKET=1 GIT_TRACE_BARE=1 git ls-remote https://git.kernel.org/pub/scm/git/git.git 2>&1 | grep version\n    packet:          git< version 2\n    packet:          git< version 2\n    packet:          git< version 2\n\nI will submit a patch for the docs to indicate that this is required, because\nright now the only mention of GIT_PROTOCOL env variable are for the file://\nand ssh:// schemes.\n\n-K\n"}]}