{"thread":{"id":"52026","subject":"Cannot shallow clone using GitHub tag refs after commit 34066f06","startedAt":"2019-10-12T22:31:06Z","lastAt":"2019-10-13T06:43:17Z","messageCount":2,"participants":["Teddy Reed","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"383979","messageId":"20191012183102.337fe74d728a4942a7de5ab7@gmail.com","threadId":"52026","inReplyTo":null,"subject":"Cannot shallow clone using GitHub tag refs after commit 34066f06","fromName":"Teddy Reed","fromEmail":"teddy.reed@gmail.com","sentAt":"2019-10-12T22:31:02Z","receivedAt":"2019-10-12T22:31:06Z","isPatch":false,"sender":{"key":"teddy.reed@gmail.com","avatar":null},"body":"Hello, I am trying to debug an issue where a shallow clone (--depth=1) fails when the reference is the tip of a tag on a GitHub repository.\n\nThe git client works on versions below 2.22.0-rc0 up until commit 34066f06, \"fetch: do not consider peeled tags as advertised tips\". That commit peels refs in the form of \"refs/tags/v2.2.2^{}\". My question is if the intention was to peel these refs and break this functionality? I do not know enough about git to understand if the behavior before commit 34066f06 was unintended.\n\nHere is a method to reproduce the previous behavior:\n\n$ mkdir test-repo\n$ cd test-repo\n$ git init\n$ git submodule add https://github.com/gflags/gflags gflags\n$ (cd gflags; git checkout v2.2.2)\n$ git add .\n$ git commit -m 'Add gflags submodule'\n\nThen with git client version 2.21.0:\n\n$ git clone test-repo test-repo2\n$ cd test-repo2\n$ git submodule update --init --depth=1\n\nSubmodule 'gflags' (https://github.com/gflags/gflags) registered for path 'gflags'\nCloning into '/tmp/test-repo2/gflags'...\nremote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0\nremote: Enumerating objects: 17, done.\nremote: Counting objects: 100% (17/17), done.\nremote: Compressing objects: 100% (9/9), done.\nremote: Total 9 (delta 8), reused 1 (delta 0), pack-reused 0\nUnpacking objects: 100% (9/9), done.\nFrom https://github.com/gflags/gflags\n * branch            e171aa2d15ed9eb17054558e0b3a6a413bb01067 -> FETCH_HEAD\nSubmodule path 'gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'\n\nAnd attempting with newer versions of the git client:\n\nCloning into '/tmp/test-repo2/gflags'...\nerror: Server does not allow request for unadvertised object e171aa2d15ed9eb17054558e0b3a6a413bb01067\nFetched in submodule path 'gflags', but it did not contain e171aa2d15ed9eb17054558e0b3a6a413bb01067. Direct fetching of that commit failed.\n\nThanks for the help!\n\n-- \nTeddy Reed <teddy.reed@gmail.com>\n"},{"id":"383985","messageId":"20191013064314.GA28018@sigill.intra.peff.net","threadId":"52026","inReplyTo":"20191012183102.337fe74d728a4942a7de5ab7@gmail.com","subject":"Re: Cannot shallow clone using GitHub tag refs after commit 34066f06","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-13T06:43:14Z","receivedAt":"2019-10-13T06:43:17Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Oct 12, 2019 at 06:31:02PM -0400, Teddy Reed wrote:\n\n> Hello, I am trying to debug an issue where a shallow clone (--depth=1)\n> fails when the reference is the tip of a tag on a GitHub repository.\n> \n> The git client works on versions below 2.22.0-rc0 up until commit\n> 34066f06, \"fetch: do not consider peeled tags as advertised tips\".\n> That commit peels refs in the form of \"refs/tags/v2.2.2^{}\". My\n> question is if the intention was to peel these refs and break this\n> functionality? I do not know enough about git to understand if the\n> behavior before commit 34066f06 was unintended.\n\nThis is a known and intended outcome of that commit. In fact, that fetch\nwas never _supposed_ to work, but it did in certain circumstances for\na few versions (2.14.x through 2.21.x).\n\nHere are a few background facts:\n\n  - upload-pack does not (and never has) considered the peeled value of\n    a ref to be \"advertised\"\n\n  - unless configured otherwise, upload-pack will generally refuse to\n    serve requests for objects it didn't advertise. This has been the\n    case since day one, _except_ for one interesting exception, which\n    I'll get into below.\n\n  - fetch-pack can realize that the server upload-pack is not\n    configured to allow unadvertised requests (by the server not\n    advertising that capability string), and refuses to even send such a\n    request to the server\n\n  - a bug in fdb69d33c4 (in Git 2.14) meant that for this specific case\n    (an object mentioned only as a peeled value), fetch-pack's check was\n    wrong, and it would ask the server for the object anyway\n\nSo prior to the bug in fdb69d33c4, fetch-pack would refuse the request.\nAfter that bug (and until the fix in 34066f06), it would make the\nrequest and the server would sometimes honor it and sometimes not.\n\nAnd that's where we get into the one exception. Since 051e4005a3\n(helping smart-http/stateless-rpc fetch race, 2011-08-05), the server\nupload-pack implicitly allows requesting reachable non-tip objects over\nhttp only.  But it does so quietly, without making a capability\nadvertisement to the client.\n\nSo whether this request works depends on your client version, the server\nversion, and the protocol in use. I didn't test each combo, but it\nshould be something like:\n\n  server    | client        | protocol     | works?\n  -------------------------------------------------\n  <1.7.6.1  | any           | http/ssh/git | no\n  >=1.7.6.1 | <2.14         | http/ssh/git | no\n  >=1.7.6.1 | >=2.14, <2.22 | http         | yes\n  >=1.7.6.1 | >=2.14, <2.22 | ssh/git      | no\n  >=1.7.6.1 | >=2.22        | http/ssh/git | no\n\nSo that's the back-story, but of course it's still annoying when it\ndoesn't work with a submodule. Where can we go from here?\n\n  - if you enable protocol v2 (by setting `protocol.version` to `2` in\n    your git config), its rules are different: it implicitly allows\n    fetching any object, reachable or not.\n\n    Arguably GitHub ought to set uploadpack.allowAnySHA1InWant for all\n    repositories, to let the v0 protocol behave the same way (after all,\n    any such object is available by making a v2 request already). The\n    fact that it hasn't is mostly history and inertia, along with some\n    open questions about whether v2 was perhaps a bit too cavalier in\n    this regard (but at this point, I think it's mostly a done deal).\n\n  - fetch-pack could assume that fetching via http means the server has\n    implicitly enabled uploadpack.allowReachableSHA1InWant, which it\n    effectively has since 1.7.6.1. Of course we might be talking to\n    another server implementation (I have no idea how JGit works there,\n    for example)\n\n  - the flip side of that is: upload-pack could always advertise the\n    \"allow-reachable-sha1-in-want\" capability when it's speaking http,\n    since it has effectively enabled it anyway.\n\n  - this is mostly of interest for submodules, which is the case for\n    your repo. Your output has this:\n\n> Cloning into '/tmp/test-repo2/gflags'...\n> error: Server does not allow request for unadvertised object e171aa2d15ed9eb17054558e0b3a6a413bb01067\n> Fetched in submodule path 'gflags', but it did not contain e171aa2d15ed9eb17054558e0b3a6a413bb01067. Direct fetching of that commit failed.\n\n   which I think is a problem in the submodule code. If a direct fetch\n   doesn't work, it's supposed to do a larger fetch as a fallback\n   (exactly to cope with this situation that a server's tips have moved\n   on). If it's not getting the object (which we know is referenced\n   directly by a tag!) that is a sign that it could be trying harder.\n\nSo I think my first suggestion above is something you can do immediately\nto make your case work. The rest are possible changes that could be made\nin Git. I do feel like one easy path forward, though, is to just assume\nwe'll flip the default to the v2 protocol at some point, and all of this\nwill go away.\n\n-Peff\n"}]}