{"thread":{"id":"47836","subject":"Can we make git clone --recurse-submodules --shallow-submodules work for commits that are not on tags or branches","startedAt":"2018-02-14T05:07:12Z","lastAt":"2018-02-14T19:18:21Z","messageCount":2,"participants":["Ciro Santilli","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"339260","messageId":"CAFXrp_cP-WaXO27VV1OcVakfmDX0b6M62Qz1j4XRCbtQJzFfnw@mail.gmail.com","threadId":"47836","inReplyTo":null,"subject":"Can we make git clone --recurse-submodules --shallow-submodules work for commits that are not on tags or branches","fromName":"Ciro Santilli","fromEmail":"ciro.santilli@gmail.com","sentAt":"2018-02-14T05:06:40Z","receivedAt":"2018-02-14T05:07:12Z","isPatch":false,"sender":{"key":"ciro.santilli@gmail.com","avatar":"https://gravatar.com/avatar/24a59ac21bfa4133ba88106940ec03cc546a14b2afd61ff6b810b06b9732a772?d=mp&s=160"},"body":"I have a git repo with large submodules, notably the Linux kernel, and\nshallow clone saves a lot of time.\n\nHowever, QEMU is also a submodule, which I don't control, and QEMU has\nsubmodules which don't point to commits that are not on any tag or\nbranch, so:\n\n git clone --recurse-submodules --shallow-submodules git://git.qemu.org/qemu.git\n\ncurrently fails with:\n\nerror: Server does not allow request for unadvertised object\n22ead3e0bfdb87516656453336160e0a37b066bf\nFetched in submodule path 'capstone', but it did not contain\n22ead3e0bfdb87516656453336160e0a37b066bf. Direct fetching of that\ncommit failed.\n\non git 2.16.1.\n\nFurthermore, I reproduce this locally with direct filesystem clones:\nhttps://github.com/cirosantilli/test-git-web-interface/blob/15335d3002a3e64fc5756b69fb832a733aa63fb9/shallow-submodule.sh#L158\nand on GitHub, so I'm guessing it is not just the settings for a\nspecific server?\n\nWould it be possible to make that work, or are there fundamental\nreasons why it is not possible?\n\nHere is my use case repo, at the point of the ugly workaround I'm\nhaving to do: https://github.com/cirosantilli/linux-kernel-module-cheat/blob/a14c95346cfd9d2e7b2e475b0981aa71d819c20b/configure#L23\n\nSome more context:\nhttps://stackoverflow.com/questions/2144406/git-shallow-submodules/47374702#47374702\n\nThis would make some embedded people happy.\n"},{"id":"339388","messageId":"CAGZ79ka6UXKyVLmdLg_M5-sB1x96g8FRzRZy=ENy5aJBQf9_QA@mail.gmail.com","threadId":"47836","inReplyTo":"CAFXrp_cP-WaXO27VV1OcVakfmDX0b6M62Qz1j4XRCbtQJzFfnw@mail.gmail.com","subject":"Re: Can we make git clone --recurse-submodules --shallow-submodules work for commits that are not on tags or branches","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-14T19:18:11Z","receivedAt":"2018-02-14T19:18:21Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 13, 2018 at 9:06 PM, Ciro Santilli <ciro.santilli@gmail.com> wrote:\n> I have a git repo with large submodules, notably the Linux kernel, and\n> shallow clone saves a lot of time.\n>\n> However, QEMU is also a submodule, which I don't control, and QEMU has\n> submodules which don't point to commits that are not on any tag or\n> branch, so:\n>\n>  git clone --recurse-submodules --shallow-submodules git://git.qemu.org/qemu.git\n>\n> currently fails with:\n>\n> error: Server does not allow request for unadvertised object\n> 22ead3e0bfdb87516656453336160e0a37b066bf\n> Fetched in submodule path 'capstone', but it did not contain\n> 22ead3e0bfdb87516656453336160e0a37b066bf. Direct fetching of that\n> commit failed.\n>\n> on git 2.16.1.\n\nIn theory this should be easy. :)\n\nIn practice not so much, unfortunately. This is because cloning will just obtain\nthe latest tip of a branch (usually master). There is no mechanism in clone\nto specify the exact sha1 that is wanted.\n\nThe wire protocol supports for asking exact sha1s, so that should be covered.\n(Caveat: it only works if the server operator enables\nuploadpack.allowReachableSHA1InWant which github has not AFAICT)\n\ngit-fetch allows to fetch arbitrary sha1, so as a workaround you can run a fetch\nafter the recursive clone by using \"git submodule update\" as that will use\nfetches after the initial clone.\n\n\n> Furthermore, I reproduce this locally with direct filesystem clones:\n> https://github.com/cirosantilli/test-git-web-interface/blob/15335d3002a3e64fc5756b69fb832a733aa63fb9/shallow-submodule.sh#L158\n> and on GitHub, so I'm guessing it is not just the settings for a\n> specific server?\n>\n> Would it be possible to make that work, or are there fundamental\n> reasons why it is not possible?\n\nThe wire protocol allows for it, so it should be possible fundamentally.\nThere is a redesign of the wire protocol going on currently, which\ncould allow for a new fetch mode that allows having just one\nconnection, which would only negotiate on the superproject and\ninfer the submodule parts from the superproject.\n\n\n> Here is my use case repo, at the point of the ugly workaround I'm\n> having to do: https://github.com/cirosantilli/linux-kernel-module-cheat/blob/a14c95346cfd9d2e7b2e475b0981aa71d819c20b/configure#L23\n>\n> Some more context:\n> https://stackoverflow.com/questions/2144406/git-shallow-submodules/47374702#47374702\n>\n> This would make some embedded people happy.\n\nYes, a lot of submodule users would be happy if that\nissue is solved.\n\nThanks for reporting,\nStefan\n"}]}