{"thread":{"id":"33361","subject":"[BUG] shallow clones over http","startedAt":"2013-04-03T04:03:52Z","lastAt":"2014-02-06T12:30:33Z","messageCount":7,"participants":["Jeff King","Tom","Ramkumar Ramachandra","Junio C Hamano","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"213021","messageId":"20130403040352.GA8284@sigill.intra.peff.net","threadId":"33361","inReplyTo":null,"subject":"[BUG] shallow clones over http","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-04-03T04:03:52Z","receivedAt":"2013-04-03T04:03:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"I'm trying to track down a protocol bug that happens with shallow clones\nover smart-http. As far as I can tell, the bug has existed in all\nversions.\n\nYou can reproduce it using the attached repository, which is a shallow\nclone of https://github.com/mileszs/ack.vim.git, like:\n\n  $ tar xzf repo.tar.gz\n  $ cd repo.git\n  $ git fetch --depth=10\n  fatal: git fetch-pack: expected shallow list\n\nIn that test my fetch actually hit github.com as the upstream full repo,\nbut you can also clone it down locally and demonstrate it with purely\nlocal copies of git (but it's more of a pain, because you have to set up\na smart http server).\n\nThe last part of the conversation looks like this:\n\n  packet:   fetch-pack< 0000\n  packet:   fetch-pack< ACK f183a345a0c10caed7684d07dabae33e007c7590 common\n  packet:   fetch-pack> have f183a345a0c10caed7684d07dabae33e007c7590\n  packet:   fetch-pack< ACK 33312d4db4e91468957b1b41dd039c5d88e85fda common\n  packet:   fetch-pack< ACK 34d0b2fbc182b31d926632d170bc07d6a6fc3f9b common\n  packet:   fetch-pack< ACK 45c802e07c60686986474b6b05b2c7048330b6b5 common\n  packet:   fetch-pack< ACK e93f693fd2a9940d6421bf9e4ddd1f535994eaa5 common\n  packet:   fetch-pack< ACK 132ee41e8e2c8c545b3aed120171e1596c9211a4 common\n  packet:   fetch-pack< ACK 973deb3145a99992638b2301cfd654721cf35d68 common\n  packet:   fetch-pack< ACK e53a88a4e72d84562493313e8911ada4def787da common\n  packet:   fetch-pack< ACK 90be0bf3eee6f7a0cb9c2377a50610f4ce738da3 common\n  packet:   fetch-pack< ACK aeab88ccf41bf216fde37983bd403d9b913391e7 common\n  packet:   fetch-pack< ACK 5f480935d3ce431c393657c3000337bcbdbd5535 common\n  packet:   fetch-pack< ACK db81e01b433501b159983ea38690aeb01eea1e6b common\n  packet:   fetch-pack< ACK 06c44b8cab93e780a29ff7f7b5b1dd41dba4b2d5 common\n  packet:   fetch-pack< ACK 65f3966becdb2d931d5afbdcc6a28008d154668a common\n  packet:   fetch-pack< ACK 10e8caef9f2ed308231ce1abc326c512e86a5d4c common\n  packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2 common\n  packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2 ready\n  packet:   fetch-pack< NAK\n  packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2\n  fatal: git fetch-pack: expected shallow list\n\nSo we see that upload-pack sends a bunch of detailed ACKs, followed by a\nNAK, and then it sends another ACK.\n\nFetch-pack is inside find_common, reading these acks. At the beginning\nof each stateless-rpc response, it expects to consume any\nshallow/unshallow lines up to a flush packet (the call to\nconsume_shallow_list). And then it reads the acks in a loop. After it\nsees the NAK, it assumes that the server is done sending the packet, and\nloops again, expecting another set of shallow/unshallow lines. But we\nget the next ACK instead, and die.\n\nSo who is wrong? Is upload-pack wrong to send an ACK after the NAK? From\nthe protocol description, it seems like it should wait after sending a\nNAK. Or is fetch-pack wrong not to handle the extra ACK without\nreceiving another shallow/unshallow segment?\n\n-Peff\n"},{"id":"213766","messageId":"loom.20130410T135840-387@post.gmane.org","threadId":"33361","inReplyTo":"20130403040352.GA8284@sigill.intra.peff.net","subject":"Re: [BUG] shallow clones over http","fromName":"Tom","fromEmail":"mail@tgries.de","sentAt":"2013-04-10T12:01:27Z","receivedAt":"2013-04-10T12:01:27Z","isPatch":false,"sender":{"key":"mail@tgries.de","avatar":null},"body":"Jeff King <peff <at> peff.net> writes:\n\n> \n> I'm trying to track down a protocol bug that happens with shallow clones\n> over smart-http. As far as I can tell, the bug has existed in all\n> versions.\n> \n> You can reproduce it using the attached repository, which is a shallow\n> clone of https://github.com/mileszs/ack.vim.git, like:\n> \n>   $ tar xzf repo.tar.gz\n>   $ cd repo.git\n>   $ git fetch --depth=10\n>   fatal: git fetch-pack: expected shallow list\n\nThe problem occurs to me also when I want to \"deepen\" a shallow clone of\nMediaWiki via https://\n\n git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git\n git pull --depth=99999\n\nfatal: git fetch-pack: expected shallow list.\n\nPerhaps it helps someone to find the reason.\n"},{"id":"213767","messageId":"CALkWK0mSiWJ_O-jqu2XZ3UvTsa4P5V0Bi8j67-dtCBR2Mb56Bg@mail.gmail.com","threadId":"33361","inReplyTo":"loom.20130410T135840-387@post.gmane.org","subject":"Re: [BUG] shallow clones over http","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-10T12:14:15Z","receivedAt":"2013-04-10T12:14:15Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Tom wrote:\n> git pull --depth=99999\n\nUnrelated nit: we now have git fetch --unshallow.\n"},{"id":"213768","messageId":"loom.20130410T144700-304@post.gmane.org","threadId":"33361","inReplyTo":"loom.20130410T135840-387@post.gmane.org","subject":"Re: [BUG] shallow clones over http","fromName":"Tom","fromEmail":"mail@tgries.de","sentAt":"2013-04-10T12:48:51Z","receivedAt":"2013-04-10T12:48:51Z","isPatch":false,"sender":{"key":"mail@tgries.de","avatar":null},"body":"The problem occurs to me also when I want to \"deepen\" a shallow clone of\nMediaWiki via https://\n\n git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git\n git pull --depth=99999\n\nfatal: git fetch-pack: expected shallow list.\n\nPerhaps it helps someone to find the reason.\n\nUPDATE:\n\nThis however works\n\n git clone --depth=2 https://gerrit.wikimedia.org/r/p/mediawiki/core.git\n git fetch --depth=5\n"},{"id":"214127","messageId":"20130413060310.GA21921@sigill.intra.peff.net","threadId":"33361","inReplyTo":"loom.20130410T144700-304@post.gmane.org","subject":"Re: [BUG] shallow clones over http","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-04-13T06:03:10Z","receivedAt":"2013-04-13T06:03:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 10, 2013 at 12:48:51PM +0000, Tom wrote:\n\n> The problem occurs to me also when I want to \"deepen\" a shallow clone of\n> MediaWiki via https://\n> \n>  git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git\n>  git pull --depth=99999\n> \n> fatal: git fetch-pack: expected shallow list.\n> \n> Perhaps it helps someone to find the reason.\n> \n> UPDATE:\n> \n> This however works\n> \n>  git clone --depth=2 https://gerrit.wikimedia.org/r/p/mediawiki/core.git\n>  git fetch --depth=5\n\nYes, I experienced the same thing. I don't think the problem is related\nto the depth per se, but rather the particular pattern of ACKs that\nupload-pack sends based on which commits are being selected.\n\n-Peff\n"},{"id":"216722","messageId":"7v38txffqn.fsf@alter.siamese.dyndns.org","threadId":"33361","inReplyTo":"20130403040352.GA8284@sigill.intra.peff.net","subject":"Re: [BUG] shallow clones over http","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-05-08T19:12:16Z","receivedAt":"2013-05-08T19:12:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> I'm trying to track down a protocol bug that happens with shallow clones\n> over smart-http. As far as I can tell, the bug has existed in all\n> versions.\n>\n> You can reproduce it using the attached repository, which is a shallow\n> clone of https://github.com/mileszs/ack.vim.git, like:\n>\n>   $ tar xzf repo.tar.gz\n>   $ cd repo.git\n>   $ git fetch --depth=10\n>   fatal: git fetch-pack: expected shallow list\n>\n> In that test my fetch actually hit github.com as the upstream full repo,\n> but you can also clone it down locally and demonstrate it with purely\n> local copies of git (but it's more of a pain, because you have to set up\n> a smart http server).\n>\n> The last part of the conversation looks like this:\n>\n>   packet:   fetch-pack< 0000\n>   packet:   fetch-pack< ACK f183a345a0c10caed7684d07dabae33e007c7590 common\n>   packet:   fetch-pack> have f183a345a0c10caed7684d07dabae33e007c7590\n>   packet:   fetch-pack< ACK 33312d4db4e91468957b1b41dd039c5d88e85fda common\n>   packet:   fetch-pack< ACK 34d0b2fbc182b31d926632d170bc07d6a6fc3f9b common\n>   packet:   fetch-pack< ACK 45c802e07c60686986474b6b05b2c7048330b6b5 common\n>   packet:   fetch-pack< ACK e93f693fd2a9940d6421bf9e4ddd1f535994eaa5 common\n>   packet:   fetch-pack< ACK 132ee41e8e2c8c545b3aed120171e1596c9211a4 common\n>   packet:   fetch-pack< ACK 973deb3145a99992638b2301cfd654721cf35d68 common\n>   packet:   fetch-pack< ACK e53a88a4e72d84562493313e8911ada4def787da common\n>   packet:   fetch-pack< ACK 90be0bf3eee6f7a0cb9c2377a50610f4ce738da3 common\n>   packet:   fetch-pack< ACK aeab88ccf41bf216fde37983bd403d9b913391e7 common\n>   packet:   fetch-pack< ACK 5f480935d3ce431c393657c3000337bcbdbd5535 common\n>   packet:   fetch-pack< ACK db81e01b433501b159983ea38690aeb01eea1e6b common\n>   packet:   fetch-pack< ACK 06c44b8cab93e780a29ff7f7b5b1dd41dba4b2d5 common\n>   packet:   fetch-pack< ACK 65f3966becdb2d931d5afbdcc6a28008d154668a common\n>   packet:   fetch-pack< ACK 10e8caef9f2ed308231ce1abc326c512e86a5d4c common\n>   packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2 common\n>   packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2 ready\n>   packet:   fetch-pack< NAK\n>   packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2\n>   fatal: git fetch-pack: expected shallow list\n>\n> So we see that upload-pack sends a bunch of detailed ACKs, followed by a\n> NAK, and then it sends another ACK.\n>\n> Fetch-pack is inside find_common, reading these acks. At the beginning\n> of each stateless-rpc response, it expects to consume any\n> shallow/unshallow lines up to a flush packet (the call to\n> consume_shallow_list). And then it reads the acks in a loop. After it\n> sees the NAK, it assumes that the server is done sending the packet, and\n> loops again, expecting another set of shallow/unshallow lines. But we\n> get the next ACK instead, and die.\n>\n> So who is wrong? Is upload-pack wrong to send an ACK after the NAK?\n\n3e63b21aced1 (upload-pack: Implement no-done capability, 2011-03-14)\nclaims that the above sequence of acks and naks is what upload-pack\nwants to show.\n\nWhat happens when you disable no-done extension handling on the\nserver end, I wonder?\n"},{"id":"234362","messageId":"CACsJy8Bx6voBepHLmmB0qfXx8MHtrBebB2tsan9iQipC-VCVyw@mail.gmail.com","threadId":"33361","inReplyTo":"7v38txffqn.fsf@alter.siamese.dyndns.org","subject":"Re: [BUG] shallow clones over http","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-06T12:30:33Z","receivedAt":"2014-02-06T12:30:33Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"(Digging back an old topic after Jeff mentioned it)\n\nOn Thu, May 9, 2013 at 2:12 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Jeff King <peff@peff.net> writes:\n>\n>> I'm trying to track down a protocol bug that happens with shallow clones\n>> over smart-http. As far as I can tell, the bug has existed in all\n>> versions.\n>>\n>> You can reproduce it using the attached repository, which is a shallow\n>> clone of https://github.com/mileszs/ack.vim.git, like:\n>>\n>>   $ tar xzf repo.tar.gz\n>>   $ cd repo.git\n>>   $ git fetch --depth=10\n>>   fatal: git fetch-pack: expected shallow list\n>>\n>> In that test my fetch actually hit github.com as the upstream full repo,\n>> but you can also clone it down locally and demonstrate it with purely\n>> local copies of git (but it's more of a pain, because you have to set up\n>> a smart http server).\n>>\n>> The last part of the conversation looks like this:\n>>\n>>   packet:   fetch-pack< 0000\n>>   packet:   fetch-pack< ACK f183a345a0c10caed7684d07dabae33e007c7590 common\n>>   packet:   fetch-pack> have f183a345a0c10caed7684d07dabae33e007c7590\n>>   packet:   fetch-pack< ACK 33312d4db4e91468957b1b41dd039c5d88e85fda common\n>>   packet:   fetch-pack< ACK 34d0b2fbc182b31d926632d170bc07d6a6fc3f9b common\n>>   packet:   fetch-pack< ACK 45c802e07c60686986474b6b05b2c7048330b6b5 common\n>>   packet:   fetch-pack< ACK e93f693fd2a9940d6421bf9e4ddd1f535994eaa5 common\n>>   packet:   fetch-pack< ACK 132ee41e8e2c8c545b3aed120171e1596c9211a4 common\n>>   packet:   fetch-pack< ACK 973deb3145a99992638b2301cfd654721cf35d68 common\n>>   packet:   fetch-pack< ACK e53a88a4e72d84562493313e8911ada4def787da common\n>>   packet:   fetch-pack< ACK 90be0bf3eee6f7a0cb9c2377a50610f4ce738da3 common\n>>   packet:   fetch-pack< ACK aeab88ccf41bf216fde37983bd403d9b913391e7 common\n>>   packet:   fetch-pack< ACK 5f480935d3ce431c393657c3000337bcbdbd5535 common\n>>   packet:   fetch-pack< ACK db81e01b433501b159983ea38690aeb01eea1e6b common\n>>   packet:   fetch-pack< ACK 06c44b8cab93e780a29ff7f7b5b1dd41dba4b2d5 common\n>>   packet:   fetch-pack< ACK 65f3966becdb2d931d5afbdcc6a28008d154668a common\n>>   packet:   fetch-pack< ACK 10e8caef9f2ed308231ce1abc326c512e86a5d4c common\n>>   packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2 common\n>>   packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2 ready\n>>   packet:   fetch-pack< NAK\n>>   packet:   fetch-pack< ACK 6b55dd91f2e7fc64c23eea57e85171cb958f9cd2\n>>   fatal: git fetch-pack: expected shallow list\n>>\n>> So we see that upload-pack sends a bunch of detailed ACKs, followed by a\n>> NAK, and then it sends another ACK.\n>>\n>> Fetch-pack is inside find_common, reading these acks. At the beginning\n>> of each stateless-rpc response, it expects to consume any\n>> shallow/unshallow lines up to a flush packet (the call to\n>> consume_shallow_list). And then it reads the acks in a loop. After it\n>> sees the NAK, it assumes that the server is done sending the packet, and\n>> loops again, expecting another set of shallow/unshallow lines. But we\n>> get the next ACK instead, and die.\n>>\n>> So who is wrong? Is upload-pack wrong to send an ACK after the NAK?\n>\n> 3e63b21aced1 (upload-pack: Implement no-done capability, 2011-03-14)\n> claims that the above sequence of acks and naks is what upload-pack\n> wants to show.\n>\n> What happens when you disable no-done extension handling on the\n> server end, I wonder?\n\nfetch succeeded when no-done was disabled. An immediate workaround\nwould be disable no-done in fetch-pack.c in a shallow repo but maybe\nwe can do better..\n-- \nDuy\n"}]}