{"thread":{"id":"54418","subject":"Partial clone fails with file:/// URI","startedAt":"2020-10-14T16:38:30Z","lastAt":"2020-10-15T02:09:55Z","messageCount":6,"participants":["Nipunn Koorapati","Taylor Blau","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"407521","messageId":"CAN8Z4-XgctFZxZoTWRpD1V9NFr34ObzG2dxUoAfuJ4NOsBDdtg@mail.gmail.com","threadId":"54418","inReplyTo":null,"subject":"Partial clone fails with file:/// URI","fromName":"Nipunn Koorapati","fromEmail":"nipunn1313@gmail.com","sentAt":"2020-10-14T16:38:15Z","receivedAt":"2020-10-14T16:38:30Z","isPatch":false,"sender":{"key":"nipunn1313@gmail.com","avatar":"https://gravatar.com/avatar/d0b19cc6499ffcae349d237d7166f5fa0fc29942783a61894ccdb5ee5246ac97?d=mp&s=160"},"body":"Hi friends,\n\nWhen I run a git clone with --filter and a file:/// URI - it fails.\n\nRepro steps:\n\nI have the `git` repo cloned and built into the CWD at v2.28.0\ngit/git clone --filter=blob:none file:///home/nipunn/src/git git2\n\nnipunn@nipunn-dbx:~/src$ git/git clone --filter=blob:none\nfile:///home/nipunn/src/git git2\nCloning into 'git2'...\nwarning: templates not found in /home/nipunn/share/git-core/templates\nremote: Enumerating objects: 177110, done.\nremote: Counting objects: 100% (177110/177110), done.\nremote: Compressing objects: 100% (61436/61436), done.\nReceiving objects: 100% (177110/177110), 62.58 MiB | 43.95 MiB/s, done.\nremote: Total 177110 (delta 114305), reused 176904 (delta 114102), pack-reused 0\nResolving deltas: 100% (114305/114305), done.\nfatal: the remote end hung up unexpectedly\nfatal: protocol error: bad pack header\nwarning: Clone succeeded, but checkout failed.\nYou can inspect what was checked out with 'git status'\nand retry with 'git restore --source=HEAD :/'\n\nIt seems to work ok with a remote URI or w/o a --filter arg.\nI was able to confirm that the \"bad pack header\" is a \"-1\" response\ncode PH_ERROR_EOF\nPerhaps the `fatal: protocol error: bad pack header` - should include\nthe error code as well.\nThanks\n--Nipunn\n"},{"id":"407523","messageId":"20201014165607.GA2044338@nand.local","threadId":"54418","inReplyTo":"CAN8Z4-XgctFZxZoTWRpD1V9NFr34ObzG2dxUoAfuJ4NOsBDdtg@mail.gmail.com","subject":"Re: Partial clone fails with file:/// URI","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-14T16:56:07Z","receivedAt":"2020-10-14T16:56:13Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Nipunn,\n\nOn Wed, Oct 14, 2020 at 05:38:15PM +0100, Nipunn Koorapati wrote:\n> Hi friends,\n>\n> When I run a git clone with --filter and a file:/// URI - it fails.\n>\n> Repro steps:\n>\n> I have the `git` repo cloned and built into the CWD at v2.28.0\n> git/git clone --filter=blob:none file:///home/nipunn/src/git git2\n\nI couldn't quite reproduce this. Here's my reproduction:\n\n  #!/bin/sh\n\n  rm -fr repo clone\n  git init repo\n\n  (\n    cd repo\n\n    git config uploadpack.allowfilter true\n    git config uploadpack.allowanysha1inwant true\n\n    for f in a b; do\n      echo $f > $f\n      git add $f\n      git commit -m \"$f: initial commit\"\n    done\n  )\n\n  git clone --filter=blob:none file://$(pwd)/repo clone\n\nThat works on v2.28.0, and v2.29.0 for me. Is git installed properly on\nyour system?\n\nThanks,\nTaylor\n"},{"id":"407526","messageId":"CAN8Z4-XGbb_z3vmJYATONmBkxBcfxwyC-_4XEOkP0sxsdT9wxw@mail.gmail.com","threadId":"54418","inReplyTo":"20201014165607.GA2044338@nand.local","subject":"Re: Partial clone fails with file:/// URI","fromName":"Nipunn Koorapati","fromEmail":"nipunn1313@gmail.com","sentAt":"2020-10-14T17:04:18Z","receivedAt":"2020-10-14T17:04:31Z","isPatch":false,"sender":{"key":"nipunn1313@gmail.com","avatar":"https://gravatar.com/avatar/d0b19cc6499ffcae349d237d7166f5fa0fc29942783a61894ccdb5ee5246ac97?d=mp&s=160"},"body":"Hi,\n\nI've been building git from source and directly using the executable. Roughly\n```\ncd git\nmake\ncd ..\ngit/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git\n```\n\nThis is what I was missing:\ngit config uploadpack.allowanysha1inwant true\n\nIf this is not set, then the checkout fails w/ a\n\nfatal: protocol error: bad pack header\n\nPerhaps the error message and/or docs could be improved here to\nsomething with more detail than \"bad pack header\"?\nNot seeing anything to suggest this in the docs\nhttps://git-scm.com/docs/partial-clone\n\nThanks for the prompt response. Appreciated!\n--Nipunn\n\nOn Wed, Oct 14, 2020 at 5:56 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> Hi Nipunn,\n>\n> On Wed, Oct 14, 2020 at 05:38:15PM +0100, Nipunn Koorapati wrote:\n> > Hi friends,\n> >\n> > When I run a git clone with --filter and a file:/// URI - it fails.\n> >\n> > Repro steps:\n> >\n> > I have the `git` repo cloned and built into the CWD at v2.28.0\n> > git/git clone --filter=blob:none file:///home/nipunn/src/git git2\n>\n> I couldn't quite reproduce this. Here's my reproduction:\n>\n>   #!/bin/sh\n>\n>   rm -fr repo clone\n>   git init repo\n>\n>   (\n>     cd repo\n>\n>     git config uploadpack.allowfilter true\n>     git config uploadpack.allowanysha1inwant true\n>\n>     for f in a b; do\n>       echo $f > $f\n>       git add $f\n>       git commit -m \"$f: initial commit\"\n>     done\n>   )\n>\n>   git clone --filter=blob:none file://$(pwd)/repo clone\n>\n> That works on v2.28.0, and v2.29.0 for me. Is git installed properly on\n> your system?\n>\n> Thanks,\n> Taylor\n"},{"id":"407530","messageId":"20201014172746.GB2044338@nand.local","threadId":"54418","inReplyTo":"CAN8Z4-XGbb_z3vmJYATONmBkxBcfxwyC-_4XEOkP0sxsdT9wxw@mail.gmail.com","subject":"Re: Partial clone fails with file:/// URI","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-14T17:27:46Z","receivedAt":"2020-10-14T17:27:53Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Oct 14, 2020 at 06:04:18PM +0100, Nipunn Koorapati wrote:\n> Hi,\n>\n> I've been building git from source and directly using the executable. Roughly\n> ```\n> cd git\n> make\n> cd ..\n> git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git\n> ```\n>\n> This is what I was missing:\n> git config uploadpack.allowanysha1inwant true\n>\n> If this is not set, then the checkout fails w/ a\n>\n> fatal: protocol error: bad pack header\n>\n> Perhaps the error message and/or docs could be improved here to\n> something with more detail than \"bad pack header\"?\n> Not seeing anything to suggest this in the docs\n> https://git-scm.com/docs/partial-clone\n\nAh. OK; I feel much better knowing what the difference between our two\nsetups is. This is a long-standing issue (I remember seeing it when I\nwas experimenting with partial clones several releases ago) so I'm glad\nthat we _don't_ have to treat this as a regression in 2.29.\n\n> Thanks for the prompt response. Appreciated!\n> --Nipunn\n\nThanks,\nTaylor\n"},{"id":"407555","messageId":"20201015012054.GA384995@coredump.intra.peff.net","threadId":"54418","inReplyTo":"CAN8Z4-XGbb_z3vmJYATONmBkxBcfxwyC-_4XEOkP0sxsdT9wxw@mail.gmail.com","subject":"Re: Partial clone fails with file:/// URI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-15T01:20:54Z","receivedAt":"2020-10-15T01:20:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 14, 2020 at 06:04:18PM +0100, Nipunn Koorapati wrote:\n\n> I've been building git from source and directly using the executable. Roughly\n> ```\n> cd git\n> make\n> cd ..\n> git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git\n> ```\n\nNote that running a built copy directly without installing may end up\nrunning a different version of Git under the hood. Your initial\n\"git/git\" will run the newly built version, but it will put the\nto-be-installed exec-path at the front of $PATH (probably\n~/libexec/git-core if you haven't configured anything). But since you\nhaven't installed anything there, that won't do anything, and instead\nany git subprocesses it runs (including the server-side upload-pack and\nany child fetches it runs to get the blobs) will be done with whatever\nversion of git is in your $PATH already.\n\nTry:\n\n  git/bin-wrappers/git clone ...\n\nwhich is a shell script that sets up the exec-path properly for running\nout of the build directory.\n\nWhen I run your commands, I get:\n\n  error: Server does not allow request for unadvertised object d16e5f041f5d3ad91129c4fe6713c71727895e9c\n  error: unable to read sha1 file of Documentation/RelNotes/2.1.4.txt (d16e5f041f5d3ad91129c4fe6713c71727895e9c)\n\nand so on for each object. Some older versions of Git did not correctly\nhandle this situation (they asked for the object even though the server\ndid not claim to allow it). It was fixed in 2b69d3116f (Merge branch\n'mm/fetch-show-error-message-on-unadvertised-object' into maint,\n2017-03-28), I think, in v2.21.3. If your system Git is older than that,\nthis might be the culprit.\n\n> This is what I was missing:\n> git config uploadpack.allowanysha1inwant true\n\nNote that you shouldn't need to do that starting with v2.29, because the\nv2 protocol is the default (which always allows any sha1 to be\nrequested).\n\n> If this is not set, then the checkout fails w/ a\n> \n> fatal: protocol error: bad pack header\n> \n> Perhaps the error message and/or docs could be improved here to\n> something with more detail than \"bad pack header\"?\n\nThe problem is that the client is hanging up when we are trying to read\nthe pack header. I think in newer versions of Git this is better, as\nwell (the server is better about sending an over-the-wire ERR packet,\nand the client is better about reading and relaying it, though there are\nstill some races around write() errors here).\n\n-Peff\n"},{"id":"407575","messageId":"CAN8Z4-V2zb4kLw9wgkWxf+Cs9NdgUu5YHnhtDZm0qiqf4WS+BA@mail.gmail.com","threadId":"54418","inReplyTo":"20201014172746.GB2044338@nand.local","subject":"Re: Partial clone fails with file:/// URI","fromName":"Nipunn Koorapati","fromEmail":"nipunn1313@gmail.com","sentAt":"2020-10-14T23:41:08Z","receivedAt":"2020-10-15T02:09:55Z","isPatch":false,"sender":{"key":"nipunn1313@gmail.com","avatar":"https://gravatar.com/avatar/d0b19cc6499ffcae349d237d7166f5fa0fc29942783a61894ccdb5ee5246ac97?d=mp&s=160"},"body":"I improved the \"bad pack header\" error message in a patch off of\nv2.28.0 (https://github.com/gitgitgadget/git/pull/755)\nHowever, rebasing onto v2.29.0, it seems like the error no longer\noccurs during checkout.\nWas there some recent improvement? Looking through the changelog, I'm\nnot finding anything obvious.\n\nIt appears that regardless of\ngit config uploadpack.allowanysha1inwant true\n\nwe find `git checkout` succeeds on v2.29.0\n\nThere is a test case in the PR above\nAlso a manual repro:\n\ncd git\ngit checkout v2.29.0-rc1\nmake\ncd ..\ngit/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git\n\nThanks\n--Nipunn\n\nOn Wed, Oct 14, 2020 at 6:27 PM Taylor Blau <me@ttaylorr.com> wrote:\n>\n> On Wed, Oct 14, 2020 at 06:04:18PM +0100, Nipunn Koorapati wrote:\n> > Hi,\n> >\n> > I've been building git from source and directly using the executable. Roughly\n> > ```\n> > cd git\n> > make\n> > cd ..\n> > git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git\n> > ```\n> >\n> > This is what I was missing:\n> > git config uploadpack.allowanysha1inwant true\n> >\n> > If this is not set, then the checkout fails w/ a\n> >\n> > fatal: protocol error: bad pack header\n> >\n> > Perhaps the error message and/or docs could be improved here to\n> > something with more detail than \"bad pack header\"?\n> > Not seeing anything to suggest this in the docs\n> > https://git-scm.com/docs/partial-clone\n>\n> Ah. OK; I feel much better knowing what the difference between our two\n> setups is. This is a long-standing issue (I remember seeing it when I\n> was experimenting with partial clones several releases ago) so I'm glad\n> that we _don't_ have to treat this as a regression in 2.29.\n>\n> > Thanks for the prompt response. Appreciated!\n> > --Nipunn\n>\n> Thanks,\n> Taylor\n"}]}