Partial clone fails with file:/// URI

Hi friends,

When I run a git clone with --filter and a file:/// URI - it fails.

Repro steps:

I have the `git` repo cloned and built into the CWD at v2.28.0 git/git clone --filter=blob:none file:///home/nipunn/src/git git2

nipunn@nipunn-dbx:~/src$ git/git clone --filter=blob:none file:///home/nipunn/src/git git2 Cloning into 'git2'... warning: templates not found in /home/nipunn/share/git-core/templates remote: Enumerating objects: 177110, done. remote: Counting objects: 100% (177110/177110), done. remote: Compressing objects: 100% (61436/61436), done. Receiving objects: 100% (177110/177110), 62.58 MiB | 43.95 MiB/s, done. remote: Total 177110 (delta 114305), reused 176904 (delta 114102), pack-reused 0 Resolving deltas: 100% (114305/114305), done. fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry with 'git restore --source=HEAD :/'

It seems to work ok with a remote URI or w/o a --filter arg. I was able to confirm that the "bad pack header" is a "-1" response code PH_ERROR_EOF Perhaps the `fatal: protocol error: bad pack header` - should include the error code as well. Thanks --Nipunn

Hi Nipunn,

On Wed, Oct 14, 2020 at 05:38:15PM +0100, Nipunn Koorapati wrote:

I couldn't quite reproduce this. Here's my reproduction:

  #!/bin/sh

  rm -fr repo clone
  git init repo

  (
    cd repo

    git config uploadpack.allowfilter true
    git config uploadpack.allowanysha1inwant true

    for f in a b; do
      echo $f > $f
      git add $f
      git commit -m "$f: initial commit"
    done
  )

  git clone --filter=blob:none file://$(pwd)/repo clone

That works on v2.28.0, and v2.29.0 for me. Is git installed properly on your system?

Thanks, Taylor

Hi,

I've been building git from source and directly using the executable. Roughly ``` cd git make cd .. git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git ```

This is what I was missing: git config uploadpack.allowanysha1inwant true

If this is not set, then the checkout fails w/ a

fatal: protocol error: bad pack header

Perhaps the error message and/or docs could be improved here to something with more detail than "bad pack header"? Not seeing anything to suggest this in the docs https://git-scm.com/docs/partial-clone

Thanks for the prompt response. Appreciated! --Nipunn

On Wed, Oct 14, 2020 at 5:56 PM Taylor Blau <me@ttaylorr.com> wrote:

On Wed, Oct 14, 2020 at 06:04:18PM +0100, Nipunn Koorapati wrote:

Ah. OK; I feel much better knowing what the difference between our two setups is. This is a long-standing issue (I remember seeing it when I was experimenting with partial clones several releases ago) so I'm glad that we _don't_ have to treat this as a regression in 2.29.

Thanks, Taylor

I improved the "bad pack header" error message in a patch off of v2.28.0 (https://github.com/gitgitgadget/git/pull/755) However, rebasing onto v2.29.0, it seems like the error no longer occurs during checkout. Was there some recent improvement? Looking through the changelog, I'm not finding anything obvious.

It appears that regardless of git config uploadpack.allowanysha1inwant true

we find `git checkout` succeeds on v2.29.0

There is a test case in the PR above Also a manual repro:

cd git git checkout v2.29.0-rc1 make cd .. git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git

Thanks --Nipunn

On Wed, Oct 14, 2020 at 6:27 PM Taylor Blau <me@ttaylorr.com> wrote:

On Wed, Oct 14, 2020 at 06:04:18PM +0100, Nipunn Koorapati wrote:

Note that running a built copy directly without installing may end up running a different version of Git under the hood. Your initial "git/git" will run the newly built version, but it will put the to-be-installed exec-path at the front of $PATH (probably ~/libexec/git-core if you haven't configured anything). But since you haven't installed anything there, that won't do anything, and instead any git subprocesses it runs (including the server-side upload-pack and any child fetches it runs to get the blobs) will be done with whatever version of git is in your $PATH already.

Try:

  git/bin-wrappers/git clone ...

which is a shell script that sets up the exec-path properly for running out of the build directory.

When I run your commands, I get:

  error: Server does not allow request for unadvertised object d16e5f041f5d3ad91129c4fe6713c71727895e9c
  error: unable to read sha1 file of Documentation/RelNotes/2.1.4.txt (d16e5f041f5d3ad91129c4fe6713c71727895e9c)

and so on for each object. Some older versions of Git did not correctly handle this situation (they asked for the object even though the server did not claim to allow it). It was fixed in 2b69d3116f (Merge branch 'mm/fetch-show-error-message-on-unadvertised-object' into maint, 2017-03-28), I think, in v2.21.3. If your system Git is older than that, this might be the culprit.

Note that you shouldn't need to do that starting with v2.29, because the v2 protocol is the default (which always allows any sha1 to be requested).

The problem is that the client is hanging up when we are trying to read the pack header. I think in newer versions of Git this is better, as well (the server is better about sending an over-the-wire ERR packet, and the client is better about reading and relaying it, though there are still some races around write() errors here).

-Peff