Unexpected cat-file --batch-check output

I'm working with some users trying to reconcile an odd mismatch observed in some Git output.

Running an ls-tree for a branch and path, limited to a single pattern within, shows this: /usr/bin/git ls-tree -z refs/heads/develop:path/to/parent – file 100644 blob 4c8d566ed80a1554a059b97f7cd533a55bbd2ea8 file

If we then run cat-file --batch-check, though, we see this: echo 'refs/heads/develop refs/heads/develop:path/to/parent/file' | /usr/bin/git cat-file --batch-check 28a05ce2e3079afcb32e4f1777b42971d7933a91 commit 259 cc10f4b278086325aab2f95df97c807c7c6cd75e commit 330

There's a newline after the branch name, inside the single quotes, followed by the same branch name plus the full path. In this output, it comes back as a commit, though. Both commands were run with refs/heads/develop at the same commit. I've checked for a .gitmodules file and while they _do_ have submodules, they're at different, non-intersecting paths to the one in question here.

I can't share the actual repository (I don't have access to it myself), but I'm hoping someone might have some ideas. I've never seen this sort of mismatch before; for every path in the repositories I do have access to that I've tried this for, the cat-file --batch-check always shows "commit" (or "tag") for the ref, and then "blob" for the ref+path. Submodules were the only thing I could think of, but that doesn't appear to be the case. Could it be a subtree instead? How would I check?

Thanks in advance for any ideas; I appreciate any help. Bryan Turner

On Mon, Oct 25, 2021 at 12:02:38PM -0700, Bryan Turner wrote:

That's definitely odd. Some things I'd try:

  - do other versions of cat-file behave differently (i.e., is it a
    regression)?

  - what does "git rev-parse refs/heads/develop:path/to/parent/file"
    say? If it comes up with 4c8d566ed80, then the problem is cat-file
    specific. If not, then it's a problem in the name resolution
    routines.

  - likewise, what does "git cat-file -t cc10f4b27808" say? I'd expect
    it to really be a commit (a bug in batch-check's formatting routines
    could show the wrong object, but I'd expect the oid to at least
    match what ls-tree showed).

  - Is there anything odd about the tree? E.g., duplicate entries, out
    of order entries, etc? Examining "ls-tree" output might help, but
    "git fsck" should also note any irregularities.

After that, I'd probably start running "cat-file --batch-check" through a debugger. I know you said you don't have access to the repository, but perhaps whoever does might be willing to run it through "fast-export --anonymize" and see if the bug persists?

-Peff

On Mon, Oct 25, 2021 at 12:18 PM Jeff King <peff@peff.net> wrote:

I don't have that specific data, but one thing I do know is that cat-file -p prints commit contents:

/usr/bin/git cat-file -p refs/heads/develop:path/to/parent/file tree c378146c918c05794e5fb1d1f6986c81ca866326 parent 6cb6016c78c4c963311ca82fc53764141b0d3bdd author ... committer ...

<Commit message starts here>

One other observation. I threw cc10f4b278086325aab2f95df97c807c7c6cd75e into Github's search, on a lark, since so much open source is there, and it actually finds that commit in multiple repositories[1][2][3]

[1] https://github.com/bitcoin-sv/bitcoin-sv/commit/cc10f4b278086325aab2f95df97c807c7c6cd75e [2] https://github.com/fakecoinbase/bitcoin-svslashbitcoin-sv/commit/cc10f4b278086325aab2f95df97c807c7c6cd75e [3] https://github.com/TuringBitchain/TuringBitchain/commit/cc10f4b278086325aab2f95df97c807c7c6cd75e

None of those repositories has a branch named "develop", and the "file" I've obscured here is not present in any of them, so while there is clearly some ancestry in this repository with open source roots, it's evolved since then. Experimenting with some of the "nearby" files that are present in those public repositories, I have not been able to reproduce the issue in any of them.

I've sent some further commands, based on your suggestions, to the users.

fast-export --anonymize might be a way forward. Thanks for suggesting it (I always forget about it); I've mentioned it to the users.

Thanks Jeff; I appreciate your time/insights!

Best regards, Bryan Turner

A few quick updates to some of the questions:

On Mon, Oct 25, 2021 at 12:18 PM Jeff King <peff@peff.net> wrote:

They're using Git 2.32 built from source on Ubuntu 20.04. I may see if they can reinstall the 2.25.1 from focal's standard repositories and see if it reproduces the issue. That said, they may not be able/willing to do it.

$ /usr/bin/git rev-parse refs/heads/develop 28a05ce2e3079afcb32e4f1777b42971d7933a91 $ /usr/bin/git rev-parse refs/heads/develop:path/to/parent/file cc10f4b278086325aab2f95df97c807c7c6cd75e

So it looks like rev-parse and cat-file --batch-check both exhibit the same behavior.

I also had them expand their cat-file --batch-check to include another file in the same "path/to/parent" directory: $ echo 'refs/heads/develop refs/heads/develop:path/to/parent/sibling refs/heads/develop:path/to/parent/file' | /usr/bin/git cat-file --batch-check 28a05ce2e3079afcb32e4f1777b42971d7933a91 commit 259 2bfe7b4b7c7cdeb9653801d99b65dfefe5780dda blob 897 cc10f4b278086325aab2f95df97c807c7c6cd75e commit 330

So the "sibling" file in the same directory comes out as a "blob", as expected.

They also ran an ls-tree for the directory without any globs: # /usr/bin/git ls-tree refs/heads/develop:path/to/parent 100644 blob 2bfe7b4b7c7cdeb9653801d99b65dfefe5780dda sibling 100644 blob 4c8d566ed80a1554a059b97f7cd533a55bbd2ea8 file

For "sibling" the blob's ID matches what cat-file --batch-check shows, as I'd expect. There are several other tree entries, one "tree" and the rest "blob", that I've omitted for brevity. All of their modes look normal.

I also had them check ls-tree for some parent levels: $ /usr/bin/git ls-tree refs/heads/develop:path -- to 040000 tree 5244cd18e3d9de9002bdfcd18e173ca55c035084 to $ /usr/bin/git ls-tree refs/heads/develop:path/to -- parent 040000 tree 2847dc49d79e8d66040047a9dd61376115bf8829 parent

Nothing out of the ordinary to my eye.

$ /usr/bin/git cat-file -t cc10f4b278086325aab2f95df97c807c7c6cd75e commit

$ /usr/bin/git fsck --no-dangling Checking object directories: 100% (256/256), done. Checking object directories: 100% (256/256), done. Checking objects: 100% (122888/122888), done.

There's one alternate. No warnings, though.

I've asked them to double-check whether they can provide me with the repository, or with an anonymized copy. At this point, it feels like there's not a lot more I can do/check without access to data that reproduces the issue so I can attach a debugger.

Thanks again, Bryan

On Tue, Oct 26, 2021 at 04:58:49PM -0700, Bryan Turner wrote:

OK, that's not too surprising, since they're using the same routines under the hood. But that does imply that the problem is in the get_oid() family, which is what's doing that name to oid lookup.

I don't recall us ever having a bug of this nature in the history of Git, nor do I think this code would have changed recently. But of course there's a first time for everything.

The parser there isn't exactly left-to-right, so perhaps this particular name is stimulating some corner case. I imagine the answer is "no", or you'd have said so already, but are there any unusual characters in the filename path? Colons, curly braces, etc?

Interesting. That again points to their being something funny either with this filename, or perhaps with the tree that contains it.

That's not too surprising. I did wonder if refs/replace or something could be at work here, but I think in that case we'd still report the expected oid. At any rate, we can probably rule that out as rev-parse is returning the same unexpected oid, which means the problem is during the name resolution (and we shouldn't respect refs/replace there at all; we would respect it while reading the outer tree, but then so would your ls-tree, etc).

Another possibility, if they would run a custom Git on their end, is to provide them with a patch that cranks up the debugging output from get_oid_with_context_1(). Though I feel like it's hard to know where to sprinkle printf()s until we know where things go wrong. Is it misinterpreting the name, and not realizing it's a tree:path name? Or is get_tree_entry() at fault? That kind of thing is much easier to figure out interactively in a debugger.

-Peff

Am 27.10.21 um 01:58 schrieb Bryan Turner:

Just a shot in the dark: what happens when you use /usr/bin/git --no-replace-objects?

-- Hannes