{"thread":{"id":"58733","subject":"smudge filters do not round trip through `git diff` / `git apply`","startedAt":"2022-11-01T20:56:05Z","lastAt":"2022-11-01T20:56:05Z","messageCount":1,"participants":["Anthony Sottile"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"466217","messageId":"CA+dzEB=acvDob10gCaWBgVcd3E5VEX+chTKsvxZ3VAUh4Dhrow@mail.gmail.com","threadId":"58733","inReplyTo":null,"subject":"smudge filters do not round trip through `git diff` / `git apply`","fromName":"Anthony Sottile","fromEmail":"asottile@umich.edu","sentAt":"2022-11-01T20:55:42Z","receivedAt":"2022-11-01T20:56:05Z","isPatch":false,"sender":{"key":"asottile@umich.edu","avatar":"https://avatars.githubusercontent.com/u/1810591?v=4"},"body":"this is boiled down from a larger problem outlined here:\nhttps://github.com/pre-commit/pre-commit/issues/776\n\nI've had some time to sit down and poke at this today -- here's my\nminimal reproduction using just `git` and `git-crypt` -- though my\nusers have reported this also seems to affect other smudge filters.\n\nI understand `git-crypt` is not associated with the git project,\nhowever it was an easy, readily-available smudge filter to demonstrate\nthe problem.  I'm using git-crypt 0.6.0 (from ubuntu 22.04).\n\n(the key material below is not sensitive -- I generated it afresh in a\ndocker container)\n\n```bash\n#!/usr/bin/env bash\nset -euxo pipefail\n\nrm -rf repo\n\ngit --version\ngit init --quiet repo -b main\ncd repo\ngit commit --allow-empty -m 'Initial empty commit'\n\n# determinstic git-crypt key so the output is stable\nbase64 -d > keyfile <<EOF\nAEdJVENSWVBUS0VZAAAAAgAAAAAAAAABAAAABAAAAAAAAAADAAAAIIBi0O4iuCHghpYj4Teb6F72\nKjTRHePBTf/6XC6fiVqvAAAABQAAAECTwWTDHfx0/Ytw3IZrVhonb5IPTr7kio27u0prnb8X25ui\n9k4UqrdRQy8ZtBERv6wnHwC8A6q7CamRZ22L4q7UAAAAAA==\nEOF\ngit-crypt unlock keyfile\n\necho 'f filter=git-crypt diff=git-crypt' > .gitattributes\ngit add .gitattributes\necho 'hello world' > f\ngit add f\nrm f && touch f\n\ntree=\"$(git write-tree)\"\n! git diff-index \\\n    --ignore-submodules \\\n    --binary \\\n    --exit-code \\\n    --no-color \\\n    --no-ext-diff \\\n    --no-textconv \\\n    \"$tree\" -- > patch\n\ngit checkout -- .\ngit apply patch || (echo FAILED && cat patch && exit 1)\n```\n\nhere's my output:\n\n```console\n$ bash t.sh\n+ rm -rf repo\n+ git --version\ngit version 2.38.1.381.gc03801e19c\n+ git init --quiet repo -b main\n+ cd repo\n+ git commit --allow-empty -m 'Initial empty commit'\n[main (root-commit) 97d9520] Initial empty commit\n+ base64 -d\n+ git-crypt unlock keyfile\n+ echo 'f filter=git-crypt diff=git-crypt'\n+ git add .gitattributes\n+ echo 'hello world'\n+ git add f\n+ rm f\n+ touch f\n++ git write-tree\n+ tree=beca08f8b3c0774060f3e28e081ac69a80a1a10d\n+ git diff-index --ignore-submodules --binary --exit-code --no-color\n--no-ext-diff --no-textconv beca08f8b3c0774060f3e28e081ac69a80a1a10d\n--\n+ git checkout -- .\n+ git apply patch\nerror: binary patch to 'f' creates incorrect result (expecting\n2f89279ce748725a41cec60d5025b22efc863b42, got\ne69de29bb2d1d6434b8b29ae775ad8c2e48c5391)\nerror: f: patch does not apply\n+ echo FAILED\nFAILED\n+ cat patch\ndiff --git a/f b/f\nindex ee7d1b67cd31482ae9bc772a0b2d016c81e1c613..2f89279ce748725a41cec60d5025b22efc863b42\n100644\nGIT binary patch\nliteral 0\nHcmV?d00001\n\nliteral 34\nqcmZQ@_Y83kiVO&0IB9<_f2(d}=e+;CXFaaIzZ@c>%yE8!<X-^jJPz&v\n\n+ exit 1\n```\n\nI traced through the execution and it appears that smudge filters are\nmaybe still running despite the `--no-textconv` setting which may\nexplain this?\n\n```\n+ GIT_TRACE=2\n+ git diff-index --ignore-submodules --binary --exit-code --no-color\n--no-ext-diff --no-textconv beca08f8b3c0774060f3e28e081ac69a80a1a10d\n--\n20:40:21.551771 git.c:455               trace: built-in: git\ndiff-index --ignore-submodules --binary --exit-code --no-color\n--no-ext-diff --no-textconv beca08f8b3c0774060f3e28e081ac69a80a1a10d\n--\n20:40:21.552190 run-command.c:668       trace: run_command: '\"git-crypt\" clean'\n20:40:21.555249 git.c:455               trace: built-in: git rev-parse --git-dir\n```\n\nas shown in the output I'm using the current primary branch revision\nof git -- though I usually use 2.34.1 (ubuntu 22.04)\n\noddly enough, using `--textconv` instead of `--no-textconv` \"fixes\" --\nbut is unsatisfactory for my use case (I don't want to rely on the\nstate of filters installed, etc.)\n\nthe error message seems to occur due to the comparison of the hash in\nthe `index a...b` line above the patch hunk\n\n`e69de29bb2d1d6434b8b29ae775ad8c2e48c5391` is the hash of an empty file:\n\n```console\n$ git hash-object -w /dev/null\ne69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n```\n\n`2f89279ce748725a41cec60d5025b22efc863b42` appears to be the hash of\nthe smudged empty file (I used base64 here since it's binary nonsense\n-- I got the contents of this blob by committing the empty file and\nthen fished the object out of the git database instead of trying to do\nthe patch dance):\n\n```console\n$ base64 -d <<< 'AEdJVENSWVBUAGoDwWO5GXWQ4B1kIQ==' | git hash-object\n-w /dev/stdin\n2f89279ce748725a41cec60d5025b22efc863b42\n```\n\nI *believe* the fix here is to avoid smudging in `git diff\n--no-textconv` -- I started a patch where I added a `HASH_NO_TEXTCONV`\nflag to `cache.h` but wasn't super sure on where to go from there and\ndecided I should ask first whether this is the right approach to take!\n\nanthony\n"}]}