{"thread":{"id":"66113","subject":"Question on textconv","startedAt":"2026-08-04T18:49:52Z","lastAt":"2026-08-06T04:11:00Z","messageCount":7,"participants":["rsbecker@nexbridge.com","D. Ben Knoble","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"549602","messageId":"017e01dd2441$476839f0$d638add0$@nexbridge.com","threadId":"66113","inReplyTo":null,"subject":"Question on textconv","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2026-08-04T18:44:22Z","receivedAt":"2026-08-04T18:49:52Z","isPatch":false,"body":"I experienced a change in how textconv works since about 2.50 and it is hard\nto get past. I would appreciate advice:\n\nWhen I define an external binary textconv, roughly like:\n\n.gitattributes:\nsimple binary diff=enscr\n\n.gitconfig\n[diff \"enscr\"]\n        textconv = run -debug ../../enscribe-conv --verbose\n        binary = true\n\nThe supplied file going to the textconv program looks like\n/tmp/git-blob-GFtIhK/simple\nand is always empty regardless of the file contents.\n\nWhen there is only one file named simple in the repository I can find it,\nbut otherwise\nany ambiguity in the name makes textconv processing impractical. Somewhere\nprior to this\nI was supplied with the actual file in the working index instead of a temp\nfile.\n\nAm I missing something?\n\nThanks,\nRandall\n\n"},{"id":"549617","messageId":"CALnO6CD+LmWNffptqp4bsoJQaq7Ah8VaKHjTTu6m-Zfm2uN+9w@mail.gmail.com","threadId":"66113","inReplyTo":"017e01dd2441$476839f0$d638add0$@nexbridge.com","subject":"Re: Question on textconv","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2026-08-04T20:52:35Z","receivedAt":"2026-08-04T20:52:49Z","isPatch":false,"body":"Hi Randall,\n\nOn Tue, Aug 4, 2026 at 2:52 PM <rsbecker@nexbridge.com> wrote:\n>\n> I experienced a change in how textconv works since about 2.50 and it is hard\n> to get past. I would appreciate advice:\n>\n> When I define an external binary textconv, roughly like:\n>\n> .gitattributes:\n> simple binary diff=enscr\n>\n> .gitconfig\n> [diff \"enscr\"]\n>         textconv = run -debug ../../enscribe-conv --verbose\n>         binary = true\n>\n> The supplied file going to the textconv program looks like\n> /tmp/git-blob-GFtIhK/simple\n> and is always empty regardless of the file contents.\n\nThat's strange. Over here (2.53.0), I get the temp file, but it has\nthe expected contents.\n\n(For example, I can debug a bit by using the shell form 'textconv =\n\"f() { echo $@; cat $@; <stuff>; }; f\"'. NB those $@ need quoted in\nproduction, but for my little test case that was more hassle than it\nwas worth. We can also see the invocation using GIT_TRACE2=1.)\n\nThis was true for me even after setting \"binary = true\" in the diff\ndriver block, which I thought was a bit unusual.\n\n> When there is only one file named simple in the repository I can find it,\n> but otherwise\n> any ambiguity in the name makes textconv processing impractical. Somewhere\n> prior to this\n> I was supplied with the actual file in the working index instead of a temp\n> file.\n>\n> Am I missing something?\n>\n> Thanks,\n> Randall\n\nI'm not sure :/\n\n-- \nD. Ben Knoble\n"},{"id":"549626","messageId":"01ac01dd245a$8b64f990$a22eecb0$@nexbridge.com","threadId":"66113","inReplyTo":"CALnO6CD+LmWNffptqp4bsoJQaq7Ah8VaKHjTTu6m-Zfm2uN+9w@mail.gmail.com","subject":"RE: Question on textconv","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2026-08-04T21:45:13Z","receivedAt":"2026-08-04T21:45:22Z","isPatch":false,"body":"On August 4, 2026 4:53 PM, D. Ben Knoble wrote:\n> On Tue, Aug 4, 2026 at 2:52 PM <rsbecker@nexbridge.com> wrote:\n> >\n> > I experienced a change in how textconv works since about 2.50 and it\n> > is hard to get past. I would appreciate advice:\n> >\n> > When I define an external binary textconv, roughly like:\n> >\n> > .gitattributes:\n> > simple binary diff=enscr\n> >\n> > .gitconfig\n> > [diff \"enscr\"]\n> >         textconv = run -debug ../../enscribe-conv --verbose\n> >         binary = true\n> >\n> > The supplied file going to the textconv program looks like\n> > /tmp/git-blob-GFtIhK/simple and is always empty regardless of the file\n> > contents.\n> \n> That's strange. Over here (2.53.0), I get the temp file, but it has the expected\n> contents.\n> \n> (For example, I can debug a bit by using the shell form 'textconv =\n> \"f() { echo $@; cat $@; <stuff>; }; f\"'. NB those $@ need quoted in production, but\n> for my little test case that was more hassle than it was worth. We can also see the\n> invocation using GIT_TRACE2=1.)\n> \n> This was true for me even after setting \"binary = true\" in the diff driver block, which\n> I thought was a bit unusual.\n> \n> > When there is only one file named simple in the repository I can find\n> > it, but otherwise any ambiguity in the name makes textconv processing\n> > impractical. Somewhere prior to this I was supplied with the actual\n> > file in the working index instead of a temp file.\n> >\n> > Am I missing something?\n> >\n> > Thanks,\n> > Randall\n> \n> I'm not sure :/\n\nCuriously, I get this only on binary files. It is almost as if leading nulls is causing an issue.\nIf I try on a text file or an ELF format file, the above works - ELF does not begin with a NULL.\n\n"},{"id":"549637","messageId":"20260805045026.GA972736@coredump.intra.peff.net","threadId":"66113","inReplyTo":"017e01dd2441$476839f0$d638add0$@nexbridge.com","subject":"Re: Question on textconv","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-08-05T04:50:26Z","receivedAt":"2026-08-05T04:50:28Z","isPatch":false,"body":"On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:\n\n> The supplied file going to the textconv program looks like\n> /tmp/git-blob-GFtIhK/simple\n> and is always empty regardless of the file contents.\n\nI can't reproduce the problem here, even for files with embedded NULs.\nHowever...\n\n> When there is only one file named simple in the repository I can find\n> it, but otherwise any ambiguity in the name makes textconv processing\n> impractical. Somewhere prior to this I was supplied with the actual\n> file in the working index instead of a temp file.\n\nThis part I can explain. We sometimes try to reuse the working tree\ninstead of generating a tempfile, as an optimization. We can only do\nthis when the working tree file is clean. But we also only bother to try\nwhen one of the diff endpoints is the index. So if we set up a sample\ntextconv like:\n\n  git config diff.foo.textconv 'echo >&2 \"got: $*\" && tr a-z A-Z <'\n  echo \"file diff=foo\" >.gitattributes\n\n  echo one >file && git add file && git commit -m one\n  echo two >file && git add file && git commit -m two\n\nThe running either \"git diff HEAD^\" or \"git diff --cached HEAD^\" will\nconvert the copy in the working tree, and you'll get:\n\n  got: /tmp/git-blob-0CLCMr/file\n  got: file\n  diff --git a/file b/file\n  index 5626abf..f719efd 100644\n  --- a/file\n  +++ b/file\n  @@ -1 +1 @@\n  -ONE\n  +TWO\n\nbut if you do \"git show HEAD\", you'll get two tempfiles:\n\n  got: /tmp/git-blob-w1binM/file\n  got: /tmp/git-blob-1nu2Rm/file\n  [same diff]\n\neven though this is the same diff! We _could_ try harder to reuse the\nworking tree copy here by checking whether the path has the same sha1 in\nthe tree and the index (and that the index entry is clean). But it only\nhelps in a few special cases, and it's not something users should rely\non (we might choose to create a tempfile anyway if the index is\nstat-dirty).\n\n-Peff\n"},{"id":"549677","messageId":"CALnO6CAhVzptUYpoHU93y5Sho3cPJgVbT81bb0ChugNCE9zsTw@mail.gmail.com","threadId":"66113","inReplyTo":"20260805045026.GA972736@coredump.intra.peff.net","subject":"Re: Question on textconv","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2026-08-05T11:36:15Z","receivedAt":"2026-08-05T11:36:27Z","isPatch":false,"body":"On Wed, Aug 5, 2026 at 12:50 AM Jeff King <peff@peff.net> wrote:\n>\n> On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:\n>\n> > The supplied file going to the textconv program looks like\n> > /tmp/git-blob-GFtIhK/simple\n> > and is always empty regardless of the file contents.\n>\n> I can't reproduce the problem here, even for files with embedded NULs.\n\nYeah, same:\n\n    dd if=/dev/zero of=file count=1\n    echo 'file diff=debug' >.gitattributes\n    git config diff.debug.textconv 'echo >&2 \"got $*\"; xxd'\n\nshows (with \"git add --intent-to-add file; git diff file\") the xxd\ndump of zeros from the working tree. Then\n\n    git commit file -mwip\n    dd if=/dev/zero of=file count=2\n    git diff file\n\nshows a diff between a temp file and the working tree (but still\ncorrectly showing the new block of zeros).\n"},{"id":"549700","messageId":"020201dd24e3$89ad1220$9d073660$@nexbridge.com","threadId":"66113","inReplyTo":"20260805045026.GA972736@coredump.intra.peff.net","subject":"RE: Question on textconv","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2026-08-05T14:05:52Z","receivedAt":"2026-08-05T14:06:05Z","isPatch":false,"body":"On August 5, 2026 12:50 AM, Jeff King wrote:\n> On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:\n> \n> > The supplied file going to the textconv program looks like\n> > /tmp/git-blob-GFtIhK/simple and is always empty regardless of the file\n> > contents.\n> \n> I can't reproduce the problem here, even for files with embedded NULs.\n> However...\n> \n> > When there is only one file named simple in the repository I can find\n> > it, but otherwise any ambiguity in the name makes textconv processing\n> > impractical. Somewhere prior to this I was supplied with the actual\n> > file in the working index instead of a temp file.\n> \n> This part I can explain. We sometimes try to reuse the working tree instead of\n> generating a tempfile, as an optimization. We can only do this when the working\n> tree file is clean. But we also only bother to try when one of the diff endpoints is\n> the index. So if we set up a sample textconv like:\n> \n>   git config diff.foo.textconv 'echo >&2 \"got: $*\" && tr a-z A-Z <'\n>   echo \"file diff=foo\" >.gitattributes\n> \n>   echo one >file && git add file && git commit -m one\n>   echo two >file && git add file && git commit -m two\n> \n> The running either \"git diff HEAD^\" or \"git diff --cached HEAD^\" will convert the\n> copy in the working tree, and you'll get:\n> \n>   got: /tmp/git-blob-0CLCMr/file\n>   got: file\n>   diff --git a/file b/file\n>   index 5626abf..f719efd 100644\n>   --- a/file\n>   +++ b/file\n>   @@ -1 +1 @@\n>   -ONE\n>   +TWO\n> \n> but if you do \"git show HEAD\", you'll get two tempfiles:\n> \n>   got: /tmp/git-blob-w1binM/file\n>   got: /tmp/git-blob-1nu2Rm/file\n>   [same diff]\n> \n> even though this is the same diff! We _could_ try harder to reuse the working tree\n> copy here by checking whether the path has the same sha1 in the tree and the\n> index (and that the index entry is clean). But it only helps in a few special cases, and\n> it's not something users should rely on (we might choose to create a tempfile\n> anyway if the index is stat-dirty).\n\nCould we extend textconv to support %f (the original path) if specified in the\ntextconv configuration? That would solve the ambiguity of what is being supplied.\n\n"},{"id":"549784","messageId":"20260806041052.GA1610686@coredump.intra.peff.net","threadId":"66113","inReplyTo":"020201dd24e3$89ad1220$9d073660$@nexbridge.com","subject":"Re: Question on textconv","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-08-06T04:10:52Z","receivedAt":"2026-08-06T04:11:00Z","isPatch":false,"body":"On Wed, Aug 05, 2026 at 10:05:52AM -0400, rsbecker@nexbridge.com wrote:\n\n> Could we extend textconv to support %f (the original path) if specified in the\n> textconv configuration? That would solve the ambiguity of what is being supplied.\n\nIn theory, yes. But there is one gotcha: there's a system for caching\ntextconv output in git-notes, and it uses only the original blob id as\nthe cache key.\n\nSo I'm not opposed to adding something like %f, as long as the patch to\ndo so handles the caching problem (even if it just refuses to cache,\nthat would be much better than returning possibly-wrong results).\n\nThat said, it sounds like you just want %f to work around a bug where\nthe content is not provided. Probably fixing the bug is a better path\nforward. Looking at the working tree file to get the contents will not\nalways be correct (e.g., if you're diffing an old tree).\n\n-Peff\n"}]}