Volume XXII, number 279Tuesday, October 6, 2026Latest message 38 minutes ago

The Git List

News and archive of git@vger.kernel.org, since April 2005

Question on textconv

7 messages between Aug 4, 2026 and Aug 6, 2026, from rsbecker@nexbridge.com, D. Ben Knoble, Jeff King.

Plain Markdown or JSON for tools and agents.

rsbecker@nexbridge.comAug 4, 2026, 18:44 UTC on lore

I experienced a change in how textconv works since about 2.50 and it is hard to get past. I would appreciate advice:

When I define an external binary textconv, roughly like:

.gitattributes: simple binary diff=enscr

.gitconfig
[diff "enscr"]
        textconv = run -debug ../../enscribe-conv --verbose
        binary = true

The supplied file going to the textconv program looks like /tmp/git-blob-GFtIhK/simple and is always empty regardless of the file contents.

When there is only one file named simple in the repository I can find it, but otherwise any ambiguity in the name makes textconv processing impractical. Somewhere prior to this I was supplied with the actual file in the working index instead of a temp file.

Am I missing something?

Thanks, Randall

D. Ben KnobleAug 4, 2026, 20:52 UTC in reply to rsbecker@nexbridge.com on lore

Re: Question on textconv

Hi Randall,
On Tue, Aug 4, 2026 at 2:52 PM <rsbecker@nexbridge.com> wrote:
Show 17 quoted lines
>
> I experienced a change in how textconv works since about 2.50 and it is hard
> to get past. I would appreciate advice:
>
> When I define an external binary textconv, roughly like:
>
> .gitattributes:
> simple binary diff=enscr
>
> .gitconfig
> [diff "enscr"]
>         textconv = run -debug ../../enscribe-conv --verbose
>         binary = true
>
> The supplied file going to the textconv program looks like
> /tmp/git-blob-GFtIhK/simple
> and is always empty regardless of the file contents.

That's strange. Over here (2.53.0), I get the temp file, but it has the expected contents.

(For example, I can debug a bit by using the shell form 'textconv = "f() { echo $@; cat $@; <stuff>; }; f"'. NB those $@ need quoted in production, but for my little test case that was more hassle than it was worth. We can also see the invocation using GIT_TRACE2=1.)

This was true for me even after setting "binary = true" in the diff driver block, which I thought was a bit unusual.

Show 11 quoted lines
> When there is only one file named simple in the repository I can find it,
> but otherwise
> any ambiguity in the name makes textconv processing impractical. Somewhere
> prior to this
> I was supplied with the actual file in the working index instead of a temp
> file.
>
> Am I missing something?
>
> Thanks,
> Randall
I'm not sure :/
-- 
D. Ben Knoble
rsbecker@nexbridge.comAug 4, 2026, 21:45 UTC in reply to D. Ben Knoble on lore

RE: Question on textconv

On August 4, 2026 4:53 PM, D. Ben Knoble wrote:
Show 41 quoted lines
> On Tue, Aug 4, 2026 at 2:52 PM <rsbecker@nexbridge.com> wrote:
> >
> > I experienced a change in how textconv works since about 2.50 and it
> > is hard to get past. I would appreciate advice:
> >
> > When I define an external binary textconv, roughly like:
> >
> > .gitattributes:
> > simple binary diff=enscr
> >
> > .gitconfig
> > [diff "enscr"]
> >         textconv = run -debug ../../enscribe-conv --verbose
> >         binary = true
> >
> > The supplied file going to the textconv program looks like
> > /tmp/git-blob-GFtIhK/simple and is always empty regardless of the file
> > contents.
> 
> That's strange. Over here (2.53.0), I get the temp file, but it has the expected
> contents.
> 
> (For example, I can debug a bit by using the shell form 'textconv =
> "f() { echo $@; cat $@; <stuff>; }; f"'. NB those $@ need quoted in production, but
> for my little test case that was more hassle than it was worth. We can also see the
> invocation using GIT_TRACE2=1.)
> 
> This was true for me even after setting "binary = true" in the diff driver block, which
> I thought was a bit unusual.
> 
> > When there is only one file named simple in the repository I can find
> > it, but otherwise any ambiguity in the name makes textconv processing
> > impractical. Somewhere prior to this I was supplied with the actual
> > file in the working index instead of a temp file.
> >
> > Am I missing something?
> >
> > Thanks,
> > Randall
> 
> I'm not sure :/

Curiously, I get this only on binary files. It is almost as if leading nulls is causing an issue. If I try on a text file or an ELF format file, the above works - ELF does not begin with a NULL.

Jeff KingAug 5, 2026, 04:50 UTC in reply to rsbecker@nexbridge.com on lore

Re: Question on textconv

On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:
> The supplied file going to the textconv program looks like
> /tmp/git-blob-GFtIhK/simple
> and is always empty regardless of the file contents.

I can't reproduce the problem here, even for files with embedded NULs. However...

> When there is only one file named simple in the repository I can find
> it, but otherwise any ambiguity in the name makes textconv processing
> impractical. Somewhere prior to this I was supplied with the actual
> file in the working index instead of a temp file.

This part I can explain. We sometimes try to reuse the working tree instead of generating a tempfile, as an optimization. We can only do this when the working tree file is clean. But we also only bother to try when one of the diff endpoints is the index. So if we set up a sample textconv like:

  git config diff.foo.textconv 'echo >&2 "got: $*" && tr a-z A-Z <'
  echo "file diff=foo" >.gitattributes
  echo one >file && git add file && git commit -m one
  echo two >file && git add file && git commit -m two

The running either "git diff HEAD^" or "git diff --cached HEAD^" will convert the copy in the working tree, and you'll get:

  got: /tmp/git-blob-0CLCMr/file
  got: file
  diff --git a/file b/file
  index 5626abf..f719efd 100644
  --- a/file
  +++ b/file
  @@ -1 +1 @@
  -ONE
  +TWO
but if you do "git show HEAD", you'll get two tempfiles:
  got: /tmp/git-blob-w1binM/file
  got: /tmp/git-blob-1nu2Rm/file
  [same diff]

even though this is the same diff! We _could_ try harder to reuse the working tree copy here by checking whether the path has the same sha1 in the tree and the index (and that the index entry is clean). But it only helps in a few special cases, and it's not something users should rely on (we might choose to create a tempfile anyway if the index is stat-dirty).

-Peff
D. Ben KnobleAug 5, 2026, 11:36 UTC in reply to Jeff King on lore

Re: Question on textconv

On Wed, Aug 5, 2026 at 12:50 AM Jeff King <peff@peff.net> wrote:
Show 8 quoted lines
>
> On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:
>
> > The supplied file going to the textconv program looks like
> > /tmp/git-blob-GFtIhK/simple
> > and is always empty regardless of the file contents.
>
> I can't reproduce the problem here, even for files with embedded NULs.
Yeah, same:
    dd if=/dev/zero of=file count=1
    echo 'file diff=debug' >.gitattributes
    git config diff.debug.textconv 'echo >&2 "got $*"; xxd'

shows (with "git add --intent-to-add file; git diff file") the xxd dump of zeros from the working tree. Then

    git commit file -mwip
    dd if=/dev/zero of=file count=2
    git diff file

shows a diff between a temp file and the working tree (but still correctly showing the new block of zeros).

rsbecker@nexbridge.comAug 5, 2026, 14:05 UTC in reply to Jeff King on lore

RE: Question on textconv

On August 5, 2026 12:50 AM, Jeff King wrote:
Show 49 quoted lines
> On Tue, Aug 04, 2026 at 02:44:22PM -0400, rsbecker@nexbridge.com wrote:
> 
> > The supplied file going to the textconv program looks like
> > /tmp/git-blob-GFtIhK/simple and is always empty regardless of the file
> > contents.
> 
> I can't reproduce the problem here, even for files with embedded NULs.
> However...
> 
> > When there is only one file named simple in the repository I can find
> > it, but otherwise any ambiguity in the name makes textconv processing
> > impractical. Somewhere prior to this I was supplied with the actual
> > file in the working index instead of a temp file.
> 
> This part I can explain. We sometimes try to reuse the working tree instead of
> generating a tempfile, as an optimization. We can only do this when the working
> tree file is clean. But we also only bother to try when one of the diff endpoints is
> the index. So if we set up a sample textconv like:
> 
>   git config diff.foo.textconv 'echo >&2 "got: $*" && tr a-z A-Z <'
>   echo "file diff=foo" >.gitattributes
> 
>   echo one >file && git add file && git commit -m one
>   echo two >file && git add file && git commit -m two
> 
> The running either "git diff HEAD^" or "git diff --cached HEAD^" will convert the
> copy in the working tree, and you'll get:
> 
>   got: /tmp/git-blob-0CLCMr/file
>   got: file
>   diff --git a/file b/file
>   index 5626abf..f719efd 100644
>   --- a/file
>   +++ b/file
>   @@ -1 +1 @@
>   -ONE
>   +TWO
> 
> but if you do "git show HEAD", you'll get two tempfiles:
> 
>   got: /tmp/git-blob-w1binM/file
>   got: /tmp/git-blob-1nu2Rm/file
>   [same diff]
> 
> even though this is the same diff! We _could_ try harder to reuse the working tree
> copy here by checking whether the path has the same sha1 in the tree and the
> index (and that the index entry is clean). But it only helps in a few special cases, and
> it's not something users should rely on (we might choose to create a tempfile
> anyway if the index is stat-dirty).

Could we extend textconv to support %f (the original path) if specified in the textconv configuration? That would solve the ambiguity of what is being supplied.

Jeff KingAug 6, 2026, 04:10 UTC in reply to rsbecker@nexbridge.com on lore

Re: Question on textconv

On Wed, Aug 05, 2026 at 10:05:52AM -0400, rsbecker@nexbridge.com wrote:
> Could we extend textconv to support %f (the original path) if specified in the
> textconv configuration? That would solve the ambiguity of what is being supplied.

In theory, yes. But there is one gotcha: there's a system for caching textconv output in git-notes, and it uses only the original blob id as the cache key.

So I'm not opposed to adding something like %f, as long as the patch to do so handles the caching problem (even if it just refuses to cache, that would be much better than returning possibly-wrong results).

That said, it sounds like you just want %f to work around a bug where the content is not provided. Probably fixing the bug is a better path forward. Looking at the working tree file to get the contents will not always be correct (e.g., if you're diffing an old tree).

-Peff

Back to recent threads