patch, 3 partsdiff: allow turning on textconv explicitly for plumbing

Some history viewers use the diff plumbing to generate diffs rather than going through the "git diff" porcelain. Currently, there is no way for them to specify that they would like to see the text-converted version of the diff.

This patch adds a "--textconv" option to allow such a plumbing user to allow text conversion. The user can then tell the viewer whether or not they would like text conversion enabled.

While it may be tempting add a configuration option rather than requiring each plumbing user to be configured to pass --textconv, that is somewhat dangerous. Text-converted diffs generally cannot be applied directly, so each plumbing user should "opt in" to generating such a diff, either by explicit request of the user or by confirming that their output will not be fed to patch.

Signed-off-by: Jeff King <peff@peff.net>
---
My ultimate goal is to see these diffs in gitk, which is implemented in
3/3. As a bonus side effect, the --no-textconv option can be used with
"git diff" or "git log" if you really don't want to see them there
(e.g., because you are abusing "git log" to produce a binary patch you
mean to apply).

I know this is not strictly a bugfix and we are in -rc, but:

  1. It is an enhancement to a previously unreleased feature, and
     shouldn't affect anything outside of that.

  2. It affects the scripting interface to textconv, so I would like to
     get it in before textconv is ever released so that it is always the
     "right way" to turn text conversion off or on.

 diff.c |    4 ++++
 diff.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

Jeff King <peff@peff.net> writes:

I'd agree with #1, especially if you said "doesn't" instead of "shouldn't".

But I am not 100% sure if the scripting part is "the right way".

If a script wants to take whatever Porcelain users are happy as the "presentation for human consumption" and pass that through as its own output to the end user, maybe it is better off reading from Porcelain, instead of reading from the plumbing (the latter of which requires making the plumbing output less reliable)?

When we later enhance textconv output from the "diff" Porcelain to benefit interactive users, it will automatically help the script that passes through the "diff" output to the end users.

You can certainly argue that this "textconv" feature that is grafted from Porcelain into plumbing is a special case in that its output is subject to change any time to help human consumption and we never strive for its stability as we do for other features in the plumbing to support machine readability by scripts. You can propagate the later enhancement of textconv diff output we'd make for Porcelain to the scripted users that reads from the plumbing that way.

But then wouldn't it be the same for these scripts that do value the "presentation meant for human consumption" over "machine readability" to read from Porcelain? That would not have to blur the distinction between the Porcelain and plumbing like the approach you are suggesting here.

Junio C Hamano <gitster@pobox.com> writes:

But I'll apply them anyway.

On Sun, Dec 07, 2008 at 07:55:12PM -0800, Junio C Hamano wrote:

But I don't think having gitk call the porcelain makes sense, either. That would enable, for example, external diff drivers which might actually spawn an entirely new window. And I don't think most people would want that from gitk.

Now, obviously it could call "git diff --no-ext-diff" to opt out of that. But I think it is backwards to call porcelain but opt out of features that you don't want. It makes more sense to start with the bare minimum (i.e., plumbing) and opt in to features that you are OK with.

And yes, you increase the risk that a script which blindly passes options to plumbing may now be asked to generate a textconv'd diff which will be useless to the user. But:

  1. that is already a risk with --ext-diff

  2. it is actually a _feature_. It is asking the user to say "is this
     a sane option to pass for my situation or not" instead of forcing
     the script to make that decision.

I'm not sure what enhancement you mean. But it is possible that gitk would not _want_ such an enhancement, and it would then have to opt out of it.

Right. _If_ it's a change that won't upset any plumbing consumers. If it is, then it needs to be a separate option so that the plumbing consumers who don't mind the change can start using it.

I don't agree that this is blurring the distinction. I don't see textconv diffs as a "porcelain feature" that has been grafted on to plumbing. I see it as a core diff feature which can be turned off or on. Porcelain has it on by default, and plumbing has it off by default. But you can ask either to change their default settings[1].

-Peff

[1]: You can argue that the current implementation is buggy in this sense, though, since some porcelains will accept "--textconv" but not change their behavior. I think blame, for example, will have that problem, because it accepts diff options but does not necessarily use them to create diffs. But I think that just means we have a bug and eventually should handle the case of "git blame --textconv" correctly.

Jeff King <peff@peff.net> writes:

In the above argument, you are assuming that you can have a complete enumeration of "plumbing consumers", so that we can tell what kind of changes to textconv output do affect them in a negative way, and what kind of changes are safe.

Yes, you can enumerate in-tree consumers, but that misses the whole point of having "plumbing", which can be used by any scripts out of tree and they can rely on stable output from the plumbing. By giving --textconv to them to use with the plumbing, you are effecively casting textconv output in stone. By definition we will never know what kind of changes to the output from the textconv filter out of tree consumers would mind.

Currently "diff" machinery uses the output of textconv filter as-is without any further embellishment, but I think it is too early to tell if that is really what we would want, or we may need some minimum postprocessing on what we receive from the textconv filter before feeding that to the textual diff machinery, because the feature has not been used in the field at all yet.

In any case, I've applied the series for an entirely different reason. The patch is the most natural way to allow users of Porcelain to disable textconv with the --no-textconv option, just like --no-ext-diff can be used to disable the external diff.

We _might_ want to add another patch to make the plumbing layer ignore (or error out) --textconv option given from the command line before 1.6.1 ships, and after we gain confidence with the stability of the feature, revert that patch to allow use of --textconv freely from the plumbing.

It always is easier to introduce a new feature in a more limited form and then later loosen it, than adding it in an unrestricted form and having later to limit it for whatever reason.

On Sun, Dec 07, 2008 at 10:52:39PM -0800, Junio C Hamano wrote:

No, I'm assuming we have an idea of what is a "reasonable" change to the feature that won't break consumers, and what is a change that will break them. And it's the same judgement we have to make all the time when thinking about how changes will impact the scriptable interface.

And furthermore, I'm proposing to take the safest path, which is to say that the scriptable interface _doesn't_ change unless each consumer decides to ask it to do so. That is, the output of "git diff-tree" is unchanged unless the caller explicitly allows textconv. So now we have given a meaning to "--textconv". If you want a _new_ textconv variant (let's say you want it to output the textconv patch with some "this isn't a real patch" munging, and also include the binary patch -- which would make it both human readable and applicable), then I think it is safe to say that is an incompatible change. And it gets called --textconv-with-patch or whatever, and porcelain can move to it by default. But the plumbing consumers still get the same thing via --textconv, and they can opt into the new format if that is useful to them.

So uptake of the new feature is slower (you have to wait for the maintainer of the script to support it) but you never break compatibility.

Right. But we _have_ to cast it in stone if we want to make it available to them at all. So I am proposing to give what exists now a name, and further enhancements will have to have a different name.

If you want to argue that it is too early to cast in stone, since we will have to carry around this implementation and the name "--textconv" forever, then I can accept that. But it is a bit annoying, since I want to use it in gitk _now_. :)

But I do run 'next' usually, so maybe it is worth pushing it off until the next release cycle.

Yes, I agree that is worthwhile and should have been there since the beginning (and I think should definitely ship in 1.6.1).

If you want to do that, the simplest thing is to simply take the "--no-textconv" part of 2/3 and drop the rest. There is not really any point in supporting --textconv just for porcelain, which already has it on by default.

-Peff

Junio C Hamano <gitster@pobox.com> writes:

I imagine that web developers may want to use a textconv filter that replaces ">" with "\n>" on the HTML files their designer-colleagues throw in the source tree to make "git log -p" of the whole project easier to follow. When the developers would want to suggest improvements to what their designer-colleagues did, however, by running "git diff --stat -p" in their dirty work tree to produce a patch (like I just did just now, visible from the mnemonic prefixes below), they would want to disable textconv temporarily to get an appliable patch with --no-textconv option.

Once we have --no-textconv, somebody would inevitably ask about its positive counterpart, --textconv option. Even though it might not make sense from patch applicability viewpoint, the option would allow the end user to explicitly ask for "git format-patch --textconv" and get a patch that can only be reviewed by humans but cannot be applied. Hence the attached update to the draft release notes to version 1.6.1 [Update #1].

You raised an intriguing possibility to use textconv in blame. It would also be useful if we allowed "git show --textconv $blob" to pass the blob via textconv filter and any other transformation controlled by the attributes mechanism.. When "git show" sees the above command line, it only knows the blob object name and not the path, so we may need to allow a new option to tell the command to pretend as if the content came from a path, perhaps with a syntax like:

	$ git show --attribute-path=a/b/c $blob
	$ git show --attribute-path=a/b/c --textconv $blob

Note that I envision that the above two commands would produce different results. The former would behave as if $blob were recorded at path a/b/c and what would be checked out (i.e. usual crlf, ident, and smudge that are in effect for the path are applied) to the standard output. The latter would apply the textconv filter that would apply to what is recorded at the given path and show that.

[Update #1]

 Documentation/RelNotes-1.6.1.txt |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

On Mon, Dec 08, 2008 at 12:27:25AM -0800, Junio C Hamano wrote:

Hrm, didn't you argue against textconv'ing non-binary contents earlier in the cycle? At any rate, I agree that is a reasonable use. Maybe a better name would have been "normalize" or "canonicalize".

Yep, exactly. I use "git diff >patch" all the time instead of format-patch.

I think that makes sense. If you are going to implement a "use this as the attribute path" feature, though, you might also want just "use these attributes" which I can imagine being simpler in some cases. Like:

  $ git show --attribute diff=whatever $blob

But on the subject of "other places to see textconv", another one I considered was git log -S. I haven't looked to see if it even checks binary files at all, but certainly if I was searching for some text content I would want to find it in the text version (and depending on the file format, there may be other binary cruft preventing you from finding it in the binary version at all).

The series I posted was to show the text in gitk diffs. However, it does nothing for finding text in gitk's "find commits introducing this string", which I assume just uses git's pickaxe.

Unfortunately, it would probably be painfully slow. But still better than nothing. One enhancement for textconv that I am considering is to name the tempfiles based on the blob sha1. Then a smart helper could cache expensive conversions if it wanted (and dumb helpers would obviously just ignore the filename).

Side note: For one use case, storing word processor documents, this approach seems sensible. But for my jpg example, it almost seems that I could use git more efficiently by having two files: one with the exif text, and one with the image content. And then I could "build" the resulting tagged jpgs by combining the two, but operations on the text would be very efficient. The downside, of course, being that the "build" step is annoying, and consumes a ton of disk space. But it just occurred to me as an alternate way to think about the use case.

I wonder if we want to word this even more strongly: --textconv is there and you can play with it if you want, but we are not promising that it is part of the stable API yet.

-Peff