Stupid quoting...

Hi,

what is the point in quoting file names and their characters in git-diff's output? And what is the recommended way of undoing the damage?

I have something like

git-diff -M -C --name-status -r master^ master | {
    while read -r flag name
    do
	case "$name" in *\\[0-3][0-7][0-7]*)
		name=$(echo -e $(echo "$name"|sed 's/\\\([0-3][0-7][0-7]\)/\\0\1/g;s/\\\([^0]\)/\\\\\1/g'))
	esac
        [...]

in order to get through the worst with utf-8 file names, and it is a complete nuisance (double quotemarks are treated later).

Is there any utility or pipe or invocation that can take a sequence of filenames as printed by git and turn them back into what they actually were in the first place?

On 6/13/07, David Kastrup <dak@gnu.org> wrote:

Just use "-z". Everything will be unquoted and separated by \0

Hi,

On Wed, 13 Jun 2007, David Kastrup wrote:

The recommended way is not using spaces to begin with. I mean, does "David" contain spaces? People seem not to see the problem, and fail to blame Microsoft for all the damage they have done, introducing that stupid, stupid concept of filenames containing spaces, and _enforcing_ it.

Please understand that the quotes are not there for you, but for processing by other programs.

However, I _suspect_ that you want to do something like

	name="$(echo $name)"

because "echo" is exactly one of the programs this quoting was invented for.

Ciao, Dscho

Johannes Schindelin wrote:

To be fair, Microsoft did not invent the concept of filenames with spaces. Even UNIX has, I believe, always allowed them, though you risked running into buggy scripts misbehaving if you used them. And really, filenames with spaces are only a nuisance in a text-based command line scripting environment, and only then because someone early on decided to use space rather than some other metacharacter as the only available delimiter between command arguments in scripts.

For regular users who aren't writing shell scripts, long-form filenames mean a document's name is the same as its title, which is hugely helpful from a user interface point of view: "February Marketing Budget" is much more self-documenting as a filename than "febmkbgt" or whatever they would have had to choose in the old days (and remember, UNIX had a 14-character filename size limit in the early days; long filenames weren't introduced until BSD.)

I view filenames as primarily for human consumption, so they should act the way humans expect names to act. Computers can just use inode numbers or file IDs to refer to files (like, say, SHA1 hashes) -- the pretty names are all for the benefit of us meat-brained entities.

Then again, that means I also think case sensitivity in filenames was a bad design choice. To use your "people's names" analogy, if you ask any random person whether "Billy-bob Thornton" and "Billy-Bob Thornton" are the same name, they'll almost always say yes; most people don't consider capital letters and lower-case letters to be different letters, just different forms of the same letters. And I know how popular *that* opinion is around here...

-Steve

Hi,

On Wed, 13 Jun 2007, Steven Grimm wrote:

I didn't say that, did I?

They _forced_ the use onto the world. That's what I was complaining about.

Okay, Steven Grimm. How do you think _I_ can tell that Steven is your name from looking at your _full_ name "Steven Grimm"? Huh?

Exactly. I split at the space.

Ciao, Dscho

Johannes Schindelin wrote:

At the risk of drawing the conversation way off topic: What's Mary Ann Summers' first name? (Hint: It's not "Mary.")

-Steve

Hi,

On Wed, 13 Jun 2007, Steven Grimm wrote:

The first first name _is_ Mary. Maybe it is not the name you shout when calling her.

But that is irrelevant. The names of Mary Ann Summers are separated by spaces. Period.

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

Why are you talking about spaces ;-)?

There are a few things to note, but the first thing is that mere spaces do not trigger quoting. A tab (HT) does, so do non ASCII characters. The second thing is that we do this quoting for various good reasons, and it is not likely to change.

As Alex mentions, the most safe way for programs to read is to read from the -z format. However, even if you are capable to do so, it may be inconvenient in some languages (mainstream languages like C and Perl are not among them). Not quoting SP is a conscious decision, as SP in filenames are rather common, more common than non ASCII and much more common than HT.

The "raw" formats "ls-files -s", "ls-tree" and "diff --raw" produce are designed to put names at the end, and typically delimited with a HT, so that "lazy" scripts can use cut (whose default delimiter is a HT) to pick out pieces from its output. And plumbing tools reading from the standard input (most notably, "update-index --stdin") know how to unquote them. In practice, not many people use non ASCII in pathnames and expect them work sanely for everybody, so loosely written scripts, as long as they cut at HT to pick out the pathname part, "mostly" work (I think traditional core git scripts are safe, I suspect some contributed ones shipped with git core may not be, Cogito used to be very unsafe but it was audited and became much safer before it got discontinued).

The pathname quoting rules in textual output was chosen primarily to make diff output safer, as one of the most important workflow git supports is e-mailable patches.

GNU patch treats HT on "+++ name"/"--- name" lines as the end of name (and after HT comes timestamp), but the timestamp part is treated as optional, which introduces ambiguities and confusion. The issue was discussed some time ago (check the list archive for discussion among I, Linus and Paul Eggert -- the GNU diff and patch maintainer) and the quoting rules we use now is consistent with what the diff and patch plan to use. The update on the GNU side may have already happened, it may not have.

When a patch appears in an e-mail, you would need to be aware that not everybody has the luxury of living in UTF-8 only world. Your commit message and cover letter may be in one encoding, the pathnames that appear in diff headers may be in your filesystem encoding, and the patch text that appear as the diff payload may be in another document specific encoding. All three could be different (worse, a patch that touch more than one file can carry different encodings in the payload part), and mixing character set in a single piece of e-mail confuses people's MUA and tends to mangle messages. Quoting non ASCII characters in pathnames, even they are perfectly valid and ordinary UTF-8 strings, is to eliminate one element in the above three as a possible source of worries.

David Kastrup wrote:

7-bit email.

Jakub Narebski <jnareb@gmail.com> writes:

I think it can be reasonably safely assumed that people using 8-bit characters in file names will not refrain from using them in the files themselves: file names usually are chosen descriptive of the contents, and so rarely are in a different language. So I don't see what quoting such characters in file names is supposed to buy with regard to diff output in 7-bit email.

On Mon, Jun 18, 2007 at 10:00:31AM +0200, David Kastrup wrote:

Not to mention the commit messages.

But more importantly, diffs aren't necessarily going through mail. When I run 'git-show', this isn't useful to me:

Hi,

On Mon, 18 Jun 2007, David Kastrup wrote:

However, please realise that chances are very good that none of these 8-bit unclean things show in the diff.

Besides, the proper fix would probably involve making none-8-bit-clean diffs binary diffs (for FORMAT_EMAIL only, of course).

But isn't that obvious? Even if the diffs are not 7-bit clean, which I consider as an error, quoting the file names is already half what is required.

Don't just throw away backwards compatibility, only because it does not fit your wishes.

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

Puh-leaze. So you prefer a behavior which makes it harder to notice problems, on the chance that it may sometimes work by accident?

If you want to process diffs, you need an 8-bit clean (and space-preserving) channel, period. This is the task of mail encapsulation, not of the diff utility.

This is so utterly absurd for people working on non-English documents that I get the expression you are pulling people's legs considering your Email address.

What is required is a reliable mail channel, and there are a lot of tools for that, from uuencode to various MIME standards and encapsulation methods. The right tool for the right job. Everything else is a mistake because it makes life harder for everyone, not just those using mail, for no good purpose.

There is no backwards compatibility involved here _at_ _all_. No current tool can process the quoted mess, not even humans (random octal escape sequences are not more readable than characters, or we never would have progressed beyond ASCII).

So you are not talking about backward compatibility, but rather gratuitous forward _incompatibility_, and nobody is better off by the latter. There is no point in making life harder for people using non-ASCII characters when there is absolutely no benefit whatsoever involved for those restricting themselves to ASCII characters.

Hi,

On Tue, 19 Jun 2007, David Kastrup wrote:

I was not talking about Git here. The specification for SMTP is not going to change just because you want it. There are still mail servers out there which speak 7-bit, and the standard requires you to cope with them.

Ciao, Dscho

On Tue, Jun 19, 2007 at 10:50:31AM +0100, Johannes Schindelin wrote:

There are standards to send 8-bit into 7-bit for email, and \xxx is in none of them. And 8-to-7 encoding for email is not git's job in any case unless git speaks SMTP directly. 8-to-7 is the mail client responsability.

  OG.

Hi,

[somehow I got the impression your mail did not make it to the list]

On Wed, 13 Jun 2007, David Kastrup wrote:

That is the common reason for quoting. I mean, really, how many files do you have which contain newlines or backslashes or tabs? Huh?

And "Guenther" (sorry, have problems with my mailer, so I simulate it in plain ASCII" does not need quotes, _even_ if containing non-ASCII characters.

So what exactly was your point again?

It is completely unridiculous. Before Microsoft -- in its infinite wisdom -- decided to create folders like "Program Files", and "Documents and Settings", and made it the _default_ (of all things) to save its ridiculous Word documents as "New Document", _nobody_ on this planet even _thought_ about including stupid whitespace in a filename.

You can tell that this is true by looking at now-ancient Unix scripts.

Um. How does that apply here? Git only does quoting so that programs like echo get it right, when passed the name? No funny \0NNN or \NNN or whatever?

echo.

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

I am talking about non-ASCII characters.

You _are_ aware that git writes out \303\274 (8 characters: 2 backslashes and 6 digits) instead of ü in a file name? And I am talking about a pure utf-8 locale.

LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=

My point was that these octal escape sequences are utterly pointless.

You are making a spectacle of yourself. Do you even read what you are replying to? When spaces became commonplace in _MacOS_, _MacOS_ was by no means Unix-based. Microsoft only followed the trend (with a delay of several years, by the way) when imitating the MacOS GUI.

git puts out funny \NNN quotes.  That's what I am complaining about.

It doesn't, since it does not interpret the \NNN escape sequences that git chooses to output.

On 6/14/07, David Kastrup <dak@gnu.org> wrote:

Have you tried that -z switch yet?

On 6/14/07, David Kastrup <dak@gnu.org> wrote:

Just checking.

Well, fix that. How do you think _should_ it be?

It's just up until now you are only complaining. No _useful_ idea came from you.

Hi,

[sorry for responding so late, your mail got stuck in the GWB-like spam filter.]

On Tue, 19 Jun 2007, David Kastrup wrote:

The problem there, of course, is that you still might want to reply to the patch, even if the name was chosen as non-ASCII (which is a sin, if you believe in UNIX).

Usually, comments are not done on the filenames, so they can be as escaped as they want in an email, as long as the commenter still recognizes their names.

There you have a point. If the name is non-ASCII, it uses a specific encoding. if the human reader has a different encoding set in her display, is it any better to display garbled characters (possibly leaving the console in a corrupted state), or to display escaped characters?

And scripts have been known to get encodings all wrong, so I think the escaping is the best way out, absent a perfect knowledge of what encoding the file name was meant for.

I am sorry. No, really, I am sorry that you received it as derision. By all means, it was _not_ meant as that. The problem was on my side, not yours: I simply did not get that you were talking about non-ASCII characters, even if you were talking about them.

I would really, really, really like to see a solution. Alas, I cannot think of one, other than _forcing_ the developers to use ASCII-only filenames.

Note that there is no convention yet in Git to state which encoding your filenames are supposed to use. And in fact, we already had a fine example in git.git why this is particularly difficult. MacOSX is too clever to be true, in that it gladly takes filenames in one encoding, but reads those filenames out in _another_ encoding. Thus, a "git add <filename>" can well end up in git-status saying that a file was deleted, and another file (actually the same, but in a different encoding) is untracked.

Again, I would be _so_ glad if you solved the problem, now that I actually understand it.

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

By the way, the pathname quoting done by "diff" does not even attempt to tackle that. I already explained why in the thread so I would not repeat myself.

Having said that, the absolute minimum that needs to be quoted are double-quote (because it is used by quoting as agreed with GNU diff/patch maintainer), backslash (used to introduce C-like quoting), newline and horizontal tab (makes "patch" confused, as it would make it ambiguous where the pathname ends), so I am not opposed to a patch that introduces a new mode, probably on by default _unless_ we are generating --format=email, that does not quote high byte values. That would solve "My UTF-8 filenames are unreadable on my terminal" problem.

Junio C Hamano <gitster@pobox.com> writes:

And ASCII-only files.  Just eradicate that dreaded Bit 7 from the world.

I think it would be ok to quote non-graphic characters with octal escape sequences. On ASCII-based systems, those are the characters 0x00 to 0x1f. They don't have a visual representation of their own, anyway. _IF_ they appear in filenames, it is certainly a case involved with excessive cleverness and/or garbage. I'd leave the rest alone.

But there is no point if the most primitive of mail readers does a better job than listing the directory will.

7-Bit terminals are the wrong thing to use for manipulating 8-bit-encoded files, period. And the escape sequences for 8-bit terminals are quite certain to start with characters in the 0x00 to 0x1f range.

David Kastrup wrote:

By the way, ls(1) has its --quoting-style=WORD option, why shouldn't git-diff and friends (including git-format-patch) have the same? And we could change the default later on...

Jakub Narebski <jnareb@gmail.com> writes:

Because interpreting a diff means interpreting both file names as well as contents. It does not make much sense to use different forms of escaping (\01a and similar) here, though in the diff command line, some additional quoting might be called for.

It is also worth noting that bash's echo -e can interpret octal escapes only when they start with \0, and the quoted 3-character forms of 0x00-0x1f incidentally do start in this manner. There is still potential for misinterpretation if an escaped character is immediately followed by a digit. Since octal ASCII digits are in the range 060 to 067, one can get around this problem by continuing to escape characters until one hits a non-octal-digit. So there is at least a reasonable builtin way for bash scripts to translate the three-digit octal escapes for 0x00 to 0x1f uniquely into the proper corresponding strings.

With regard to escaping: unless used unarmored in Email (a bad idea) or on a terminal, it might be easiest (for post-processors) to completely refrain from escaping (in effect ignoring the non-printability of characters) and just apply a minimal level of quoting on the file names.

On Tue, Jun 19, 2007 at 23:19:39 -0700, Junio C Hamano wrote:

I saw bazaar folks discussing this MacOSX issue. Basically in MacOSX filenames are *unicode* strings (just as they are in Windows, btw). Unicode, for compatibility reasons allows expressing many characters in multiple forms -- composed and decomposed. For example 'á' can be expressed as '\u00e1' ('\xc3\xa1' in utf-8) or as 'a\u0301' ('a\xcc\x81' in utf-8).

MaxOSX opts to, in accord with unicode standard, treat such representations as equal and it does so by normalizing all filenames to one form. I don't know whether it uses compatibility normalization and I believe it uses the decomposed form (which makes the issue immediately obvious, because most programs work in composed form).

IMHO it should be the default even for email format. Most projects that use non-ascii filenames probably have all members using same locale. And for such group, it will just work. Also usually the file names, content and commit messages will usually be in the same (though project-specific) encoding, so if charset in content-type is set to that, people with different locale able to represent the same characters will still see the names correctly. For other people, the MUA will probably print some escape anyway (it will not screw up the terminal -- it usually knows what it can safely pass to it).

söndag 24 juni 2007 skrev Jan Hudec:

different

I can't talk about "most" here, only local conditions, i.e. northern Europe where both the legacy ISO encodings are very common with a steady increase in UTF-8 usage, in the Linux community. People using OSS in windows almost exclusively get the windows-1252 (for most practical purposes the same as ISO-8859-1).

Even a *very* small set of random people you will wind up with people having different locales.

-- robin

Robin Rosenberg <robin.rosenberg.lists@dewire.com> writes:

More problematic is the case where pathnames and contents are in different encodings, even for the same language.

For example, my mbox files that store messages I receive from people in Japan have contents in ISO-2022 as that is the longstanding standard encoding used for e-mail over there, but the pathname encoding used by the system I have that mbox file on is EUC-JP.

If I were to create a patch between two versions of such a file, the diff header would show the pathname encoded in one, and the changed contents would ben shown in another. As long as you treat "git diff" output as binary blob, that would work just fine, but when you have to transmit such a diff in e-mail as an in-line patch, you would have troubles.

Junio C Hamano <gitster@pobox.com> writes:

ASCII-armoring of what amounts to binary files is the task of the mail software. Also working with encodings. Escaping characters in the diff headers but not in the file contents is not going to achieve anything useful, anyway.

With the proper mailing software, you can get your diff across the line in a manner where the other side can make use of it. This is not the case for unarmored mail with ^ escapes in them, since the receiving side can't distinguish them from "real" ^ characters.

David Kastrup <dak@gnu.org> writes:

You misunderstood me. The issue is not about transmitting without corruption. Armoring would make it impossible to COMMENTING on the patch INLINE.

And that is where the pathname quoting git diff does originally comes from.

On Sun, Jun 24, 2007 at 05:19:12AM -0700, Junio C Hamano wrote:

Then how about quoted-printable?

The point is that you're _already_ screwed by the fact that there can be up to three different encodings in a patch (commit message, pathnames, and file contents) but we only know one of them (the commit message). With the other two, trying to convert encodings is pointless, since we don't know the starting point. So we can either output them as-is as binary, or use some sort of quoting mechanism.

The quoting that happens now is:
  - sometimes unnecessary, and hurts people who are _not_ sending the
    diff through the mail
  - not recognized by any widely-used un-quoter. I can't comment on your
    diff very well if it changes the file "\a/f\303\263\303\266", and
    there's no viewer that will let me read that in a sane way.
    I think David's point is that by doing the quoting at the MIME
    level (using 8bit, or 7bit with QP), the recipient's MUA can at
    least show the binary characters.  Sure, that will totally break if
    you are using a bad mismatch of encodings, but there's nothing we
    can do to fix that, not knowing what the encodings are. At least it
    _will_ work in the case that your encodings are the same.

The only argument I see _for_ the current quoting is for parsing by non-mail programs (like patch or git-apply); in that case, it would seem only necessary only to quote tab, newline, backslash, and double quote. But at least those retain their human-readability.

-Peff

On Sun, Jun 24, 2007 at 13:14:45 +0200, Robin Rosenberg wrote:

A small set of *random* people will likely have different locales. But a project that would use non-ascii filenames would probably use some particular language and thus be run by people that all speak that language -- which means they are not random at all and probably will use the same locale.

söndag 24 juni 2007 skrev Jan Hudec:

Europe 

in 

having 

language --

locale.

I was still in referernce to those "local conditions" at that point. It was not meant as a universal statement. Substitutute that for "A small bunch of swedish speaking people from Stockholm".

-- robin

Robin Rosenberg <robin.rosenberg.lists@dewire.com> writes:

A wrong encoding is a wrong encoding. Escaping the characters in transition will not magically make the encodings adapt. Escaping characters buys us exactly zilch _unless_ the _channel_ is not 8-bit clean. In which case we should use a normal mail-armoring/attachment/inline data wrapper.

In fact, when using editors with some heuristics regarding character sets (like Emacs), leaving 8-bit characters intact gives the editor a chance to guess the correct character set even if it is not the default on the receiving end.

Escaping the characters, in contrast, just hides 8-bit usage away in transition. An escaped character in the wrong encoding will get reconstituted into the wrong encoding.

söndag 24 juni 2007 skrev David Kastrup:

Please don't quote me when the content is not in reference to me or what I've written.

-- robin

Robin Rosenberg <robin.rosenberg.lists@dewire.com> writes:

No idea how that has happened: I actually intended to refer to something different. Sorry.