{"thread":{"id":"27475","subject":"Whitespace and ' '","startedAt":"2011-05-26T21:12:31Z","lastAt":"2011-05-30T12:52:38Z","messageCount":15,"participants":["Linus Torvalds","Junio C Hamano","Matthieu Moy","Andreas Schwab","Daniel Nyström"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"168825","messageId":"BANLkTik7eJ=BC9Bekqu-W1-r0cheCjC+wg@mail.gmail.com","threadId":"27475","inReplyTo":null,"subject":"Whitespace and ' '","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2011-05-26T21:12:31Z","receivedAt":"2011-05-26T21:12:31Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"So I just got a patch that had a hidden   in it (unicode: u00a0,\nutf-8: \\0302\\0240) and sad to say my terminal window shows no\nindication of that what-so-ever. It looks exactly like a regular\nspace, except gcc won't actually accept it.\n\nSo I'm wondering whether git shouldn't be escaping things like that in\ndiffs, for example. Looking at binary noise (using \"git diff --text\"),\nI notice we have other things that don't get escaped either, although\nat least then \"less\" will escape it for us. Again, for  , less\ndoesn't show it any different than a regular space either.\n\nYeah, I'm sure there is some programming font out there that makes it\nvisible, but I do wonder whether git should help.\n\nI even used \"--whitespace=fix\" to apply the patch, but git doesn't\neven recognize the thing as whitespace, so that didn't make any real\ndifference.\n\n Linus\n"},{"id":"168836","messageId":"7vzkm9unu0.fsf@alter.siamese.dyndns.org","threadId":"27475","inReplyTo":"BANLkTik7eJ=BC9Bekqu-W1-r0cheCjC+wg@mail.gmail.com","subject":"Re: Whitespace and ' '","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-26T21:57:43Z","receivedAt":"2011-05-26T21:57:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> So I just got a patch that had a hidden   in it (unicode: u00a0,\n> utf-8: \\0302\\0240) and sad to say my terminal window shows no\n> indication of that what-so-ever. It looks exactly like a regular\n> space, except gcc won't actually accept it.\n\nI have been annoyed by these   in mailing list traffic in general,\nnot just in patches. In my MUA, they are shown as inverted whitespace.\n\nWe could certainly add a new whitespace error class (nbsp?) to diagnose\nand fix to \"git apply\". I do not think people deliberately put these\nthings to their editor, so there probably is no urgent need to add the\nsame diagnosis to \"git diff\", but these days they tend to share the common\ncode, so that may come for free.\n\nBut in the longer term, I think we should find what MUA causes this\nbreakage and yell at them. It might be some mail relays, but I am not sure\nwhere these come from. I often see alternating real whitespace and nbsp\nin \"> \" indented quotes.\n"},{"id":"168837","messageId":"BANLkTi=hYR4ow1eMR3rHkMuVRsHJ=TFDZA@mail.gmail.com","threadId":"27475","inReplyTo":"7vzkm9unu0.fsf@alter.siamese.dyndns.org","subject":"Re: Whitespace and ' '","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2011-05-26T22:08:44Z","receivedAt":"2011-05-26T22:08:44Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Thu, May 26, 2011 at 2:57 PM, Junio C Hamano wrote:\n>\n> But in the longer term, I think we should find what MUA causes this\n> breakage and yell at them. It might be some mail relays, but I am not sure\n> where these come from.  I often see alternating real whitespace and nbsp\n> in \"> \" indented quotes.\n\nIn this case, it was\n\n Content-Type: text/plain; charset=utf-8\n Content-Disposition: inline\n Content-Transfer-Encoding: 8bit\n User-Agent: Mutt/1.5.21 (2010-09-15)\n\nso there was nothing odd in the MUA. And everything I have (gmail in\nweb browser, less, gnome-terminal) treats the nbsp as a regular space,\nso it doesn't show up as any inverted block or anything like that for\nme until I tried to compile it.\n\nIt could easily have been there in the patch originally and wasn't\ncompile-tested. On some keyboard layouts, AltGr+space gives a nbsp.\n\n Linus\n"},{"id":"168838","messageId":"vpqpqn5m7cc.fsf@bauges.imag.fr","threadId":"27475","inReplyTo":"7vzkm9unu0.fsf@alter.siamese.dyndns.org","subject":"Re: Whitespace and ' '","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-05-26T22:21:07Z","receivedAt":"2011-05-26T22:21:07Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> I do not think people deliberately put these things to their editor,\n\nI never put them in C code, but I do often have some in text and LaTeX\nfiles: the French typography rules uses more spaces than the english\nones, in places where one shouldn't break lines (before :, !, ? and\ninside « French quotes »). Real typographers would distinguish short\nnon-breaking spaces and normal ones, but I usually don't ;-).\n\nMy editor shows them as a space underlined in slight color. One can\nreproduce this in a terminal with things like\n\necho -e '\\033[3;4m ' # no color\n\nor\n\necho -e '\\033[1;4m\\033[1;31m ' # red underline space\n\nThat's lighter than full red background, hence acceptable as a warning\nthat people who purposely use   can ignore.\n\n> But in the longer term, I think we should find what MUA causes this\n> breakage and yell at them.\n\nA common typo for French people is to type\n\nfoo | bar\n ^\n  \n\nsince the | needs Alt-gr on azerty keyboard, and Alt-gr + Space gives a\n . Particularly painfull to diagnose since the shell answers e.g.\n\nzsh: command not found:  bar\n\n{ and } also requires Alt-gr, so I can imagine someone typing\n\nif (foo) {\n ^\n  \n\nby mistake.\n\n> It might be some mail relays, but I am not sure where these come from.\n> I often see alternating real whitespace and nbsp in \"> \" indented\n> quotes.\n\nI often see them too, I guess the mailer does this to prevent other\n(broken) mailers from rewrapping, but I don't like it either.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"168882","messageId":"m262owhyuy.fsf@igel.home","threadId":"27475","inReplyTo":"BANLkTi=hYR4ow1eMR3rHkMuVRsHJ=TFDZA@mail.gmail.com","subject":"Re: Whitespace and ' '","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2011-05-27T16:50:13Z","receivedAt":"2011-05-27T16:50:13Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Linus Torvalds writes:\n\n> On Thu, May 26, 2011 at 2:57 PM, Junio C Hamano wrote:\n>>\n>> But in the longer term, I think we should find what MUA causes this\n>> breakage and yell at them. It might be some mail relays, but I am not sure\n>> where these come from.  I often see alternating real whitespace and nbsp\n ^^ Like this...\n>> in \"> \" indented quotes.\n>\n> In this case, it was\n>\n> Content-Type: text/plain; charset=utf-8\n> Content-Disposition: inline\n> Content-Transfer-Encoding: 8bit\n> User-Agent: Mutt/1.5.21 (2010-09-15)\n>\n> so there was nothing odd in the MUA.\n\nIs this a gmail thing?\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"168883","messageId":"BANLkTimPfN6LQBhWj6rW3Zcm9JHPsMWsjA@mail.gmail.com","threadId":"27475","inReplyTo":"m262owhyuy.fsf@igel.home","subject":"Re: Whitespace and ' '","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2011-05-27T17:01:17Z","receivedAt":"2011-05-27T17:01:17Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Fri, May 27, 2011 at 9:50 AM, Andreas Schwab wrote:\n>\n> Is this a gmail thing?\n\nGmail on the sending side - quite possibly part of a reflowing thing,\nespecially with quoting. I send patches as attachments because gmail\nis crap in this regard (and yes, I hate it, and I go back to alpine\nfor any bigger issues).\n\nBut on the receiving side? Sounds unlikely, since I've used gmail for\nthe last year to receive patches, and this is the first time I've seen\nit.\n\nAnd the sent email that had the issue this time implied mutt, not gmail.\n\nRegardless, what took me by surprise was how (a) the patch applied\nfine, (b) it _looked_ fine in all the normal tools, and (c) it just\ndidn't work.\n\nNow, very arguably this is not a git issue at all. Having the odd nbsp\nbe more visible in my other tools would have been fine - either 'less'\nshowing it (the way it shows other control characters) or my terminal\nmaking some visual distinction. That said, I think it's something that\ngit could perhaps protect against a bit.\n\n Linus\n"},{"id":"168889","messageId":"7vipswro57.fsf@alter.siamese.dyndns.org","threadId":"27475","inReplyTo":"BANLkTimPfN6LQBhWj6rW3Zcm9JHPsMWsjA@mail.gmail.com","subject":"Re: Whitespace and ' '","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-27T18:31:32Z","receivedAt":"2011-05-27T18:31:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> Now, very arguably this is not a git issue at all. Having the odd nbsp\n> be more visible in my other tools would have been fine - either 'less'\n> showing it (the way it shows other control characters) or my terminal\n> making some visual distinction. That said, I think it's something that\n> git could perhaps protect against a bit.\n\nYeah. I would have expected that \"git log -p\" would at least show after\nthe fact, relying on \"less\" being aware about it, but that does not seem\nto be the case. Setting my pager to \"cat -e\" is the only workaround I can\nthink of right now (shows \"M-BM- \" there), but of course that wouldn't be\npractical at all. We want to notice this during patch application.\n"},{"id":"168891","messageId":"7vboyorm4i.fsf@alter.siamese.dyndns.org","threadId":"27475","inReplyTo":"7vipswro57.fsf@alter.siamese.dyndns.org","subject":"Re: Whitespace and ' '","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-27T19:15:09Z","receivedAt":"2011-05-27T19:15:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Linus Torvalds writes:\n>\n>> Now, very arguably this is not a git issue at all. Having the odd nbsp\n>> be more visible in my other tools would have been fine - either 'less'\n>> showing it (the way it shows other control characters) or my terminal\n>> making some visual distinction. That said, I think it's something that\n>> git could perhaps protect against a bit.\n>\n> Yeah. I would have expected that \"git log -p\" would at least show after\n> the fact, relying on \"less\" being aware about it, but that does not seem\n> to be the case. Setting my pager to \"cat -e\" is the only workaround I can\n> think of right now (shows \"M-BM- \" there), but of course that wouldn't be\n> practical at all. We want to notice this during patch application.\n\nThe \"diff\" side would look like this. I am tempted to change all the\n\"char *line\" to \"unsigned char *line\", before updating \"apply\", though...\n\n .gitattributes | 2 +-\n ws.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++------\n 2 files changed, 68 insertions(+), 9 deletions(-)\n\ndiff --git a/.gitattributes b/.gitattributes\nindex 5e98806..b4459b0 100644\n--- a/.gitattributes\n+++ b/.gitattributes\n@@ -1,3 +1,3 @@\n * whitespace=!indent,trail,space\n-*.[ch] whitespace=indent,trail,space\n+*.[ch] whitespace=indent,trail,space,nbsp\n *.sh whitespace=indent,trail,space\ndiff --git a/ws.c b/ws.c\nindex 01762cc..4766d6f 100644\n--- a/ws.c\n+++ b/ws.c\n@@ -20,7 +20,7 @@ static struct whitespace_rule {\n \t{ \"blank-at-eol\", WS_BLANK_AT_EOL, 0 },\n \t{ \"blank-at-eof\", WS_BLANK_AT_EOF, 0 },\n \t{ \"tab-in-indent\", WS_TAB_IN_INDENT, 0, 1 },\n-\t{ \"nbsp\", WS_NSBP, 0, 0 },\n+\t{ \"nbsp\", WS_NBSP, 0, 0 },\n };\n \n unsigned parse_whitespace_rule(const char *string)\n@@ -147,6 +147,49 @@ char *whitespace_error_string(unsigned ws)\n \treturn strbuf_detach(&err, NULL);\n }\n \n+static inline int is_nbsp(const char *at_)\n+{\n+\tconst unsigned char *at = (const unsigned char *)at_;\n+\treturn at[0] == 0xc2 && at[1] == 0xa0;\n+}\n+\n+/*\n+ * Show line while highlighting nbsp \" \" (c2a0) if ws is set\n+ */\n+static void emit_with_nbsp_hilite(FILE *stream,\n+\t\t\t\t const char *set, const char *reset,\n+\t\t\t\t const char *ws,\n+\t\t\t\t const char *line, int len)\n+{\n+\tif (!len)\n+\t\treturn;\n+\twhile (len) {\n+\t\tint ok; /* leading segment without problematic nbsp */\n+\t\tif (!ws) {\n+\t\t\tok = len;\n+\t\t} else {\n+\t\t\tfor (ok = 0; ok < len; ok++) {\n+\t\t\t\tif (ok < len - 1 && is_nbsp(line + ok))\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t\tif (ok) {\n+\t\t\tfputs(set, stream);\n+\t\t\tfwrite(line, ok, 1, stream);\n+\t\t\tfputs(reset, stream);\n+\t\t}\n+\t\tline += ok;\n+\t\tlen -= ok;\n+\t\tif (len) {\n+\t\t\tfputs(ws, stream);\n+\t\t\tfwrite(line, 2, 1, stream);\n+\t\t\tfputs(reset, stream);\n+\t\t\tline += 2;\n+\t\t\tlen -= 2;\n+\t\t}\n+\t}\n+}\n+\n /* If stream is non-NULL, emits the line after checking. */\n static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\t\t\tFILE *stream, const char *set,\n@@ -170,6 +213,24 @@ static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\tlen--;\n \t}\n \n+\t/* Check for nbsp in UTF-8 (c2a0) */\n+\tif (ws_rule & WS_NBSP) {\n+\t\tfor (i = 1; i < len; i++) {\n+\t\t\tswitch (line[i] & 0xff) {\n+\t\t\tcase 0xc2:\n+\t\t\t\tbreak;\n+\t\t\tcase 0xa0:\n+\t\t\t\tif ((line[i-1] & 0xff) == 0xc2) {\n+\t\t\t\t\tresult |= WS_NBSP;\n+\t\t\t\t\tcontinue;\n+\t\t\t\t}\n+\t\t\t\t/* fallthru */\n+\t\t\tdefault:\n+\t\t\t\ti++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \t/* Check for trailing whitespace. */\n \tif (ws_rule & WS_BLANK_AT_EOL) {\n \t\tfor (i = len - 1; i >= 0; i--) {\n@@ -230,13 +291,11 @@ static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\t * The non-highlighted part ends at \"trailing_whitespace\".\n \t\t */\n \n-\t\t/* Emit non-highlighted (middle) segment. */\n-\t\tif (trailing_whitespace - written > 0) {\n-\t\t\tfputs(set, stream);\n-\t\t\tfwrite(line + written,\n-\t\t\t trailing_whitespace - written, 1, stream);\n-\t\t\tfputs(reset, stream);\n-\t\t}\n+\t\t/* Emit middle segment, highlighting nbsp as needed */\n+\t\temit_with_nbsp_hilite(stream, set, reset,\n+\t\t\t\t (result & WS_NBSP) ? ws : NULL,\n+\t\t\t\t line + written,\n+\t\t\t\t trailing_whitespace - written);\n \n \t\t/* Highlight errors in trailing whitespace. */\n \t\tif (trailing_whitespace != len) {\n"},{"id":"168900","messageId":"BANLkTinwOr5Yzp_N6BNyNK5Y1FcVtdtbUw@mail.gmail.com","threadId":"27475","inReplyTo":"7vboyorm4i.fsf@alter.siamese.dyndns.org","subject":"Re: Whitespace and ' '","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2011-05-27T21:14:54Z","receivedAt":"2011-05-27T21:14:54Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Fri, May 27, 2011 at 12:15 PM, Junio C Hamano wrote:\n>\n> The \"diff\" side would look like this.  I am tempted to change all the\n> \"char *line\" to \"unsigned char *line\", before updating \"apply\", though...\n\nWithout actually testing it, it looks reasonable.\n\nI do wonder whether the highlighting might be something that is\nindependent of whitespace issues. What about mis-formed or\nnon-printable characters? Right now we depend on \"less\" doing the\nright thing, but I can make \"git diff\" generate random escape\nsequences etc by just putting them in the file, so if I do\nGIT_PAGER=cat I can make \"git diff\" really mess up my terminal if I\nwanted to.\n\nAgain, I'm not convinced git should really care, but I'm also not\nconvinced that sbsp is necessarily all about the git whitespace\nfixups.\n\n Linus\n"},{"id":"168902","messageId":"7vy61rrcae.fsf@alter.siamese.dyndns.org","threadId":"27475","inReplyTo":"BANLkTinwOr5Yzp_N6BNyNK5Y1FcVtdtbUw@mail.gmail.com","subject":"Re: Whitespace and ' '","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-27T22:47:37Z","receivedAt":"2011-05-27T22:47:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> Again, I'm not convinced git should really care, but I'm also not\n> convinced that sbsp is necessarily all about the git whitespace\n> fixups.\n\nI am not convinced git should care, either, but if nobody else helps us,\nwe need to help ourselves ;-).\n"},{"id":"168903","messageId":"7vr57jrc8c.fsf_-_@alter.siamese.dyndns.org","threadId":"27475","inReplyTo":"7vy61rrcae.fsf@alter.siamese.dyndns.org","subject":"[PATCH 1/3] ws.c: add a helper to format comma separated messages","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-27T22:48:51Z","receivedAt":"2011-05-27T22:48:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"We can find more than one class of whitespace errors on a single line,\nand we concatenate a message per class in a strbuf, separated with a\ncolon. Use a small helper function to make the code easier to read.\n\nSigned-off-by: Junio C Hamano \n---\n * Just a trivial clean-up before waging a \"War on nbsp\"\n\n ws.c | 43 +++++++++++++++++++------------------------\n 1 files changed, 19 insertions(+), 24 deletions(-)\n\ndiff --git a/ws.c b/ws.c\nindex 9fb9b14..3058be4 100644\n--- a/ws.c\n+++ b/ws.c\n@@ -116,36 +116,31 @@ unsigned whitespace_rule(const char *pathname)\n \t}\n }\n \n+static void add_err_item(struct strbuf *err, const char *message)\n+{\n+\tif (err->len)\n+\t\tstrbuf_addstr(err, \", \");\n+\tstrbuf_addstr(err, message);\n+}\n+\n /* The returned string should be freed by the caller. */\n char *whitespace_error_string(unsigned ws)\n {\n \tstruct strbuf err = STRBUF_INIT;\n-\tif ((ws & WS_TRAILING_SPACE) == WS_TRAILING_SPACE)\n-\t\tstrbuf_addstr(&err, \"trailing whitespace\");\n-\telse {\n+\tif ((ws & WS_TRAILING_SPACE) == WS_TRAILING_SPACE) {\n+\t\tadd_err_item(&err, \"trailing whitespace\");\n+\t} else {\n \t\tif (ws & WS_BLANK_AT_EOL)\n-\t\t\tstrbuf_addstr(&err, \"trailing whitespace\");\n-\t\tif (ws & WS_BLANK_AT_EOF) {\n-\t\t\tif (err.len)\n-\t\t\t\tstrbuf_addstr(&err, \", \");\n-\t\t\tstrbuf_addstr(&err, \"new blank line at EOF\");\n-\t\t}\n-\t}\n-\tif (ws & WS_SPACE_BEFORE_TAB) {\n-\t\tif (err.len)\n-\t\t\tstrbuf_addstr(&err, \", \");\n-\t\tstrbuf_addstr(&err, \"space before tab in indent\");\n-\t}\n-\tif (ws & WS_INDENT_WITH_NON_TAB) {\n-\t\tif (err.len)\n-\t\t\tstrbuf_addstr(&err, \", \");\n-\t\tstrbuf_addstr(&err, \"indent with spaces\");\n-\t}\n-\tif (ws & WS_TAB_IN_INDENT) {\n-\t\tif (err.len)\n-\t\t\tstrbuf_addstr(&err, \", \");\n-\t\tstrbuf_addstr(&err, \"tab in indent\");\n+\t\t\tadd_err_item(&err, \"trailing whitespace\");\n+\t\tif (ws & WS_BLANK_AT_EOF)\n+\t\t\tadd_err_item(&err, \"new blank line at EOF\");\n \t}\n+\tif (ws & WS_SPACE_BEFORE_TAB)\n+\t\tadd_err_item(&err, \"space before tab in indent\");\n+\tif (ws & WS_INDENT_WITH_NON_TAB)\n+\t\tadd_err_item(&err, \"indent with spaces\");\n+\tif (ws & WS_TAB_IN_INDENT)\n+\t\tadd_err_item(&err, \"tab in indent\");\n \treturn strbuf_detach(&err, NULL);\n }\n \n-- \n1.7.5.3.503.g893a4\n"},{"id":"168904","messageId":"7vmxi7rc7o.fsf_-_@alter.siamese.dyndns.org","threadId":"27475","inReplyTo":"7vy61rrcae.fsf@alter.siamese.dyndns.org","subject":"[PATCH 2/3] War on nbsp: a bit of retreat","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-27T22:49:15Z","receivedAt":"2011-05-27T22:49:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Before introducing a new whitespace breakage class \"nbsp\" to catch\ncompiler-breaking use of nbsp (UTF-8 c2a0) in place of SP, update the\ncurrent code to treat a nbsp just like SP that takes one display column.\n\nIndenting with 6 nbsp is not an error under the indent-with-non-tab rule,\nas it only consumes 6 display columns even though it may occupy 12 bytes.\n\nSigned-off-by: Junio C Hamano \n---\n t/t4019-diff-wserror.sh | 93 ++++++++++++++++++++++++++++++++---------------\n ws.c | 38 ++++++++++++++++----\n 2 files changed, 94 insertions(+), 37 deletions(-)\n\ndiff --git a/t/t4019-diff-wserror.sh b/t/t4019-diff-wserror.sh\nindex a501975..665f693 100755\n--- a/t/t4019-diff-wserror.sh\n+++ b/t/t4019-diff-wserror.sh\n@@ -14,6 +14,9 @@ test_expect_success setup '\n \techo \"With trailing SP \" >>F &&\n \techo \"Carriage ReturnQ\" | tr Q \"\\015\" >>F &&\n \techo \"No problem\" >>F &&\n+\techo \"  Enough NBSP and Space\" >>F &&\n+\techo \"     Bit of NBSP and Space\" >>F &&\n+\techo \"NBSP At End  \" >>F &&\n \techo >>F\n \n '\n@@ -47,8 +50,10 @@ test_expect_success default '\n \tgrep HT error >/dev/null &&\n \tgrep With error >/dev/null &&\n \tgrep Return error >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep Enough normal >/dev/null &&\n+\tgrep No normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End error >/dev/null\n '\n \n test_expect_success 'default (attribute)' '\n@@ -61,8 +66,10 @@ test_expect_success 'default (attribute)' '\n \tgrep HT error >/dev/null &&\n \tgrep With error >/dev/null &&\n \tgrep Return error >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough error >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End error >/dev/null\n '\n \n test_expect_success 'default, tabwidth=10 (attribute)' '\n@@ -75,8 +82,10 @@ test_expect_success 'default, tabwidth=10 (attribute)' '\n \tgrep HT error >/dev/null &&\n \tgrep With error >/dev/null &&\n \tgrep Return error >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End error >/dev/null\n '\n \n test_expect_success 'no check (attribute)' '\n@@ -89,8 +98,10 @@ test_expect_success 'no check (attribute)' '\n \tgrep HT normal >/dev/null &&\n \tgrep With normal >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End normal >/dev/null\n '\n \n test_expect_success 'no check, tabwidth=10 (attribute), must be irrelevant' '\n@@ -103,8 +114,10 @@ test_expect_success 'no check, tabwidth=10 (attribute), must be irrelevant' '\n \tgrep HT normal >/dev/null &&\n \tgrep With normal >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End normal >/dev/null\n '\n \n test_expect_success 'without -trail' '\n@@ -117,8 +130,10 @@ test_expect_success 'without -trail' '\n \tgrep HT error >/dev/null &&\n \tgrep With normal >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End normal >/dev/null\n '\n \n test_expect_success 'without -trail (attribute)' '\n@@ -131,8 +146,10 @@ test_expect_success 'without -trail (attribute)' '\n \tgrep HT error >/dev/null &&\n \tgrep With normal >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End normal >/dev/null\n '\n \n test_expect_success 'without -space' '\n@@ -145,8 +162,10 @@ test_expect_success 'without -space' '\n \tgrep HT normal >/dev/null &&\n \tgrep With error >/dev/null &&\n \tgrep Return error >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End error >/dev/null\n '\n \n test_expect_success 'without -space (attribute)' '\n@@ -159,8 +178,10 @@ test_expect_success 'without -space (attribute)' '\n \tgrep HT normal >/dev/null &&\n \tgrep With error >/dev/null &&\n \tgrep Return error >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End error >/dev/null\n '\n \n test_expect_success 'with indent-non-tab only' '\n@@ -173,8 +194,10 @@ test_expect_success 'with indent-non-tab only' '\n \tgrep HT normal >/dev/null &&\n \tgrep With normal >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough error >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End normal >/dev/null\n '\n \n test_expect_success 'with indent-non-tab only (attribute)' '\n@@ -187,8 +210,10 @@ test_expect_success 'with indent-non-tab only (attribute)' '\n \tgrep HT normal >/dev/null &&\n \tgrep With normal >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough error >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End normal >/dev/null\n '\n \n test_expect_success 'with indent-non-tab only, tabwidth=10' '\n@@ -201,8 +226,10 @@ test_expect_success 'with indent-non-tab only, tabwidth=10' '\n \tgrep HT normal >/dev/null &&\n \tgrep With normal >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End normal >/dev/null\n '\n \n test_expect_success 'with indent-non-tab only, tabwidth=10 (attribute)' '\n@@ -215,8 +242,10 @@ test_expect_success 'with indent-non-tab only, tabwidth=10 (attribute)' '\n \tgrep HT normal >/dev/null &&\n \tgrep With normal >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End normal >/dev/null\n '\n \n test_expect_success 'with cr-at-eol' '\n@@ -229,8 +258,10 @@ test_expect_success 'with cr-at-eol' '\n \tgrep HT error >/dev/null &&\n \tgrep With error >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End error >/dev/null\n '\n \n test_expect_success 'with cr-at-eol (attribute)' '\n@@ -243,8 +274,10 @@ test_expect_success 'with cr-at-eol (attribute)' '\n \tgrep HT error >/dev/null &&\n \tgrep With error >/dev/null &&\n \tgrep Return normal >/dev/null &&\n-\tgrep No normal >/dev/null\n-\n+\tgrep No normal >/dev/null &&\n+\tgrep Enough normal >/dev/null &&\n+\tgrep Bit normal >/dev/null &&\n+\tgrep End error >/dev/null\n '\n \n test_expect_success 'trailing empty lines (1)' '\ndiff --git a/ws.c b/ws.c\nindex 3058be4..68c7599 100644\n--- a/ws.c\n+++ b/ws.c\n@@ -144,6 +144,12 @@ char *whitespace_error_string(unsigned ws)\n \treturn strbuf_detach(&err, NULL);\n }\n \n+static int is_nbsp(const char *at_)\n+{\n+\tunsigned const char *at = (unsigned const char *)at_;\n+\treturn at[0] == 0xc2 && at[1] == 0xa0;\n+}\n+\n /* If stream is non-NULL, emits the line after checking. */\n static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\t\t\tFILE *stream, const char *set,\n@@ -154,7 +160,7 @@ static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \tint trailing_whitespace = -1;\n \tint trailing_newline = 0;\n \tint trailing_carriage_return = 0;\n-\tint i;\n+\tint i, col_offset;\n \n \t/* Logic is simpler if we temporarily ignore the trailing newline. */\n \tif (len > 0 && line[len - 1] == '\\n') {\n@@ -170,11 +176,16 @@ static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t/* Check for trailing whitespace. */\n \tif (ws_rule & WS_BLANK_AT_EOL) {\n \t\tfor (i = len - 1; i >= 0; i--) {\n-\t\t\tif (isspace(line[i])) {\n+\t\t\tint is_space = isspace(line[i]);\n+\n+\t\t\tif (!is_space && i && is_nbsp(&line[i-1])) {\n+\t\t\t\tis_space = 1;\n+\t\t\t\ti--;\n+\t\t\t}\n+\t\t\tif (is_space) {\n \t\t\t\ttrailing_whitespace = i;\n \t\t\t\tresult |= WS_BLANK_AT_EOL;\n-\t\t\t}\n-\t\t\telse\n+\t\t\t} else\n \t\t\t\tbreak;\n \t\t}\n \t}\n@@ -183,9 +194,14 @@ static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\ttrailing_whitespace = len;\n \n \t/* Check indentation */\n-\tfor (i = 0; i < trailing_whitespace; i++) {\n+\tfor (i = col_offset = 0; i < trailing_whitespace; i++) {\n \t\tif (line[i] == ' ')\n \t\t\tcontinue;\n+\t\tif (i + 1 < trailing_whitespace && is_nbsp(&line[i])) {\n+\t\t\ti++;\n+\t\t\tcol_offset++;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (line[i] != '\\t')\n \t\t\tbreak;\n \t\tif ((ws_rule & WS_SPACE_BEFORE_TAB) && written < i) {\n@@ -208,10 +224,12 @@ static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\t\tfwrite(line + written, i - written + 1, 1, stream);\n \t\t}\n \t\twritten = i + 1;\n+\t\tcol_offset = 0;\n \t}\n \n \t/* Check for indent using non-tab. */\n-\tif ((ws_rule & WS_INDENT_WITH_NON_TAB) && i - written >= ws_tab_width(ws_rule)) {\n+\tif ((ws_rule & WS_INDENT_WITH_NON_TAB) &&\n+\t i - written - col_offset >= ws_tab_width(ws_rule)) {\n \t\tresult |= WS_INDENT_WITH_NON_TAB;\n \t\tif (stream) {\n \t\t\tfputs(ws, stream);\n@@ -270,7 +288,13 @@ int ws_blank_line(const char *line, int len, unsigned ws_rule)\n \t * for now we just use this stupid definition.\n \t */\n \twhile (len-- > 0) {\n-\t\tif (!isspace(*line))\n+\t\tif (isspace(*line))\n+\t\t\t;\n+\t\telse if (len && is_nbsp(line)) {\n+\t\t\tline++;\n+\t\t\tlen--;\n+\t\t}\n+\t\telse\n \t\t\treturn 0;\n \t\tline++;\n \t}\n-- \n1.7.5.3.503.g893a4\n"},{"id":"168905","messageId":"7vfwnzrc4h.fsf_-_@alter.siamese.dyndns.org","threadId":"27475","inReplyTo":"7vy61rrcae.fsf@alter.siamese.dyndns.org","subject":"[PATCH 3/3] War on nbsp: Add \"nbsp\" whitespace breakage class","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-27T22:51:10Z","receivedAt":"2011-05-27T22:51:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Not even \"less\" shows nbsp as anything special nor unusual, so eyeballing\nwith \"git log -p\" after applying a patch that accidentally had it where a\nregular SP should be, breaking compilation, would not help.\n\nThis only handles \"diff\", not \"apply\" yet.\n\nSigned-off-by: Junio C Hamano \n---\n * This is the moral equivalent of the earlier patch, but based on the two\n clean-ups. If we do not consider nbsp an error, at least we should\n count it just like an ordinary SP that takes one display column, and if\n we do not like indenting with non-TAB or trailing whitespaces, we\n should complain loudly, which was the topic of 2/3.\n\n cache.h | 18 ++++++-----\n t/t4019-diff-wserror.sh | 8 ++--\n ws.c | 72 ++++++++++++++++++++++++++++++++++++++++++----\n 3 files changed, 79 insertions(+), 19 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex dd34fed..4f58587 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1122,17 +1122,19 @@ void shift_tree_by(const unsigned char *, const unsigned char *, unsigned char *\n /*\n * whitespace rules.\n * used by both diff and apply\n- * last two digits are tab width\n+ * last 6-bits are tab width\n */\n-#define WS_BLANK_AT_EOL 0100\n-#define WS_SPACE_BEFORE_TAB 0200\n-#define WS_INDENT_WITH_NON_TAB 0400\n-#define WS_CR_AT_EOL 01000\n-#define WS_BLANK_AT_EOF 02000\n-#define WS_TAB_IN_INDENT 04000\n+#define WS_TAB_WIDTH_MASK 077\n+#define WS_BLANK_AT_EOL (1<< 6)\n+#define WS_SPACE_BEFORE_TAB (1<< 7)\n+#define WS_INDENT_WITH_NON_TAB (1<< 8)\n+#define WS_CR_AT_EOL (1<< 9)\n+#define WS_BLANK_AT_EOF (1<<10)\n+#define WS_TAB_IN_INDENT (1<<11)\n+#define WS_NBSP (1<<12)\n #define WS_TRAILING_SPACE (WS_BLANK_AT_EOL|WS_BLANK_AT_EOF)\n #define WS_DEFAULT_RULE (WS_TRAILING_SPACE|WS_SPACE_BEFORE_TAB|8)\n-#define WS_TAB_WIDTH_MASK 077\n+\n extern unsigned whitespace_rule_cfg;\n extern unsigned whitespace_rule(const char *);\n extern unsigned parse_whitespace_rule(const char *);\ndiff --git a/t/t4019-diff-wserror.sh b/t/t4019-diff-wserror.sh\nindex 665f693..8c7fea2 100755\n--- a/t/t4019-diff-wserror.sh\n+++ b/t/t4019-diff-wserror.sh\n@@ -56,7 +56,7 @@ test_expect_success default '\n \tgrep End error >/dev/null\n '\n \n-test_expect_success 'default (attribute)' '\n+test_expect_success 'default (attribute) -- must check all available rule' '\n \n \ttest_might_fail git config --unset core.whitespace &&\n \techo \"F whitespace\" >.gitattributes &&\n@@ -68,7 +68,7 @@ test_expect_success 'default (attribute)' '\n \tgrep Return error >/dev/null &&\n \tgrep No normal >/dev/null &&\n \tgrep Enough error >/dev/null &&\n-\tgrep Bit normal >/dev/null &&\n+\tgrep Bit error >/dev/null &&\n \tgrep End error >/dev/null\n '\n \n@@ -83,8 +83,8 @@ test_expect_success 'default, tabwidth=10 (attribute)' '\n \tgrep With error >/dev/null &&\n \tgrep Return error >/dev/null &&\n \tgrep No normal >/dev/null &&\n-\tgrep Enough normal >/dev/null &&\n-\tgrep Bit normal >/dev/null &&\n+\tgrep Enough error >/dev/null &&\n+\tgrep Bit error >/dev/null &&\n \tgrep End error >/dev/null\n '\n \ndiff --git a/ws.c b/ws.c\nindex 68c7599..53e263d 100644\n--- a/ws.c\n+++ b/ws.c\n@@ -20,6 +20,7 @@ static struct whitespace_rule {\n \t{ \"blank-at-eol\", WS_BLANK_AT_EOL, 0 },\n \t{ \"blank-at-eof\", WS_BLANK_AT_EOF, 0 },\n \t{ \"tab-in-indent\", WS_TAB_IN_INDENT, 0, 1 },\n+\t{ \"nbsp\", WS_NBSP, 0, 0 },\n };\n \n unsigned parse_whitespace_rule(const char *string)\n@@ -141,6 +142,8 @@ char *whitespace_error_string(unsigned ws)\n \t\tadd_err_item(&err, \"indent with spaces\");\n \tif (ws & WS_TAB_IN_INDENT)\n \t\tadd_err_item(&err, \"tab in indent\");\n+\tif (ws & WS_NBSP)\n+\t\tadd_err_item(&err, \"  in source\");\n \treturn strbuf_detach(&err, NULL);\n }\n \n@@ -150,6 +153,45 @@ static int is_nbsp(const char *at_)\n \treturn at[0] == 0xc2 && at[1] == 0xa0;\n }\n \n+/*\n+ * Show line while highlighting nbsp \" \" (c2a0) if ws is set\n+ */\n+static void emit_with_nbsp_hilite(FILE *stream,\n+\t\t\t\t const char *set, const char *reset,\n+\t\t\t\t const char *ws,\n+\t\t\t\t const char *line, int len)\n+{\n+\tif (!len)\n+\t\treturn;\n+\twhile (len) {\n+\t\t/* number of bytes in the leading segment w/o nbsp error */\n+\t\tint ok;\n+\t\tif (!ws) {\n+\t\t\tok = len;\n+\t\t} else {\n+\t\t\tfor (ok = 0; ok < len; ok++) {\n+\t\t\t\tif (ok < len - 1 && is_nbsp(line + ok))\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t\tif (ok) {\n+\t\t\tfputs(set, stream);\n+\t\t\tfwrite(line, ok, 1, stream);\n+\t\t\tfputs(reset, stream);\n+\t\t}\n+\t\tline += ok;\n+\t\tlen -= ok;\n+\t\tif (len) {\n+\t\t\t/* do not bother bundling consecutive ones */\n+\t\t\tfputs(ws, stream);\n+\t\t\tfwrite(line, 2, 1, stream);\n+\t\t\tfputs(reset, stream);\n+\t\t\tline += 2;\n+\t\t\tlen -= 2;\n+\t\t}\n+\t}\n+}\n+\n /* If stream is non-NULL, emits the line after checking. */\n static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\t\t\tFILE *stream, const char *set,\n@@ -173,6 +215,24 @@ static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\tlen--;\n \t}\n \n+\t/* Check for nbsp in UTF-8 (c2a0) */\n+\tif (ws_rule & WS_NBSP) {\n+\t\tfor (i = 1; i < len; i++) {\n+\t\t\tswitch (line[i] & 0xff) {\n+\t\t\tcase 0xc2:\n+\t\t\t\tbreak;\n+\t\t\tcase 0xa0:\n+\t\t\t\tif ((line[i-1] & 0xff) == 0xc2) {\n+\t\t\t\t\tresult |= WS_NBSP;\n+\t\t\t\t\tcontinue;\n+\t\t\t\t}\n+\t\t\t\t/* fallthru */\n+\t\t\tdefault:\n+\t\t\t\ti++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \t/* Check for trailing whitespace. */\n \tif (ws_rule & WS_BLANK_AT_EOL) {\n \t\tfor (i = len - 1; i >= 0; i--) {\n@@ -245,13 +305,11 @@ static unsigned ws_check_emit_1(const char *line, int len, unsigned ws_rule,\n \t\t * The non-highlighted part ends at \"trailing_whitespace\".\n \t\t */\n \n-\t\t/* Emit non-highlighted (middle) segment. */\n-\t\tif (trailing_whitespace - written > 0) {\n-\t\t\tfputs(set, stream);\n-\t\t\tfwrite(line + written,\n-\t\t\t trailing_whitespace - written, 1, stream);\n-\t\t\tfputs(reset, stream);\n-\t\t}\n+\t\t/* Emit middle segment, highlighting nbsp as needed */\n+\t\temit_with_nbsp_hilite(stream, set, reset,\n+\t\t\t\t (result & WS_NBSP) ? ws : NULL,\n+\t\t\t\t line + written,\n+\t\t\t\t trailing_whitespace - written);\n \n \t\t/* Highlight errors in trailing whitespace. */\n \t\tif (trailing_whitespace != len) {\n-- \n1.7.5.3.503.g893a4\n"},{"id":"168906","messageId":"7vboynr4oh.fsf_-_@alter.siamese.dyndns.org","threadId":"27475","inReplyTo":"7vy61rrcae.fsf@alter.siamese.dyndns.org","subject":"[PATCH 4/3] War on nbsp: teach \"git apply\" to check and fix nbsp","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-28T01:31:58Z","receivedAt":"2011-05-28T01:31:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This still does not work to apply in reverse \"git apply -R\", but I thought\nit is a good place to stop, as it is dubious if this series makes much\nsense to begin with.\n\nSigned-off-by: Junio C Hamano \n---\n ws.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++----------------\n 1 files changed, 54 insertions(+), 18 deletions(-)\n\ndiff --git a/ws.c b/ws.c\nindex 53e263d..4413f95 100644\n--- a/ws.c\n+++ b/ws.c\n@@ -374,6 +374,7 @@ void ws_fix_copy(struct strbuf *dst, const char *src, int len, unsigned ws_rule,\n \tint last_tab_in_indent = -1;\n \tint last_space_in_indent = -1;\n \tint need_fix_leading_space = 0;\n+\tint col_offset = 0;\n \n \t/*\n \t * Strip trailing whitespace\n@@ -387,10 +388,17 @@ void ws_fix_copy(struct strbuf *dst, const char *src, int len, unsigned ws_rule,\n \t\t\t\tlen--;\n \t\t\t}\n \t\t}\n-\t\tif (0 < len && isspace(src[len - 1])) {\n-\t\t\twhile (0 < len && isspace(src[len-1]))\n-\t\t\t\tlen--;\n-\t\t\tfixed = 1;\n+\t\tif (0 < len) {\n+\t\t\tint orig_len = len;\n+\t\t\twhile (len) {\n+\t\t\t\tif (isspace(src[len - 1]))\n+\t\t\t\t\tlen--;\n+\t\t\t\telse if (1 < len && is_nbsp(&src[len - 2]))\n+\t\t\t\t\tlen -= 2;\n+\t\t\t\telse\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tfixed = (orig_len != len);\n \t\t}\n \t}\n \n@@ -404,13 +412,23 @@ void ws_fix_copy(struct strbuf *dst, const char *src, int len, unsigned ws_rule,\n \t\t\tif ((ws_rule & WS_SPACE_BEFORE_TAB) &&\n \t\t\t 0 <= last_space_in_indent)\n \t\t\t need_fix_leading_space = 1;\n-\t\t} else if (ch == ' ') {\n-\t\t\tlast_space_in_indent = i;\n-\t\t\tif ((ws_rule & WS_INDENT_WITH_NON_TAB) &&\n-\t\t\t ws_tab_width(ws_rule) <= i - last_tab_in_indent)\n-\t\t\t\tneed_fix_leading_space = 1;\n-\t\t} else\n+\t\t\tcol_offset = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (ch == ' ') {\n+\t\t\t;\n+\t\t} else if ((i < len - 1) && is_nbsp(src + i)) {\n+\t\t\ti++;\n+\t\t\tcol_offset++;\n+\t\t} else {\n \t\t\tbreak;\n+\t\t}\n+\t\tlast_space_in_indent = i;\n+\n+\t\tif ((ws_rule & WS_INDENT_WITH_NON_TAB) &&\n+\t\t ws_tab_width(ws_rule) <= (i - col_offset) - last_tab_in_indent)\n+\t\t\tneed_fix_leading_space = 1;\n \t}\n \n \tif (need_fix_leading_space) {\n@@ -432,15 +450,20 @@ void ws_fix_copy(struct strbuf *dst, const char *src, int len, unsigned ws_rule,\n \t\t */\n \t\tfor (i = 0; i < last; i++) {\n \t\t\tchar ch = src[i];\n-\t\t\tif (ch != ' ') {\n+\n+\t\t\tif (ch == ' ') {\n+\t\t\t\t;\n+\t\t\t} else if ((i < last - 1) && is_nbsp(src + i)) {\n+\t\t\t\ti++;\n+\t\t\t} else {\n \t\t\t\tconsecutive_spaces = 0;\n \t\t\t\tstrbuf_addch(dst, ch);\n-\t\t\t} else {\n-\t\t\t\tconsecutive_spaces++;\n-\t\t\t\tif (consecutive_spaces == ws_tab_width(ws_rule)) {\n-\t\t\t\t\tstrbuf_addch(dst, '\\t');\n-\t\t\t\t\tconsecutive_spaces = 0;\n-\t\t\t\t}\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tconsecutive_spaces++;\n+\t\t\tif (consecutive_spaces == ws_tab_width(ws_rule)) {\n+\t\t\t\tstrbuf_addch(dst, '\\t');\n+\t\t\t\tconsecutive_spaces = 0;\n \t\t\t}\n \t\t}\n \t\twhile (0 < consecutive_spaces--)\n@@ -465,7 +488,20 @@ void ws_fix_copy(struct strbuf *dst, const char *src, int len, unsigned ws_rule,\n \t\tfixed = 1;\n \t}\n \n-\tstrbuf_add(dst, src, len);\n+\tif (ws_rule & WS_NBSP) {\n+\t\twhile (len--) {\n+\t\t\tif (len && is_nbsp(src)) {\n+\t\t\t\tsrc++;\n+\t\t\t\tlen--;\n+\t\t\t\tstrbuf_addch(dst, ' ');\n+\t\t\t} else {\n+\t\t\t\tstrbuf_addch(dst, *src);\n+\t\t\t}\n+\t\t\tsrc++;\n+\t\t}\n+\t} else {\n+\t\tstrbuf_add(dst, src, len);\n+\t}\n \tif (add_cr_to_tail)\n \t\tstrbuf_addch(dst, '\\r');\n \tif (add_nl_to_tail)\n-- \n1.7.5.3.503.g893a4\n"},{"id":"168992","messageId":"BANLkTinNwfuiFtg+y66zmgTJWSoWbB0wXw@mail.gmail.com","threadId":"27475","inReplyTo":"7vy61rrcae.fsf@alter.siamese.dyndns.org","subject":"Re: Whitespace and ' '","fromName":"Daniel Nyström","fromEmail":"daniel.nystrom@timeterminal.se","sentAt":"2011-05-30T12:52:38Z","receivedAt":"2011-05-30T12:52:38Z","isPatch":false,"sender":{"key":"daniel.nystrom@timeterminal.se","avatar":null},"body":"2011/5/28 Junio C Hamano :\n>> Again, I'm not convinced git should really care, but I'm also not\n>> convinced that sbsp is necessarily all about the git whitespace\n>> fixups.\n>\n> I am not convinced git should care, either, but if nobody else helps us,\n> we need to help ourselves ;-).\n\nOr write a standalone tool through which patches (and code for that\nmatter) could be piped, with the only purpose of showing \"misleading\"\nunicode characters and the likes?\n\nI can see even wider use of such a tool.\n"}]}