{"thread":{"id":"56657","subject":"[PATCH 0/3] Fun with cpp word regex","startedAt":"2021-10-07T06:51:06Z","lastAt":"2021-10-24T09:56:49Z","messageCount":24,"participants":["Johannes Sixt via GitGitGadget","Ævar Arnfjörð Bjarmason","Johannes Sixt"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"438122","messageId":"pull.1054.git.1633589461.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":null,"subject":"[PATCH 0/3] Fun with cpp word regex","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T06:50:57Z","receivedAt":"2021-10-07T06:51:06Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"The cpp word regex driver is a bit too loose and can match too much text\nwhere the intent is to match only a number. The first patch fixes that.\n\nThe other two patches add support for digit separators and the spaceship\noperator <=> (generalized comparison operator).\n\nI left out support for hexadecimal floating point constants because that\nwould require to tighten the regex even more to avoid that entire\nexpressions are treated as single tokens.\n\nJohannes Sixt (3):\n userdiff: tighten cpp word regex\n userdiff: permit the digit-separating single-quote in numbers\n userdiff: learn the C++ spaceship operator\n\n userdiff.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\n\nbase-commit: 225bc32a989d7a22fa6addafd4ce7dcd04675dbf\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1054%2Fj6t%2Ffun-with-cpp-word-regex-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1054/j6t/fun-with-cpp-word-regex-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1054\n-- \ngitgitgadget\n"},{"id":"438123","messageId":"a47ab9ba20ec931a9d7a844633319c486dd699a1.1633589461.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.git.1633589461.gitgitgadget@gmail.com","subject":"[PATCH 1/3] userdiff: tighten cpp word regex","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T06:50:58Z","receivedAt":"2021-10-07T06:51:08Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nGenerally, word regex can be written such that they match tokens\nliberally and need not model the actual syntax because it can be assumed\nthat the regex will only be applied to syntactically correct text.\n\nThe regex for cpp (C/C++) is too liberal, though. It regards these\nsequences as single tokens:\n\n 1+2\n 1.5-e+2+f\n\nand the following amalgams as one token:\n\n .l as in str.length\n .f as in str.find\n\nTighten the regex in the following way:\n\n- Accept + and - only in one position in the exponent. + and - are no\n longer regarded as the sign of a number and are treated by the\n catcher-all that is not visible in the driver's regex.\n\n- Accept a leading decimal point only when it is followed by a digit.\n\nFor readability, factor hex- and binary numbers into an own term.\n\nAs a drive-by, this fixes that floatingpoint numbers such as 12E5\n(with upper-case E) were split into two tokens.\n\nSigned-off-by: Johannes Sixt \n---\n userdiff.c | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/userdiff.c b/userdiff.c\nindex d9b2ba752f0..ce2a9230703 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -54,8 +54,14 @@ PATTERNS(\"cpp\",\n \t /* functions/methods, variables, and compounds at top level */\n \t \"^((::[[:space:]]*)?[A-Za-z_].*)$\",\n \t /* -- */\n+\t /* identifiers and keywords */\n \t \"[a-zA-Z_][a-zA-Z0-9_]*\"\n-\t \"|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*\"\n+\t /* decimal and octal integers as well as floatingpoint numbers */\n+\t \"|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n+\t /* hexadecimal and binary integers */\n+\t \"|0[xXbB][0-9a-fA-F]+[lLuU]*\"\n+\t /* floatingpoint numbers that begin with a decimal point */\n+\t \"|\\\\.[0-9]+([Ee][-+]?[0-9]+)?[fFlL]?\"\n \t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n-- \ngitgitgadget\n\n"},{"id":"438124","messageId":"9d1c05f5f414b77a4edd741b738f47d4b9e3c56a.1633589461.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.git.1633589461.gitgitgadget@gmail.com","subject":"[PATCH 2/3] userdiff: permit the digit-separating single-quote in numbers","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T06:50:59Z","receivedAt":"2021-10-07T06:51:10Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nSince C++17, the single-quote can be used as digit separator:\n\n 3.141'592'654\n 1'000'000\n 0xdead'beaf\n\nMake it known to the word regex of the cpp driver, so that numbers are not\nsplit into separate tokens at the single-quotes.\n\nSigned-off-by: Johannes Sixt \n---\n userdiff.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/userdiff.c b/userdiff.c\nindex ce2a9230703..1b640c7df79 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -57,11 +57,11 @@ PATTERNS(\"cpp\",\n \t /* identifiers and keywords */\n \t \"[a-zA-Z_][a-zA-Z0-9_]*\"\n \t /* decimal and octal integers as well as floatingpoint numbers */\n-\t \"|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n+\t \"|[0-9][0-9.']*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n \t /* hexadecimal and binary integers */\n-\t \"|0[xXbB][0-9a-fA-F]+[lLuU]*\"\n+\t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n \t /* floatingpoint numbers that begin with a decimal point */\n-\t \"|\\\\.[0-9]+([Ee][-+]?[0-9]+)?[fFlL]?\"\n+\t \"|\\\\.[0-9']+([Ee][-+]?[0-9]+)?[fFlL]?\"\n \t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n-- \ngitgitgadget\n\n"},{"id":"438125","messageId":"df66485e7f019dd81f53f73fd3772bf0b678059d.1633589461.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.git.1633589461.gitgitgadget@gmail.com","subject":"[PATCH 3/3] userdiff: learn the C++ spaceship operator","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T06:51:00Z","receivedAt":"2021-10-07T06:51:12Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nSince C++20, the language has a generalized comparison operator. Teach\nthe cpp driver not to separate it into <= and > tokens.\n\nSigned-off-by: Johannes Sixt \n---\n userdiff.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/userdiff.c b/userdiff.c\nindex 1b640c7df79..13cec0b48db 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -62,7 +62,7 @@ PATTERNS(\"cpp\",\n \t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n \t /* floatingpoint numbers that begin with a decimal point */\n \t \"|\\\\.[0-9']+([Ee][-+]?[0-9]+)?[fFlL]?\"\n-\t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n+\t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*|<=>\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n \t \"!^[ \\t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\\n\"\n-- \ngitgitgadget\n"},{"id":"438127","messageId":"87o881gpq7.fsf@evledraar.gmail.com","threadId":"56657","inReplyTo":"pull.1054.git.1633589461.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/3] Fun with cpp word regex","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-07T09:14:00Z","receivedAt":"2021-10-07T09:15:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 07 2021, Johannes Sixt via GitGitGadget wrote:\n\n> The cpp word regex driver is a bit too loose and can match too much text\n> where the intent is to match only a number. The first patch fixes that.\n>\n> The other two patches add support for digit separators and the spaceship\n> operator <=> (generalized comparison operator).\n>\n> I left out support for hexadecimal floating point constants because that\n> would require to tighten the regex even more to avoid that entire\n> expressions are treated as single tokens.\n>\n> Johannes Sixt (3):\n> userdiff: tighten cpp word regex\n> userdiff: permit the digit-separating single-quote in numbers\n> userdiff: learn the C++ spaceship operator\n>\n> userdiff.c | 10 ++++++++--\n> 1 file changed, 8 insertions(+), 2 deletions(-)\n\nI haven't dug into the C++ syntax here, I trust you know what you're\ndoing there.\n\nBut some tests in t/t4034/cpp/* would be great, and seem from the files\neasy to add. E.g. wouldn't changing:\n\n ab a>=b\n\n\nto:\n\n ab a>=b a<=>b\n\nGive you an updated regression test for your 3/3? Similar changes can be\ndone for 1-2/3.\n"},{"id":"438179","messageId":"ebfe018b-b4e7-0dc8-e535-5cac9cf3d106@kdbg.org","threadId":"56657","inReplyTo":"87o881gpq7.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/3] Fun with cpp word regex","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2021-10-07T16:40:52Z","receivedAt":"2021-10-07T16:40:56Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 07.10.21 um 11:14 schrieb Ævar Arnfjörð Bjarmason:\n> But some tests in t/t4034/cpp/* would be great, and seem from the files\n> easy to add. E.g. wouldn't changing:\n\nAh, I didn't notice that we actually have tests for word-diff. I'll look\ninto it and resubmit.\n\n-- Hannes\n"},{"id":"438321","messageId":"pull.1054.v2.git.1633720197.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.git.1633589461.gitgitgadget@gmail.com","subject":"[PATCH v2 0/5] Fun with cpp word regex","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-08T19:09:52Z","receivedAt":"2021-10-08T19:10:02Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"The cpp word regex driver is a bit too loose and can match too much text\nwhere the intent is to match only a number.\n\nThe first patch makes the cpp word regex tests more effective.\n\nThe second patch adds problematic test cases. The third patch fixes these\nproblems.\n\nThe final two patches add support for digit separators and the spaceship\noperator <=> (generalized comparison operator).\n\nI left out support for hexadecimal floating point constants because that\nwould require to tighten the regex even more to avoid that entire\nexpressions are treated as single tokens.\n\nChanges since V1:\n\n * Tests, tests, tests.\n * Polished commit messages.\n\nJohannes Sixt (5):\n t4034/cpp: actually test that operator tokens are not split\n t4034: add tests showing problematic cpp tokenizations\n userdiff-cpp: tighten word regex\n userdiff-cpp: permit the digit-separating single-quote in numbers\n userdiff-cpp: learn the C++ spaceship operator\n\n t/t4034/cpp/expect | 63 +++++++++++++++++++++++-----------------------\n t/t4034/cpp/post | 47 +++++++++++++++++++++-------------\n t/t4034/cpp/pre | 41 +++++++++++++++++++-----------\n userdiff.c | 10 ++++++--\n 4 files changed, 94 insertions(+), 67 deletions(-)\n\n\nbase-commit: 225bc32a989d7a22fa6addafd4ce7dcd04675dbf\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1054%2Fj6t%2Ffun-with-cpp-word-regex-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1054/j6t/fun-with-cpp-word-regex-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1054\n\nRange-diff vs v1:\n\n -: ----------- > 1: dd9f82ba712 t4034/cpp: actually test that operator tokens are not split\n -: ----------- > 2: 5a84fc9cf71 t4034: add tests showing problematic cpp tokenizations\n 1: a47ab9ba20e ! 3: d4ebe45fddc userdiff: tighten cpp word regex\n @@ Metadata\n Author: Johannes Sixt \n \n ## Commit message ##\n - userdiff: tighten cpp word regex\n + userdiff-cpp: tighten word regex\n \n Generally, word regex can be written such that they match tokens\n liberally and need not model the actual syntax because it can be assumed\n @@ Commit message\n \n .l as in str.length\n .f as in str.find\n + .e as in str.erase\n \n Tighten the regex in the following way:\n \n @@ Commit message\n \n For readability, factor hex- and binary numbers into an own term.\n \n - As a drive-by, this fixes that floatingpoint numbers such as 12E5\n + As a drive-by, this fixes that floating point numbers such as 12E5\n (with upper-case E) were split into two tokens.\n \n Signed-off-by: Johannes Sixt \n \n + ## t/t4034/cpp/expect ##\n +@@\n + --- a/pre\n + +++ b/post\n + @@ -1,30 +1,30 @@\n +-Foo() : x(0&&1&42) { foo0bar(x.f.Find); }\n ++Foo() : x(0&&1&42) { foo0bar(x.findFind); }\n + cout<<\"Hello World!?\\n\"<(1 -1e10+1e10 0xabcdef) 'xy'\n ++(1 -+1e10 0xabcdef) 'xy'\n + // long double\n + 3.141592653e-10l3.141592654e+10l\n + // float\n +-120E5fE6f\n ++120E5f120E6f\n + // hex\n +-0xdeadbeaf+80xdeadBeaf+7ULL\n ++0xdeadbeaf0xdeadBeaf+8ULL7ULL\n + // octal\n + 0123456701234560\n + // binary\n + 0b10000b1100+e1\n + // expression\n +-1.5-e+2+f1.5-e+3+f\n ++1.5-e+23+f\n + // another one\n +-str.e+65.e+75\n +-[a] b->->*v d.e.*e\n ++str.e+6575\n ++[a] b->->*v d..*e\n + ~!a !~b c+++ d--- e**f g&&&h\n + a**=b c//=d e%%=f\n + a+++b c---d\n +@@ t/t4034/cpp/expect: a==!=b c!==d\n + a^^=b c||=d e&&&=f\n + a|||b\n + a?:b\n +-a===b c+=+d e-=fe-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n ++a===b c+=+d e-=-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n + a,b\n + a:::b\n +\n ## userdiff.c ##\n @@ userdiff.c: PATTERNS(\"cpp\",\n \t /* functions/methods, variables, and compounds at top level */\n 2: 9d1c05f5f41 ! 4: dd75d19cee9 userdiff: permit the digit-separating single-quote in numbers\n @@ Metadata\n Author: Johannes Sixt \n \n ## Commit message ##\n - userdiff: permit the digit-separating single-quote in numbers\n + userdiff-cpp: permit the digit-separating single-quote in numbers\n \n Since C++17, the single-quote can be used as digit separator:\n \n @@ Commit message\n 1'000'000\n 0xdead'beaf\n \n - Make it known to the word regex of the cpp driver, so that numbers are not\n - split into separate tokens at the single-quotes.\n + Make it known to the word regex of the cpp driver, so that numbers are\n + not split into separate tokens at the single-quotes.\n \n Signed-off-by: Johannes Sixt \n \n + ## t/t4034/cpp/expect ##\n +@@\n + diff --git a/pre b/post\n +-index 1229cdb..3feae6f 100644\n ++index 60f3640..f6fbf7b 100644\n + --- a/pre\n + +++ b/post\n + @@ -1,30 +1,30 @@\n +@@ t/t4034/cpp/expect: Foo() : x(0&&1&42) { foo0bar\n + cout<<\"Hello World!?\\n\"<(1 -+1e10 0xabcdef) 'xy'\n + // long double\n +-3.141592653e-10l3.141592654e+10l\n ++3.141'592'653e-10l3.141'592'654e+10l\n + // float\n + 120E5f120E6f\n + // hex\n +-0xdeadbeaf0xdeadBeaf+8ULL7ULL\n ++0xdead'beaf0xdead'Beaf+8ULL7ULL\n + // octal\n +-0123456701234560\n ++0123'45670123'4560\n + // binary\n +-0b10000b1100+e1\n ++0b10'000b11'00+e1\n + // expression\n + 1.5-e+23+f\n + // another one\n +\n + ## t/t4034/cpp/post ##\n +@@ t/t4034/cpp/post: Foo() : x(0&42) { bar(x.Find); }\n + cout<<\"Hello World?\\n\"<\n \n ## Commit message ##\n - userdiff: learn the C++ spaceship operator\n + userdiff-cpp: learn the C++ spaceship operator\n \n - Since C++20, the language has a generalized comparison operator. Teach\n - the cpp driver not to separate it into <= and > tokens.\n + Since C++20, the language has a generalized comparison operator <=>.\n + Teach the cpp driver not to separate it into <= and > tokens.\n \n Signed-off-by: Johannes Sixt \n \n + ## t/t4034/cpp/expect ##\n +@@\n + diff --git a/pre b/post\n +-index 60f3640..f6fbf7b 100644\n ++index 144cd98..244f79c 100644\n + --- a/pre\n + +++ b/post\n + @@ -1,30 +1,30 @@\n +@@ t/t4034/cpp/expect: str.e+6575\n + a**=b c//=d e%%=f\n + a+++b c---d\n + a<<<<=b c>>>>=d\n +-a<<=b c<=<d e>>=f g>=>h\n ++a<<=b c<=<d e>>=f g>=>h i<=<=>j\n + a==!=b c!==d\n + a^^=b c||=d e&&&=f\n + a|||b\n +\n + ## t/t4034/cpp/post ##\n +@@ t/t4034/cpp/post: str.e+75\n + a*=b c/=d e%=f\n + a++b c--d\n + a<<=b c>>=d\n +-a<=b c=f g>h\n ++a<=b c=f g>h i<=>j\n + a!=b c=d\n + a^=b c|=d e&=f\n + a|b\n +\n + ## t/t4034/cpp/pre ##\n +@@ t/t4034/cpp/pre: str.e+65\n + a*b c/d e%f\n + a+b c-d\n + a<>d\n +-af g>=h\n ++af g>=h i<=j\n + a==b c!=d\n + a^b c|d e&&f\n + a||b\n +\n ## userdiff.c ##\n @@ userdiff.c: PATTERNS(\"cpp\",\n \t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n\n-- \ngitgitgadget\n"},{"id":"438322","messageId":"dd9f82ba712f54245f19df95926050543d25b769.1633720197.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v2.git.1633720197.gitgitgadget@gmail.com","subject":"[PATCH v2 1/5] t4034/cpp: actually test that operator tokens are not split","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-08T19:09:53Z","receivedAt":"2021-10-08T19:10:05Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\n8d96e7288f2b (t4034: bulk verify builtin word regex sanity, 2010-12-18)\nadded many tests with the intent to verify that operators consisting of\nmore than one symbol are kept together. These are tested by probing a\ntransition from, e.g., a!=b to x!=y, which results in the word-diff\n\n [-a-]{+x+}!=[-b-]{+y+}\n\nBut that proves only that the letters and operators are separate tokens.\nTo prove that != is an unseparable token, we have to probe a transition\nfrom, e.g., a=b to a!=b having a word-diff\n\n a[-=-]{+!=+}b\n\nthat proves that the ! is not separate from the =.\n\nIn the post-image, add to or remove from operators a character that\nturns it into another valid operator.\n\nChange the identifiers used around operators such that the diff\nalgorithm does not have an incentive to match, e.g., a\n---\n t/t4034/cpp/expect | 45 +++++++++++++++------------------------------\n t/t4034/cpp/post | 29 +++++++++++++----------------\n t/t4034/cpp/pre | 25 +++++++++++--------------\n 3 files changed, 39 insertions(+), 60 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 37d1ea25870..41976971b93 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -1,36 +1,21 @@\n diff --git a/pre b/post\n-index 23d5c8a..7e8c026 100644\n+index c5672a2..4229868 100644\n --- a/pre\n +++ b/post\n-@@ -1,19 +1,19 @@\n+@@ -1,16 +1,16 @@\n Foo() : x(0&&1&42) { bar(x); }\n cout<<\"Hello World!?\\n\"<(1) (-1e10) (0xabcdef) 'xy'\n-[ax] ax->b ay x.by\n-!ax ~a ax x++ ax-- ax*b ay x&b\n-ay\n-x*b ay x/b ay x%b\n-ay\n-x+b ay x-b\n-ay\n-x<<b ay x>>b\n-ay\n-x<b ay x<=b ay x>b ay x>=b\n-ay\n-x==b ay x!=b\n-ay\n-x&b\n-ay\n-x^b\n-ay\n-x|b\n-ay\n-x&&b\n-ay\n-x||b\n-ay\n-x?by:z\n-ax=b ay x+=b ay x-=b ay x*=b ay x/=b ay x%=b ay x<<=b ay x>>=b ay x&=b ay x^=b ay x|=b\n-ay\n-x,y\n-ax::by\n+[a] b->->*v d.e.*e\n+~!a !~b c+++ d--- e**f g&&&h\n+a**=b c//=d e%%=f\n+a+++b c---d\n+a<<<<=b c>>>>=d\n+a<<=b c<=<d e>>=f g>=>h\n+a==!=b c!==d\n+a^^=b c||=d e&&&=f\n+a|||b\n+a?:b\n+a===b c+=+d e-=fe-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n+a,b\n+a:::b\ndiff --git a/t/t4034/cpp/post b/t/t4034/cpp/post\nindex 7e8c026cefb..4229868ae62 100644\n--- a/t/t4034/cpp/post\n+++ b/t/t4034/cpp/post\n@@ -1,19 +1,16 @@\n Foo() : x(0&42) { bar(x); }\n cout<<\"Hello World?\\n\"<y x.y\n-!x ~x x++ x-- x*y x&y\n-x*y x/y x%y\n-x+y x-y\n-x<>y\n-xy x>=y\n-x==y x!=y\n-x&y\n-x^y\n-x|y\n-x&&y\n-x||y\n-x?y:z\n-x=y x+=y x-=y x*=y x/=y x%=y x<<=y x>>=y x&=y x^=y x|=y\n-x,y\n-x::y\n+[a] b->*v d.*e\n+~!a !~b c+ d- e**f g&&h\n+a*=b c/=d e%=f\n+a++b c--d\n+a<<=b c>>=d\n+a<=b c=f g>h\n+a!=b c=d\n+a^=b c|=d e&=f\n+a|b\n+a?:b\n+a==b c+d e-f g*h i/j k%l m<>p q&r s^t u|v\n+a,b\n+a:b\ndiff --git a/t/t4034/cpp/pre b/t/t4034/cpp/pre\nindex 23d5c8adf54..c5672a24cfc 100644\n--- a/t/t4034/cpp/pre\n+++ b/t/t4034/cpp/pre\n@@ -1,19 +1,16 @@\n Foo():x(0&&1){}\n cout<<\"Hello World!\\n\"<b a.b\n-!a ~a a++ a-- a*b a&b\n-a*b a/b a%b\n-a+b a-b\n-a<>b\n-ab a>=b\n-a==b a!=b\n-a&b\n-a^b\n-a|b\n-a&&b\n+[a] b->v d.e\n+!a ~b c++ d-- e*f g&h\n+a*b c/d e%f\n+a+b c-d\n+a<>d\n+af g>=h\n+a==b c!=d\n+a^b c|d e&&f\n a||b\n-a?b:z\n-a=b a+=b a-=b a*=b a/=b a%=b a<<=b a>>=b a&=b a^=b a|=b\n-a,y\n+a?b\n+a=b c+=d e-=f g*=h i/=j k%=l m<<=n o>>=p q&=r s^=t u|=v\n+a,b\n a::b\n-- \ngitgitgadget\n\n"},{"id":"438323","messageId":"5a84fc9cf715aec258d9cda2dd7d2e8eff2dc66c.1633720197.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v2.git.1633720197.gitgitgadget@gmail.com","subject":"[PATCH v2 2/5] t4034: add tests showing problematic cpp tokenizations","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-08T19:09:54Z","receivedAt":"2021-10-08T19:10:05Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nThe word regex is too loose and matches long streaks of characters\nthat should actually be separate tokens. Add these problematic test\ncases. Separate the lines with text that will remain identical in the\npre- and post-image so that the diff algorithm will not lump removals\nand additions of consecutive lines together. This makes the expected\noutput easier to read.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 22 ++++++++++++++++++----\n t/t4034/cpp/post | 18 ++++++++++++++++--\n t/t4034/cpp/pre | 16 +++++++++++++++-\n 3 files changed, 49 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 41976971b93..63e53a61e62 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -1,11 +1,25 @@\n diff --git a/pre b/post\n-index c5672a2..4229868 100644\n+index 1229cdb..3feae6f 100644\n --- a/pre\n +++ b/post\n-@@ -1,16 +1,16 @@\n-Foo() : x(0&&1&42) { bar(x); }\n+@@ -1,30 +1,30 @@\n+Foo() : x(0&&1&42) { foo0bar(x.f.Find); }\n cout<<\"Hello World!?\\n\"<(1) (-1e10) (0xabcdef) 'xy'\n+(1 -1e10+1e10 0xabcdef) 'xy'\n+// long double\n+3.141592653e-10l3.141592654e+10l\n+// float\n+120E5fE6f\n+// hex\n+0xdeadbeaf+80xdeadBeaf+7ULL\n+// octal\n+0123456701234560\n+// binary\n+0b10000b1100+e1\n+// expression\n+1.5-e+2+f1.5-e+3+f\n+// another one\n+str.e+65.e+75\n [a] b->->*v d.e.*e\n ~!a !~b c+++ d--- e**f g&&&h\n a**=b c//=d e%%=f\ndiff --git a/t/t4034/cpp/post b/t/t4034/cpp/post\nindex 4229868ae62..3feae6f430f 100644\n--- a/t/t4034/cpp/post\n+++ b/t/t4034/cpp/post\n@@ -1,6 +1,20 @@\n-Foo() : x(0&42) { bar(x); }\n+Foo() : x(0&42) { bar(x.Find); }\n cout<<\"Hello World?\\n\"<*v d.*e\n ~!a !~b c+ d- e**f g&&h\n a*=b c/=d e%=f\ndiff --git a/t/t4034/cpp/pre b/t/t4034/cpp/pre\nindex c5672a24cfc..1229cdb59d1 100644\n--- a/t/t4034/cpp/pre\n+++ b/t/t4034/cpp/pre\n@@ -1,6 +1,20 @@\n-Foo():x(0&&1){}\n+Foo():x(0&&1){ foo0( x.find); }\n cout<<\"Hello World!\\n\"<v d.e\n !a ~b c++ d-- e*f g&h\n a*b c/d e%f\n-- \ngitgitgadget\n\n"},{"id":"438324","messageId":"d4ebe45fddc4e357b7f4c5299b26c0cd905fd3f2.1633720197.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v2.git.1633720197.gitgitgadget@gmail.com","subject":"[PATCH v2 3/5] userdiff-cpp: tighten word regex","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-08T19:09:55Z","receivedAt":"2021-10-08T19:10:08Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nGenerally, word regex can be written such that they match tokens\nliberally and need not model the actual syntax because it can be assumed\nthat the regex will only be applied to syntactically correct text.\n\nThe regex for cpp (C/C++) is too liberal, though. It regards these\nsequences as single tokens:\n\n 1+2\n 1.5-e+2+f\n\nand the following amalgams as one token:\n\n .l as in str.length\n .f as in str.find\n .e as in str.erase\n\nTighten the regex in the following way:\n\n- Accept + and - only in one position in the exponent. + and - are no\n longer regarded as the sign of a number and are treated by the\n catcher-all that is not visible in the driver's regex.\n\n- Accept a leading decimal point only when it is followed by a digit.\n\nFor readability, factor hex- and binary numbers into an own term.\n\nAs a drive-by, this fixes that floating point numbers such as 12E5\n(with upper-case E) were split into two tokens.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 16 ++++++++--------\n userdiff.c | 8 +++++++-\n 2 files changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 63e53a61e62..46c9460a968 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -3,24 +3,24 @@\n --- a/pre\n +++ b/post\n @@ -1,30 +1,30 @@\n-Foo() : x(0&&1&42) { foo0bar(x.f.Find); }\n+Foo() : x(0&&1&42) { foo0bar(x.findFind); }\n cout<<\"Hello World!?\\n\"<(1 -1e10+1e10 0xabcdef) 'xy'\n+(1 -+1e10 0xabcdef) 'xy'\n // long double\n 3.141592653e-10l3.141592654e+10l\n // float\n-120E5fE6f\n+120E5f120E6f\n // hex\n-0xdeadbeaf+80xdeadBeaf+7ULL\n+0xdeadbeaf0xdeadBeaf+8ULL7ULL\n // octal\n 0123456701234560\n // binary\n 0b10000b1100+e1\n // expression\n-1.5-e+2+f1.5-e+3+f\n+1.5-e+23+f\n // another one\n-str.e+65.e+75\n-[a] b->->*v d.e.*e\n+str.e+6575\n+[a] b->->*v d..*e\n ~!a !~b c+++ d--- e**f g&&&h\n a**=b c//=d e%%=f\n a+++b c---d\n@@ -30,6 +30,6 @@ a==!=b c!==d\n a^^=b c||=d e&&&=f\n a|||b\n a?:b\n-a===b c+=+d e-=fe-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n+a===b c+=+d e-=-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n a,b\n a:::b\ndiff --git a/userdiff.c b/userdiff.c\nindex d9b2ba752f0..ce2a9230703 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -54,8 +54,14 @@ PATTERNS(\"cpp\",\n \t /* functions/methods, variables, and compounds at top level */\n \t \"^((::[[:space:]]*)?[A-Za-z_].*)$\",\n \t /* -- */\n+\t /* identifiers and keywords */\n \t \"[a-zA-Z_][a-zA-Z0-9_]*\"\n-\t \"|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*\"\n+\t /* decimal and octal integers as well as floatingpoint numbers */\n+\t \"|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n+\t /* hexadecimal and binary integers */\n+\t \"|0[xXbB][0-9a-fA-F]+[lLuU]*\"\n+\t /* floatingpoint numbers that begin with a decimal point */\n+\t \"|\\\\.[0-9]+([Ee][-+]?[0-9]+)?[fFlL]?\"\n \t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n-- \ngitgitgadget\n\n"},{"id":"438325","messageId":"dd75d19cee9a911de93d19585c888c57fe8c3aee.1633720197.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v2.git.1633720197.gitgitgadget@gmail.com","subject":"[PATCH v2 4/5] userdiff-cpp: permit the digit-separating single-quote in numbers","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-08T19:09:56Z","receivedAt":"2021-10-08T19:10:11Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nSince C++17, the single-quote can be used as digit separator:\n\n 3.141'592'654\n 1'000'000\n 0xdead'beaf\n\nMake it known to the word regex of the cpp driver, so that numbers are\nnot split into separate tokens at the single-quotes.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 10 +++++-----\n t/t4034/cpp/post | 8 ++++----\n t/t4034/cpp/pre | 8 ++++----\n userdiff.c | 6 +++---\n 4 files changed, 16 insertions(+), 16 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 46c9460a968..a3a234f5461 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -1,5 +1,5 @@\n diff --git a/pre b/post\n-index 1229cdb..3feae6f 100644\n+index 60f3640..f6fbf7b 100644\n --- a/pre\n +++ b/post\n @@ -1,30 +1,30 @@\n@@ -7,15 +7,15 @@ Foo() : x(0&&1&42) { foo0bar\n cout<<\"Hello World!?\\n\"<(1 -+1e10 0xabcdef) 'xy'\n // long double\n-3.141592653e-10l3.141592654e+10l\n+3.141'592'653e-10l3.141'592'654e+10l\n // float\n 120E5f120E6f\n // hex\n-0xdeadbeaf0xdeadBeaf+8ULL7ULL\n+0xdead'beaf0xdead'Beaf+8ULL7ULL\n // octal\n-0123456701234560\n+0123'45670123'4560\n // binary\n-0b10000b1100+e1\n+0b10'000b11'00+e1\n // expression\n 1.5-e+23+f\n // another one\ndiff --git a/t/t4034/cpp/post b/t/t4034/cpp/post\nindex 3feae6f430f..f6fbf7bc04c 100644\n--- a/t/t4034/cpp/post\n+++ b/t/t4034/cpp/post\n@@ -2,15 +2,15 @@ Foo() : x(0&42) { bar(x.Find); }\n cout<<\"Hello World?\\n\"<%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n-- \ngitgitgadget\n\n"},{"id":"438326","messageId":"43a701f5ffd899ae56b2db0fb865e37dd2bb4e07.1633720197.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v2.git.1633720197.gitgitgadget@gmail.com","subject":"[PATCH v2 5/5] userdiff-cpp: learn the C++ spaceship operator","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-08T19:09:57Z","receivedAt":"2021-10-08T19:10:12Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nSince C++20, the language has a generalized comparison operator <=>.\nTeach the cpp driver not to separate it into <= and > tokens.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 4 ++--\n t/t4034/cpp/post | 2 +-\n t/t4034/cpp/pre | 2 +-\n userdiff.c | 2 +-\n 4 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex a3a234f5461..bf3cd2abc74 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -1,5 +1,5 @@\n diff --git a/pre b/post\n-index 60f3640..f6fbf7b 100644\n+index 144cd98..244f79c 100644\n --- a/pre\n +++ b/post\n @@ -1,30 +1,30 @@\n@@ -25,7 +25,7 @@ str.e+6575\n a**=b c//=d e%%=f\n a+++b c---d\n a<<<<=b c>>>>=d\n-a<<=b c<=<d e>>=f g>=>h\n+a<<=b c<=<d e>>=f g>=>h i<=<=>j\n a==!=b c!==d\n a^^=b c||=d e&&&=f\n a|||b\ndiff --git a/t/t4034/cpp/post b/t/t4034/cpp/post\nindex f6fbf7bc04c..244f79c9900 100644\n--- a/t/t4034/cpp/post\n+++ b/t/t4034/cpp/post\n@@ -20,7 +20,7 @@ str.e+75\n a*=b c/=d e%=f\n a++b c--d\n a<<=b c>>=d\n-a<=b c=f g>h\n+a<=b c=f g>h i<=>j\n a!=b c=d\n a^=b c|=d e&=f\n a|b\ndiff --git a/t/t4034/cpp/pre b/t/t4034/cpp/pre\nindex 60f3640d773..144cd980d6b 100644\n--- a/t/t4034/cpp/pre\n+++ b/t/t4034/cpp/pre\n@@ -20,7 +20,7 @@ str.e+65\n a*b c/d e%f\n a+b c-d\n a<>d\n-af g>=h\n+af g>=h i<=j\n a==b c!=d\n a^b c|d e&&f\n a||b\ndiff --git a/userdiff.c b/userdiff.c\nindex 1b640c7df79..13cec0b48db 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -62,7 +62,7 @@ PATTERNS(\"cpp\",\n \t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n \t /* floatingpoint numbers that begin with a decimal point */\n \t \"|\\\\.[0-9']+([Ee][-+]?[0-9]+)?[fFlL]?\"\n-\t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n+\t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*|<=>\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n \t \"!^[ \\t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\\n\"\n-- \ngitgitgadget\n"},{"id":"438341","messageId":"87r1cvmg0c.fsf@evledraar.gmail.com","threadId":"56657","inReplyTo":"pull.1054.v2.git.1633720197.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/5] Fun with cpp word regex","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-08T20:07:18Z","receivedAt":"2021-10-08T20:13:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 08 2021, Johannes Sixt via GitGitGadget wrote:\n\n> The cpp word regex driver is a bit too loose and can match too much text\n> where the intent is to match only a number.\n>\n> The first patch makes the cpp word regex tests more effective.\n>\n> The second patch adds problematic test cases. The third patch fixes these\n> problems.\n>\n> The final two patches add support for digit separators and the spaceship\n> operator <=> (generalized comparison operator).\n>\n> I left out support for hexadecimal floating point constants because that\n> would require to tighten the regex even more to avoid that entire\n> expressions are treated as single tokens.\n>\n> Changes since V1:\n>\n> * Tests, tests, tests.\n> * Polished commit messages.\n\nI've read this much improved v2 over, thanks a lot. This looks much\nbetter.\n\nJust some general comments, in my mind this is more than good enough\nalready and doesn't require a re-roll, just food for thought:\n\n * I wonder if it isn't time to split up \"cpp\" into a \"c\" driver,\n e.g. git.git's .gitattributes has \"cpp\" for *.[ch] files, but as C++\n adds more syntax sugar.\n\n So e.g. if you use \"<=>\" after this series we'll tokenize it\n differently in *.c files, but it's a C++-only operator, on the other\n hand probably nobody cares that much...\n\n * I found myself back-porting some of your tests (manually mostly),\n maybe you disagree, but in cases like 123'123, <=> etc. I'd find it\n easier to follow if we first added the test data, and then the\n changed behavior.\n\n Because after all, we're going to change how we highlight existing\n data, so testing for that would be informative.\n\n * This pre-dates your much improved tests, but these test files could\n really use some test comments, as in:\n\n /* Now that we're going to understand the \"'\" character somehow, will any of this change? */\n /* We haven't written code like this since the 1960's ... */\n /* Run & free */\n\n I.e. we don't just highlight code the compiler likes to eat, but also\n comments. So particularly for smaller tokens that also occur in\n natural language like \"'\" and \"&\" are we getting expected results?\n\n It looked like it from skimming your changes, i.e. the ' rule is\n checked by surrounding digits, just might be handy to test it ... :)\n\n> Johannes Sixt (5):\n> t4034/cpp: actually test that operator tokens are not split\n> t4034: add tests showing problematic cpp tokenizations\n> userdiff-cpp: tighten word regex\n> userdiff-cpp: permit the digit-separating single-quote in numbers\n> userdiff-cpp: learn the C++ spaceship operator\n>\n> t/t4034/cpp/expect | 63 +++++++++++++++++++++++-----------------------\n> t/t4034/cpp/post | 47 +++++++++++++++++++++-------------\n> t/t4034/cpp/pre | 41 +++++++++++++++++++-----------\n> userdiff.c | 10 ++++++--\n> 4 files changed, 94 insertions(+), 67 deletions(-)\n>\n>\n> base-commit: 225bc32a989d7a22fa6addafd4ce7dcd04675dbf\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1054%2Fj6t%2Ffun-with-cpp-word-regex-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1054/j6t/fun-with-cpp-word-regex-v2\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1054\n>\n> Range-diff vs v1:\n>\n> -: ----------- > 1: dd9f82ba712 t4034/cpp: actually test that operator tokens are not split\n> -: ----------- > 2: 5a84fc9cf71 t4034: add tests showing problematic cpp tokenizations\n> 1: a47ab9ba20e ! 3: d4ebe45fddc userdiff: tighten cpp word regex\n> @@ Metadata\n> Author: Johannes Sixt \n> \n> ## Commit message ##\n> - userdiff: tighten cpp word regex\n> + userdiff-cpp: tighten word regex\n> \n> Generally, word regex can be written such that they match tokens\n> liberally and need not model the actual syntax because it can be assumed\n> @@ Commit message\n> \n> .l as in str.length\n> .f as in str.find\n> + .e as in str.erase\n> \n> Tighten the regex in the following way:\n> \n> @@ Commit message\n> \n> For readability, factor hex- and binary numbers into an own term.\n> \n> - As a drive-by, this fixes that floatingpoint numbers such as 12E5\n> + As a drive-by, this fixes that floating point numbers such as 12E5\n> (with upper-case E) were split into two tokens.\n> \n> Signed-off-by: Johannes Sixt \n> \n> + ## t/t4034/cpp/expect ##\n> +@@\n> + --- a/pre\n> + +++ b/post\n> + @@ -1,30 +1,30 @@\n> +-Foo() : x(0&&1&42) { foo0bar(x.f.Find); }\n> ++Foo() : x(0&&1&42) { foo0bar(x.findFind); }\n> + cout<<\"Hello World!?\\n\"< +-(1 -1e10+1e10 0xabcdef) 'xy'\n> ++(1 -+1e10 0xabcdef) 'xy'\n> + // long double\n> + 3.141592653e-10l3.141592654e+10l\n> + // float\n> +-120E5fE6f\n> ++120E5f120E6f\n> + // hex\n> +-0xdeadbeaf+80xdeadBeaf+7ULL\n> ++0xdeadbeaf0xdeadBeaf+8ULL7ULL\n> + // octal\n> + 0123456701234560\n> + // binary\n> + 0b10000b1100+e1\n> + // expression\n> +-1.5-e+2+f1.5-e+3+f\n> ++1.5-e+23+f\n> + // another one\n> +-str.e+65.e+75\n> +-[a] b->->*v d.e.*e\n> ++str.e+6575\n> ++[a] b->->*v d..*e\n> + ~!a !~b c+++ d--- e**f g&&&h\n> + a**=b c//=d e%%=f\n> + a+++b c---d\n> +@@ t/t4034/cpp/expect: a==!=b c!==d\n> + a^^=b c||=d e&&&=f\n> + a|||b\n> + a?:b\n> +-a===b c+=+d e-=fe-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n> ++a===b c+=+d e-=-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n> + a,b\n> + a:::b\n> +\n> ## userdiff.c ##\n> @@ userdiff.c: PATTERNS(\"cpp\",\n> \t /* functions/methods, variables, and compounds at top level */\n> 2: 9d1c05f5f41 ! 4: dd75d19cee9 userdiff: permit the digit-separating single-quote in numbers\n> @@ Metadata\n> Author: Johannes Sixt \n> \n> ## Commit message ##\n> - userdiff: permit the digit-separating single-quote in numbers\n> + userdiff-cpp: permit the digit-separating single-quote in numbers\n> \n> Since C++17, the single-quote can be used as digit separator:\n> \n> @@ Commit message\n> 1'000'000\n> 0xdead'beaf\n> \n> - Make it known to the word regex of the cpp driver, so that numbers are not\n> - split into separate tokens at the single-quotes.\n> + Make it known to the word regex of the cpp driver, so that numbers are\n> + not split into separate tokens at the single-quotes.\n> \n> Signed-off-by: Johannes Sixt \n> \n> + ## t/t4034/cpp/expect ##\n> +@@\n> + diff --git a/pre b/post\n> +-index 1229cdb..3feae6f 100644\n> ++index 60f3640..f6fbf7b 100644\n> + --- a/pre\n> + +++ b/post\n> + @@ -1,30 +1,30 @@\n> +@@ t/t4034/cpp/expect: Foo() : x(0&&1&42) { foo0bar\n> + cout<<\"Hello World!?\\n\"< + (1 -+1e10 0xabcdef) 'xy'\n> + // long double\n> +-3.141592653e-10l3.141592654e+10l\n> ++3.141'592'653e-10l3.141'592'654e+10l\n> + // float\n> + 120E5f120E6f\n> + // hex\n> +-0xdeadbeaf0xdeadBeaf+8ULL7ULL\n> ++0xdead'beaf0xdead'Beaf+8ULL7ULL\n> + // octal\n> +-0123456701234560\n> ++0123'45670123'4560\n> + // binary\n> +-0b10000b1100+e1\n> ++0b10'000b11'00+e1\n> + // expression\n> + 1.5-e+23+f\n> + // another one\n> +\n> + ## t/t4034/cpp/post ##\n> +@@ t/t4034/cpp/post: Foo() : x(0&42) { bar(x.Find); }\n> + cout<<\"Hello World?\\n\"< + (1 +1e10 0xabcdef) 'y'\n> + // long double\n> +-3.141592654e+10l\n> ++3.141'592'654e+10l\n> + // float\n> + 120E6f\n> + // hex\n> +-0xdeadBeaf+7ULL\n> ++0xdead'Beaf+7ULL\n> + // octal\n> +-01234560\n> ++0123'4560\n> + // binary\n> +-0b1100+e1\n> ++0b11'00+e1\n> + // expression\n> + 1.5-e+3+f\n> + // another one\n> +\n> + ## t/t4034/cpp/pre ##\n> +@@ t/t4034/cpp/pre: Foo():x(0&&1){ foo0( x.find); }\n> + cout<<\"Hello World!\\n\"< + 1 -1e10 0xabcdef 'x'\n> + // long double\n> +-3.141592653e-10l\n> ++3.141'592'653e-10l\n> + // float\n> + 120E5f\n> + // hex\n> +-0xdeadbeaf+8ULL\n> ++0xdead'beaf+8ULL\n> + // octal\n> +-01234567\n> ++0123'4567\n> + // binary\n> +-0b1000+e1\n> ++0b10'00+e1\n> + // expression\n> + 1.5-e+2+f\n> + // another one\n> +\n> ## userdiff.c ##\n> @@ userdiff.c: PATTERNS(\"cpp\",\n> \t /* identifiers and keywords */\n> 3: df66485e7f0 ! 5: 43a701f5ffd userdiff: learn the C++ spaceship operator\n> @@ Metadata\n> Author: Johannes Sixt \n> \n> ## Commit message ##\n> - userdiff: learn the C++ spaceship operator\n> + userdiff-cpp: learn the C++ spaceship operator\n> \n> - Since C++20, the language has a generalized comparison operator. Teach\n> - the cpp driver not to separate it into <= and > tokens.\n> + Since C++20, the language has a generalized comparison operator <=>.\n> + Teach the cpp driver not to separate it into <= and > tokens.\n> \n> Signed-off-by: Johannes Sixt \n> \n> + ## t/t4034/cpp/expect ##\n> +@@\n> + diff --git a/pre b/post\n> +-index 60f3640..f6fbf7b 100644\n> ++index 144cd98..244f79c 100644\n> + --- a/pre\n> + +++ b/post\n> + @@ -1,30 +1,30 @@\n> +@@ t/t4034/cpp/expect: str.e+6575\n> + a**=b c//=d e%%=f\n> + a+++b c---d\n> + a<<<<=b c>>>>=d\n> +-a<<=b c<=<d e>>=f g>=>h\n> ++a<<=b c<=<d e>>=f g>=>h i<=<=>j\n> + a==!=b c!==d\n> + a^^=b c||=d e&&&=f\n> + a|||b\n> +\n> + ## t/t4034/cpp/post ##\n> +@@ t/t4034/cpp/post: str.e+75\n> + a*=b c/=d e%=f\n> + a++b c--d\n> + a<<=b c>>=d\n> +-a<=b c=f g>h\n> ++a<=b c=f g>h i<=>j\n> + a!=b c=d\n> + a^=b c|=d e&=f\n> + a|b\n> +\n> + ## t/t4034/cpp/pre ##\n> +@@ t/t4034/cpp/pre: str.e+65\n> + a*b c/d e%f\n> + a+b c-d\n> + a<>d\n> +-af g>=h\n> ++af g>=h i<=j\n> + a==b c!=d\n> + a^b c|d e&&f\n> + a||b\n> +\n> ## userdiff.c ##\n> @@ userdiff.c: PATTERNS(\"cpp\",\n> \t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n\n"},{"id":"438372","messageId":"25363715-dc39-1f18-a937-f715b106f529@kdbg.org","threadId":"56657","inReplyTo":"87r1cvmg0c.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/5] Fun with cpp word regex","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2021-10-08T22:11:43Z","receivedAt":"2021-10-08T22:11:47Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 08.10.21 um 22:07 schrieb Ævar Arnfjörð Bjarmason:\n> * I wonder if it isn't time to split up \"cpp\" into a \"c\" driver,\n> e.g. git.git's .gitattributes has \"cpp\" for *.[ch] files, but as C++\n> adds more syntax sugar.\n> \n> So e.g. if you use \"<=>\" after this series we'll tokenize it\n> differently in *.c files, but it's a C++-only operator, on the other\n> hand probably nobody cares that much...\n\nYes, it is that: <=> won't appear in a correct C file (outside of\ncomments), so no-one will care. As far as tokenization is concerned, C\nis a subset of C++. I don't think we need to separate the drivers.\n\n> * I found myself back-porting some of your tests (manually mostly),\n> maybe you disagree, but in cases like 123'123, <=> etc. I'd find it\n> easier to follow if we first added the test data, and then the\n> changed behavior.\n> \n> Because after all, we're going to change how we highlight existing\n> data, so testing for that would be informative.\n\nGood point. I'll work a bit more on that.\n\n> * This pre-dates your much improved tests, but these test files could\n> really use some test comments, as in:\n> \n> /* Now that we're going to understand the \"'\" character somehow, will any of this change? */\n> /* We haven't written code like this since the 1960's ... */\n> /* Run & free */\n> \n> I.e. we don't just highlight code the compiler likes to eat, but also\n> comments. So particularly for smaller tokens that also occur in\n> natural language like \"'\" and \"&\" are we getting expected results?\n\nComments are free text. Anything can happen. There is no such thing as\n\"correct tokenization\" in comments. Not interested.\n\nThank you for the review.\n-- Hannes\n"},{"id":"438384","messageId":"87mtnjm416.fsf@evledraar.gmail.com","threadId":"56657","inReplyTo":"25363715-dc39-1f18-a937-f715b106f529@kdbg.org","subject":"Re: [PATCH v2 0/5] Fun with cpp word regex","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-09T00:00:12Z","receivedAt":"2021-10-09T00:32:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Oct 09 2021, Johannes Sixt wrote:\n\n> Am 08.10.21 um 22:07 schrieb Ævar Arnfjörð Bjarmason:\n\n[re-arranged a bit]\n\n> [...]As far as tokenization is concerned, C\n> is a subset of C++. I don't think we need to separate the drivers.\n>\n>> * I found myself back-porting some of your tests (manually mostly),\n>> maybe you disagree, but in cases like 123'123, <=> etc. I'd find it\n>> easier to follow if we first added the test data, and then the\n>> changed behavior.\n>> \n>> Because after all, we're going to change how we highlight existing\n>> data, so testing for that would be informative.\n>\n> Good point. I'll work a bit more on that.\n\nGreat!\n\n>> * I wonder if it isn't time to split up \"cpp\" into a \"c\" driver,\n>> e.g. git.git's .gitattributes has \"cpp\" for *.[ch] files, but as C++\n>> adds more syntax sugar.\n>> \n>> So e.g. if you use \"<=>\" after this series we'll tokenize it\n>> differently in *.c files, but it's a C++-only operator, on the other\n>> hand probably nobody cares that much...\n>\n> Yes, it is that: <=> won't appear in a correct C file (outside of\n> comments), so no-one will care. [...]\n\n..mmm..\n\n>> * This pre-dates your much improved tests, but these test files could\n>> really use some test comments, as in:\n>> \n>> /* Now that we're going to understand the \"'\" character somehow, will any of this change? */\n>> /* We haven't written code like this since the 1960's ... */\n>> /* Run & free */\n>> \n>> I.e. we don't just highlight code the compiler likes to eat, but also\n>> comments. So particularly for smaller tokens that also occur in\n>> natural language like \"'\" and \"&\" are we getting expected results?\n>\n> Comments are free text. Anything can happen. There is no such thing as\n> \"correct tokenization\" in comments. Not interested.\n\nSure there is, just because the problem is fuzzy doesn't mean there\naren't more and less correct things to do.\n\nBut most importantly the output of \"git diff\" is made for human\nconsumption, people who use --word-diff are going to be looking at code\nthat contains comments, embedded natural language in C-strings etc.\n\nI've got no reason to think that your changes here make it worse, but\njust as a general matter we absolutely should consider that one of the\ntop priorities when it comes to these language drivers.\n\nE.g. in some languages (like CJK) it's common for characters or words\nnot to have any unicode whitespace between them, so even a word-diff\nmode for C can benefit from recognizing those character ranges and\nsplitting them appropriately, or at least trying.\n\nSo to take a comment of yours where you changed a comment at random:\n \n $ git log -U0 --oneline -1 --word-diff -p af920e36977 --word-diff-regex='[ a-zA-Z]*'\n [...]\n [- Note that this seemingly redundant second declaration is required-]{+ Note that this redundant forward declaration is required+}\n\nDon't you think that would suck v.s. the now-behavior of:\n\n [...]\n Note that this[-seemingly-] redundant [-second-]{+forward+} declaration is required\n\nThe former is exactly the sort of thing you'd get in CJK languages with\na word-diff driver thought the line we should stop at was the same as a\ncomiler tokenizer.\n\nAnyway, the cpp driver seems to do just fine on CJK.\n\nI'm just saying that as a general thing it's definitely a priority for\nthese drivers to not only handle the narrow cases of tokens a compiler\nwould know about. Text that people commonly use should be presented in\nsome way that isn't line noise.\n\nFor an example of something we do a bit badly with the cpp driver is\nparts of my 66f5f6dca95 (C style: use standard style for \"TRANSLATORS\"\ncomments, 2017-05-11).\n\nI.e. there I was changing a comment format, and added a full stop to a\nsentence, the word-diff is:\n\n /*\n {+*+} TRANSLATORS: here is a comment that explains the string to\n {+*+} be translated, that follows immediately after [-it-]{+it.+}\n */\n\nEven though it has nothing to do with C syntax per-se that would be much more useful as:\n\n /*\n {+*+} TRANSLATORS: here is a comment that explains the string to\n {+*+} be translated, that follows immediately after it{+.+}\n */\n\nI.e. treating a \".\" at the end of a word specially isn't C or C++\nsyntax, but it's absolutely input that the cpp driver *is* getting and\nshould be if possible be handling well.\n\nI just did that by experimenting with\n--word-diff-regex='([A-Za-z:]*|\\*|\\.)', that example is unchanged with\nyour series, but maybe low-hanging fruit....\n\nThanks for working on this, just an unsolicited braindump :)\n"},{"id":"438419","messageId":"pull.1054.v3.git.1633885384.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v2.git.1633720197.gitgitgadget@gmail.com","subject":"[PATCH v3 0/6] Fun with cpp word regex","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-10T17:02:58Z","receivedAt":"2021-10-10T17:03:12Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"The cpp word regex driver is a bit too loose and can match too much text\nwhere the intent is to match only a number.\n\nThe first patch makes the cpp word regex tests more effective.\n\nThe second patch adds problematic test cases. The third patch fixes these\nproblems.\n\nThe remaining three patches add support for digit separators and the\nspaceship operator <=> (generalized comparison operator).\n\nI left out support for hexadecimal floating point constants because that\nwould require to tighten the regex even more to avoid that entire\nexpressions are treated as single tokens.\n\nChanges since V2:\n\n * Add test cases for the new features in a separate commit so that the new\n behavior is better visible.\n * Don't treat .' as in '.' as a token.\n\nChanges since V1:\n\n * Tests, tests, tests.\n * Polished commit messages.\n\nJohannes Sixt (6):\n t4034/cpp: actually test that operator tokens are not split\n t4034: add tests showing problematic cpp tokenizations\n userdiff-cpp: tighten word regex\n userdiff-cpp: prepare test cases with yet unsupported features\n userdiff-cpp: permit the digit-separating single-quote in numbers\n userdiff-cpp: learn the C++ spaceship operator\n\n t/t4034/cpp/expect | 63 +++++++++++++++++++++++-----------------------\n t/t4034/cpp/post | 47 +++++++++++++++++++++-------------\n t/t4034/cpp/pre | 41 +++++++++++++++++++-----------\n userdiff.c | 10 ++++++--\n 4 files changed, 94 insertions(+), 67 deletions(-)\n\n\nbase-commit: 225bc32a989d7a22fa6addafd4ce7dcd04675dbf\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1054%2Fj6t%2Ffun-with-cpp-word-regex-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1054/j6t/fun-with-cpp-word-regex-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1054\n\nRange-diff vs v2:\n\n 1: dd9f82ba712 = 1: dd9f82ba712 t4034/cpp: actually test that operator tokens are not split\n 2: 5a84fc9cf71 = 2: 5a84fc9cf71 t4034: add tests showing problematic cpp tokenizations\n 3: d4ebe45fddc = 3: d4ebe45fddc userdiff-cpp: tighten word regex\n 4: dd75d19cee9 ! 4: c9f58b5e82f userdiff-cpp: permit the digit-separating single-quote in numbers\n @@ Metadata\n Author: Johannes Sixt \n \n ## Commit message ##\n - userdiff-cpp: permit the digit-separating single-quote in numbers\n + userdiff-cpp: prepare test cases with yet unsupported features\n \n - Since C++17, the single-quote can be used as digit separator:\n -\n - 3.141'592'654\n - 1'000'000\n - 0xdead'beaf\n -\n - Make it known to the word regex of the cpp driver, so that numbers are\n - not split into separate tokens at the single-quotes.\n + We are going to add support for C++'s digit-separating single-quote and\n + the spaceship operator. By adding the test cases in this separate\n + commit, the effect on the word highlighting will become more obvious\n + as the features are implemented and the file cpp/expect is updated.\n \n Signed-off-by: Johannes Sixt \n \n @@ t/t4034/cpp/expect\n @@\n diff --git a/pre b/post\n -index 1229cdb..3feae6f 100644\n -+index 60f3640..f6fbf7b 100644\n ++index 144cd98..64e78af 100644\n --- a/pre\n +++ b/post\n @@ -1,30 +1,30 @@\n -@@ t/t4034/cpp/expect: Foo() : x(0&&1&42) { foo0bar\n + Foo() : x(0&&1&42) { foo0bar(x.findFind); }\n cout<<\"Hello World!?\\n\"<(1 -+1e10 0xabcdef) 'xy'\n +-(1 -+1e10 0xabcdef) 'xy'\n ++(1 -+1e10 0xabcdef) 'x.'\n // long double\n -3.141592653e-10l3.141592654e+10l\n -+3.141'592'653e-10l3.141'592'654e+10l\n ++3.141'592'653e-10l654e+10l\n // float\n 120E5f120E6f\n // hex\n -0xdeadbeaf0xdeadBeaf+8ULL7ULL\n -+0xdead'beaf0xdead'Beaf+8ULL7ULL\n ++0xdead'beafBeaf+8ULL7ULL\n // octal\n -0123456701234560\n -+0123'45670123'4560\n ++0123'45674560\n // binary\n -0b10000b1100+e1\n -+0b10'000b11'00+e1\n ++0b100b11'00+e1\n // expression\n 1.5-e+23+f\n // another one\n +@@ t/t4034/cpp/expect: str.e+6575\n + a**=b c//=d e%%=f\n + a+++b c---d\n + a<<<<=b c>>>>=d\n +-a<<=b c<=<d e>>=f g>=>h\n ++a<<=b c<=<d e>>=f g>=>h i<=>j\n + a==!=b c!==d\n + a^^=b c||=d e&&&=f\n + a|||b\n \n ## t/t4034/cpp/post ##\n -@@ t/t4034/cpp/post: Foo() : x(0&42) { bar(x.Find); }\n +@@\n + Foo() : x(0&42) { bar(x.Find); }\n cout<<\"Hello World?\\n\"<>=d\n +-a<=b c=f g>h\n ++a<=b c=f g>h i<=>j\n + a!=b c=d\n + a^=b c|=d e&=f\n + a|b\n \n ## t/t4034/cpp/pre ##\n @@ t/t4034/cpp/pre: Foo():x(0&&1){ foo0( x.find); }\n @@ t/t4034/cpp/pre: Foo():x(0&&1){ foo0( x.find); }\n // expression\n 1.5-e+2+f\n // another one\n -\n - ## userdiff.c ##\n -@@ userdiff.c: PATTERNS(\"cpp\",\n - \t /* identifiers and keywords */\n - \t \"[a-zA-Z_][a-zA-Z0-9_]*\"\n - \t /* decimal and octal integers as well as floatingpoint numbers */\n --\t \"|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n -+\t \"|[0-9][0-9.']*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n - \t /* hexadecimal and binary integers */\n --\t \"|0[xXbB][0-9a-fA-F]+[lLuU]*\"\n -+\t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n - \t /* floatingpoint numbers that begin with a decimal point */\n --\t \"|\\\\.[0-9]+([Ee][-+]?[0-9]+)?[fFlL]?\"\n -+\t \"|\\\\.[0-9']+([Ee][-+]?[0-9]+)?[fFlL]?\"\n - \t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n - PATTERNS(\"csharp\",\n - \t /* Keywords */\n +@@ t/t4034/cpp/pre: str.e+65\n + a*b c/d e%f\n + a+b c-d\n + a<>d\n +-af g>=h\n ++af g>=h i<=j\n + a==b c!=d\n + a^b c|d e&&f\n + a||b\n -: ----------- > 5: 037c743d9e3 userdiff-cpp: permit the digit-separating single-quote in numbers\n 5: 43a701f5ffd ! 6: cc9dc967f10 userdiff-cpp: learn the C++ spaceship operator\n @@ Commit message\n Signed-off-by: Johannes Sixt \n \n ## t/t4034/cpp/expect ##\n -@@\n - diff --git a/pre b/post\n --index 60f3640..f6fbf7b 100644\n -+index 144cd98..244f79c 100644\n - --- a/pre\n - +++ b/post\n - @@ -1,30 +1,30 @@\n @@ t/t4034/cpp/expect: str.e+6575\n a**=b c//=d e%%=f\n a+++b c---d\n a<<<<=b c>>>>=d\n --a<<=b c<=<d e>>=f g>=>h\n +-a<<=b c<=<d e>>=f g>=>h i<=>j\n +a<<=b c<=<d e>>=f g>=>h i<=<=>j\n a==!=b c!==d\n a^^=b c||=d e&&&=f\n a|||b\n \n - ## t/t4034/cpp/post ##\n -@@ t/t4034/cpp/post: str.e+75\n - a*=b c/=d e%=f\n - a++b c--d\n - a<<=b c>>=d\n --a<=b c=f g>h\n -+a<=b c=f g>h i<=>j\n - a!=b c=d\n - a^=b c|=d e&=f\n - a|b\n -\n - ## t/t4034/cpp/pre ##\n -@@ t/t4034/cpp/pre: str.e+65\n - a*b c/d e%f\n - a+b c-d\n - a<>d\n --af g>=h\n -+af g>=h i<=j\n - a==b c!=d\n - a^b c|d e&&f\n - a||b\n -\n ## userdiff.c ##\n @@ userdiff.c: PATTERNS(\"cpp\",\n \t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n \t /* floatingpoint numbers that begin with a decimal point */\n - \t \"|\\\\.[0-9']+([Ee][-+]?[0-9]+)?[fFlL]?\"\n + \t \"|\\\\.[0-9][0-9']*([Ee][-+]?[0-9]+)?[fFlL]?\"\n -\t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n +\t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*|<=>\"),\n PATTERNS(\"csharp\",\n\n-- \ngitgitgadget\n"},{"id":"438420","messageId":"dd9f82ba712f54245f19df95926050543d25b769.1633885384.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v3.git.1633885384.gitgitgadget@gmail.com","subject":"[PATCH v3 1/6] t4034/cpp: actually test that operator tokens are not split","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-10T17:02:59Z","receivedAt":"2021-10-10T17:03:13Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\n8d96e7288f2b (t4034: bulk verify builtin word regex sanity, 2010-12-18)\nadded many tests with the intent to verify that operators consisting of\nmore than one symbol are kept together. These are tested by probing a\ntransition from, e.g., a!=b to x!=y, which results in the word-diff\n\n [-a-]{+x+}!=[-b-]{+y+}\n\nBut that proves only that the letters and operators are separate tokens.\nTo prove that != is an unseparable token, we have to probe a transition\nfrom, e.g., a=b to a!=b having a word-diff\n\n a[-=-]{+!=+}b\n\nthat proves that the ! is not separate from the =.\n\nIn the post-image, add to or remove from operators a character that\nturns it into another valid operator.\n\nChange the identifiers used around operators such that the diff\nalgorithm does not have an incentive to match, e.g., a\n---\n t/t4034/cpp/expect | 45 +++++++++++++++------------------------------\n t/t4034/cpp/post | 29 +++++++++++++----------------\n t/t4034/cpp/pre | 25 +++++++++++--------------\n 3 files changed, 39 insertions(+), 60 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 37d1ea25870..41976971b93 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -1,36 +1,21 @@\n diff --git a/pre b/post\n-index 23d5c8a..7e8c026 100644\n+index c5672a2..4229868 100644\n --- a/pre\n +++ b/post\n-@@ -1,19 +1,19 @@\n+@@ -1,16 +1,16 @@\n Foo() : x(0&&1&42) { bar(x); }\n cout<<\"Hello World!?\\n\"<(1) (-1e10) (0xabcdef) 'xy'\n-[ax] ax->b ay x.by\n-!ax ~a ax x++ ax-- ax*b ay x&b\n-ay\n-x*b ay x/b ay x%b\n-ay\n-x+b ay x-b\n-ay\n-x<<b ay x>>b\n-ay\n-x<b ay x<=b ay x>b ay x>=b\n-ay\n-x==b ay x!=b\n-ay\n-x&b\n-ay\n-x^b\n-ay\n-x|b\n-ay\n-x&&b\n-ay\n-x||b\n-ay\n-x?by:z\n-ax=b ay x+=b ay x-=b ay x*=b ay x/=b ay x%=b ay x<<=b ay x>>=b ay x&=b ay x^=b ay x|=b\n-ay\n-x,y\n-ax::by\n+[a] b->->*v d.e.*e\n+~!a !~b c+++ d--- e**f g&&&h\n+a**=b c//=d e%%=f\n+a+++b c---d\n+a<<<<=b c>>>>=d\n+a<<=b c<=<d e>>=f g>=>h\n+a==!=b c!==d\n+a^^=b c||=d e&&&=f\n+a|||b\n+a?:b\n+a===b c+=+d e-=fe-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n+a,b\n+a:::b\ndiff --git a/t/t4034/cpp/post b/t/t4034/cpp/post\nindex 7e8c026cefb..4229868ae62 100644\n--- a/t/t4034/cpp/post\n+++ b/t/t4034/cpp/post\n@@ -1,19 +1,16 @@\n Foo() : x(0&42) { bar(x); }\n cout<<\"Hello World?\\n\"<y x.y\n-!x ~x x++ x-- x*y x&y\n-x*y x/y x%y\n-x+y x-y\n-x<>y\n-xy x>=y\n-x==y x!=y\n-x&y\n-x^y\n-x|y\n-x&&y\n-x||y\n-x?y:z\n-x=y x+=y x-=y x*=y x/=y x%=y x<<=y x>>=y x&=y x^=y x|=y\n-x,y\n-x::y\n+[a] b->*v d.*e\n+~!a !~b c+ d- e**f g&&h\n+a*=b c/=d e%=f\n+a++b c--d\n+a<<=b c>>=d\n+a<=b c=f g>h\n+a!=b c=d\n+a^=b c|=d e&=f\n+a|b\n+a?:b\n+a==b c+d e-f g*h i/j k%l m<>p q&r s^t u|v\n+a,b\n+a:b\ndiff --git a/t/t4034/cpp/pre b/t/t4034/cpp/pre\nindex 23d5c8adf54..c5672a24cfc 100644\n--- a/t/t4034/cpp/pre\n+++ b/t/t4034/cpp/pre\n@@ -1,19 +1,16 @@\n Foo():x(0&&1){}\n cout<<\"Hello World!\\n\"<b a.b\n-!a ~a a++ a-- a*b a&b\n-a*b a/b a%b\n-a+b a-b\n-a<>b\n-ab a>=b\n-a==b a!=b\n-a&b\n-a^b\n-a|b\n-a&&b\n+[a] b->v d.e\n+!a ~b c++ d-- e*f g&h\n+a*b c/d e%f\n+a+b c-d\n+a<>d\n+af g>=h\n+a==b c!=d\n+a^b c|d e&&f\n a||b\n-a?b:z\n-a=b a+=b a-=b a*=b a/=b a%=b a<<=b a>>=b a&=b a^=b a|=b\n-a,y\n+a?b\n+a=b c+=d e-=f g*=h i/=j k%=l m<<=n o>>=p q&=r s^=t u|=v\n+a,b\n a::b\n-- \ngitgitgadget\n\n"},{"id":"438421","messageId":"5a84fc9cf715aec258d9cda2dd7d2e8eff2dc66c.1633885384.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v3.git.1633885384.gitgitgadget@gmail.com","subject":"[PATCH v3 2/6] t4034: add tests showing problematic cpp tokenizations","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-10T17:03:00Z","receivedAt":"2021-10-10T17:03:13Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nThe word regex is too loose and matches long streaks of characters\nthat should actually be separate tokens. Add these problematic test\ncases. Separate the lines with text that will remain identical in the\npre- and post-image so that the diff algorithm will not lump removals\nand additions of consecutive lines together. This makes the expected\noutput easier to read.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 22 ++++++++++++++++++----\n t/t4034/cpp/post | 18 ++++++++++++++++--\n t/t4034/cpp/pre | 16 +++++++++++++++-\n 3 files changed, 49 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 41976971b93..63e53a61e62 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -1,11 +1,25 @@\n diff --git a/pre b/post\n-index c5672a2..4229868 100644\n+index 1229cdb..3feae6f 100644\n --- a/pre\n +++ b/post\n-@@ -1,16 +1,16 @@\n-Foo() : x(0&&1&42) { bar(x); }\n+@@ -1,30 +1,30 @@\n+Foo() : x(0&&1&42) { foo0bar(x.f.Find); }\n cout<<\"Hello World!?\\n\"<(1) (-1e10) (0xabcdef) 'xy'\n+(1 -1e10+1e10 0xabcdef) 'xy'\n+// long double\n+3.141592653e-10l3.141592654e+10l\n+// float\n+120E5fE6f\n+// hex\n+0xdeadbeaf+80xdeadBeaf+7ULL\n+// octal\n+0123456701234560\n+// binary\n+0b10000b1100+e1\n+// expression\n+1.5-e+2+f1.5-e+3+f\n+// another one\n+str.e+65.e+75\n [a] b->->*v d.e.*e\n ~!a !~b c+++ d--- e**f g&&&h\n a**=b c//=d e%%=f\ndiff --git a/t/t4034/cpp/post b/t/t4034/cpp/post\nindex 4229868ae62..3feae6f430f 100644\n--- a/t/t4034/cpp/post\n+++ b/t/t4034/cpp/post\n@@ -1,6 +1,20 @@\n-Foo() : x(0&42) { bar(x); }\n+Foo() : x(0&42) { bar(x.Find); }\n cout<<\"Hello World?\\n\"<*v d.*e\n ~!a !~b c+ d- e**f g&&h\n a*=b c/=d e%=f\ndiff --git a/t/t4034/cpp/pre b/t/t4034/cpp/pre\nindex c5672a24cfc..1229cdb59d1 100644\n--- a/t/t4034/cpp/pre\n+++ b/t/t4034/cpp/pre\n@@ -1,6 +1,20 @@\n-Foo():x(0&&1){}\n+Foo():x(0&&1){ foo0( x.find); }\n cout<<\"Hello World!\\n\"<v d.e\n !a ~b c++ d-- e*f g&h\n a*b c/d e%f\n-- \ngitgitgadget\n\n"},{"id":"438422","messageId":"c9f58b5e82f2de4d4c3c9cee651e295ed935f63b.1633885384.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v3.git.1633885384.gitgitgadget@gmail.com","subject":"[PATCH v3 4/6] userdiff-cpp: prepare test cases with yet unsupported features","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-10T17:03:02Z","receivedAt":"2021-10-10T17:03:15Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nWe are going to add support for C++'s digit-separating single-quote and\nthe spaceship operator. By adding the test cases in this separate\ncommit, the effect on the word highlighting will become more obvious\nas the features are implemented and the file cpp/expect is updated.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 14 +++++++-------\n t/t4034/cpp/post | 12 ++++++------\n t/t4034/cpp/pre | 10 +++++-----\n 3 files changed, 18 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 46c9460a968..3d37ddac42c 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -1,21 +1,21 @@\n diff --git a/pre b/post\n-index 1229cdb..3feae6f 100644\n+index 144cd98..64e78af 100644\n --- a/pre\n +++ b/post\n @@ -1,30 +1,30 @@\n Foo() : x(0&&1&42) { foo0bar(x.findFind); }\n cout<<\"Hello World!?\\n\"<(1 -+1e10 0xabcdef) 'xy'\n+(1 -+1e10 0xabcdef) 'x.'\n // long double\n-3.141592653e-10l3.141592654e+10l\n+3.141'592'653e-10l654e+10l\n // float\n 120E5f120E6f\n // hex\n-0xdeadbeaf0xdeadBeaf+8ULL7ULL\n+0xdead'beafBeaf+8ULL7ULL\n // octal\n-0123456701234560\n+0123'45674560\n // binary\n-0b10000b1100+e1\n+0b100b11'00+e1\n // expression\n 1.5-e+23+f\n // another one\n@@ -25,7 +25,7 @@ str.e+6575\n a**=b c//=d e%%=f\n a+++b c---d\n a<<<<=b c>>>>=d\n-a<<=b c<=<d e>>=f g>=>h\n+a<<=b c<=<d e>>=f g>=>h i<=>j\n a==!=b c!==d\n a^^=b c||=d e&&&=f\n a|||b\ndiff --git a/t/t4034/cpp/post b/t/t4034/cpp/post\nindex 3feae6f430f..64e78afbfb5 100644\n--- a/t/t4034/cpp/post\n+++ b/t/t4034/cpp/post\n@@ -1,16 +1,16 @@\n Foo() : x(0&42) { bar(x.Find); }\n cout<<\"Hello World?\\n\"<>=d\n-a<=b c=f g>h\n+a<=b c=f g>h i<=>j\n a!=b c=d\n a^=b c|=d e&=f\n a|b\ndiff --git a/t/t4034/cpp/pre b/t/t4034/cpp/pre\nindex 1229cdb59d1..144cd980d6b 100644\n--- a/t/t4034/cpp/pre\n+++ b/t/t4034/cpp/pre\n@@ -2,15 +2,15 @@ Foo():x(0&&1){ foo0( x.find); }\n cout<<\"Hello World!\\n\"<>d\n-af g>=h\n+af g>=h i<=j\n a==b c!=d\n a^b c|d e&&f\n a||b\n-- \ngitgitgadget\n\n"},{"id":"438423","messageId":"d4ebe45fddc4e357b7f4c5299b26c0cd905fd3f2.1633885384.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v3.git.1633885384.gitgitgadget@gmail.com","subject":"[PATCH v3 3/6] userdiff-cpp: tighten word regex","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-10T17:03:01Z","receivedAt":"2021-10-10T17:03:15Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nGenerally, word regex can be written such that they match tokens\nliberally and need not model the actual syntax because it can be assumed\nthat the regex will only be applied to syntactically correct text.\n\nThe regex for cpp (C/C++) is too liberal, though. It regards these\nsequences as single tokens:\n\n 1+2\n 1.5-e+2+f\n\nand the following amalgams as one token:\n\n .l as in str.length\n .f as in str.find\n .e as in str.erase\n\nTighten the regex in the following way:\n\n- Accept + and - only in one position in the exponent. + and - are no\n longer regarded as the sign of a number and are treated by the\n catcher-all that is not visible in the driver's regex.\n\n- Accept a leading decimal point only when it is followed by a digit.\n\nFor readability, factor hex- and binary numbers into an own term.\n\nAs a drive-by, this fixes that floating point numbers such as 12E5\n(with upper-case E) were split into two tokens.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 16 ++++++++--------\n userdiff.c | 8 +++++++-\n 2 files changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 63e53a61e62..46c9460a968 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -3,24 +3,24 @@\n --- a/pre\n +++ b/post\n @@ -1,30 +1,30 @@\n-Foo() : x(0&&1&42) { foo0bar(x.f.Find); }\n+Foo() : x(0&&1&42) { foo0bar(x.findFind); }\n cout<<\"Hello World!?\\n\"<(1 -1e10+1e10 0xabcdef) 'xy'\n+(1 -+1e10 0xabcdef) 'xy'\n // long double\n 3.141592653e-10l3.141592654e+10l\n // float\n-120E5fE6f\n+120E5f120E6f\n // hex\n-0xdeadbeaf+80xdeadBeaf+7ULL\n+0xdeadbeaf0xdeadBeaf+8ULL7ULL\n // octal\n 0123456701234560\n // binary\n 0b10000b1100+e1\n // expression\n-1.5-e+2+f1.5-e+3+f\n+1.5-e+23+f\n // another one\n-str.e+65.e+75\n-[a] b->->*v d.e.*e\n+str.e+6575\n+[a] b->->*v d..*e\n ~!a !~b c+++ d--- e**f g&&&h\n a**=b c//=d e%%=f\n a+++b c---d\n@@ -30,6 +30,6 @@ a==!=b c!==d\n a^^=b c||=d e&&&=f\n a|||b\n a?:b\n-a===b c+=+d e-=fe-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n+a===b c+=+d e-=-f g*=*h i/=/j k%=%l m<<=<<n o>>=>>p q&=&r s^=^t u|=|v\n a,b\n a:::b\ndiff --git a/userdiff.c b/userdiff.c\nindex d9b2ba752f0..ce2a9230703 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -54,8 +54,14 @@ PATTERNS(\"cpp\",\n \t /* functions/methods, variables, and compounds at top level */\n \t \"^((::[[:space:]]*)?[A-Za-z_].*)$\",\n \t /* -- */\n+\t /* identifiers and keywords */\n \t \"[a-zA-Z_][a-zA-Z0-9_]*\"\n-\t \"|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*\"\n+\t /* decimal and octal integers as well as floatingpoint numbers */\n+\t \"|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n+\t /* hexadecimal and binary integers */\n+\t \"|0[xXbB][0-9a-fA-F]+[lLuU]*\"\n+\t /* floatingpoint numbers that begin with a decimal point */\n+\t \"|\\\\.[0-9]+([Ee][-+]?[0-9]+)?[fFlL]?\"\n \t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n-- \ngitgitgadget\n\n"},{"id":"438424","messageId":"037c743d9e317ced040bf76bb979c6e17583de8a.1633885384.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v3.git.1633885384.gitgitgadget@gmail.com","subject":"[PATCH v3 5/6] userdiff-cpp: permit the digit-separating single-quote in numbers","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-10T17:03:03Z","receivedAt":"2021-10-10T17:03:16Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nSince C++17, the single-quote can be used as digit separator:\n\n 3.141'592'654\n 1'000'000\n 0xdead'beaf\n\nMake it known to the word regex of the cpp driver, so that numbers are\nnot split into separate tokens at the single-quotes.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 8 ++++----\n userdiff.c | 6 +++---\n 2 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 3d37ddac42c..b90b3f207bf 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -7,15 +7,15 @@ Foo() : x(0&&1&42) { foo0bar\n cout<<\"Hello World!?\\n\"<(1 -+1e10 0xabcdef) 'x.'\n // long double\n-3.141'592'653e-10l654e+10l\n+3.141'592'653e-10l3.141'592'654e+10l\n // float\n 120E5f120E6f\n // hex\n-0xdead'beafBeaf+8ULL7ULL\n+0xdead'beaf0xdead'Beaf+8ULL7ULL\n // octal\n-0123'45674560\n+0123'45670123'4560\n // binary\n-0b100b11'00+e1\n+0b10'000b11'00+e1\n // expression\n 1.5-e+23+f\n // another one\ndiff --git a/userdiff.c b/userdiff.c\nindex ce2a9230703..5072d12e51e 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -57,11 +57,11 @@ PATTERNS(\"cpp\",\n \t /* identifiers and keywords */\n \t \"[a-zA-Z_][a-zA-Z0-9_]*\"\n \t /* decimal and octal integers as well as floatingpoint numbers */\n-\t \"|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n+\t \"|[0-9][0-9.']*([Ee][-+]?[0-9]+)?[fFlLuU]*\"\n \t /* hexadecimal and binary integers */\n-\t \"|0[xXbB][0-9a-fA-F]+[lLuU]*\"\n+\t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n \t /* floatingpoint numbers that begin with a decimal point */\n-\t \"|\\\\.[0-9]+([Ee][-+]?[0-9]+)?[fFlL]?\"\n+\t \"|\\\\.[0-9][0-9']*([Ee][-+]?[0-9]+)?[fFlL]?\"\n \t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n-- \ngitgitgadget\n\n"},{"id":"438425","messageId":"cc9dc967f10b6e1cb1d97e815496151e32f19af8.1633885384.git.gitgitgadget@gmail.com","threadId":"56657","inReplyTo":"pull.1054.v3.git.1633885384.gitgitgadget@gmail.com","subject":"[PATCH v3 6/6] userdiff-cpp: learn the C++ spaceship operator","fromName":"Johannes Sixt via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-10T17:03:04Z","receivedAt":"2021-10-10T17:03:18Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"From: Johannes Sixt \n\nSince C++20, the language has a generalized comparison operator <=>.\nTeach the cpp driver not to separate it into <= and > tokens.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 2 +-\n userdiff.c | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex b90b3f207bf..5ff4ce477b4 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -25,7 +25,7 @@ str.e+6575\n a**=b c//=d e%%=f\n a+++b c---d\n a<<<<=b c>>>>=d\n-a<<=b c<=<d e>>=f g>=>h i<=>j\n+a<<=b c<=<d e>>=f g>=>h i<=<=>j\n a==!=b c!==d\n a^^=b c||=d e&&&=f\n a|||b\ndiff --git a/userdiff.c b/userdiff.c\nindex 5072d12e51e..96adddd6f9a 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -62,7 +62,7 @@ PATTERNS(\"cpp\",\n \t \"|0[xXbB][0-9a-fA-F']+[lLuU]*\"\n \t /* floatingpoint numbers that begin with a decimal point */\n \t \"|\\\\.[0-9][0-9']*([Ee][-+]?[0-9]+)?[fFlL]?\"\n-\t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*\"),\n+\t \"|[-+*/<>%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*|<=>\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n \t \"!^[ \\t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\\n\"\n-- \ngitgitgadget\n"},{"id":"438427","messageId":"27a4476a-4743-b798-b3fa-ce068472143b@kdbg.org","threadId":"56657","inReplyTo":"87mtnjm416.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/5] Fun with cpp word regex","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2021-10-10T20:15:42Z","receivedAt":"2021-10-10T20:15:46Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 09.10.21 um 02:00 schrieb Ævar Arnfjörð Bjarmason:\n> For an example of something we do a bit badly with the cpp driver is\n> parts of my 66f5f6dca95 (C style: use standard style for \"TRANSLATORS\"\n> comments, 2017-05-11).\n> \n> I.e. there I was changing a comment format, and added a full stop to a\n> sentence, the word-diff is:\n> \n> /*\n> {+*+} TRANSLATORS: here is a comment that explains the string to\n> {+*+} be translated, that follows immediately after [-it-]{+it.+}\n> */\n> \n> Even though it has nothing to do with C syntax per-se that would be much more useful as:\n> \n> /*\n> {+*+} TRANSLATORS: here is a comment that explains the string to\n> {+*+} be translated, that follows immediately after it{+.+}\n> */\n> \n> I.e. treating a \".\" at the end of a word specially isn't C or C++\n> syntax, but it's absolutely input that the cpp driver *is* getting and\n> should be if possible be handling well.\n\nFWIW, I wondered why the cpp driver should not handle this case as you\nexpect it, and the answer is that it actually does when this comment\nappears in a C file. This particular word-diff occurs in\nDocumentation/CodingGuidelines and that is not covered by the cpp driver.\n\n-- Hannes\n"},{"id":"439484","messageId":"a5a05e33-3ad9-e78c-69c0-a466eadeffec@kdbg.org","threadId":"56657","inReplyTo":"pull.1054.v3.git.1633885384.gitgitgadget@gmail.com","subject":"[PATCH 7/6] userdiff-cpp: back out the digit-separators in numbers","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2021-10-24T09:56:43Z","receivedAt":"2021-10-24T09:56:49Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"The implementation of digit-separating single-quotes introduced a\nnote-worthy regression: the change of a character literal with a\ndigit would splice the digit and the closing single-quote. For\nexample, the change from 'a' to '2' is now tokenized as\n'[-a'-]{+2'+} instead of '[-a-]{+2+}'.\n\nThe options to fix the regression are:\n\n- Tighten the regular expression such that the single-quote can only\n occur between digits (that would match the official syntax).\n\n- Remove support for digit separators.\n\nI chose to remove support, because\n\n- I have not seen a lot of code make use of digit separators.\n\n- If code does use digit separators, then the numbers are typically\n long. If a change in one of the segments occurs, it is actually\n better visible if only that segment is highlighted as the word\n that changed instead of the whole long number.\n\nThis choice does introduce another minor regression, though, which\nis highlighted in the test case: when a change occurs in the second\nor later segment of a hexadecimal number where the segment begins\nwith a digit, but also has letters, the segment is mistaken as\nconsisting of a number and an identifier. I can live with that.\n\nSigned-off-by: Johannes Sixt \n---\n t/t4034/cpp/expect | 12 ++++++------\n t/t4034/cpp/post | 10 +++++-----\n t/t4034/cpp/pre | 8 ++++----\n userdiff.c | 6 +++---\n 4 files changed, 18 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect\nindex 5ff4ce477b..dc500ae092 100644\n--- a/t/t4034/cpp/expect\n+++ b/t/t4034/cpp/expect\n@@ -1,21 +1,21 @@\n diff --git a/pre b/post\n-index 144cd98..64e78af 100644\n+index a1a09b7..f1b6f3c 100644\n --- a/pre\n +++ b/post\n @@ -1,30 +1,30 @@\n Foo() : x(0&&1&42) { foo0bar(x.findFind); }\n cout<<\"Hello World!?\\n\"<(1 -+1e10 0xabcdef) 'x.'\n+(1 -+1e10 0xabcdef) 'x2'\n // long double\n-3.141'592'653e-10l3.141'592'654e+10l\n+3.141592653e-10l3.141592654e+10l\n // float\n 120E5f120E6f\n // hex\n-0xdead'beaf0xdead'Beaf+8ULL7ULL\n+0xdead0xdeaf'1eaFeaf+8ULL7ULL\n // octal\n-0123'45670123'4560\n+0123456701234560\n // binary\n-0b10'000b11'00+e1\n+0b10000b1100+e1\n // expression\n 1.5-e+23+f\n // another one\ndiff --git a/t/t4034/cpp/post b/t/t4034/cpp/post\nindex 64e78afbfb..f1b6f3c228 100644\n--- a/t/t4034/cpp/post\n+++ b/t/t4034/cpp/post\n@@ -1,16 +1,16 @@\n Foo() : x(0&42) { bar(x.Find); }\n cout<<\"Hello World?\\n\"<%&^|=!]=|--|\\\\+\\\\+|<<=?|>>=?|&&|\\\\|\\\\||::|->\\\\*?|\\\\.\\\\*|<=>\"),\n PATTERNS(\"csharp\",\n \t /* Keywords */\n-- \n2.33.0.129.g739793498e\n"}]}