git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 1/6] t4034/cpp: actually test that operator tokens are not split

From
Johannes Sixt via GitGitGadget <gitgitgadget@gmail.com>
Date
Oct 10, 2021, 17:02 UTC
Message-ID
<dd9f82ba712f54245f19df95926050543d25b769.1633885384.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1054.v3.git.1633885384.gitgitgadget@gmail.com>
From: Johannes Sixt <j6t@kdbg.org>

8d96e7288f2b (t4034: bulk verify builtin word regex sanity, 2010-12-18) added many tests with the intent to verify that operators consisting of more than one symbol are kept together. These are tested by probing a transition from, e.g., a!=b to x!=y, which results in the word-diff

  [-a-]{+x+}!=[-b-]{+y+}

But that proves only that the letters and operators are separate tokens. To prove that != is an unseparable token, we have to probe a transition from, e.g., a=b to a!=b having a word-diff

  a[-=-]{+!=+}b
that proves that the ! is not separate from the =.

In the post-image, add to or remove from operators a character that turns it into another valid operator.

Change the identifiers used around operators such that the diff algorithm does not have an incentive to match, e.g., a<b in one spot in the pre-image with a<b elsewhere in the post-image.

Adjust the expected output to match the new differences. Notice that there are some undesirable tokenizations around e, ., and -. This will be addressed in a later change.

Signed-off-by: Johannes Sixt <j6t@kdbg.org>
---
 t/t4034/cpp/expect | 45 +++++++++++++++------------------------------
 t/t4034/cpp/post   | 29 +++++++++++++----------------
 t/t4034/cpp/pre    | 25 +++++++++++--------------
 3 files changed, 39 insertions(+), 60 deletions(-)
diff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect
index 37d1ea25870..41976971b93 100644
--- a/t/t4034/cpp/expect
+++ b/t/t4034/cpp/expect
@@ -1,36 +1,21 @@
 <BOLD>diff --git a/pre b/post<RESET>
-<BOLD>index 23d5c8a..7e8c026 100644<RESET>
+<BOLD>index c5672a2..4229868 100644<RESET>
 <BOLD>--- a/pre<RESET>
 <BOLD>+++ b/post<RESET>
-<CYAN>@@ -1,19 +1,19 @@<RESET>
+<CYAN>@@ -1,16 +1,16 @@<RESET>
 Foo() : x(0<RED>&&1<RESET><GREEN>&42<RESET>) { <GREEN>bar(x);<RESET> }
 cout<<"Hello World<RED>!<RESET><GREEN>?<RESET>\n"<<endl;
 <GREEN>(<RESET>1<GREEN>) (<RESET>-1e10<GREEN>) (<RESET>0xabcdef<GREEN>)<RESET> '<RED>x<RESET><GREEN>y<RESET>'
-[<RED>a<RESET><GREEN>x<RESET>] <RED>a<RESET><GREEN>x<RESET>-><RED>b a<RESET><GREEN>y x<RESET>.<RED>b<RESET><GREEN>y<RESET>
-!<RED>a<RESET><GREEN>x<RESET> ~<RED>a a<RESET><GREEN>x x<RESET>++ <RED>a<RESET><GREEN>x<RESET>-- <RED>a<RESET><GREEN>x<RESET>*<RED>b a<RESET><GREEN>y x<RESET>&<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>*<RED>b a<RESET><GREEN>y x<RESET>/<RED>b a<RESET><GREEN>y x<RESET>%<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>+<RED>b a<RESET><GREEN>y x<RESET>-<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET><<<RED>b a<RESET><GREEN>y x<RESET>>><RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET><<RED>b a<RESET><GREEN>y x<RESET><=<RED>b a<RESET><GREEN>y x<RESET>><RED>b a<RESET><GREEN>y x<RESET>>=<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>==<RED>b a<RESET><GREEN>y x<RESET>!=<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>&<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>^<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>|<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>&&<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>||<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>?<RED>b<RESET><GREEN>y<RESET>:z
-<RED>a<RESET><GREEN>x<RESET>=<RED>b a<RESET><GREEN>y x<RESET>+=<RED>b a<RESET><GREEN>y x<RESET>-=<RED>b a<RESET><GREEN>y x<RESET>*=<RED>b a<RESET><GREEN>y x<RESET>/=<RED>b a<RESET><GREEN>y x<RESET>%=<RED>b a<RESET><GREEN>y x<RESET><<=<RED>b a<RESET><GREEN>y x<RESET>>>=<RED>b a<RESET><GREEN>y x<RESET>&=<RED>b a<RESET><GREEN>y x<RESET>^=<RED>b a<RESET><GREEN>y x<RESET>|=<RED>b<RESET>
-<RED>a<RESET><GREEN>y<RESET>
-<GREEN>x<RESET>,y
-<RED>a<RESET><GREEN>x<RESET>::<RED>b<RESET><GREEN>y<RESET>
+[a] b<RED>-><RESET><GREEN>->*<RESET>v d<RED>.e<RESET><GREEN>.*e<RESET>
+<GREEN>~<RESET>!a <GREEN>!<RESET>~b c<RED>++<RESET><GREEN>+<RESET> d<RED>--<RESET><GREEN>-<RESET> e*<GREEN>*<RESET>f g<RED>&<RESET><GREEN>&&<RESET>h
+a<RED>*<RESET><GREEN>*=<RESET>b c<RED>/<RESET><GREEN>/=<RESET>d e<RED>%<RESET><GREEN>%=<RESET>f
+a<RED>+<RESET><GREEN>++<RESET>b c<RED>-<RESET><GREEN>--<RESET>d
+a<RED><<<RESET><GREEN><<=<RESET>b c<RED>>><RESET><GREEN>>>=<RESET>d
+a<RED><<RESET><GREEN><=<RESET>b c<RED><=<RESET><GREEN><<RESET>d e<RED>><RESET><GREEN>>=<RESET>f g<RED>>=<RESET><GREEN>><RESET>h
+a<RED>==<RESET><GREEN>!=<RESET>b c<RED>!=<RESET><GREEN>=<RESET>d
+a<RED>^<RESET><GREEN>^=<RESET>b c<RED>|<RESET><GREEN>|=<RESET>d e<RED>&&<RESET><GREEN>&=<RESET>f
+a<RED>||<RESET><GREEN>|<RESET>b
+a?<GREEN>:<RESET>b
+a<RED>=<RESET><GREEN>==<RESET>b c<RED>+=<RESET><GREEN>+<RESET>d <RED>e-=f<RESET><GREEN>e-f<RESET> g<RED>*=<RESET><GREEN>*<RESET>h i<RED>/=<RESET><GREEN>/<RESET>j k<RED>%=<RESET><GREEN>%<RESET>l m<RED><<=<RESET><GREEN><<<RESET>n o<RED>>>=<RESET><GREEN>>><RESET>p q<RED>&=<RESET><GREEN>&<RESET>r s<RED>^=<RESET><GREEN>^<RESET>t u<RED>|=<RESET><GREEN>|<RESET>v
+a,b<RESET>
+a<RED>::<RESET><GREEN>:<RESET>b
diff --git a/t/t4034/cpp/post b/t/t4034/cpp/post
index 7e8c026cefb..4229868ae62 100644
--- a/t/t4034/cpp/post
+++ b/t/t4034/cpp/post
@@ -1,19 +1,16 @@
 Foo() : x(0&42) { bar(x); }
 cout<<"Hello World?\n"<<endl;
 (1) (-1e10) (0xabcdef) 'y'
-[x] x->y x.y
-!x ~x x++ x-- x*y x&y
-x*y x/y x%y
-x+y x-y
-x<<y x>>y
-x<y x<=y x>y x>=y
-x==y x!=y
-x&y
-x^y
-x|y
-x&&y
-x||y
-x?y:z
-x=y x+=y x-=y x*=y x/=y x%=y x<<=y x>>=y x&=y x^=y x|=y
-x,y
-x::y
+[a] b->*v d.*e
+~!a !~b c+ d- e**f g&&h
+a*=b c/=d e%=f
+a++b c--d
+a<<=b c>>=d
+a<=b c<d e>=f g>h
+a!=b c=d
+a^=b c|=d e&=f
+a|b
+a?:b
+a==b c+d e-f g*h i/j k%l m<<n o>>p q&r s^t u|v
+a,b
+a:b
diff --git a/t/t4034/cpp/pre b/t/t4034/cpp/pre
index 23d5c8adf54..c5672a24cfc 100644
--- a/t/t4034/cpp/pre
+++ b/t/t4034/cpp/pre
@@ -1,19 +1,16 @@
 Foo():x(0&&1){}
 cout<<"Hello World!\n"<<endl;
 1 -1e10 0xabcdef 'x'
-[a] a->b a.b
-!a ~a a++ a-- a*b a&b
-a*b a/b a%b
-a+b a-b
-a<<b a>>b
-a<b a<=b a>b a>=b
-a==b a!=b
-a&b
-a^b
-a|b
-a&&b
+[a] b->v d.e
+!a ~b c++ d-- e*f g&h
+a*b c/d e%f
+a+b c-d
+a<<b c>>d
+a<b c<=d e>f g>=h
+a==b c!=d
+a^b c|d e&&f
 a||b
-a?b:z
-a=b a+=b a-=b a*=b a/=b a%=b a<<=b a>>=b a&=b a^=b a|=b
-a,y
+a?b
+a=b c+=d e-=f g*=h i/=j k%=l m<<=n o>>=p q&=r s^=t u|=v
+a,b
 a::b

-- 
gitgitgadget
Previous: Johannes Sixt via GitGitGadgetNext: Johannes Sixt via GitGitGadget
Message 18 of 24 in “Fun with cpp word regex”
  1. 0/3 Fun with cpp word regexJohannes Sixt via GitGitGadget, Oct 7, 2021
  2. 1/3 userdiff: tighten cpp word regexJohannes Sixt via GitGitGadget, Oct 7, 2021
  3. 2/3 userdiff: permit the digit-separating single-quote in numbersJohannes Sixt via GitGitGadget, Oct 7, 2021
  4. 3/3 userdiff: learn the C++ spaceship operatorJohannes Sixt via GitGitGadget, Oct 7, 2021
  5. Ævar Arnfjörð BjarmasonOct 7, 2021
  6. Johannes SixtOct 7, 2021
  7. 0/5 Fun with cpp word regexJohannes Sixt via GitGitGadget, Oct 8, 2021
  8. 1/5 t4034/cpp: actually test that operator tokens are not splitJohannes Sixt via GitGitGadget, Oct 8, 2021
  9. 2/5 t4034: add tests showing problematic cpp tokenizationsJohannes Sixt via GitGitGadget, Oct 8, 2021
  10. 3/5 userdiff-cpp: tighten word regexJohannes Sixt via GitGitGadget, Oct 8, 2021
  11. 4/5 userdiff-cpp: permit the digit-separating single-quote in numbersJohannes Sixt via GitGitGadget, Oct 8, 2021
  12. 5/5 userdiff-cpp: learn the C++ spaceship operatorJohannes Sixt via GitGitGadget, Oct 8, 2021
  13. Ævar Arnfjörð BjarmasonOct 8, 2021
  14. Johannes SixtOct 8, 2021
  15. Ævar Arnfjörð BjarmasonOct 9, 2021
  16. Johannes SixtOct 10, 2021
  17. 0/6 Fun with cpp word regexJohannes Sixt via GitGitGadget, Oct 10, 2021
  18. 1/6 t4034/cpp: actually test that operator tokens are not splitJohannes Sixt via GitGitGadget, Oct 10, 2021
  19. 2/6 t4034: add tests showing problematic cpp tokenizationsJohannes Sixt via GitGitGadget, Oct 10, 2021
  20. 4/6 userdiff-cpp: prepare test cases with yet unsupported featuresJohannes Sixt via GitGitGadget, Oct 10, 2021
  21. 3/6 userdiff-cpp: tighten word regexJohannes Sixt via GitGitGadget, Oct 10, 2021
  22. 5/6 userdiff-cpp: permit the digit-separating single-quote in numbersJohannes Sixt via GitGitGadget, Oct 10, 2021
  23. 6/6 userdiff-cpp: learn the C++ spaceship operatorJohannes Sixt via GitGitGadget, Oct 10, 2021
  24. 7/6 userdiff-cpp: back out the digit-separators in numbersJohannes Sixt, Oct 24, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.