git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/5] Fun with cpp word regex

From
Johannes Sixt <j6t@kdbg.org>
Date
Oct 8, 2021, 22:11 UTC
Message-ID
<25363715-dc39-1f18-a937-f715b106f529@kdbg.org>
In-Reply-To
<87r1cvmg0c.fsf@evledraar.gmail.com>
Am 08.10.21 um 22:07 schrieb Ævar Arnfjörð Bjarmason:
Show 7 quoted lines
>  * I wonder if it isn't time to split up "cpp" into a "c" driver,
>    e.g. git.git's .gitattributes has "cpp" for *.[ch] files, but as C++
>    adds more syntax sugar.
> 
>    So e.g. if you use "<=>" after this series we'll tokenize it
>    differently in *.c files, but it's a C++-only operator, on the other
>    hand probably nobody cares that much...

Yes, it is that: <=> won't appear in a correct C file (outside of comments), so no-one will care. As far as tokenization is concerned, C is a subset of C++. I don't think we need to separate the drivers.

Show 7 quoted lines
>  * I found myself back-porting some of your tests (manually mostly),
>    maybe you disagree, but in cases like 123'123, <=> etc. I'd find it
>    easier to follow if we first added the test data, and then the
>    changed behavior.
> 
>    Because after all, we're going to change how we highlight existing
>    data, so testing for that would be informative.
Good point. I'll work a bit more on that.
Show 10 quoted lines
>  * This pre-dates your much improved tests, but these test files could
>    really use some test comments, as in:
> 
>    /* Now that we're going to understand the "'" character somehow, will any of this change? */
>    /* We haven't written code like this since the 1960's ... */
>    /* Run & free */
> 
>    I.e. we don't just highlight code the compiler likes to eat, but also
>    comments. So particularly for smaller tokens that also occur in
>    natural language like "'" and "&" are we getting expected results?

Comments are free text. Anything can happen. There is no such thing as "correct tokenization" in comments. Not interested.

Thank you for the review. -- Hannes

Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 14 of 24 in “Fun with cpp word regex”
  1. 0/3 Fun with cpp word regexJohannes Sixt via GitGitGadget, Oct 7, 2021
  2. 1/3 userdiff: tighten cpp word regexJohannes Sixt via GitGitGadget, Oct 7, 2021
  3. 2/3 userdiff: permit the digit-separating single-quote in numbersJohannes Sixt via GitGitGadget, Oct 7, 2021
  4. 3/3 userdiff: learn the C++ spaceship operatorJohannes Sixt via GitGitGadget, Oct 7, 2021
  5. Ævar Arnfjörð BjarmasonOct 7, 2021
  6. Johannes SixtOct 7, 2021
  7. 0/5 Fun with cpp word regexJohannes Sixt via GitGitGadget, Oct 8, 2021
  8. 1/5 t4034/cpp: actually test that operator tokens are not splitJohannes Sixt via GitGitGadget, Oct 8, 2021
  9. 2/5 t4034: add tests showing problematic cpp tokenizationsJohannes Sixt via GitGitGadget, Oct 8, 2021
  10. 3/5 userdiff-cpp: tighten word regexJohannes Sixt via GitGitGadget, Oct 8, 2021
  11. 4/5 userdiff-cpp: permit the digit-separating single-quote in numbersJohannes Sixt via GitGitGadget, Oct 8, 2021
  12. 5/5 userdiff-cpp: learn the C++ spaceship operatorJohannes Sixt via GitGitGadget, Oct 8, 2021
  13. Ævar Arnfjörð BjarmasonOct 8, 2021
  14. Johannes SixtOct 8, 2021
  15. Ævar Arnfjörð BjarmasonOct 9, 2021
  16. Johannes SixtOct 10, 2021
  17. 0/6 Fun with cpp word regexJohannes Sixt via GitGitGadget, Oct 10, 2021
  18. 1/6 t4034/cpp: actually test that operator tokens are not splitJohannes Sixt via GitGitGadget, Oct 10, 2021
  19. 2/6 t4034: add tests showing problematic cpp tokenizationsJohannes Sixt via GitGitGadget, Oct 10, 2021
  20. 4/6 userdiff-cpp: prepare test cases with yet unsupported featuresJohannes Sixt via GitGitGadget, Oct 10, 2021
  21. 3/6 userdiff-cpp: tighten word regexJohannes Sixt via GitGitGadget, Oct 10, 2021
  22. 5/6 userdiff-cpp: permit the digit-separating single-quote in numbersJohannes Sixt via GitGitGadget, Oct 10, 2021
  23. 6/6 userdiff-cpp: learn the C++ spaceship operatorJohannes Sixt via GitGitGadget, Oct 10, 2021
  24. 7/6 userdiff-cpp: back out the digit-separators in numbersJohannes Sixt, Oct 24, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.