{"thread":{"id":"66864","subject":"[PATCH] utf8: make display_mode_esc_sequence_len() take a length","startedAt":"2026-11-17T16:50:41Z","lastAt":"2026-11-18T19:29:47Z","messageCount":15,"participants":["Junio C Hamano","Dirk Gouders","Andy Koppe"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"558631","messageId":"xmqqmrr7fllq.fsf@gitster.g","threadId":"66864","inReplyTo":null,"subject":"[PATCH] utf8: make display_mode_esc_sequence_len() take a length","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-17T16:50:41Z","receivedAt":"2026-11-17T16:50:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"display_mode_esc_sequence_len() scans for an ANSI display mode\nescape sequence (e.g. \"ESC [ 1 ; 31 m\"), relying on NUL termination\nto stop.  Some callers, however, use a counted <ptr, len> buffer or\nneed to stop at a specific position.\n\nTeach it to take a buffer length and never read at or beyond\nptr[len], returning the length of a complete sequence at the start\nor 0 otherwise.\n\nUpdate the callers to pass the remaining byte count.  For\nstrbuf_add_wrapped_text(), which takes a NUL-terminated string,\ncompute the end once upfront instead of calling strlen() each time.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n * Copy of https://lore.kernel.org/git/xmqqse0zfn2n.fsf@gitster.g/\n   with a proposed commit log message.\n\n pretty.c |  3 ++-\n utf8.c   | 27 ++++++++++++++++++---------\n utf8.h   |  2 +-\n 3 files changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/pretty.c b/pretty.c\nindex 0bc8ad8a9a..a20fd61909 100644\n--- a/pretty.c\n+++ b/pretty.c\n@@ -1834,7 +1834,8 @@ static size_t format_and_pad_commit(struct strbuf *sb, /* in UTF-8 */\n \t\t\twhile (p > sb->buf && ch - p < 10 && *p != '\\033')\n \t\t\t\tp--;\n \t\t\tif (*p != '\\033' ||\n-\t\t\t    ch + 1 - p != display_mode_esc_sequence_len(p))\n+\t\t\t    ch + 1 - p != display_mode_esc_sequence_len(p,\n+\t\t\t\t\t\t\t\tch + 1 - p))\n \t\t\t\tbreak;\n \t\t\t/*\n \t\t\t * got a good ansi sequence, put it back to\ndiff --git a/utf8.c b/utf8.c\nindex 96460cc414..544c5329f3 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -16,16 +16,23 @@ struct interval {\n \tucs_char_t last;\n };\n \n-size_t display_mode_esc_sequence_len(const char *s)\n+/*\n+ * Return the length of the ANSI display mode escape sequence (e.g.,\n+ * \"\\033[1;31m\") that appears at the beginning of the \"len\" bytes\n+ * of string \"s\", or 0 if the bytes do not begin with a complete\n+ * such sequence.  Never looks at s[len] or beyond.\n+ */\n+size_t display_mode_esc_sequence_len(const char *s, size_t len)\n {\n-\tconst char *p = s;\n-\tif (*p++ != '\\033')\n+\tconst char *p = s, *end = s + len;\n+\n+\tif (end <= p || *p++ != '\\033')\n \t\treturn 0;\n-\tif (*p++ != '[')\n+\tif (end <= p || *p++ != '[')\n \t\treturn 0;\n-\twhile (isdigit(*p) || *p == ';')\n+\twhile (p < end && (isdigit(*p) || *p == ';'))\n \t\tp++;\n-\tif (*p++ != 'm')\n+\tif (end <= p || *p++ != 'm')\n \t\treturn 0;\n \treturn p - s;\n }\n@@ -218,7 +225,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)\n \t\tsize_t skip;\n \n \t\twhile (skip_ansi &&\n-\t\t       (skip = display_mode_esc_sequence_len(string)) != 0)\n+\t\t       (skip = display_mode_esc_sequence_len(string,\n+\t\t\t\t\t\t\t     orig + len - string)) != 0)\n \t\t\tstring += skip;\n \n \t\tglyph_width = utf8_width(&string, NULL);\n@@ -279,6 +287,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,\n {\n \tint indent, w, assume_utf8 = 1;\n \tconst char *bol, *space, *start = text;\n+\tconst char *end = text + strlen(text);\n \tsize_t orig_len = buf->len;\n \n \tif (width <= 0) {\n@@ -299,7 +308,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,\n \t\tchar c;\n \t\tsize_t skip;\n \n-\t\twhile ((skip = display_mode_esc_sequence_len(text)))\n+\t\twhile ((skip = display_mode_esc_sequence_len(text, end - text)))\n \t\t\ttext += skip;\n \n \t\tc = *text;\n@@ -378,7 +387,7 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,\n \t\tint glyph_width;\n \t\tsize_t n;\n \n-\t\twhile ((n = display_mode_esc_sequence_len(src))) {\n+\t\twhile ((n = display_mode_esc_sequence_len(src, end - src))) {\n \t\t\tstrbuf_add(&dst, src, n);\n \t\t\tsrc += n;\n \t\t}\ndiff --git a/utf8.h b/utf8.h\nindex cf8ecb0f21..fb2227a9e2 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -5,7 +5,7 @@ struct strbuf;\n \n typedef unsigned int ucs_char_t;  /* assuming 32bit int */\n \n-size_t display_mode_esc_sequence_len(const char *s);\n+size_t display_mode_esc_sequence_len(const char *s, size_t len);\n int utf8_width(const char **start, size_t *remainder_p);\n int utf8_strnwidth(const char *string, size_t len, int skip_ansi);\n int utf8_strwidth(const char *string);\n-- \n2.56.1-740-gf38658f3d0\n\n\n"},{"id":"558646","messageId":"xmqqqzgjcjes.fsf@gitster.g","threadId":"66864","inReplyTo":"xmqqmrr7fllq.fsf@gitster.g","subject":"[PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-17T20:06:19Z","receivedAt":"2026-11-17T20:06:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"display_mode_esc_sequence_len() scans for an ANSI display mode\nescape sequence (e.g. \"ESC [ 1 ; 31 m\") at the start of a string,\nrelying on NUL termination to stop.  Some callers, however, work on\na counted buffer: utf8_strnwidth() takes <string, len> and must not\nscan past it, while strbuf_utf8_replace() and\nstrbuf_add_wrapped_text() know (or can cheaply learn) the end.\n\nAdd display_mode_esc_sequence_len_counted() taking <ptr, len>,\nwhich never reads at or beyond ptr[len], and make\ndisplay_mode_esc_sequence_len() a thin wrapper for NUL-terminated\nstrings.\n\nUse the counted variant in utf8_strnwidth() for correctness, and in\nstrbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the\nend once upfront in the latter) to avoid measuring the remaining\ntext in each iteration.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n * As there are new patches that want to add new callers to this\n   function, let's keep the original with the same API while adding\n   a new helper function that allows us to limit the scan.\n\n utf8.c | 35 ++++++++++++++++++++++++++---------\n utf8.h |  1 +\n 2 files changed, 27 insertions(+), 9 deletions(-)\n\ndiff --git a/utf8.c b/utf8.c\nindex 96460cc414..d0546ab31f 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -16,20 +16,33 @@ struct interval {\n \tucs_char_t last;\n };\n \n-size_t display_mode_esc_sequence_len(const char *s)\n+/*\n+ * Return the length of the ANSI display mode escape sequence (e.g.,\n+ * \"\\033[1;31m\") that appears at the beginning of the \"len\" bytes\n+ * of string \"s\", or 0 if the bytes do not begin with a complete\n+ * such sequence.  Never looks at s[len] or beyond.\n+ */\n+size_t display_mode_esc_sequence_len_counted(const char *s, size_t len)\n {\n-\tconst char *p = s;\n-\tif (*p++ != '\\033')\n+\tconst char *p = s, *end = s + len;\n+\n+\tif (end <= p || *p++ != '\\033')\n \t\treturn 0;\n-\tif (*p++ != '[')\n+\tif (end <= p || *p++ != '[')\n \t\treturn 0;\n-\twhile (isdigit(*p) || *p == ';')\n+\twhile (p < end && (isdigit(*p) || *p == ';'))\n \t\tp++;\n-\tif (*p++ != 'm')\n+\tif (end <= p || *p++ != 'm')\n \t\treturn 0;\n \treturn p - s;\n }\n \n+/* Same as above, but for a NUL-terminated string \"s\". */\n+size_t display_mode_esc_sequence_len(const char *s)\n+{\n+\treturn display_mode_esc_sequence_len_counted(s, strlen(s));\n+}\n+\n /* auxiliary function for binary search in interval table */\n static int bisearch(ucs_char_t ucs, const struct interval *table, int max)\n {\n@@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)\n \t\tsize_t skip;\n \n \t\twhile (skip_ansi &&\n-\t\t       (skip = display_mode_esc_sequence_len(string)) != 0)\n+\t\t       (skip = display_mode_esc_sequence_len_counted(string,\n+\t\t\t\t\t\t\torig + len - string)) != 0)\n \t\t\tstring += skip;\n \n \t\tglyph_width = utf8_width(&string, NULL);\n@@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,\n {\n \tint indent, w, assume_utf8 = 1;\n \tconst char *bol, *space, *start = text;\n+\tconst char *end = text + strlen(text);\n \tsize_t orig_len = buf->len;\n \n \tif (width <= 0) {\n@@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,\n \t\tchar c;\n \t\tsize_t skip;\n \n-\t\twhile ((skip = display_mode_esc_sequence_len(text)))\n+\t\twhile ((skip = display_mode_esc_sequence_len_counted(text,\n+\t\t\t\t\t\t\t\tend - text)))\n \t\t\ttext += skip;\n \n \t\tc = *text;\n@@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,\n \t\tint glyph_width;\n \t\tsize_t n;\n \n-\t\twhile ((n = display_mode_esc_sequence_len(src))) {\n+\t\twhile ((n = display_mode_esc_sequence_len_counted(src,\n+\t\t\t\t\t\t\t\t  end - src))) {\n \t\t\tstrbuf_add(&dst, src, n);\n \t\t\tsrc += n;\n \t\t}\ndiff --git a/utf8.h b/utf8.h\nindex cf8ecb0f21..57ca2ab270 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -6,6 +6,7 @@ struct strbuf;\n typedef unsigned int ucs_char_t;  /* assuming 32bit int */\n \n size_t display_mode_esc_sequence_len(const char *s);\n+size_t display_mode_esc_sequence_len_counted(const char *s, size_t len);\n int utf8_width(const char **start, size_t *remainder_p);\n int utf8_strnwidth(const char *string, size_t len, int skip_ansi);\n int utf8_strwidth(const char *string);\n-- \n2.56.1-740-gf38658f3d0\n\n\n"},{"id":"558649","messageId":"ghy0arkv39.fsf@gouders.net","threadId":"66864","inReplyTo":"xmqqqzgjcjes.fsf@gitster.g","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Dirk Gouders","fromEmail":"dirk@gouders.net","sentAt":"2026-11-17T21:26:50Z","receivedAt":"2026-11-17T21:26:50Z","isPatch":true,"sender":{"key":"dirk@gouders.net","avatar":"https://avatars.githubusercontent.com/u/81326422?v=4"},"body":"On Tue, Nov 17 2026, Junio C Hamano wrote:\n\n> display_mode_esc_sequence_len() scans for an ANSI display mode\n> escape sequence (e.g. \"ESC [ 1 ; 31 m\") at the start of a string,\n> relying on NUL termination to stop.  Some callers, however, work on\n> a counted buffer: utf8_strnwidth() takes <string, len> and must not\n> scan past it, while strbuf_utf8_replace() and\n> strbuf_add_wrapped_text() know (or can cheaply learn) the end.\n>\n> Add display_mode_esc_sequence_len_counted() taking <ptr, len>,\n> which never reads at or beyond ptr[len], and make\n> display_mode_esc_sequence_len() a thin wrapper for NUL-terminated\n> strings.\n>\n> Use the counted variant in utf8_strnwidth() for correctness, and in\n> strbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the\n> end once upfront in the latter) to avoid measuring the remaining\n> text in each iteration.\n>\n> Signed-off-by: Junio C Hamano <gitster@pobox.com>\n> ---\n>  * As there are new patches that want to add new callers to this\n>    function, let's keep the original with the same API while adding\n>    a new helper function that allows us to limit the scan.\n>\n>  utf8.c | 35 ++++++++++++++++++++++++++---------\n>  utf8.h |  1 +\n>  2 files changed, 27 insertions(+), 9 deletions(-)\n>\n> diff --git a/utf8.c b/utf8.c\n> index 96460cc414..d0546ab31f 100644\n> --- a/utf8.c\n> +++ b/utf8.c\n> @@ -16,20 +16,33 @@ struct interval {\n>  \tucs_char_t last;\n>  };\n>  \n> -size_t display_mode_esc_sequence_len(const char *s)\n> +/*\n> + * Return the length of the ANSI display mode escape sequence (e.g.,\n> + * \"\\033[1;31m\") that appears at the beginning of the \"len\" bytes\n> + * of string \"s\", or 0 if the bytes do not begin with a complete\n> + * such sequence.  Never looks at s[len] or beyond.\n> + */\n> +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len)\n>  {\n> -\tconst char *p = s;\n> -\tif (*p++ != '\\033')\n> +\tconst char *p = s, *end = s + len;\n\nYou have better knowledge of the users of this function: would an early\nreturn perhaps handle a significant amount of cases?\n\n        /* The shortest SGR sequence is \"\\033[m\". */\n        if (len < 3)\n                return 0;\n\n\nAnd, why not add some tests (the untested function I showed had\nan off-by-one error):\n\ndiff --git a/Makefile b/Makefile\nindex 4c84eea6e2..3c8dc9ca40 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec\n CLAR_TEST_SUITES += u-trailer\n CLAR_TEST_SUITES += u-urlmatch-normalization\n CLAR_TEST_SUITES += u-utf8-width\n+CLAR_TEST_SUITES += u-utf8-ansi\n CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)\n CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))\n CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o\ndiff --git a/t/meson.build b/t/meson.build\nindex 1cf1d95f70..e224df58c8 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -27,6 +27,7 @@ clar_test_suites = [\n   'unit-tests/u-trailer.c',\n   'unit-tests/u-urlmatch-normalization.c',\n   'unit-tests/u-utf8-width.c',\n+  'unit-tests/u-utf8-ansi.c',\n ]\n \n clar_sources = [\ndiff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c\nnew file mode 100644\nindex 0000000000..c19d8bce78\n--- /dev/null\n+++ b/t/unit-tests/u-utf8-ansi.c\n@@ -0,0 +1,14 @@\n+#include \"unit-test.h\"\n+#include \"utf8.h\"\n+#include \"strbuf.h\"\n+\n+/*\n+ * Test len limited functions for ANSI SGR lengths.\n+ */\n+void test_utf8_ansi__sgr_len(void)\n+{\n+\tcl_assert_equal_i(0, display_mode_esc_sequence_len_counted(\"\\033[m\", 2));\n+\tcl_assert_equal_i(3, display_mode_esc_sequence_len_counted(\"\\033[m\", 3));\n+\tcl_assert_equal_i(5, display_mode_esc_sequence_len_counted(\"\\033[31m\", 5));\n+\tcl_assert_equal_i(7, display_mode_esc_sequence_len_counted(\"\\033[0;31m\", 7));\n+}\n\nRegards,\n\nDirk\n\n> +\tif (end <= p || *p++ != '\\033')\n>  \t\treturn 0;\n> -\tif (*p++ != '[')\n> +\tif (end <= p || *p++ != '[')\n>  \t\treturn 0;\n> -\twhile (isdigit(*p) || *p == ';')\n> +\twhile (p < end && (isdigit(*p) || *p == ';'))\n>  \t\tp++;\n> -\tif (*p++ != 'm')\n> +\tif (end <= p || *p++ != 'm')\n>  \t\treturn 0;\n>  \treturn p - s;\n>  }\n>  \n> +/* Same as above, but for a NUL-terminated string \"s\". */\n> +size_t display_mode_esc_sequence_len(const char *s)\n> +{\n> +\treturn display_mode_esc_sequence_len_counted(s, strlen(s));\n> +}\n> +\n>  /* auxiliary function for binary search in interval table */\n>  static int bisearch(ucs_char_t ucs, const struct interval *table, int max)\n>  {\n> @@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)\n>  \t\tsize_t skip;\n>  \n>  \t\twhile (skip_ansi &&\n> -\t\t       (skip = display_mode_esc_sequence_len(string)) != 0)\n> +\t\t       (skip = display_mode_esc_sequence_len_counted(string,\n> +\t\t\t\t\t\t\torig + len - string)) != 0)\n>  \t\t\tstring += skip;\n>  \n>  \t\tglyph_width = utf8_width(&string, NULL);\n> @@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,\n>  {\n>  \tint indent, w, assume_utf8 = 1;\n>  \tconst char *bol, *space, *start = text;\n> +\tconst char *end = text + strlen(text);\n>  \tsize_t orig_len = buf->len;\n>  \n>  \tif (width <= 0) {\n> @@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,\n>  \t\tchar c;\n>  \t\tsize_t skip;\n>  \n> -\t\twhile ((skip = display_mode_esc_sequence_len(text)))\n> +\t\twhile ((skip = display_mode_esc_sequence_len_counted(text,\n> +\t\t\t\t\t\t\t\tend - text)))\n>  \t\t\ttext += skip;\n>  \n>  \t\tc = *text;\n> @@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,\n>  \t\tint glyph_width;\n>  \t\tsize_t n;\n>  \n> -\t\twhile ((n = display_mode_esc_sequence_len(src))) {\n> +\t\twhile ((n = display_mode_esc_sequence_len_counted(src,\n> +\t\t\t\t\t\t\t\t  end - src))) {\n>  \t\t\tstrbuf_add(&dst, src, n);\n>  \t\t\tsrc += n;\n>  \t\t}\n> diff --git a/utf8.h b/utf8.h\n> index cf8ecb0f21..57ca2ab270 100644\n> --- a/utf8.h\n> +++ b/utf8.h\n> @@ -6,6 +6,7 @@ struct strbuf;\n>  typedef unsigned int ucs_char_t;  /* assuming 32bit int */\n>  \n>  size_t display_mode_esc_sequence_len(const char *s);\n> +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len);\n>  int utf8_width(const char **start, size_t *remainder_p);\n>  int utf8_strnwidth(const char *string, size_t len, int skip_ansi);\n>  int utf8_strwidth(const char *string);\n\n"},{"id":"558652","messageId":"xmqqbj7ncexk.fsf@gitster.g","threadId":"66864","inReplyTo":"ghy0arkv39.fsf@gouders.net","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-17T21:43:03Z","receivedAt":"2026-11-17T21:43:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dirk Gouders <dirk@gouders.net> writes:\n\n> You have better knowledge of the users of this function: would an early\n> return perhaps handle a significant amount of cases?\n>\n>         /* The shortest SGR sequence is \"\\033[m\". */\n>         if (len < 3)\n>                 return 0;\n\nDo you have any benchmarks?  I don't.\n\nI find it of dubious performance merit to run this comparison for a\nlong string as it is eaten one UTF-8 characters at a time by the\ncaller, and when the length is actually shorter than 3, then\nrunnning the remainder of the function for a few steps should not be\n10x more expensive than the above comparison, no?\n\n> And, why not add some tests (the untested function I showed had\n> an off-by-one error):\n\nSounds good.  Should I just steal what you sent out?\n\n> diff --git a/Makefile b/Makefile\n> index 4c84eea6e2..3c8dc9ca40 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec\n>  CLAR_TEST_SUITES += u-trailer\n>  CLAR_TEST_SUITES += u-urlmatch-normalization\n>  CLAR_TEST_SUITES += u-utf8-width\n> +CLAR_TEST_SUITES += u-utf8-ansi\n>  CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)\n>  CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))\n>  CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o\n> diff --git a/t/meson.build b/t/meson.build\n> index 1cf1d95f70..e224df58c8 100644\n> --- a/t/meson.build\n> +++ b/t/meson.build\n> @@ -27,6 +27,7 @@ clar_test_suites = [\n>    'unit-tests/u-trailer.c',\n>    'unit-tests/u-urlmatch-normalization.c',\n>    'unit-tests/u-utf8-width.c',\n> +  'unit-tests/u-utf8-ansi.c',\n>  ]\n>  \n>  clar_sources = [\n> diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c\n> new file mode 100644\n> index 0000000000..c19d8bce78\n> --- /dev/null\n> +++ b/t/unit-tests/u-utf8-ansi.c\n> @@ -0,0 +1,14 @@\n> +#include \"unit-test.h\"\n> +#include \"utf8.h\"\n> +#include \"strbuf.h\"\n> +\n> +/*\n> + * Test len limited functions for ANSI SGR lengths.\n> + */\n> +void test_utf8_ansi__sgr_len(void)\n> +{\n> +\tcl_assert_equal_i(0, display_mode_esc_sequence_len_counted(\"\\033[m\", 2));\n> +\tcl_assert_equal_i(3, display_mode_esc_sequence_len_counted(\"\\033[m\", 3));\n> +\tcl_assert_equal_i(5, display_mode_esc_sequence_len_counted(\"\\033[31m\", 5));\n> +\tcl_assert_equal_i(7, display_mode_esc_sequence_len_counted(\"\\033[0;31m\", 7));\n> +}\n\nThanks.\n\n"},{"id":"558654","messageId":"ghtslfkte4.fsf@gouders.net","threadId":"66864","inReplyTo":"xmqqbj7ncexk.fsf@gitster.g","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Dirk Gouders","fromEmail":"dirk@gouders.net","sentAt":"2026-11-17T22:03:31Z","receivedAt":"2026-11-17T22:03:31Z","isPatch":true,"sender":{"key":"dirk@gouders.net","avatar":"https://avatars.githubusercontent.com/u/81326422?v=4"},"body":"On Tue, Nov 17 2026, Junio C Hamano wrote:\n\n> Dirk Gouders <dirk@gouders.net> writes:\n>\n>> You have better knowledge of the users of this function: would an early\n>> return perhaps handle a significant amount of cases?\n>>\n>>         /* The shortest SGR sequence is \"\\033[m\". */\n>>         if (len < 3)\n>>                 return 0;\n>\n> Do you have any benchmarks?  I don't.\n>\n> I find it of dubious performance merit to run this comparison for a\n> long string as it is eaten one UTF-8 characters at a time by the\n> caller, and when the length is actually shorter than 3, then\n> runnning the remainder of the function for a few steps should not be\n> 10x more expensive than the above comparison, no?\n\nMeanwhile, I thought: even if the test would be a good idea, the\n\"\\033\"-test probably is the one that handles most cases.\n\nMaybe I will look closer, but not now.  Correctness is more important\nand I need to work on v2.\n\n>> And, why not add some tests (the untested function I showed had\n>> an off-by-one error):\n>\n> Sounds good.  Should I just steal what you sent out?\n\nThat wouldn't be stealing, of course you can use it.\n\nRegards,\n\nDirk\n\n>> diff --git a/Makefile b/Makefile\n>> index 4c84eea6e2..3c8dc9ca40 100644\n>> --- a/Makefile\n>> +++ b/Makefile\n>> @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec\n>>  CLAR_TEST_SUITES += u-trailer\n>>  CLAR_TEST_SUITES += u-urlmatch-normalization\n>>  CLAR_TEST_SUITES += u-utf8-width\n>> +CLAR_TEST_SUITES += u-utf8-ansi\n>>  CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)\n>>  CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))\n>>  CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o\n>> diff --git a/t/meson.build b/t/meson.build\n>> index 1cf1d95f70..e224df58c8 100644\n>> --- a/t/meson.build\n>> +++ b/t/meson.build\n>> @@ -27,6 +27,7 @@ clar_test_suites = [\n>>    'unit-tests/u-trailer.c',\n>>    'unit-tests/u-urlmatch-normalization.c',\n>>    'unit-tests/u-utf8-width.c',\n>> +  'unit-tests/u-utf8-ansi.c',\n>>  ]\n>>  \n>>  clar_sources = [\n>> diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c\n>> new file mode 100644\n>> index 0000000000..c19d8bce78\n>> --- /dev/null\n>> +++ b/t/unit-tests/u-utf8-ansi.c\n>> @@ -0,0 +1,14 @@\n>> +#include \"unit-test.h\"\n>> +#include \"utf8.h\"\n>> +#include \"strbuf.h\"\n>> +\n>> +/*\n>> + * Test len limited functions for ANSI SGR lengths.\n>> + */\n>> +void test_utf8_ansi__sgr_len(void)\n>> +{\n>> +\tcl_assert_equal_i(0, display_mode_esc_sequence_len_counted(\"\\033[m\", 2));\n>> +\tcl_assert_equal_i(3, display_mode_esc_sequence_len_counted(\"\\033[m\", 3));\n>> +\tcl_assert_equal_i(5, display_mode_esc_sequence_len_counted(\"\\033[31m\", 5));\n>> +\tcl_assert_equal_i(7, display_mode_esc_sequence_len_counted(\"\\033[0;31m\", 7));\n>> +}\n>\n> Thanks.\n\n"},{"id":"558659","messageId":"CAHWeT-Zt=1MjD=fbOiAiQtb28Q4DAR_wF12iymXmeXzGyyLgbQ@mail.gmail.com","threadId":"66864","inReplyTo":"xmqqqzgjcjes.fsf@gitster.g","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Andy Koppe","fromEmail":"andy.koppe@gmail.com","sentAt":"2026-11-17T23:08:20Z","receivedAt":"2026-11-17T23:08:20Z","isPatch":true,"sender":{"key":"andy.koppe@gmail.com","avatar":"https://avatars.githubusercontent.com/u/223411?v=4"},"body":"Junio C Hamano wrote:\n\n> -       while (isdigit(*p) || *p == ';')\n> +       while (p < end && (isdigit(*p) || *p == ';'))\n\nWhile you're there, could you add colons (':') to that?\n\nECMA-48 §5.4.2 defines them as sub-separators within the\nsemicolon-separated parameters of control sequences. Colons are\nreferred to as \"3/10\" there, aka ASCII 0x3A:\n\nhttps://ecma-international.org/wp-content/uploads/ECMA-48_5th_edition_june_1991.pdf#page=26\n\nMaybe more practically relevant, xterm and its many\nmore-or-less-compatible brethren accept colons in SGR sequences:\n\n\"xterm allows either colons (standard) or semicolons (legacy) to\nseparate the subparameters (but after the first colon, colons must be\nused)\"\nhttps://invisible-island.net/xterm/ctlseqs/ctlseqs.html\n\nAndy\n\n"},{"id":"558662","messageId":"xmqq33szavmy.fsf@gitster.g","threadId":"66864","inReplyTo":"CAHWeT-Zt=1MjD=fbOiAiQtb28Q4DAR_wF12iymXmeXzGyyLgbQ@mail.gmail.com","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-17T23:25:09Z","receivedAt":"2026-11-17T23:25:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andy Koppe <andy.koppe@gmail.com> writes:\n\n> Junio C Hamano wrote:\n>\n>> -       while (isdigit(*p) || *p == ';')\n>> +       while (p < end && (isdigit(*p) || *p == ';'))\n>\n> While you're there, could you add colons (':') to that?\n\nDo we generate such a sequence that uses subseparators?\n\nIIRC, this code is not counting the display width of end-user\nsupplied string.  Rather, it gets string we paint in a buffer and\nthen counts how much display column it occupies.\n\n"},{"id":"558670","messageId":"xmqqqzgiapp2.fsf@gitster.g","threadId":"66864","inReplyTo":"ghtslfkte4.fsf@gouders.net","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-18T01:33:29Z","receivedAt":"2026-11-18T01:33:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dirk Gouders <dirk@gouders.net> writes:\n\n> Meanwhile, I thought: even if the test would be a good idea, the\n> \"\\033\"-test probably is the one that handles most cases.\n>\n> Maybe I will look closer, but not now.  Correctness is more important\n> and I need to work on v2.\n\nIt would be nice if you can take this patch over and make it your\n[PATCH v2 1/2].\n\nThanks.\n\n\n"},{"id":"558678","messageId":"CAHWeT-ZcOqm9PXzM5W18+ktJ4gBdPNyx+g_2ZCaW25ufKH4g5A@mail.gmail.com","threadId":"66864","inReplyTo":"xmqq33szavmy.fsf@gitster.g","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Andy Koppe","fromEmail":"andy.koppe@gmail.com","sentAt":"2026-11-18T06:59:07Z","receivedAt":"2026-11-18T06:59:07Z","isPatch":true,"sender":{"key":"andy.koppe@gmail.com","avatar":"https://avatars.githubusercontent.com/u/223411?v=4"},"body":"Junio C Hamano wrote:\n>\n> Andy Koppe <andy.koppe@gmail.com> writes:\n>\n> > Junio C Hamano wrote:\n> >\n> >> -       while (isdigit(*p) || *p == ';')\n> >> +       while (p < end && (isdigit(*p) || *p == ';'))\n> >\n> > While you're there, could you add colons (':') to that?\n>\n> Do we generate such a sequence that uses subseparators?\n>\n> IIRC, this code is not counting the display width of end-user\n> supplied string.  Rather, it gets string we paint in a buffer and\n> then counts how much display column it occupies.\n\nYou're right that Git itself doesn't generate them (although future\nenhancements might change that).\n\nHowever, display_mode_esc_sequence_len() already operates on expanded\ncustom formats and message text, e.g. in rewrap_message_tail() via\nstrbuf_wrap(). And on stdin in git-column. They could all contain SGR\nsequences with colons, whose width currently isn't ignored as\nintended. (And my proposed make_line_colors_self_contained() wouldn't\nreset or replay them.)\n\nThe check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.\n\nAndy\n\n"},{"id":"558679","messageId":"gh1p8ilhe6.fsf@gouders.net","threadId":"66864","inReplyTo":"xmqqqzgiapp2.fsf@gitster.g","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Dirk Gouders","fromEmail":"dirk@gouders.net","sentAt":"2026-11-18T07:37:21Z","receivedAt":"2026-11-18T07:37:21Z","isPatch":true,"sender":{"key":"dirk@gouders.net","avatar":"https://avatars.githubusercontent.com/u/81326422?v=4"},"body":"On Tue, Nov 17 2026, Junio C Hamano wrote:\n\n> Dirk Gouders <dirk@gouders.net> writes:\n>\n>> Meanwhile, I thought: even if the test would be a good idea, the\n>> \"\\033\"-test probably is the one that handles most cases.\n>>\n>> Maybe I will look closer, but not now.  Correctness is more important\n>> and I need to work on v2.\n>\n> It would be nice if you can take this patch over and make it your\n> [PATCH v2 1/2].\n\nOK, I will do so, thank you.\n\nRegards,\n\nDirk\n\n"},{"id":"558680","messageId":"ghwlqak2j9.fsf@gouders.net","threadId":"66864","inReplyTo":"CAHWeT-ZcOqm9PXzM5W18+ktJ4gBdPNyx+g_2ZCaW25ufKH4g5A@mail.gmail.com","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Dirk Gouders","fromEmail":"dirk@gouders.net","sentAt":"2026-11-18T07:43:38Z","receivedAt":"2026-11-18T07:43:38Z","isPatch":true,"sender":{"key":"dirk@gouders.net","avatar":"https://avatars.githubusercontent.com/u/81326422?v=4"},"body":"On Wed, Nov 18 2026, Andy Koppe wrote:\n\n> Junio C Hamano wrote:\n>>\n>> Andy Koppe <andy.koppe@gmail.com> writes:\n>>\n>> > Junio C Hamano wrote:\n>> >\n>> >> -       while (isdigit(*p) || *p == ';')\n>> >> +       while (p < end && (isdigit(*p) || *p == ';'))\n>> >\n>> > While you're there, could you add colons (':') to that?\n>>\n>> Do we generate such a sequence that uses subseparators?\n>>\n>> IIRC, this code is not counting the display width of end-user\n>> supplied string.  Rather, it gets string we paint in a buffer and\n>> then counts how much display column it occupies.\n>\n> You're right that Git itself doesn't generate them (although future\n> enhancements might change that).\n>\n> However, display_mode_esc_sequence_len() already operates on expanded\n> custom formats and message text, e.g. in rewrap_message_tail() via\n> strbuf_wrap(). And on stdin in git-column. They could all contain SGR\n> sequences with colons, whose width currently isn't ignored as\n> intended. (And my proposed make_line_colors_self_contained() wouldn't\n> reset or replay them.)\n>\n> The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.\n\nI would suggest to not mix these changes simply for bisection purposes.\n\nAnd as far as I know, we would need an FSM to identify valid sequences\nif we are going to allow colons.\n\nRegards,\n\nDirk\n\n"},{"id":"558691","messageId":"CAHWeT-Yh8yTnG7rLjvO+XBtU=bZHJ3WWoQQsPfvJO013P2GSPA@mail.gmail.com","threadId":"66864","inReplyTo":"ghwlqak2j9.fsf@gouders.net","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Andy Koppe","fromEmail":"andy.koppe@gmail.com","sentAt":"2026-11-18T09:48:02Z","receivedAt":"2026-11-18T09:48:02Z","isPatch":true,"sender":{"key":"andy.koppe@gmail.com","avatar":"https://avatars.githubusercontent.com/u/223411?v=4"},"body":"Dirk Gouders <wrote:\n>\n> On Wed, Nov 18 2026, Andy Koppe wrote:\n>\n> > Junio C Hamano wrote:\n> >>\n> >> Andy Koppe <andy.koppe@gmail.com> writes:\n> >>\n> >> > Junio C Hamano wrote:\n> >> >\n> >> >> -       while (isdigit(*p) || *p == ';')\n> >> >> +       while (p < end && (isdigit(*p) || *p == ';'))\n> >> >\n> >> > While you're there, could you add colons (':') to that?\n> >>\n> >> Do we generate such a sequence that uses subseparators?\n> >>\n> >> IIRC, this code is not counting the display width of end-user\n> >> supplied string.  Rather, it gets string we paint in a buffer and\n> >> then counts how much display column it occupies.\n> >\n> > You're right that Git itself doesn't generate them (although future\n> > enhancements might change that).\n> >\n> > However, display_mode_esc_sequence_len() already operates on expanded\n> > custom formats and message text, e.g. in rewrap_message_tail() via\n> > strbuf_wrap(). And on stdin in git-column. They could all contain SGR\n> > sequences with colons, whose width currently isn't ignored as\n> > intended. (And my proposed make_line_colors_self_contained() wouldn't\n> > reset or replay them.)\n> >\n> > The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.\n>\n> I would suggest to not mix these changes simply for bisection purposes.\n\nFair, and there's testing work as well of course. It was silly of me\nto suggest just sneaking that change in.\n\n> And as far as I know, we would need an FSM to identify valid sequences\n> if we are going to allow colons.\n\nNo. The parameter field can be any mix of digits, colons and\nsemicolons. Semicolons separate parameters, colons separate\nsubparameters, and individual (sub)parameters can be empty, which is\nequivalent to zero. If the field consists of nothing but colons,\nthat's one parameter with multiple empty subparameters.\n\nWhether any particular SGR sequence means anything to a terminal is\nanother matter of course, but git doesn't try to determine that for\ncolon-less sequences either, and nor should it. Well-behaved terminals\nignore valid sequences they don't understand.\n\nAndy\n\n"},{"id":"558735","messageId":"ghik1uassi.fsf@gouders.net","threadId":"66864","inReplyTo":"CAHWeT-Yh8yTnG7rLjvO+XBtU=bZHJ3WWoQQsPfvJO013P2GSPA@mail.gmail.com","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Dirk Gouders","fromEmail":"dirk@gouders.net","sentAt":"2026-11-18T18:38:53Z","receivedAt":"2026-11-18T18:38:53Z","isPatch":true,"sender":{"key":"dirk@gouders.net","avatar":"https://avatars.githubusercontent.com/u/81326422?v=4"},"body":"On Wed, Nov 18 2026, Andy Koppe wrote:\n\n>> Dirk Gouders <wrote:\n\n>> And as far as I know, we would need an FSM to identify valid sequences\n>> if we are going to allow colons.\n\n> No. The parameter field can be any mix of digits, colons and\n> semicolons.  Semicolons separate parameters, colons separate\n> subparameters, and individual (sub)parameters can be empty, which is\n> equivalent to zero. If the field consists of nothing but colons,\n> that's one parameter with multiple empty subparameters.\n\nMy statement was a bit misleading, because\ndisplay_mode_esc_sequence_len() already is an FSM\naccepting the language \"\\033\\[[0-9;]+m\".  My concern was that I doubt\nthat simply adding \":\" to the alphabet and as valid input to the state\n\"parameter\"  (without introducing one or more new states) would still\nlet it accept only valid SGR sequences.\n\nIt seems I can learn something: is \"\\033[:;m\" a valid SGR sequence?\n\nBut, such a discussion is difficult without defining what language you\nwant to accept.  I guess the two links were meant as a definition and I\nhave to confess I didn't yet read them.\n\nRegards,\n\nDirk\n\n"},{"id":"558741","messageId":"CAHWeT-ZjZHFE1BhRSwv6cvK4=rk0ZXNak8uG1y75Pgy+vNVQdQ@mail.gmail.com","threadId":"66864","inReplyTo":"ghik1uassi.fsf@gouders.net","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Andy Koppe","fromEmail":"andy.koppe@gmail.com","sentAt":"2026-11-18T19:12:34Z","receivedAt":"2026-11-18T19:12:34Z","isPatch":true,"sender":{"key":"andy.koppe@gmail.com","avatar":"https://avatars.githubusercontent.com/u/223411?v=4"},"body":"Dirk Gouders wrote:\n>\n> On Wed, Nov 18 2026, Andy Koppe wrote:\n\n> >> And as far as I know, we would need an FSM to identify valid sequences\n> >> if we are going to allow colons.\n>\n> > No. The parameter field can be any mix of digits, colons and\n> > semicolons.  Semicolons separate parameters, colons separate\n> > subparameters, and individual (sub)parameters can be empty, which is\n> > equivalent to zero. If the field consists of nothing but colons,\n> > that's one parameter with multiple empty subparameters.\n>\n> My statement was a bit misleading, because\n> display_mode_esc_sequence_len() already is an FSM\n> accepting the language \"\\033\\[[0-9;]+m\".  My concern was that I doubt\n> that simply adding \":\" to the alphabet and as valid input to the state\n> \"parameter\"  (without introducing one or more new states) would still\n> let it accept only valid SGR sequences.\n\n\"\\033\\[[0-9:;]*m\" is the complete regex for SGRs. (Of course, it's not\nthat simple if you wanted to actually collect the (sub)parameter\nvalues, but git doesn't need to.)\n\n> It seems I can learn something: is \"\\033[:;m\" a valid SGR sequence?\n\nYep. That's two parameters, with the first one consisting of two empty\nsubparameters, and the second one an empty parameter.\n\n> But, such a discussion is difficult without defining what language you\n> want to accept.  I guess the two links were meant as a definition and I\n> have to confess I didn't yet read them.\n\nTo be fair, ECMA-48 isn't the most readable, but once you realise that\n3/0 stands for 0x30 (aka '0') and 3/11 for 0x3B (aka ';'), it's not\ntoo bad.\n\nAndy\n\n"},{"id":"558744","messageId":"ghecciaqfo.fsf@gouders.net","threadId":"66864","inReplyTo":"CAHWeT-ZjZHFE1BhRSwv6cvK4=rk0ZXNak8uG1y75Pgy+vNVQdQ@mail.gmail.com","subject":"Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()","fromName":"Dirk Gouders","fromEmail":"dirk@gouders.net","sentAt":"2026-11-18T19:29:47Z","receivedAt":"2026-11-18T19:29:47Z","isPatch":true,"sender":{"key":"dirk@gouders.net","avatar":"https://avatars.githubusercontent.com/u/81326422?v=4"},"body":"On Wed, Nov 18 2026, Andy Koppe wrote:\n\n> Dirk Gouders wrote:\n>>\n>> On Wed, Nov 18 2026, Andy Koppe wrote:\n>\n>> >> And as far as I know, we would need an FSM to identify valid sequences\n>> >> if we are going to allow colons.\n>>\n>> > No. The parameter field can be any mix of digits, colons and\n>> > semicolons.  Semicolons separate parameters, colons separate\n>> > subparameters, and individual (sub)parameters can be empty, which is\n>> > equivalent to zero. If the field consists of nothing but colons,\n>> > that's one parameter with multiple empty subparameters.\n>>\n>> My statement was a bit misleading, because\n>> display_mode_esc_sequence_len() already is an FSM\n>> accepting the language \"\\033\\[[0-9;]+m\".  My concern was that I doubt\n>> that simply adding \":\" to the alphabet and as valid input to the state\n>> \"parameter\"  (without introducing one or more new states) would still\n>> let it accept only valid SGR sequences.\n>\n> \"\\033\\[[0-9:;]*m\" is the complete regex for SGRs. (Of course, it's not\n> that simple if you wanted to actually collect the (sub)parameter\n> values, but git doesn't need to.)\n>\n>> It seems I can learn something: is \"\\033[:;m\" a valid SGR sequence?\n>\n> Yep. That's two parameters, with the first one consisting of two empty\n> subparameters, and the second one an empty parameter.\n\nOK, I didn't know that, thank you.\n\n>> But, such a discussion is difficult without defining what language you\n>> want to accept.  I guess the two links were meant as a definition and I\n>> have to confess I didn't yet read them.\n>\n> To be fair, ECMA-48 isn't the most readable, but once you realise that\n> 3/0 stands for 0x30 (aka '0') and 3/11 for 0x3B (aka ';'), it's not\n> too bad.\n\nThat notation is fine for me, I will definitely read that document.\n\nRegards,\n\nDirk\n\n"}]}