threads / patch / 66864

patchutf8: make display_mode_esc_sequence_len() take a length

Subject: [PATCH] utf8: make display_mode_esc_sequence_len() take a length

## tl;dr

A bounds fix for display_mode_esc_sequence_len() has gone through several drafts, with Junio Hamano settling on a counted variant alongside a NUL-terminated wrapper. Read the story.

replies: 14people: 3as markdown or json

Junio C Hamano· Nov 17, 2026, 16:50 UTC · lore

display_mode_esc_sequence_len() scans for an ANSI display mode escape sequence (e.g. "ESC [ 1 ; 31 m"), relying on NUL termination to stop. Some callers, however, use a counted <ptr, len> buffer or need to stop at a specific position.

Teach it to take a buffer length and never read at or beyond ptr[len], returning the length of a complete sequence at the start or 0 otherwise.

Update the callers to pass the remaining byte count. For strbuf_add_wrapped_text(), which takes a NUL-terminated string, compute the end once upfront instead of calling strlen() each time.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 * Copy of https://lore.kernel.org/git/xmqqse0zfn2n.fsf@gitster.g/
   with a proposed commit log message.
 pretty.c |  3 ++-
 utf8.c   | 27 ++++++++++++++++++---------
 utf8.h   |  2 +-
 3 files changed, 21 insertions(+), 11 deletions(-)
Show changes to 3 files +21 −11

pretty.c, utf8.c, utf8.h

diff --git a/pretty.c b/pretty.c
index 0bc8ad8a9a..a20fd61909 100644
--- a/pretty.c
+++ b/pretty.c
@@ -1834,7 +1834,8 @@ static size_t format_and_pad_commit(struct strbuf *sb, /* in UTF-8 */
 			while (p > sb->buf && ch - p < 10 && *p != '\033')
 				p--;
 			if (*p != '\033' ||
-			    ch + 1 - p != display_mode_esc_sequence_len(p))
+			    ch + 1 - p != display_mode_esc_sequence_len(p,
+								ch + 1 - p))
 				break;
 			/*
 			 * got a good ansi sequence, put it back to
diff --git a/utf8.c b/utf8.c
index 96460cc414..544c5329f3 100644
--- a/utf8.c
+++ b/utf8.c
@@ -16,16 +16,23 @@ struct interval {
 	ucs_char_t last;
 };
 
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the "len" bytes
+ * of string "s", or 0 if the bytes do not begin with a complete
+ * such sequence.  Never looks at s[len] or beyond.
+ */
+size_t display_mode_esc_sequence_len(const char *s, size_t len)
 {
-	const char *p = s;
-	if (*p++ != '\033')
+	const char *p = s, *end = s + len;
+
+	if (end <= p || *p++ != '\033')
 		return 0;
-	if (*p++ != '[')
+	if (end <= p || *p++ != '[')
 		return 0;
-	while (isdigit(*p) || *p == ';')
+	while (p < end && (isdigit(*p) || *p == ';'))
 		p++;
-	if (*p++ != 'm')
+	if (end <= p || *p++ != 'm')
 		return 0;
 	return p - s;
 }
@@ -218,7 +225,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
 		size_t skip;
 
 		while (skip_ansi &&
-		       (skip = display_mode_esc_sequence_len(string)) != 0)
+		       (skip = display_mode_esc_sequence_len(string,
+							     orig + len - string)) != 0)
 			string += skip;
 
 		glyph_width = utf8_width(&string, NULL);
@@ -279,6 +287,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
 {
 	int indent, w, assume_utf8 = 1;
 	const char *bol, *space, *start = text;
+	const char *end = text + strlen(text);
 	size_t orig_len = buf->len;
 
 	if (width <= 0) {
@@ -299,7 +308,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
 		char c;
 		size_t skip;
 
-		while ((skip = display_mode_esc_sequence_len(text)))
+		while ((skip = display_mode_esc_sequence_len(text, end - text)))
 			text += skip;
 
 		c = *text;
@@ -378,7 +387,7 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
 		int glyph_width;
 		size_t n;
 
-		while ((n = display_mode_esc_sequence_len(src))) {
+		while ((n = display_mode_esc_sequence_len(src, end - src))) {
 			strbuf_add(&dst, src, n);
 			src += n;
 		}
diff --git a/utf8.h b/utf8.h
index cf8ecb0f21..fb2227a9e2 100644
--- a/utf8.h
+++ b/utf8.h
@@ -5,7 +5,7 @@ struct strbuf;
 
 typedef unsigned int ucs_char_t;  /* assuming 32bit int */
 
-size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len(const char *s, size_t len);
 int utf8_width(const char **start, size_t *remainder_p);
 int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
 int utf8_strwidth(const char *string);
-- 
2.56.1-740-gf38658f3d0
Junio C Hamano· Nov 17, 2026, 20:06 UTC · re: Junio C Hamano · lore

[PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

display_mode_esc_sequence_len() scans for an ANSI display mode escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string, relying on NUL termination to stop. Some callers, however, work on a counted buffer: utf8_strnwidth() takes <string, len> and must not scan past it, while strbuf_utf8_replace() and strbuf_add_wrapped_text() know (or can cheaply learn) the end.

Add display_mode_esc_sequence_len_counted() taking <ptr, len>, which never reads at or beyond ptr[len], and make display_mode_esc_sequence_len() a thin wrapper for NUL-terminated strings.

Use the counted variant in utf8_strnwidth() for correctness, and in strbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the end once upfront in the latter) to avoid measuring the remaining text in each iteration.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 * As there are new patches that want to add new callers to this
   function, let's keep the original with the same API while adding
   a new helper function that allows us to limit the scan.
 utf8.c | 35 ++++++++++++++++++++++++++---------
 utf8.h |  1 +
 2 files changed, 27 insertions(+), 9 deletions(-)
Show changes to 2 files +27 −9

utf8.c, utf8.h

diff --git a/utf8.c b/utf8.c
index 96460cc414..d0546ab31f 100644
--- a/utf8.c
+++ b/utf8.c
@@ -16,20 +16,33 @@ struct interval {
 	ucs_char_t last;
 };
 
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the "len" bytes
+ * of string "s", or 0 if the bytes do not begin with a complete
+ * such sequence.  Never looks at s[len] or beyond.
+ */
+size_t display_mode_esc_sequence_len_counted(const char *s, size_t len)
 {
-	const char *p = s;
-	if (*p++ != '\033')
+	const char *p = s, *end = s + len;
+
+	if (end <= p || *p++ != '\033')
 		return 0;
-	if (*p++ != '[')
+	if (end <= p || *p++ != '[')
 		return 0;
-	while (isdigit(*p) || *p == ';')
+	while (p < end && (isdigit(*p) || *p == ';'))
 		p++;
-	if (*p++ != 'm')
+	if (end <= p || *p++ != 'm')
 		return 0;
 	return p - s;
 }
 
+/* Same as above, but for a NUL-terminated string "s". */
+size_t display_mode_esc_sequence_len(const char *s)
+{
+	return display_mode_esc_sequence_len_counted(s, strlen(s));
+}
+
 /* auxiliary function for binary search in interval table */
 static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
 {
@@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
 		size_t skip;
 
 		while (skip_ansi &&
-		       (skip = display_mode_esc_sequence_len(string)) != 0)
+		       (skip = display_mode_esc_sequence_len_counted(string,
+							orig + len - string)) != 0)
 			string += skip;
 
 		glyph_width = utf8_width(&string, NULL);
@@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
 {
 	int indent, w, assume_utf8 = 1;
 	const char *bol, *space, *start = text;
+	const char *end = text + strlen(text);
 	size_t orig_len = buf->len;
 
 	if (width <= 0) {
@@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
 		char c;
 		size_t skip;
 
-		while ((skip = display_mode_esc_sequence_len(text)))
+		while ((skip = display_mode_esc_sequence_len_counted(text,
+								end - text)))
 			text += skip;
 
 		c = *text;
@@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
 		int glyph_width;
 		size_t n;
 
-		while ((n = display_mode_esc_sequence_len(src))) {
+		while ((n = display_mode_esc_sequence_len_counted(src,
+								  end - src))) {
 			strbuf_add(&dst, src, n);
 			src += n;
 		}
diff --git a/utf8.h b/utf8.h
index cf8ecb0f21..57ca2ab270 100644
--- a/utf8.h
+++ b/utf8.h
@@ -6,6 +6,7 @@ struct strbuf;
 typedef unsigned int ucs_char_t;  /* assuming 32bit int */
 
 size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len_counted(const char *s, size_t len);
 int utf8_width(const char **start, size_t *remainder_p);
 int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
 int utf8_strwidth(const char *string);
-- 
2.56.1-740-gf38658f3d0
Dirk Gouders· Nov 17, 2026, 21:26 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

On Tue, Nov 17 2026, Junio C Hamano wrote:
Show 47 quoted lines
> display_mode_esc_sequence_len() scans for an ANSI display mode
> escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string,
> relying on NUL termination to stop.  Some callers, however, work on
> a counted buffer: utf8_strnwidth() takes <string, len> and must not
> scan past it, while strbuf_utf8_replace() and
> strbuf_add_wrapped_text() know (or can cheaply learn) the end.
>
> Add display_mode_esc_sequence_len_counted() taking <ptr, len>,
> which never reads at or beyond ptr[len], and make
> display_mode_esc_sequence_len() a thin wrapper for NUL-terminated
> strings.
>
> Use the counted variant in utf8_strnwidth() for correctness, and in
> strbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the
> end once upfront in the latter) to avoid measuring the remaining
> text in each iteration.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
>  * As there are new patches that want to add new callers to this
>    function, let's keep the original with the same API while adding
>    a new helper function that allows us to limit the scan.
>
>  utf8.c | 35 ++++++++++++++++++++++++++---------
>  utf8.h |  1 +
>  2 files changed, 27 insertions(+), 9 deletions(-)
>
> diff --git a/utf8.c b/utf8.c
> index 96460cc414..d0546ab31f 100644
> --- a/utf8.c
> +++ b/utf8.c
> @@ -16,20 +16,33 @@ struct interval {
>  	ucs_char_t last;
>  };
>  
> -size_t display_mode_esc_sequence_len(const char *s)
> +/*
> + * Return the length of the ANSI display mode escape sequence (e.g.,
> + * "\033[1;31m") that appears at the beginning of the "len" bytes
> + * of string "s", or 0 if the bytes do not begin with a complete
> + * such sequence.  Never looks at s[len] or beyond.
> + */
> +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len)
>  {
> -	const char *p = s;
> -	if (*p++ != '\033')
> +	const char *p = s, *end = s + len;

You have better knowledge of the users of this function: would an early return perhaps handle a significant amount of cases?

        /* The shortest SGR sequence is "\033[m". */
        if (len < 3)
                return 0;

And, why not add some tests (the untested function I showed had an off-by-one error):

Show changes to 3 files +16 −0

Makefile, t/meson.build, t/unit-tests/u-utf8-ansi.c

diff --git a/Makefile b/Makefile
index 4c84eea6e2..3c8dc9ca40 100644
--- a/Makefile
+++ b/Makefile
@@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec
 CLAR_TEST_SUITES += u-trailer
 CLAR_TEST_SUITES += u-urlmatch-normalization
 CLAR_TEST_SUITES += u-utf8-width
+CLAR_TEST_SUITES += u-utf8-ansi
 CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
 CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
 CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
diff --git a/t/meson.build b/t/meson.build
index 1cf1d95f70..e224df58c8 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -27,6 +27,7 @@ clar_test_suites = [
   'unit-tests/u-trailer.c',
   'unit-tests/u-urlmatch-normalization.c',
   'unit-tests/u-utf8-width.c',
+  'unit-tests/u-utf8-ansi.c',
 ]
 
 clar_sources = [
diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
new file mode 100644
index 0000000000..c19d8bce78
--- /dev/null
+++ b/t/unit-tests/u-utf8-ansi.c
@@ -0,0 +1,14 @@
+#include "unit-test.h"
+#include "utf8.h"
+#include "strbuf.h"
+
+/*
+ * Test len limited functions for ANSI SGR lengths.
+ */
+void test_utf8_ansi__sgr_len(void)
+{
+	cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2));
+	cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3));
+	cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5));
+	cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7));
+}

Regards,

Dirk

> +	if (end <= p || *p++ != '\033')
>  		return 0;
> -	if (*p++ != '[')
> +	if (end <= p || *p++ != '[')
>  		return 0;
> -	while (isdigit(*p) || *p == ';')
> +	while (p < end && (isdigit(*p) || *p == ';'))
>  		p++;
> -	if (*p++ != 'm')
> +	if (end <= p || *p++ != 'm')
>  		return 0;
>  	return p - s;
>  }
>  
> +/* Same as above, but for a NUL-terminated string "s". */
> +size_t display_mode_esc_sequence_len(const char *s)
> +{
> +	return display_mode_esc_sequence_len_counted(s, strlen(s));
> +}
> +
>  /* auxiliary function for binary search in interval table */
>  static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
>  {
> @@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
>  		size_t skip;
>  
>  		while (skip_ansi &&
> -		       (skip = display_mode_esc_sequence_len(string)) != 0)
> +		       (skip = display_mode_esc_sequence_len_counted(string,
> +							orig + len - string)) != 0)
>  			string += skip;
>  
>  		glyph_width = utf8_width(&string, NULL);
> @@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
>  {
>  	int indent, w, assume_utf8 = 1;
>  	const char *bol, *space, *start = text;
> +	const char *end = text + strlen(text);
>  	size_t orig_len = buf->len;
>  
>  	if (width <= 0) {
> @@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
>  		char c;
>  		size_t skip;
>  
> -		while ((skip = display_mode_esc_sequence_len(text)))
> +		while ((skip = display_mode_esc_sequence_len_counted(text,
> +								end - text)))
>  			text += skip;
>  
>  		c = *text;
> @@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
>  		int glyph_width;
>  		size_t n;
>  
> -		while ((n = display_mode_esc_sequence_len(src))) {
> +		while ((n = display_mode_esc_sequence_len_counted(src,
> +								  end - src))) {
>  			strbuf_add(&dst, src, n);
>  			src += n;
>  		}
> diff --git a/utf8.h b/utf8.h
> index cf8ecb0f21..57ca2ab270 100644
> --- a/utf8.h
> +++ b/utf8.h
> @@ -6,6 +6,7 @@ struct strbuf;
>  typedef unsigned int ucs_char_t;  /* assuming 32bit int */
>  
>  size_t display_mode_esc_sequence_len(const char *s);
> +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len);
>  int utf8_width(const char **start, size_t *remainder_p);
>  int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
>  int utf8_strwidth(const char *string);
Junio C Hamano· Nov 17, 2026, 21:43 UTC · re: Dirk Gouders · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

Dirk Gouders <dirk@gouders.net> writes:
Show 6 quoted lines
> You have better knowledge of the users of this function: would an early
> return perhaps handle a significant amount of cases?
>
>         /* The shortest SGR sequence is "\033[m". */
>         if (len < 3)
>                 return 0;
Do you have any benchmarks?  I don't.

I find it of dubious performance merit to run this comparison for a long string as it is eaten one UTF-8 characters at a time by the caller, and when the length is actually shorter than 3, then runnning the remainder of the function for a few steps should not be 10x more expensive than the above comparison, no?

> And, why not add some tests (the untested function I showed had
> an off-by-one error):
Sounds good.  Should I just steal what you sent out?
Show 44 quoted lines
> diff --git a/Makefile b/Makefile
> index 4c84eea6e2..3c8dc9ca40 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec
>  CLAR_TEST_SUITES += u-trailer
>  CLAR_TEST_SUITES += u-urlmatch-normalization
>  CLAR_TEST_SUITES += u-utf8-width
> +CLAR_TEST_SUITES += u-utf8-ansi
>  CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
>  CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
>  CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
> diff --git a/t/meson.build b/t/meson.build
> index 1cf1d95f70..e224df58c8 100644
> --- a/t/meson.build
> +++ b/t/meson.build
> @@ -27,6 +27,7 @@ clar_test_suites = [
>    'unit-tests/u-trailer.c',
>    'unit-tests/u-urlmatch-normalization.c',
>    'unit-tests/u-utf8-width.c',
> +  'unit-tests/u-utf8-ansi.c',
>  ]
>  
>  clar_sources = [
> diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
> new file mode 100644
> index 0000000000..c19d8bce78
> --- /dev/null
> +++ b/t/unit-tests/u-utf8-ansi.c
> @@ -0,0 +1,14 @@
> +#include "unit-test.h"
> +#include "utf8.h"
> +#include "strbuf.h"
> +
> +/*
> + * Test len limited functions for ANSI SGR lengths.
> + */
> +void test_utf8_ansi__sgr_len(void)
> +{
> +	cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2));
> +	cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3));
> +	cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5));
> +	cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7));
> +}
Thanks.
Dirk Gouders· Nov 17, 2026, 22:03 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

On Tue, Nov 17 2026, Junio C Hamano wrote:
Show 16 quoted lines
> Dirk Gouders <dirk@gouders.net> writes:
>
>> You have better knowledge of the users of this function: would an early
>> return perhaps handle a significant amount of cases?
>>
>>         /* The shortest SGR sequence is "\033[m". */
>>         if (len < 3)
>>                 return 0;
>
> Do you have any benchmarks?  I don't.
>
> I find it of dubious performance merit to run this comparison for a
> long string as it is eaten one UTF-8 characters at a time by the
> caller, and when the length is actually shorter than 3, then
> runnning the remainder of the function for a few steps should not be
> 10x more expensive than the above comparison, no?

Meanwhile, I thought: even if the test would be a good idea, the "\033"-test probably is the one that handles most cases.

Maybe I will look closer, but not now. Correctness is more important and I need to work on v2.

>> And, why not add some tests (the untested function I showed had
>> an off-by-one error):
>
> Sounds good.  Should I just steal what you sent out?
That wouldn't be stealing, of course you can use it.
Regards,
Dirk
Show 46 quoted lines
>> diff --git a/Makefile b/Makefile
>> index 4c84eea6e2..3c8dc9ca40 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec
>>  CLAR_TEST_SUITES += u-trailer
>>  CLAR_TEST_SUITES += u-urlmatch-normalization
>>  CLAR_TEST_SUITES += u-utf8-width
>> +CLAR_TEST_SUITES += u-utf8-ansi
>>  CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
>>  CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
>>  CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
>> diff --git a/t/meson.build b/t/meson.build
>> index 1cf1d95f70..e224df58c8 100644
>> --- a/t/meson.build
>> +++ b/t/meson.build
>> @@ -27,6 +27,7 @@ clar_test_suites = [
>>    'unit-tests/u-trailer.c',
>>    'unit-tests/u-urlmatch-normalization.c',
>>    'unit-tests/u-utf8-width.c',
>> +  'unit-tests/u-utf8-ansi.c',
>>  ]
>>  
>>  clar_sources = [
>> diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
>> new file mode 100644
>> index 0000000000..c19d8bce78
>> --- /dev/null
>> +++ b/t/unit-tests/u-utf8-ansi.c
>> @@ -0,0 +1,14 @@
>> +#include "unit-test.h"
>> +#include "utf8.h"
>> +#include "strbuf.h"
>> +
>> +/*
>> + * Test len limited functions for ANSI SGR lengths.
>> + */
>> +void test_utf8_ansi__sgr_len(void)
>> +{
>> +	cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2));
>> +	cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3));
>> +	cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5));
>> +	cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7));
>> +}
>
> Thanks.
Junio C Hamano· Nov 18, 2026, 01:33 UTC · re: Dirk Gouders · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

Dirk Gouders <dirk@gouders.net> writes:
Show 5 quoted lines
> Meanwhile, I thought: even if the test would be a good idea, the
> "\033"-test probably is the one that handles most cases.
>
> Maybe I will look closer, but not now.  Correctness is more important
> and I need to work on v2.

It would be nice if you can take this patch over and make it your [PATCH v2 1/2].

Thanks.
Dirk Gouders· Nov 18, 2026, 07:37 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

On Tue, Nov 17 2026, Junio C Hamano wrote:
Show 10 quoted lines
> Dirk Gouders <dirk@gouders.net> writes:
>
>> Meanwhile, I thought: even if the test would be a good idea, the
>> "\033"-test probably is the one that handles most cases.
>>
>> Maybe I will look closer, but not now.  Correctness is more important
>> and I need to work on v2.
>
> It would be nice if you can take this patch over and make it your
> [PATCH v2 1/2].
OK, I will do so, thank you.
Regards,
Dirk
Andy Koppe· Nov 17, 2026, 23:08 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

Junio C Hamano wrote:
> -       while (isdigit(*p) || *p == ';')
> +       while (p < end && (isdigit(*p) || *p == ';'))
While you're there, could you add colons (':') to that?

ECMA-48 §5.4.2 defines them as sub-separators within the semicolon-separated parameters of control sequences. Colons are referred to as "3/10" there, aka ASCII 0x3A:

https://ecma-international.org/wp-content/uploads/ECMA-48_5th_edition_june_1991.pdf#page=26

Maybe more practically relevant, xterm and its many more-or-less-compatible brethren accept colons in SGR sequences:

"xterm allows either colons (standard) or semicolons (legacy) to separate the subparameters (but after the first colon, colons must be used)" https://invisible-island.net/xterm/ctlseqs/ctlseqs.html

Andy
Junio C Hamano· Nov 17, 2026, 23:25 UTC · re: Andy Koppe · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

Andy Koppe <andy.koppe@gmail.com> writes:
Show 6 quoted lines
> Junio C Hamano wrote:
>
>> -       while (isdigit(*p) || *p == ';')
>> +       while (p < end && (isdigit(*p) || *p == ';'))
>
> While you're there, could you add colons (':') to that?
Do we generate such a sequence that uses subseparators?

IIRC, this code is not counting the display width of end-user supplied string. Rather, it gets string we paint in a buffer and then counts how much display column it occupies.

Andy Koppe· Nov 18, 2026, 06:59 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

Junio C Hamano wrote:
Show 15 quoted lines
>
> Andy Koppe <andy.koppe@gmail.com> writes:
>
> > Junio C Hamano wrote:
> >
> >> -       while (isdigit(*p) || *p == ';')
> >> +       while (p < end && (isdigit(*p) || *p == ';'))
> >
> > While you're there, could you add colons (':') to that?
>
> Do we generate such a sequence that uses subseparators?
>
> IIRC, this code is not counting the display width of end-user
> supplied string.  Rather, it gets string we paint in a buffer and
> then counts how much display column it occupies.

You're right that Git itself doesn't generate them (although future enhancements might change that).

However, display_mode_esc_sequence_len() already operates on expanded custom formats and message text, e.g. in rewrap_message_tail() via strbuf_wrap(). And on stdin in git-column. They could all contain SGR sequences with colons, whose width currently isn't ignored as intended. (And my proposed make_line_colors_self_contained() wouldn't reset or replay them.)

The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.
Andy
Dirk Gouders· Nov 18, 2026, 07:43 UTC · re: Andy Koppe · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

On Wed, Nov 18 2026, Andy Koppe wrote:
Show 28 quoted lines
> Junio C Hamano wrote:
>>
>> Andy Koppe <andy.koppe@gmail.com> writes:
>>
>> > Junio C Hamano wrote:
>> >
>> >> -       while (isdigit(*p) || *p == ';')
>> >> +       while (p < end && (isdigit(*p) || *p == ';'))
>> >
>> > While you're there, could you add colons (':') to that?
>>
>> Do we generate such a sequence that uses subseparators?
>>
>> IIRC, this code is not counting the display width of end-user
>> supplied string.  Rather, it gets string we paint in a buffer and
>> then counts how much display column it occupies.
>
> You're right that Git itself doesn't generate them (although future
> enhancements might change that).
>
> However, display_mode_esc_sequence_len() already operates on expanded
> custom formats and message text, e.g. in rewrap_message_tail() via
> strbuf_wrap(). And on stdin in git-column. They could all contain SGR
> sequences with colons, whose width currently isn't ignored as
> intended. (And my proposed make_line_colors_self_contained() wouldn't
> reset or replay them.)
>
> The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.
I would suggest to not mix these changes simply for bisection purposes.

And as far as I know, we would need an FSM to identify valid sequences if we are going to allow colons.

Regards,
Dirk
Andy Koppe· Nov 18, 2026, 09:48 UTC · re: Dirk Gouders · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

Dirk Gouders <wrote:
Show 33 quoted lines
>
> On Wed, Nov 18 2026, Andy Koppe wrote:
>
> > Junio C Hamano wrote:
> >>
> >> Andy Koppe <andy.koppe@gmail.com> writes:
> >>
> >> > Junio C Hamano wrote:
> >> >
> >> >> -       while (isdigit(*p) || *p == ';')
> >> >> +       while (p < end && (isdigit(*p) || *p == ';'))
> >> >
> >> > While you're there, could you add colons (':') to that?
> >>
> >> Do we generate such a sequence that uses subseparators?
> >>
> >> IIRC, this code is not counting the display width of end-user
> >> supplied string.  Rather, it gets string we paint in a buffer and
> >> then counts how much display column it occupies.
> >
> > You're right that Git itself doesn't generate them (although future
> > enhancements might change that).
> >
> > However, display_mode_esc_sequence_len() already operates on expanded
> > custom formats and message text, e.g. in rewrap_message_tail() via
> > strbuf_wrap(). And on stdin in git-column. They could all contain SGR
> > sequences with colons, whose width currently isn't ignored as
> > intended. (And my proposed make_line_colors_self_contained() wouldn't
> > reset or replay them.)
> >
> > The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.
>
> I would suggest to not mix these changes simply for bisection purposes.

Fair, and there's testing work as well of course. It was silly of me to suggest just sneaking that change in.

> And as far as I know, we would need an FSM to identify valid sequences
> if we are going to allow colons.

No. The parameter field can be any mix of digits, colons and semicolons. Semicolons separate parameters, colons separate subparameters, and individual (sub)parameters can be empty, which is equivalent to zero. If the field consists of nothing but colons, that's one parameter with multiple empty subparameters.

Whether any particular SGR sequence means anything to a terminal is another matter of course, but git doesn't try to determine that for colon-less sequences either, and nor should it. Well-behaved terminals ignore valid sequences they don't understand.

Andy
Dirk Gouders· Nov 18, 2026, 18:38 UTC · re: Andy Koppe · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

On Wed, Nov 18 2026, Andy Koppe wrote:
>> Dirk Gouders <wrote:
>> And as far as I know, we would need an FSM to identify valid sequences
>> if we are going to allow colons.
Show 5 quoted lines
> No. The parameter field can be any mix of digits, colons and
> semicolons.  Semicolons separate parameters, colons separate
> subparameters, and individual (sub)parameters can be empty, which is
> equivalent to zero. If the field consists of nothing but colons,
> that's one parameter with multiple empty subparameters.

My statement was a bit misleading, because display_mode_esc_sequence_len() already is an FSM accepting the language "\033\[[0-9;]+m". My concern was that I doubt that simply adding ":" to the alphabet and as valid input to the state "parameter" (without introducing one or more new states) would still let it accept only valid SGR sequences.

It seems I can learn something: is "\033[:;m" a valid SGR sequence?

But, such a discussion is difficult without defining what language you want to accept. I guess the two links were meant as a definition and I have to confess I didn't yet read them.

Regards,
Dirk
Andy Koppe· Nov 18, 2026, 19:12 UTC · re: Dirk Gouders · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

Dirk Gouders wrote:
>
> On Wed, Nov 18 2026, Andy Koppe wrote:
Show 15 quoted lines
> >> And as far as I know, we would need an FSM to identify valid sequences
> >> if we are going to allow colons.
>
> > No. The parameter field can be any mix of digits, colons and
> > semicolons.  Semicolons separate parameters, colons separate
> > subparameters, and individual (sub)parameters can be empty, which is
> > equivalent to zero. If the field consists of nothing but colons,
> > that's one parameter with multiple empty subparameters.
>
> My statement was a bit misleading, because
> display_mode_esc_sequence_len() already is an FSM
> accepting the language "\033\[[0-9;]+m".  My concern was that I doubt
> that simply adding ":" to the alphabet and as valid input to the state
> "parameter"  (without introducing one or more new states) would still
> let it accept only valid SGR sequences.

"\033\[[0-9:;]*m" is the complete regex for SGRs. (Of course, it's not that simple if you wanted to actually collect the (sub)parameter values, but git doesn't need to.)

> It seems I can learn something: is "\033[:;m" a valid SGR sequence?

Yep. That's two parameters, with the first one consisting of two empty subparameters, and the second one an empty parameter.

> But, such a discussion is difficult without defining what language you
> want to accept.  I guess the two links were meant as a definition and I
> have to confess I didn't yet read them.

To be fair, ECMA-48 isn't the most readable, but once you realise that 3/0 stands for 0x30 (aka '0') and 3/11 for 0x3B (aka ';'), it's not too bad.

Andy
Dirk Gouders· Nov 18, 2026, 19:29 UTC · re: Andy Koppe · lore

Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()

On Wed, Nov 18 2026, Andy Koppe wrote:
Show 28 quoted lines
> Dirk Gouders wrote:
>>
>> On Wed, Nov 18 2026, Andy Koppe wrote:
>
>> >> And as far as I know, we would need an FSM to identify valid sequences
>> >> if we are going to allow colons.
>>
>> > No. The parameter field can be any mix of digits, colons and
>> > semicolons.  Semicolons separate parameters, colons separate
>> > subparameters, and individual (sub)parameters can be empty, which is
>> > equivalent to zero. If the field consists of nothing but colons,
>> > that's one parameter with multiple empty subparameters.
>>
>> My statement was a bit misleading, because
>> display_mode_esc_sequence_len() already is an FSM
>> accepting the language "\033\[[0-9;]+m".  My concern was that I doubt
>> that simply adding ":" to the alphabet and as valid input to the state
>> "parameter"  (without introducing one or more new states) would still
>> let it accept only valid SGR sequences.
>
> "\033\[[0-9:;]*m" is the complete regex for SGRs. (Of course, it's not
> that simple if you wanted to actually collect the (sub)parameter
> values, but git doesn't need to.)
>
>> It seems I can learn something: is "\033[:;m" a valid SGR sequence?
>
> Yep. That's two parameters, with the first one consisting of two empty
> subparameters, and the second one an empty parameter.
OK, I didn't know that, thank you.
Show 7 quoted lines
>> But, such a discussion is difficult without defining what language you
>> want to accept.  I guess the two links were meant as a definition and I
>> have to confess I didn't yet read them.
>
> To be fair, ECMA-48 isn't the most readable, but once you realise that
> 3/0 stands for 0x30 (aka '0') and 3/11 for 0x3B (aka ';'), it's not
> too bad.
That notation is fine for me, I will definitely read that document.
Regards,
Dirk

← back to recent threads