{"thread":{"id":"65406","subject":"[PATCH 0/2] t: work around bugs in Dash v0.5.13","startedAt":"2026-04-01T10:43:04Z","lastAt":"2026-05-07T07:37:37Z","messageCount":10,"participants":["Patrick Steinhardt","Eric Sunshine","Herbert Xu"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"540627","messageId":"20260401-pks-tests-with-dash-v1-0-d70b5040aa5d@pks.im","threadId":"65406","inReplyTo":null,"subject":"[PATCH 0/2] t: work around bugs in Dash v0.5.13","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-04-01T10:42:57Z","receivedAt":"2026-04-01T10:43:04Z","isPatch":true,"body":"Hi,\n\nwhile testing my `set -e` change for our test suite I was also playing\naround with Dash v0.5.13.1 and noticed that multiple of our tests fail\nwith that version. As it turns out, some of these test failures are\nactual bugs in Dash itself that have been introduced in v0.5.13.\n\nThere's ultimately two bugs:\n\n - Dash inserts CTLMBCHAR byte sequences into heredocs around multibyte\n characters. This bug still exists with the current \"master\" branch\n of Dash.\n\n - Dash may swallow some bytes when reading data, which has already\n been fixed.\n\nI've Cc'd Herbert, maintainer of Dash.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (2):\n t: work around multibyte bug in quoted heredocs with Dash v0.5.13\n t9300: work around partial read bug in Dash v0.5.13\n\n t/t0300-credentials.sh | 2 +-\n t/t3430-rebase-merges.sh | 4 ++--\n t/t3902-quoted.sh | 14 +++++++-------\n t/t4014-format-patch.sh | 8 ++++----\n t/t4201-shortlog.sh | 2 +-\n t/t9001-send-email.sh | 6 +++---\n t/t9300-fast-import.sh | 32 ++++++++++++++------------------\n 7 files changed, 32 insertions(+), 36 deletions(-)\n\n\n---\nbase-commit: 270e10ad6dda3379ea0da7efd11e4fbf2cd7a325\nchange-id: 20260401-pks-tests-with-dash-ea71cbb36958\n\n"},{"id":"540628","messageId":"20260401-pks-tests-with-dash-v1-1-d70b5040aa5d@pks.im","threadId":"65406","inReplyTo":"20260401-pks-tests-with-dash-v1-0-d70b5040aa5d@pks.im","subject":"[PATCH 1/2] t: work around multibyte bug in quoted heredocs with Dash v0.5.13","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-04-01T10:42:58Z","receivedAt":"2026-04-01T10:43:05Z","isPatch":true,"body":"When executing our test suite with Dash v0.5.13.2 one can observe\nseveral test failures that all have the same symptoms: we have a quoted\nheredoc that contains multibyte characters, but the final data does not\nmatch what we actually wanted to write. One such example is in t0300,\nwhere we see the diffs like the following:\n\n --- expect-stdout\t2026-04-01 07:25:45.249919440 +0000\n +++ stdout\t2026-04-01 07:25:45.254919509 +0000\n @@ -1,5 +1,5 @@\n protocol=https\n host=example.com\n -path=perú.git\n +path=perú.git\n username=foo\n password=bar\n\nWhile seemingly the same, the data that we've written via the heredoc\ncontains some invisible bytes. The expected hex representation of the\nstring is:\n\n 7065 72c3 ba2e 6769 74 per...git\n\nBut what we actually get instead is this string:\n\n 7065 7285 02c3 ba02 852e 6769 74 per.......git\n\nWhat's important to note here is that the multibyte character exists in\nboth versions. But in the broken version we see that the bytes are\nwrapped in a sequence of \"85 02\" and \"02 85\". This is the CTLMBCHAR byte\nsequence of Dash, which it uses internally to quote multibyte sequences.\n\nAs it turns out, this bug was introduced in c5bf970 (expand: Add\nmulti-byte support to pmatch, 2024-06-02), which adds multibyte support\nto more contexts of Dash. One of these contexts seems to be in heredocs,\nand Dash _does_ correctly unquote these multibyte sequences when using\nan unquoted heredoc. But the bug seems to be that this unquoting does\nnot happen in quoted heredocs, and the bug still exists on the latest\n\"master\" branch.\n\nFor now, work around the bug by using unquoted heredocs instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0300-credentials.sh | 2 +-\n t/t3430-rebase-merges.sh | 4 ++--\n t/t3902-quoted.sh | 14 +++++++-------\n t/t4014-format-patch.sh | 8 ++++----\n t/t4201-shortlog.sh | 2 +-\n t/t9001-send-email.sh | 6 +++---\n 6 files changed, 18 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh\nindex 07aa834d33..fda6760955 100755\n--- a/t/t0300-credentials.sh\n+++ b/t/t0300-credentials.sh\n@@ -675,7 +675,7 @@ test_expect_success 'match percent-encoded values' '\n test_expect_success 'match percent-encoded UTF-8 values in path' '\n \ttest_config credential.https://example.com.useHttpPath true &&\n \ttest_config credential.https://example.com/perú.git.helper \"$HELPER\" &&\n-\tcheck fill <<-\\EOF\n+\tcheck fill <<-EOF\n \turl=https://example.com/per%C3%BA.git\n \t--\n \tprotocol=https\ndiff --git a/t/t3430-rebase-merges.sh b/t/t3430-rebase-merges.sh\nindex cc627e34a7..d3dffbb830 100755\n--- a/t/t3430-rebase-merges.sh\n+++ b/t/t3430-rebase-merges.sh\n@@ -507,9 +507,9 @@ test_expect_success 'octopus merges' '\n \tgit rebase -i --force-rebase -r HEAD^^ &&\n \ttest \"Hank\" = \"$(git show -s --format=%an HEAD)\" &&\n \ttest \"$before\" != $(git rev-parse HEAD) &&\n-\ttest_cmp_graph HEAD^^.. <<-\\EOF\n+\ttest_cmp_graph HEAD^^.. <<-EOF\n \t*-. Tüntenfüsch\n-\t|\\ \\\n+\t|\\\\ \\\\\n \t| | * three\n \t| * | two\n \t| |/\ndiff --git a/t/t3902-quoted.sh b/t/t3902-quoted.sh\nindex f528008c36..c3b45d991e 100755\n--- a/t/t3902-quoted.sh\n+++ b/t/t3902-quoted.sh\n@@ -60,16 +60,16 @@ With SP in it\n \"\\346\\277\\261\\351\\207\\216\\347\\264\\224\"\n EOF\n \n-cat >expect.raw <<\\EOF\n+cat >expect.raw <expect <<'EOF'\n+cat >expect <\n@@ -1300,7 +1300,7 @@ test_expect_success 'format-patch wraps extremely long from-header (non-ASCII wi\n \ttest_cmp expect actual\n '\n \n-cat >expect <<'EOF'\n+cat >expect <expect <<'EOF'\n+cat >expect <patch &&\n-\tcat >expect <<-\\EOF &&\n+\tcat >expect <<-EOF &&\n \tFrom: C O Mitter \n \tContent-Type: text/plain; charset=UTF-8\n \ndiff --git a/t/t4201-shortlog.sh b/t/t4201-shortlog.sh\nindex 5f23fc147b..d73c9f5204 100755\n--- a/t/t4201-shortlog.sh\n+++ b/t/t4201-shortlog.sh\n@@ -105,7 +105,7 @@ test_expect_success 'output from user-defined format is re-wrapped' '\n '\n \n test_expect_success !MINGW,ICONV 'shortlog wrapping' '\n-\tcat >expect <<\\EOF &&\n+\tcat >expect <email-using-8bit <<\\EOF\n+cat >email-using-8bit <\n From: author@example.com\n@@ -1735,7 +1735,7 @@ test_expect_success $PREREQ '--8bit-encoding overrides sendemail.8bitEncoding' '\n '\n \n test_expect_success $PREREQ 'setup expect' '\n-\tcat >email-using-8bit <<-\\EOF\n+\tcat >email-using-8bit <<-EOF\n \tFrom fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001\n \tMessage-ID: \n \tFrom: author@example.com\n@@ -1764,7 +1764,7 @@ test_expect_success $PREREQ '--8bit-encoding also treats subject' '\n '\n \n test_expect_success $PREREQ 'setup expect' '\n-\tcat >email-using-8bit <<-\\EOF\n+\tcat >email-using-8bit <<-EOF\n \tFrom fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001\n \tMessage-ID: \n \tFrom: A U Thor \n\n-- \n2.53.0.1185.g05d4b7b318.dirty\n\n"},{"id":"540629","messageId":"20260401-pks-tests-with-dash-v1-2-d70b5040aa5d@pks.im","threadId":"65406","inReplyTo":"20260401-pks-tests-with-dash-v1-0-d70b5040aa5d@pks.im","subject":"[PATCH 2/2] t9300: work around partial read bug in Dash v0.5.13","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-04-01T10:42:59Z","receivedAt":"2026-04-01T10:43:09Z","isPatch":true,"body":"When executing t9300 with Dash v0.5.13.1 we can see that the test hangs\ncompletely with the following (condensed) trace:\n\n git fast-import\n + error=1\n + read output\n + cat input\n + echo checkpoint\n + echo progress checkpoint\n + test rogress checkpoint = progress checkpoint\n + test rogress checkpoint = UNEXPECTED\n + echo cruft: rogress checkpoint\n cruft: rogress checkpoint\n + read output\n + test = progress checkpoint\n + test = UNEXPECTED\n + echo cruft:\n cruft:\n + read output\n\nBasically, what's happening here is that we spawn git-fast-import(1) and\nwait for it to output a certain string, \"progress checkpoint\". Curiously\nthough, what we end up reading is \"rogress checkpoint\" -- so the first\nbyte of the expected string is missing.\n\nSame as in the preceding commit, this seems to be a bug in Dash itself\nthat bisects to c5bf970 (expand: Add multi-byte support to pmatch,\n2024-06-02). But other than in the preceding commit, this bug has\nalready been fixed upstream in 079059a (input: Fix heap-buffer-overflow\nin preadbuffer on long lines, 2026-02-11), which is part of v0.5.13.2.\n\nFor now though, work around the bug by waiting for the expected output\nin a different way. There is no good reason why one version should work\nbetter than the other, but at least the new version doesn't exhibit the\nbug. And, if you ask me, it's also slightly easier to read.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t9300-fast-import.sh | 32 ++++++++++++++------------------\n 1 file changed, 14 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 5685cce6fe..479437760b 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3635,25 +3635,21 @@ background_import_then_checkpoint () {\n \t\techo \"progress checkpoint\"\n \t) >&8 &\n \n-\terror=1 ;# assume the worst\n-\twhile read output <&9\n-\tdo\n-\t\tif test \"$output\" = \"progress checkpoint\"\n-\t\tthen\n-\t\t\terror=0\n-\t\t\tbreak\n-\t\telif test \"$output\" = \"UNEXPECTED\"\n-\t\tthen\n-\t\t\tbreak\n-\t\tfi\n-\t\t# otherwise ignore cruft\n-\t\techo >&2 \"cruft: $output\"\n-\tdone\n+\tlast=$(\n+\t\twhile read output <&9\n+\t\tdo\n+\t\t\tif test \"$output\" = \"progress checkpoint\" || test \"$output\" = \"UNEXPECTED\"\n+\t\t\tthen\n+\t\t\t\techo \"$output\"\n+\t\t\t\tbreak\n+\t\t\telse\n+\t\t\t\t# otherwise ignore cruft\n+\t\t\t\techo >&2 \"cruft: $output\"\n+\t\t\tfi\n+\t\tdone\n+\t)\n \n-\tif test $error -eq 1\n-\tthen\n-\t\tfalse\n-\tfi\n+\ttest \"$last\" = \"progress checkpoint\"\n }\n \n background_import_still_running () {\n\n-- \n2.53.0.1185.g05d4b7b318.dirty\n\n"},{"id":"540651","messageId":"CAPig+cQHUk5abmc+CeJ0tPWs5z7XMq9AxHKZRdvEgULF4zTg0g@mail.gmail.com","threadId":"65406","inReplyTo":"20260401-pks-tests-with-dash-v1-1-d70b5040aa5d@pks.im","subject":"Re: [PATCH 1/2] t: work around multibyte bug in quoted heredocs with Dash v0.5.13","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2026-04-01T16:21:41Z","receivedAt":"2026-04-01T16:21:54Z","isPatch":true,"body":"On Wed, Apr 1, 2026 at 6:59 AM Patrick Steinhardt wrote:\n> When executing our test suite with Dash v0.5.13.2 one can observe\n> several test failures that all have the same symptoms: we have a quoted\n> heredoc that contains multibyte characters, but the final data does not\n> match what we actually wanted to write. One such example is in t0300,\n> where we see the diffs like the following:\n> [...]\n> For now, work around the bug by using unquoted heredocs instead.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh\n> @@ -675,7 +675,7 @@ test_expect_success 'match percent-encoded values' '\n> test_expect_success 'match percent-encoded UTF-8 values in path' '\n> test_config credential.https://example.com.useHttpPath true &&\n> test_config credential.https://example.com/perú.git.helper \"$HELPER\" &&\n> - check fill <<-\\EOF\n> + check fill <<-EOF\n> url=https://example.com/per%C3%BA.git\n\nOof, this is the sort of change which cries out for an in-code\ncomment, since we can't expect that future reviewers will remember\nthis specific workaround in this specific script, and we can't expect\nthat people working on this code will think to check the history to\nlearn why the heredoc is unquoted. Without such a comment, someone\nworking on this file for some purpose or another (or even a GSoC\napplicant looking for a microproject) will come along and \"modernize\"\nthis script by adding back the heredoc quoting which this patch\nremoves.\n\nTo prevent someone from \"breaking\" your \"fix\", at minimum, please add\nan explanatory comment somewhere near the top of the script which\npretty much duplicates the information from the commit message:\n\n # *DONTQUOTE*: ...explanation of dash bug ...\n\nand reference that comment at each location you \"fix\":\n\n # NOTE: Don't quote heredoc; see *DONTQUOTE* above.\n check fill <<-EOF\n ...\n"},{"id":"540702","messageId":"ac4CVegAlxdf0zHq@pks.im","threadId":"65406","inReplyTo":"CAPig+cQHUk5abmc+CeJ0tPWs5z7XMq9AxHKZRdvEgULF4zTg0g@mail.gmail.com","subject":"Re: [PATCH 1/2] t: work around multibyte bug in quoted heredocs with Dash v0.5.13","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-04-02T05:44:53Z","receivedAt":"2026-04-02T05:45:01Z","isPatch":true,"body":"On Wed, Apr 01, 2026 at 12:21:41PM -0400, Eric Sunshine wrote:\n> On Wed, Apr 1, 2026 at 6:59 AM Patrick Steinhardt wrote:\n> > When executing our test suite with Dash v0.5.13.2 one can observe\n> > several test failures that all have the same symptoms: we have a quoted\n> > heredoc that contains multibyte characters, but the final data does not\n> > match what we actually wanted to write. One such example is in t0300,\n> > where we see the diffs like the following:\n> > [...]\n> > For now, work around the bug by using unquoted heredocs instead.\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh\n> > @@ -675,7 +675,7 @@ test_expect_success 'match percent-encoded values' '\n> > test_expect_success 'match percent-encoded UTF-8 values in path' '\n> > test_config credential.https://example.com.useHttpPath true &&\n> > test_config credential.https://example.com/perú.git.helper \"$HELPER\" &&\n> > - check fill <<-\\EOF\n> > + check fill <<-EOF\n> > url=https://example.com/per%C3%BA.git\n> \n> Oof, this is the sort of change which cries out for an in-code\n> comment, since we can't expect that future reviewers will remember\n> this specific workaround in this specific script, and we can't expect\n> that people working on this code will think to check the history to\n> learn why the heredoc is unquoted. Without such a comment, someone\n> working on this file for some purpose or another (or even a GSoC\n> applicant looking for a microproject) will come along and \"modernize\"\n> this script by adding back the heredoc quoting which this patch\n> removes.\n> \n> To prevent someone from \"breaking\" your \"fix\", at minimum, please add\n> an explanatory comment somewhere near the top of the script which\n> pretty much duplicates the information from the commit message:\n> \n> # *DONTQUOTE*: ...explanation of dash bug ...\n> \n> and reference that comment at each location you \"fix\":\n> \n> # NOTE: Don't quote heredoc; see *DONTQUOTE* above.\n> check fill <<-EOF\n> ...\n\nTrue indeed. I'll just add a short comment to the individual callsites.\nThanks!\n\nPatrick\n"},{"id":"540710","messageId":"20260402-pks-tests-with-dash-v2-0-cd7ab11dabc0@pks.im","threadId":"65406","inReplyTo":"20260401-pks-tests-with-dash-v1-0-d70b5040aa5d@pks.im","subject":"[PATCH v2 0/2] t: work around bugs in Dash v0.5.13","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-04-02T06:51:17Z","receivedAt":"2026-04-02T06:51:25Z","isPatch":true,"body":"Hi,\n\nwhile testing my `set -e` change for our test suite I was also playing\naround with Dash v0.5.13.1 and noticed that multiple of our tests fail\nwith that version. As it turns out, some of these test failures are\nactual bugs in Dash itself that have been introduced in v0.5.13.\n\nThere's ultimately two bugs:\n\n - Dash inserts CTLMBCHAR byte sequences into heredocs around multibyte\n characters. This bug still exists with the current \"master\" branch\n of Dash.\n\n - Dash may swallow some bytes when reading data, which has already\n been fixed.\n\nI've Cc'd Herbert, maintainer of Dash.\n\nChanges in v2:\n - Add comment to heredocs explaining why they should never be quoted.\n - Link to v1: https://patch.msgid.link/20260401-pks-tests-with-dash-v1-0-d70b5040aa5d@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (2):\n t: work around multibyte bug in quoted heredocs with Dash v0.5.13\n t9300: work around partial read bug in Dash v0.5.13\n\n t/t0300-credentials.sh | 4 +++-\n t/t3430-rebase-merges.sh | 6 ++++--\n t/t3902-quoted.sh | 16 +++++++++-------\n t/t4014-format-patch.sh | 16 ++++++++++++----\n t/t4201-shortlog.sh | 4 +++-\n t/t9001-send-email.sh | 12 +++++++++---\n t/t9300-fast-import.sh | 32 ++++++++++++++------------------\n 7 files changed, 54 insertions(+), 36 deletions(-)\n\nRange-diff versus v1:\n\n1: 192b5b4330 ! 1: 5367e05e0f t: work around multibyte bug in quoted heredocs with Dash v0.5.13\n @@ t/t0300-credentials.sh: test_expect_success 'match percent-encoded values' '\n \ttest_config credential.https://example.com.useHttpPath true &&\n \ttest_config credential.https://example.com/perú.git.helper \"$HELPER\" &&\n -\tcheck fill <<-\\EOF\n ++\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++\t# that contain multibyte chars.\n +\tcheck fill <<-EOF\n \turl=https://example.com/per%C3%BA.git\n \t--\n @@ t/t3430-rebase-merges.sh: test_expect_success 'octopus merges' '\n \ttest \"Hank\" = \"$(git show -s --format=%an HEAD)\" &&\n \ttest \"$before\" != $(git rev-parse HEAD) &&\n -\ttest_cmp_graph HEAD^^.. <<-\\EOF\n ++\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++\t# that contain multibyte chars.\n +\ttest_cmp_graph HEAD^^.. <<-EOF\n \t*-. Tüntenfüsch\n -\t|\\ \\\n @@ t/t3902-quoted.sh: With SP in it\n EOF\n \n -cat >expect.raw <<\\EOF\n ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++# that contain multibyte chars.\n +cat >expect.raw <expect <<'EOF'\n ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++# that contain multibyte chars.\n +cat >expect <expect <<'EOF'\n ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++# that contain multibyte chars.\n +cat >expect <expect <<'EOF'\n ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++# that contain multibyte chars.\n +cat >expect <patch &&\n -\tcat >expect <<-\\EOF &&\n ++\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++\t# that contain multibyte chars.\n +\tcat >expect <<-EOF &&\n \tFrom: C O Mitter \n \tContent-Type: text/plain; charset=UTF-8\n @@ t/t4201-shortlog.sh: test_expect_success 'output from user-defined format is re-\n \n test_expect_success !MINGW,ICONV 'shortlog wrapping' '\n -\tcat >expect <<\\EOF &&\n ++\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++\t# that contain multibyte chars.\n +\tcat >expect <email-using-8bit <<\\EOF\n ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++# that contain multibyte chars.\n +cat >email-using-8bit <\n @@ t/t9001-send-email.sh: test_expect_success $PREREQ '--8bit-encoding overrides se\n \n test_expect_success $PREREQ 'setup expect' '\n -\tcat >email-using-8bit <<-\\EOF\n ++\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++\t# that contain multibyte chars.\n +\tcat >email-using-8bit <<-EOF\n \tFrom fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001\n \tMessage-ID: \n @@ t/t9001-send-email.sh: test_expect_success $PREREQ '--8bit-encoding also treats\n \n test_expect_success $PREREQ 'setup expect' '\n -\tcat >email-using-8bit <<-\\EOF\n ++\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n ++\t# that contain multibyte chars.\n +\tcat >email-using-8bit <<-EOF\n \tFrom fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001\n \tMessage-ID: \n2: db99459b3f = 2: 364d37fcc7 t9300: work around partial read bug in Dash v0.5.13\n\n---\nbase-commit: 270e10ad6dda3379ea0da7efd11e4fbf2cd7a325\nchange-id: 20260401-pks-tests-with-dash-ea71cbb36958\n\n"},{"id":"540711","messageId":"20260402-pks-tests-with-dash-v2-1-cd7ab11dabc0@pks.im","threadId":"65406","inReplyTo":"20260402-pks-tests-with-dash-v2-0-cd7ab11dabc0@pks.im","subject":"[PATCH v2 1/2] t: work around multibyte bug in quoted heredocs with Dash v0.5.13","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-04-02T06:51:18Z","receivedAt":"2026-04-02T06:51:26Z","isPatch":true,"body":"When executing our test suite with Dash v0.5.13.2 one can observe\nseveral test failures that all have the same symptoms: we have a quoted\nheredoc that contains multibyte characters, but the final data does not\nmatch what we actually wanted to write. One such example is in t0300,\nwhere we see the diffs like the following:\n\n --- expect-stdout\t2026-04-01 07:25:45.249919440 +0000\n +++ stdout\t2026-04-01 07:25:45.254919509 +0000\n @@ -1,5 +1,5 @@\n protocol=https\n host=example.com\n -path=perú.git\n +path=perú.git\n username=foo\n password=bar\n\nWhile seemingly the same, the data that we've written via the heredoc\ncontains some invisible bytes. The expected hex representation of the\nstring is:\n\n 7065 72c3 ba2e 6769 74 per...git\n\nBut what we actually get instead is this string:\n\n 7065 7285 02c3 ba02 852e 6769 74 per.......git\n\nWhat's important to note here is that the multibyte character exists in\nboth versions. But in the broken version we see that the bytes are\nwrapped in a sequence of \"85 02\" and \"02 85\". This is the CTLMBCHAR byte\nsequence of Dash, which it uses internally to quote multibyte sequences.\n\nAs it turns out, this bug was introduced in c5bf970 (expand: Add\nmulti-byte support to pmatch, 2024-06-02), which adds multibyte support\nto more contexts of Dash. One of these contexts seems to be in heredocs,\nand Dash _does_ correctly unquote these multibyte sequences when using\nan unquoted heredoc. But the bug seems to be that this unquoting does\nnot happen in quoted heredocs, and the bug still exists on the latest\n\"master\" branch.\n\nFor now, work around the bug by using unquoted heredocs instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0300-credentials.sh | 4 +++-\n t/t3430-rebase-merges.sh | 6 ++++--\n t/t3902-quoted.sh | 16 +++++++++-------\n t/t4014-format-patch.sh | 16 ++++++++++++----\n t/t4201-shortlog.sh | 4 +++-\n t/t9001-send-email.sh | 12 +++++++++---\n 6 files changed, 40 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh\nindex 07aa834d33..64ead1571a 100755\n--- a/t/t0300-credentials.sh\n+++ b/t/t0300-credentials.sh\n@@ -675,7 +675,9 @@ test_expect_success 'match percent-encoded values' '\n test_expect_success 'match percent-encoded UTF-8 values in path' '\n \ttest_config credential.https://example.com.useHttpPath true &&\n \ttest_config credential.https://example.com/perú.git.helper \"$HELPER\" &&\n-\tcheck fill <<-\\EOF\n+\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+\t# that contain multibyte chars.\n+\tcheck fill <<-EOF\n \turl=https://example.com/per%C3%BA.git\n \t--\n \tprotocol=https\ndiff --git a/t/t3430-rebase-merges.sh b/t/t3430-rebase-merges.sh\nindex cc627e34a7..84b2d0e664 100755\n--- a/t/t3430-rebase-merges.sh\n+++ b/t/t3430-rebase-merges.sh\n@@ -507,9 +507,11 @@ test_expect_success 'octopus merges' '\n \tgit rebase -i --force-rebase -r HEAD^^ &&\n \ttest \"Hank\" = \"$(git show -s --format=%an HEAD)\" &&\n \ttest \"$before\" != $(git rev-parse HEAD) &&\n-\ttest_cmp_graph HEAD^^.. <<-\\EOF\n+\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+\t# that contain multibyte chars.\n+\ttest_cmp_graph HEAD^^.. <<-EOF\n \t*-. Tüntenfüsch\n-\t|\\ \\\n+\t|\\\\ \\\\\n \t| | * three\n \t| * | two\n \t| |/\ndiff --git a/t/t3902-quoted.sh b/t/t3902-quoted.sh\nindex f528008c36..8660ec5cb0 100755\n--- a/t/t3902-quoted.sh\n+++ b/t/t3902-quoted.sh\n@@ -60,16 +60,18 @@ With SP in it\n \"\\346\\277\\261\\351\\207\\216\\347\\264\\224\"\n EOF\n \n-cat >expect.raw <<\\EOF\n+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+# that contain multibyte chars.\n+cat >expect.raw <expect <<'EOF'\n+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+# that contain multibyte chars.\n+cat >expect <\n@@ -1300,7 +1302,9 @@ test_expect_success 'format-patch wraps extremely long from-header (non-ASCII wi\n \ttest_cmp expect actual\n '\n \n-cat >expect <<'EOF'\n+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+# that contain multibyte chars.\n+cat >expect <expect <<'EOF'\n+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+# that contain multibyte chars.\n+cat >expect <patch &&\n-\tcat >expect <<-\\EOF &&\n+\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+\t# that contain multibyte chars.\n+\tcat >expect <<-EOF &&\n \tFrom: C O Mitter \n \tContent-Type: text/plain; charset=UTF-8\n \ndiff --git a/t/t4201-shortlog.sh b/t/t4201-shortlog.sh\nindex 5f23fc147b..9f41d56d9a 100755\n--- a/t/t4201-shortlog.sh\n+++ b/t/t4201-shortlog.sh\n@@ -105,7 +105,9 @@ test_expect_success 'output from user-defined format is re-wrapped' '\n '\n \n test_expect_success !MINGW,ICONV 'shortlog wrapping' '\n-\tcat >expect <<\\EOF &&\n+\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+\t# that contain multibyte chars.\n+\tcat >expect <email-using-8bit <<\\EOF\n+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+# that contain multibyte chars.\n+cat >email-using-8bit <\n From: author@example.com\n@@ -1735,7 +1737,9 @@ test_expect_success $PREREQ '--8bit-encoding overrides sendemail.8bitEncoding' '\n '\n \n test_expect_success $PREREQ 'setup expect' '\n-\tcat >email-using-8bit <<-\\EOF\n+\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+\t# that contain multibyte chars.\n+\tcat >email-using-8bit <<-EOF\n \tFrom fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001\n \tMessage-ID: \n \tFrom: author@example.com\n@@ -1764,7 +1768,9 @@ test_expect_success $PREREQ '--8bit-encoding also treats subject' '\n '\n \n test_expect_success $PREREQ 'setup expect' '\n-\tcat >email-using-8bit <<-\\EOF\n+\t# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs\n+\t# that contain multibyte chars.\n+\tcat >email-using-8bit <<-EOF\n \tFrom fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001\n \tMessage-ID: \n \tFrom: A U Thor \n\n-- \n2.53.0.1323.g189a785ab5.dirty\n\n"},{"id":"540712","messageId":"20260402-pks-tests-with-dash-v2-2-cd7ab11dabc0@pks.im","threadId":"65406","inReplyTo":"20260402-pks-tests-with-dash-v2-0-cd7ab11dabc0@pks.im","subject":"[PATCH v2 2/2] t9300: work around partial read bug in Dash v0.5.13","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-04-02T06:51:19Z","receivedAt":"2026-04-02T06:51:28Z","isPatch":true,"body":"When executing t9300 with Dash v0.5.13.1 we can see that the test hangs\ncompletely with the following (condensed) trace:\n\n git fast-import\n + error=1\n + read output\n + cat input\n + echo checkpoint\n + echo progress checkpoint\n + test rogress checkpoint = progress checkpoint\n + test rogress checkpoint = UNEXPECTED\n + echo cruft: rogress checkpoint\n cruft: rogress checkpoint\n + read output\n + test = progress checkpoint\n + test = UNEXPECTED\n + echo cruft:\n cruft:\n + read output\n\nBasically, what's happening here is that we spawn git-fast-import(1) and\nwait for it to output a certain string, \"progress checkpoint\". Curiously\nthough, what we end up reading is \"rogress checkpoint\" -- so the first\nbyte of the expected string is missing.\n\nSame as in the preceding commit, this seems to be a bug in Dash itself\nthat bisects to c5bf970 (expand: Add multi-byte support to pmatch,\n2024-06-02). But other than in the preceding commit, this bug has\nalready been fixed upstream in 079059a (input: Fix heap-buffer-overflow\nin preadbuffer on long lines, 2026-02-11), which is part of v0.5.13.2.\n\nFor now though, work around the bug by waiting for the expected output\nin a different way. There is no good reason why one version should work\nbetter than the other, but at least the new version doesn't exhibit the\nbug. And, if you ask me, it's also slightly easier to read.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t9300-fast-import.sh | 32 ++++++++++++++------------------\n 1 file changed, 14 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 5685cce6fe..479437760b 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3635,25 +3635,21 @@ background_import_then_checkpoint () {\n \t\techo \"progress checkpoint\"\n \t) >&8 &\n \n-\terror=1 ;# assume the worst\n-\twhile read output <&9\n-\tdo\n-\t\tif test \"$output\" = \"progress checkpoint\"\n-\t\tthen\n-\t\t\terror=0\n-\t\t\tbreak\n-\t\telif test \"$output\" = \"UNEXPECTED\"\n-\t\tthen\n-\t\t\tbreak\n-\t\tfi\n-\t\t# otherwise ignore cruft\n-\t\techo >&2 \"cruft: $output\"\n-\tdone\n+\tlast=$(\n+\t\twhile read output <&9\n+\t\tdo\n+\t\t\tif test \"$output\" = \"progress checkpoint\" || test \"$output\" = \"UNEXPECTED\"\n+\t\t\tthen\n+\t\t\t\techo \"$output\"\n+\t\t\t\tbreak\n+\t\t\telse\n+\t\t\t\t# otherwise ignore cruft\n+\t\t\t\techo >&2 \"cruft: $output\"\n+\t\t\tfi\n+\t\tdone\n+\t)\n \n-\tif test $error -eq 1\n-\tthen\n-\t\tfalse\n-\tfi\n+\ttest \"$last\" = \"progress checkpoint\"\n }\n \n background_import_still_running () {\n\n-- \n2.53.0.1323.g189a785ab5.dirty\n\n"},{"id":"542832","messageId":"afwpyiK9mh23c-JV@gondor.apana.org.au","threadId":"65406","inReplyTo":"20260402-pks-tests-with-dash-v2-1-cd7ab11dabc0@pks.im","subject":"[PATCH] parser: Fix multi-byte output in here-doc with quoted delimiter","fromName":"Herbert Xu","fromEmail":"herbert@gondor.apana.org.au","sentAt":"2026-05-07T05:57:30Z","receivedAt":"2026-05-07T05:57:35Z","isPatch":true,"body":"On Thu, Apr 02, 2026 at 08:51:18AM +0200, Patrick Steinhardt wrote:\n> When executing our test suite with Dash v0.5.13.2 one can observe\n> several test failures that all have the same symptoms: we have a quoted\n> heredoc that contains multibyte characters, but the final data does not\n> match what we actually wanted to write. One such example is in t0300,\n> where we see the diffs like the following:\n> \n> --- expect-stdout\t2026-04-01 07:25:45.249919440 +0000\n> +++ stdout\t2026-04-01 07:25:45.254919509 +0000\n> @@ -1,5 +1,5 @@\n> protocol=https\n> host=example.com\n> -path=perú.git\n> +path=perú.git\n> username=foo\n> password=bar\n\nThanks for the report.\n\nThis patch should fix the problem. Please let me know if there are\nany more oustanding issues.\n\n---8<---\nFor a here-document with a quoted delimiter, multi-byte characters\nshould be written out as is with no escaping. Fix this by checking\nfor syntax == SQSYNTAX (the only time readtoken1 gets called with\nSQSYNTAX is for such a here-document) before calling getmbc in\nreadtoken1.\n\nReported-by: Patrick Steinhardt \nFixes: b12f136cc704 (\"builtin: Process multi-byte characters in read(1)\")\nSigned-off-by: Herbert Xu \n\ndiff --git a/src/parser.c b/src/parser.c\nindex bea4148..412e876 100644\n--- a/src/parser.c\n+++ b/src/parser.c\n@@ -998,9 +998,13 @@ static char *dollarsq_escape(char *out)\n STATIC int\n readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)\n {\n-\tstruct synstack synbase = { .syntax = syntax };\n+\tstruct synstack synbase = {\n+\t\t.dblquote = syntax == DQSYNTAX,\n+\t\t.syntax = syntax,\n+\t};\n \tint chkeofmark = checkkwd & CHKEOFMARK;\n \tstruct synstack *synstack = &synbase;\n+\tbool sqheredoc = syntax == SQSYNTAX;\n \tstruct nodelist *bqlist = NULL;\n \tint dollarsq = 0;\n \tint c = firstc;\n@@ -1009,9 +1013,6 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)\n \tsize_t len;\n \tchar *out;\n \n-\tif (syntax == DQSYNTAX)\n-\t\tsynstack->dblquote = 1;\n-\n \tSTARTSTACKSTR(out);\n \tloop: {\t/* for each line, until end of word */\n #if ATTY\n@@ -1035,7 +1036,8 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)\n \t\t\t\t out);\n \t\t\tfieldsplitting = synstack->syntax == BASESYNTAX &&\n \t\t\t\t\t !synstack->varnest ? 4 : 0;\n-\t\t\tml = getmbc(c, out, fieldsplitting);\n+\t\t\tml = getmbc(c, out, fieldsplitting |\n+\t\t\t\t\t (sqheredoc ? 2 : 0));\n \t\t\tif (ml == 1) {\n \t\t\t\tif (out == stackblock())\n \t\t\t\t\treturn TBLANK;\n-- \nEmail: Herbert Xu \nHome Page: http://gondor.apana.org.au/~herbert/\nPGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt\n"},{"id":"542833","messageId":"afxBOHhvXC7VxG3G@gondor.apana.org.au","threadId":"65406","inReplyTo":"afwpyiK9mh23c-JV@gondor.apana.org.au","subject":"Re: [PATCH] parser: Fix multi-byte output in here-doc with quoted delimiter","fromName":"Herbert Xu","fromEmail":"herbert@gondor.apana.org.au","sentAt":"2026-05-07T07:37:28Z","receivedAt":"2026-05-07T07:37:37Z","isPatch":true,"body":"On Thu, May 07, 2026 at 01:57:30PM +0800, Herbert Xu wrote:\n> On Thu, Apr 02, 2026 at 08:51:18AM +0200, Patrick Steinhardt wrote:\n> > When executing our test suite with Dash v0.5.13.2 one can observe\n> > several test failures that all have the same symptoms: we have a quoted\n> > heredoc that contains multibyte characters, but the final data does not\n> > match what we actually wanted to write. One such example is in t0300,\n> > where we see the diffs like the following:\n> > \n> > --- expect-stdout\t2026-04-01 07:25:45.249919440 +0000\n> > +++ stdout\t2026-04-01 07:25:45.254919509 +0000\n> > @@ -1,5 +1,5 @@\n> > protocol=https\n> > host=example.com\n> > -path=perú.git\n> > +path=perú.git\n> > username=foo\n> > password=bar\n> \n> Thanks for the report.\n> \n> This patch should fix the problem. Please let me know if there are\n> any more oustanding issues.\n\nOops, I forgot to cc the mailing list. Sorry for the resend.\n\n---8<---\nFor a here-document with a quoted delimiter, multi-byte characters\nshould be written out as is with no escaping. Fix this by checking\nfor syntax == SQSYNTAX (the only time readtoken1 gets called with\nSQSYNTAX is for such a here-document) before calling getmbc in\nreadtoken1.\n\nReported-by: Patrick Steinhardt \nFixes: b12f136cc704 (\"builtin: Process multi-byte characters in read(1)\")\nSigned-off-by: Herbert Xu \n\ndiff --git a/src/parser.c b/src/parser.c\nindex bea4148..412e876 100644\n--- a/src/parser.c\n+++ b/src/parser.c\n@@ -998,9 +998,13 @@ static char *dollarsq_escape(char *out)\n STATIC int\n readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)\n {\n-\tstruct synstack synbase = { .syntax = syntax };\n+\tstruct synstack synbase = {\n+\t\t.dblquote = syntax == DQSYNTAX,\n+\t\t.syntax = syntax,\n+\t};\n \tint chkeofmark = checkkwd & CHKEOFMARK;\n \tstruct synstack *synstack = &synbase;\n+\tbool sqheredoc = syntax == SQSYNTAX;\n \tstruct nodelist *bqlist = NULL;\n \tint dollarsq = 0;\n \tint c = firstc;\n@@ -1009,9 +1013,6 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)\n \tsize_t len;\n \tchar *out;\n \n-\tif (syntax == DQSYNTAX)\n-\t\tsynstack->dblquote = 1;\n-\n \tSTARTSTACKSTR(out);\n \tloop: {\t/* for each line, until end of word */\n #if ATTY\n@@ -1035,7 +1036,8 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)\n \t\t\t\t out);\n \t\t\tfieldsplitting = synstack->syntax == BASESYNTAX &&\n \t\t\t\t\t !synstack->varnest ? 4 : 0;\n-\t\t\tml = getmbc(c, out, fieldsplitting);\n+\t\t\tml = getmbc(c, out, fieldsplitting |\n+\t\t\t\t\t (sqheredoc ? 2 : 0));\n \t\t\tif (ml == 1) {\n \t\t\t\tif (out == stackblock())\n \t\t\t\t\treturn TBLANK;\n-- \nEmail: Herbert Xu \nHome Page: http://gondor.apana.org.au/~herbert/\nPGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt\n"}]}