From: Herbert Xu Date: Thu, 07 May 2026 05:57:30 GMT Subject: [PATCH] parser: Fix multi-byte output in here-doc with quoted delimiter Message-ID: In-Reply-To: <20260402-pks-tests-with-dash-v2-1-cd7ab11dabc0@pks.im> On Thu, Apr 02, 2026 at 08:51:18AM +0200, Patrick Steinhardt wrote: > When executing our test suite with Dash v0.5.13.2 one can observe > several test failures that all have the same symptoms: we have a quoted > heredoc that contains multibyte characters, but the final data does not > match what we actually wanted to write. One such example is in t0300, > where we see the diffs like the following: > > --- expect-stdout 2026-04-01 07:25:45.249919440 +0000 > +++ stdout 2026-04-01 07:25:45.254919509 +0000 > @@ -1,5 +1,5 @@ > protocol=https > host=example.com > -path=perú.git > +path=perú.git > username=foo > password=bar Thanks for the report. This patch should fix the problem. Please let me know if there are any more oustanding issues. ---8<--- For a here-document with a quoted delimiter, multi-byte characters should be written out as is with no escaping. Fix this by checking for syntax == SQSYNTAX (the only time readtoken1 gets called with SQSYNTAX is for such a here-document) before calling getmbc in readtoken1. Reported-by: Patrick Steinhardt Fixes: b12f136cc704 ("builtin: Process multi-byte characters in read(1)") Signed-off-by: Herbert Xu diff --git a/src/parser.c b/src/parser.c index bea4148..412e876 100644 --- a/src/parser.c +++ b/src/parser.c @@ -998,9 +998,13 @@ static char *dollarsq_escape(char *out) STATIC int readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs) { - struct synstack synbase = { .syntax = syntax }; + struct synstack synbase = { + .dblquote = syntax == DQSYNTAX, + .syntax = syntax, + }; int chkeofmark = checkkwd & CHKEOFMARK; struct synstack *synstack = &synbase; + bool sqheredoc = syntax == SQSYNTAX; struct nodelist *bqlist = NULL; int dollarsq = 0; int c = firstc; @@ -1009,9 +1013,6 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs) size_t len; char *out; - if (syntax == DQSYNTAX) - synstack->dblquote = 1; - STARTSTACKSTR(out); loop: { /* for each line, until end of word */ #if ATTY @@ -1035,7 +1036,8 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs) out); fieldsplitting = synstack->syntax == BASESYNTAX && !synstack->varnest ? 4 : 0; - ml = getmbc(c, out, fieldsplitting); + ml = getmbc(c, out, fieldsplitting | + (sqheredoc ? 2 : 0)); if (ml == 1) { if (out == stackblock()) return TBLANK; -- Email: Herbert Xu Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt