git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 2/3] mailinfo.c::convert_to_utf8: reuse strlen info

From
MÅMartin Ågren <martin.agren@gmail.com>
Date
Apr 19, 2020, 12:29 UTC
Message-ID
<CAN0heSqi8KvbfSD80UA9s_aj9dGYTvbeW4GXVY59014_Sm-HmQ@mail.gmail.com>
In-Reply-To
<e3e542d38818b762f8d6d6b50bc42e01b070772b.1587289680.git.congdanhqx@gmail.com>
On Sun, 19 Apr 2020 at 13:03, Đoàn Trần Công Danh <congdanhqx@gmail.com> wrote:
Show 25 quoted lines
>
> We're passing buffer from strbuf to reencode_string,
> which will call strlen(3) on that buffer,
> and discard the length of newly created buffer.
> Then, we compute the length of the return buffer to attach to strbuf.
>
> During this process, we introduce a discrimination between mail
> originally written in utf-8 and other encoding.
>
> * if the email was written in utf-8, we leave it as is. If there is
>   a NUL character in that line, we complains loudly:
>
>         error: a NUL byte in commit log message not allowed.
>
> * if the email was written in other encoding, we truncate the data as
>   the NUL character in that line, then we used the truncated line for
>   the metadata.
>
> We can do better by reusing all the available information,
> and call the underlying lower level function that will be called
> indirectly by reencode_string. By doing this, we will also postpone
> the NUL character processing to the commit step, which will
> complains about the faulty metadata.
>
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
This makes sense to me.

I think the subject could be adapted though? Now it's not about "reusing info" anymore, it's about using *other*, *better* info. Maybe

  mailinfo.c: avoid strlen on strings that might contain NUL
? Could probably be improved further..
Show 25 quoted lines
> --- a/mailinfo.c
> +++ b/mailinfo.c
> @@ -447,19 +447,21 @@ static int convert_to_utf8(struct mailinfo *mi,
>                            struct strbuf *line, const char *charset)
>  {
>         char *out;
> +       size_t out_len;
>
>         if (!mi->metainfo_charset || !charset || !*charset)
>                 return 0;
>
>         if (same_encoding(mi->metainfo_charset, charset))
>                 return 0;
> -       out = reencode_string(line->buf, mi->metainfo_charset, charset);
> +       out = reencode_string_len(line->buf, line->len,
> +                                 mi->metainfo_charset, charset, &out_len);
>         if (!out) {
>                 mi->input_error = -1;
>                 return error("cannot convert from %s to %s",
>                              charset, mi->metainfo_charset);
>         }
> -       strbuf_attach(line, out, strlen(out), strlen(out));
> +       strbuf_attach(line, out, out_len, out_len);
>         return 0;
>  }
Same diff as before, ok.
Show 6 quoted lines
> +write_nul_patch() {
> +       space=' '
> +       qNUL=
> +       case "$1" in
> +               subject) qNUL='=00' ;;
> +       esac
Here it's case/esac...
Show 14 quoted lines
> +       cat <<-EOF
> +       From ec7364544f690c560304f5a5de9428ea3b978b26 Mon Sep 17 00:00:00 2001
> +       From: A U Thor <author@example.com>
> +       Date: Sun, 19 Apr 2020 13:42:07 +0700
> +       Subject: [PATCH] =?ISO-8859-1?q?=C4=CB${qNUL}=D1=CF=D6?=
> +       MIME-Version: 1.0
> +       Content-Type: text/plain; charset=ISO-8859-1
> +       Content-Transfer-Encoding: 8bit
> +
> +       EOF
> +       if test "$1" = body
> +       then
> +               printf "%s\0%s\n" abc def
> +       fi
Here it's if/fi. Looks a bit inconsistent.

I suppose you tried to have a case for "body" above but couldn't get it to work? Somehow, it would seem more consistent to have a qSubject and qBody and handle them the same way, but maybe that's not possible? Maybe using q_to_nul to create qBody containing a NUL?

Show 9 quoted lines
> +       cat <<-\EOF
> +       ---
> +       diff --git a/afile b/afile
> +       new file mode 100644
> +       index 0000000000..e69de29bb2
> +       --$space
> +       2.26.1
> +       EOF
> +}
I think this helper function should use &&-chaining.
Show 14 quoted lines
> +
>  test_expect_success setup '
>         # Note the missing "+++" line:
>         cat >bad-patch.diff <<-\EOF &&
> @@ -32,4 +62,18 @@ test_expect_success 'try to apply corrupted patch' '
>         test_i18ncmp expected actual
>  '
>
> +test_expect_success "NUL in commit message's body" '
> +       test_when_finished "git am --abort" &&
> +       write_nul_patch body >body.patch &&
> +       test_must_fail git am body.patch 2>err &&
> +       grep "a NUL byte in commit log message not allowed" err
> +'
Makes sense.
Show 7 quoted lines
> +
> +test_expect_failure "NUL in commit message's header" '
> +       test_when_finished "git am --abort" &&
> +       write_nul_patch subject >subject.patch &&
> +       test_must_fail git am subject.patch 2>err &&
> +       grep "a NUL byte in Subject is not allowed" err
> +'
Interesting. :-)
Martin
Previous: Đoàn Trần Công DanhNext: Danh Doan
Message 32 of 39 in “mailinfo.c::convert_to_utf8: reuse strlen info”
  1. mailinfo.c::convert_to_utf8: reuse strlen infoĐoàn Trần Công Danh, Apr 18, 2020
  2. Martin ÅgrenApr 18, 2020
  3. 0/6 strbuf: simplify `strbuf_attach()` usageMartin Ågren, Apr 18, 2020
  4. 1/6 am: use `strbuf_attach()` correctlyMartin Ågren, Apr 18, 2020
  5. 2/6 strbuf_attach: correctly pass in `strlen() + 1` for `alloc`Martin Ågren, Apr 18, 2020
  6. 3/6 strbuf: use `strbuf_attach()` correctlyMartin Ågren, Apr 18, 2020
  7. 4/6 fast-import: avoid awkward use of `strbuf_attach()`Martin Ågren, Apr 18, 2020
  8. 5/6 rerere: avoid awkward use of `strbuf_attach()`Martin Ågren, Apr 18, 2020
  9. 6/6 strbuf: simplify `strbuf_attach()` usageMartin Ågren, Apr 18, 2020
  10. Martin ÅgrenApr 19, 2020
  11. 0/4 strbuf: fix doc for `strbuf_attach()` and avoid itMartin Ågren, Apr 19, 2020
  12. 1/4 strbuf: fix doc for `strbuf_attach()`Martin Ågren, Apr 19, 2020
  13. Junio C HamanoApr 20, 2020
  14. Martin ÅgrenApr 21, 2020
  15. 2/4 strbuf: introduce `strbuf_attachstr_len()`Martin Ågren, Apr 19, 2020
  16. 3/4 strbuf: introduce `strbuf_attachstr()`Martin Ågren, Apr 19, 2020
  17. Junio C HamanoApr 20, 2020
  18. Martin ÅgrenApr 21, 2020
  19. 4/4 strbuf_attach: prefer `strbuf_attachstr_len()`Martin Ågren, Apr 19, 2020
  20. Junio C HamanoApr 18, 2020
  21. Danh DoanApr 19, 2020
  22. Martin ÅgrenApr 19, 2020
  23. Junio C HamanoApr 19, 2020
  24. 0/3 mailinfo: disallow and complains about NUL characterĐoàn Trần Công Danh, Apr 19, 2020
  25. 1/3 t4254: merge 2 steps of a single testĐoàn Trần Công Danh, Apr 19, 2020
  26. Martin ÅgrenApr 19, 2020
  27. Danh DoanApr 19, 2020
  28. 3/3 mailinfo: disallow NUL character in mail's headerĐoàn Trần Công Danh, Apr 19, 2020
  29. Martin ÅgrenApr 19, 2020
  30. Danh DoanApr 19, 2020
  31. 2/3 mailinfo.c::convert_to_utf8: reuse strlen infoĐoàn Trần Công Danh, Apr 19, 2020
  32. Martin ÅgrenApr 19, 2020
  33. Danh DoanApr 19, 2020
  34. Junio C HamanoApr 20, 2020
  35. Danh DoanApr 20, 2020
  36. 0/3 Disallow NUL character in mailinfoĐoàn Trần Công Danh, Apr 20, 2020
  37. 1/3 t4254: merge 2 steps of a single testĐoàn Trần Công Danh, Apr 20, 2020
  38. 2/3 mailinfo.c: avoid strlen on strings that can contains NULĐoàn Trần Công Danh, Apr 20, 2020
  39. 3/3 mailinfo: disallow NUL character in mail's headerĐoàn Trần Công Danh, Apr 20, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.