git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH] mailinfo: strip CR from base64/quoted-printable email

From
Đoàn Trần Công Danh <congdanhqx@gmail.com>
Date
Apr 21, 2021, 01:34 UTC
Message-ID
<20210421013404.17383-1-congdanhqx@gmail.com>

When an SMTP server receives an 8-bit email message, possibly with only LF as line ending, some of those servers decide to change said LF to CRLF.

Some other SMTP servers, when receives an 8-bit email message, decide to encoding such message in base64 and/or quoted-printable instead.

If an email is transfered through those 2 email servers in order, the final recipients will receive an email contains a patch mungled with CRLF encoded inside another encoding. Thus, such CR couldn't be dropped by mailsplit. Such accidents have been observed in the wild [1].

Let's guess if such CR was added automatically and strip them in mailinfo.

[1]: https://nmbug.notmuchmail.org/nmweb/show/m2lf9ejegj.fsf%40guru.guru-group.fi
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 I'm not sure if guessing the heuristic to strip CR is a good approach.
 I think it's better to pass --keep-cr down from git-am.
 Let's say --keep-cr=<yes|no|auto>
 mailinfo.c             | 20 +++++++++++++++++---
 t/t5100-mailinfo.sh    |  5 +++++
 t/t5100/cr-base64.mbox | 22 ++++++++++++++++++++++
 t/t5100/info1000       |  5 +++++
 t/t5100/msg1000        |  2 ++
 t/t5100/patch1000      | 22 ++++++++++++++++++++++
 6 files changed, 73 insertions(+), 3 deletions(-)
 create mode 100644 t/t5100/cr-base64.mbox
 create mode 100644 t/t5100/info1000
 create mode 100644 t/t5100/msg1000
 create mode 100644 t/t5100/patch1000
diff --git a/mailinfo.c b/mailinfo.c
index 5681d9130d..dbff867f42 100644
--- a/mailinfo.c
+++ b/mailinfo.c
@@ -988,16 +988,27 @@ static int handle_boundary(struct mailinfo *mi, struct strbuf *line)
 }
 
 static void handle_filter_flowed(struct mailinfo *mi, struct strbuf *line,
-				 struct strbuf *prev)
+				 struct strbuf *prev, int *keep_cr)
 {
 	size_t len = line->len;
 	const char *rest;
 
 	if (!mi->format_flowed) {
+		if (*keep_cr == -1 && len >= 2)
+			*keep_cr = !(line->buf[len - 2] == '\r' &&
+				     line->buf[len - 1] == '\n');
+		if (!*keep_cr && len >= 2 &&
+		    line->buf[len - 2] == '\r' &&
+		    line->buf[len - 1] == '\n') {
+			strbuf_setlen(line, len - 2);
+			strbuf_addch(line, '\n');
+			len--;
+		}
 		handle_filter(mi, line);
 		return;
 	}
 
+	*keep_cr = 1;
 	if (line->buf[len - 1] == '\n') {
 		len--;
 		if (len && line->buf[len - 1] == '\r')
@@ -1036,6 +1047,7 @@ static void handle_filter_flowed(struct mailinfo *mi, struct strbuf *line,
 static void handle_body(struct mailinfo *mi, struct strbuf *line)
 {
 	struct strbuf prev = STRBUF_INIT;
+	int keep_cr = -1;
 
 	/* Skip up to the first boundary */
 	if (*(mi->content_top)) {
@@ -1081,7 +1093,7 @@ static void handle_body(struct mailinfo *mi, struct strbuf *line)
 						strbuf_addbuf(&prev, sb);
 						break;
 					}
-				handle_filter_flowed(mi, sb, &prev);
+				handle_filter_flowed(mi, sb, &prev, &keep_cr);
 			}
 			/*
 			 * The partial chunk is saved in "prev" and will be
@@ -1091,7 +1103,9 @@ static void handle_body(struct mailinfo *mi, struct strbuf *line)
 			break;
 		}
 		default:
-			handle_filter_flowed(mi, line, &prev);
+			/* CR in plain message was processed in mailsplit */
+			keep_cr = 1;
+			handle_filter_flowed(mi, line, &prev, &keep_cr);
 		}
 
 		if (mi->input_error)
diff --git a/t/t5100-mailinfo.sh b/t/t5100-mailinfo.sh
index 147e616533..9ccc11d16a 100755
--- a/t/t5100-mailinfo.sh
+++ b/t/t5100-mailinfo.sh
@@ -228,4 +228,9 @@ test_expect_success 'mailinfo handles unusual header whitespace' '
 	test_cmp expect actual
 '
 
+test_expect_success 'mailinfo strip CR after decode base64' '
+	cp $DATA/cr-base64.mbox 1000 &&
+	check_mailinfo 1000 ""
+'
+
 test_done
diff --git a/t/t5100/cr-base64.mbox b/t/t5100/cr-base64.mbox
new file mode 100644
index 0000000000..6ea9806a6b
--- /dev/null
+++ b/t/t5100/cr-base64.mbox
@@ -0,0 +1,22 @@
+From: A U Thor <mail@example.com>
+To: list@example.org
+Subject: [PATCH v2] sample
+Date: Mon,  3 Aug 2020 22:40:55 +0700
+Message-Id: <msg-id@example.com>
+Content-Type: text/plain; charset="utf-8"
+Content-Transfer-Encoding: base64
+
+T24gZGlmZmVyZW50IGRpc3RybywgcHl0ZXN0IGlzIHN1ZmZpeGVkIHdpdGggZGlmZmVyZW50IHBh
+dHRlcm5zLg0KDQotLS0NCiBjb25maWd1cmUgfCAyICstDQogMSBmaWxlIGNoYW5nZWQsIDEgaW5z
+ZXJ0aW9uKCspLCAxIGRlbGV0aW9uKC0pDQoNCmRpZmYgLS1naXQgYS9jb25maWd1cmUgYi9jb25m
+aWd1cmUNCmluZGV4IGRiMzUzOGIzLi5mN2MxYzA5NSAxMDA3NTUNCi0tLSBhL2NvbmZpZ3VyZQ0K
+KysrIGIvY29uZmlndXJlDQpAQCAtODE0LDcgKzgxNCw3IEBAIGlmIFsgJGhhdmVfcHl0aG9uMyAt
+ZXEgMSBdOyB0aGVuDQogICAgIHByaW50ZiAiQ2hlY2tpbmcgZm9yIHB5dGhvbjMgcHl0ZXN0ICg+
+PSAzLjApLi4uICINCiAgICAgY29uZj0kKG1rdGVtcCkNCiAgICAgcHJpbnRmICJbcHl0ZXN0XVxu
+bWludmVyc2lvbj0zLjBcbiIgPiAkY29uZg0KLSAgICBpZiBweXRlc3QtMyAtYyAkY29uZiAtLXZl
+cnNpb24gPi9kZXYvbnVsbCAyPiYxOyB0aGVuDQorICAgIGlmICIkcHl0aG9uIiAtbSBweXRlc3Qg
+LWMgJGNvbmYgLS12ZXJzaW9uID4vZGV2L251bGwgMj4mMTsgdGhlbg0KICAgICAgICAgcHJpbnRm
+ICJZZXMuXG4iDQogICAgICAgICBoYXZlX3B5dGhvbjNfcHl0ZXN0PTENCiAgICAgZWxzZQ0KLS0g
+DQoyLjI4LjANCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
+CmV4YW1wbGUgbWFpbGluZyBsaXN0IC0tIGxpc3RAZXhhbXBsZS5vcmcKVG8gdW5zdWJzY3JpYmUg
+c2VuZCBhbiBlbWFpbCB0byBsaXN0LWxlYXZlQGV4YW1wbGUub3JnCg==
diff --git a/t/t5100/info1000 b/t/t5100/info1000
new file mode 100644
index 0000000000..dab2228b70
--- /dev/null
+++ b/t/t5100/info1000
@@ -0,0 +1,5 @@
+Author: A U Thor
+Email: mail@example.com
+Subject: sample
+Date: Mon, 3 Aug 2020 22:40:55 +0700
+
diff --git a/t/t5100/msg1000 b/t/t5100/msg1000
new file mode 100644
index 0000000000..5e8e860aae
--- /dev/null
+++ b/t/t5100/msg1000
@@ -0,0 +1,2 @@
+On different distro, pytest is suffixed with different patterns.
+
diff --git a/t/t5100/patch1000 b/t/t5100/patch1000
new file mode 100644
index 0000000000..51c4fb4cb5
--- /dev/null
+++ b/t/t5100/patch1000
@@ -0,0 +1,22 @@
+---
+ configure | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/configure b/configure
+index db3538b3..f7c1c095 100755
+--- a/configure
++++ b/configure
+@@ -814,7 +814,7 @@ if [ $have_python3 -eq 1 ]; then
+     printf "Checking for python3 pytest (>= 3.0)... "
+     conf=$(mktemp)
+     printf "[pytest]\nminversion=3.0\n" > $conf
+-    if pytest-3 -c $conf --version >/dev/null 2>&1; then
++    if "$python" -m pytest -c $conf --version >/dev/null 2>&1; then
+         printf "Yes.\n"
+         have_python3_pytest=1
+     else
+-- 
+2.28.0
+_______________________________________________
+example mailing list -- list@example.org
+To unsubscribe send an email to list-leave@example.org

-- 
2.31.1.192.g0881477623
Next: Junio C Hamano
Message 1 of 35 in “mailinfo: strip CR from base64/quoted-printable email”
  1. mailinfo: strip CR from base64/quoted-printable emailĐoàn Trần Công Danh, Apr 21, 2021
  2. Junio C HamanoApr 21, 2021
  3. brian m. carlsonApr 21, 2021
  4. Đoàn Trần Công DanhApr 21, 2021
  5. brian m. carlsonApr 22, 2021
  6. 0/5 Teach am/mailinfo to process quoted CRĐoàn Trần Công Danh, May 4, 2021
  7. 1/5 mailinfo: avoid magic number in option parsingĐoàn Trần Công Danh, May 4, 2021
  8. 2/5 mailinfo: warn if CR found in base64/quoted-printable emailĐoàn Trần Công Danh, May 4, 2021
  9. Junio C HamanoMay 5, 2021
  10. 3/5 mailinfo: skip quoted CR on user's wishĐoàn Trần Công Danh, May 4, 2021
  11. Junio C HamanoMay 5, 2021
  12. Đoàn Trần Công DanhMay 5, 2021
  13. 4/5 mailinfo: strip quoted CR on users' wishĐoàn Trần Công Danh, May 4, 2021
  14. Junio C HamanoMay 5, 2021
  15. 5/5 am: learn to process quoted lines that ends with CRLFĐoàn Trần Công Danh, May 4, 2021
  16. Junio C HamanoMay 5, 2021
  17. 0/6 Teach am/mailinfo to process quoted CRĐoàn Trần Công Danh, May 6, 2021
  18. 1/6 mailinfo: load default metainfo_charset lazilyĐoàn Trần Công Danh, May 6, 2021
  19. 2/6 mailinfo: stop parsing options manuallyĐoàn Trần Công Danh, May 6, 2021
  20. Junio C HamanoMay 8, 2021
  21. 3/6 mailinfo: warn if CR found in decoded base64/QP emailĐoàn Trần Công Danh, May 6, 2021
  22. Junio C HamanoMay 8, 2021
  23. 4/6 mailinfo: allow squelching quoted CR warningĐoàn Trần Công Danh, May 6, 2021
  24. 5/6 mailinfo: allow stripping quoted CR without warningĐoàn Trần Công Danh, May 6, 2021
  25. 6/6 am: learn to process quoted lines that ends with CRLFĐoàn Trần Công Danh, May 6, 2021
  26. Junio C HamanoMay 8, 2021
  27. 0/6 Teach am/mailinfo to process quoted CRĐoàn Trần Công Danh, May 9, 2021
  28. 1/6 mailinfo: load default metainfo_charset lazilyĐoàn Trần Công Danh, May 9, 2021
  29. 2/6 mailinfo: stop parsing options manuallyĐoàn Trần Công Danh, May 9, 2021
  30. 3/6 mailinfo: warn if CRLF found in decoded base64/QP emailĐoàn Trần Công Danh, May 9, 2021
  31. 4/6 mailinfo: allow squelching quoted CRLF warningĐoàn Trần Công Danh, May 9, 2021
  32. 5/6 mailinfo: allow stripping quoted CR without warningĐoàn Trần Công Danh, May 9, 2021
  33. 6/6 am: learn to process quoted lines that ends with CRLFĐoàn Trần Công Danh, May 9, 2021
  34. 2/6 mailinfo: stop parse options manuallyĐoàn Trần Công Danh, May 6, 2021
  35. Đoàn Trần Công DanhMay 6, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.