{"thread":{"id":"52167","subject":"[PATCH 0/3] Linux with musl libc improvement","startedAt":"2019-10-31T09:26:36Z","lastAt":"2019-11-12T04:17:35Z","messageCount":89,"participants":["Doan Tran Cong Danh","Johannes Schindelin","Jeff King","brian m. carlson","Danh Doan","Junio C Hamano","Eric Sunshine","Phillip Wood"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"385205","messageId":"20191031092618.29073-1-congdanhqx@gmail.com","threadId":"52167","inReplyTo":null,"subject":"[PATCH 0/3] Linux with musl libc improvement","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-10-31T09:26:15Z","receivedAt":"2019-10-31T09:26:36Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"This series of patch tries to ease the life of musl libc user (by running\nconfigure once) and clear out those last failure on musl.\n\nDoan Tran Cong Danh (3):\n t0028: eliminate non-standard usage of printf\n configure.ac: define ICONV_OMITS_BOM if necessary\n sequencer: reencode to utf-8 before arrange rebase's todo list\n\n configure.ac | 22 ++++++++++++++++++++++\n sequencer.c | 2 +-\n t/t0028-working-tree-encoding.sh | 4 ++--\n 3 files changed, 25 insertions(+), 3 deletions(-)\n\n-- \n2.24.0.rc1.3.g89530838a3.dirty\n\n"},{"id":"385206","messageId":"20191031092618.29073-2-congdanhqx@gmail.com","threadId":"52167","inReplyTo":"20191031092618.29073-1-congdanhqx@gmail.com","subject":"[PATCH 1/3] t0028: eliminate non-standard usage of printf","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-10-31T09:26:16Z","receivedAt":"2019-10-31T09:26:44Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"man 1p printf:\n In addition to the escape sequences shown in the Base Definitions\n volume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\\\',\n '\\a', '\\b', '\\f', '\\n', '\\r', '\\t', '\\v'), \"\\ddd\", where ddd is a\n one, two, or three-digit octal number, shall be written as a byte\n with the numeric value specified by the octal number.\n\nprintf '\\xfe\\xff' in an extension of some libc.\n\nWith dash:\n$ printf '\\xfe\\xff' | xxd\n00000000: 5c78 6665 5c78 6666 \\xfe\\xff\n\nCorrect its usage.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n\nNotes:\n Despite that dash's printf doesn't accept \\x escape sequence.\n \n My glibc box (with sh linked to dash) can run the test just fine.\n But my musl box couldn't run the test, (because the header).\n\n t/t0028-working-tree-encoding.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh\nindex 7aa0945d8d..bfc4fb9af5 100755\n--- a/t/t0028-working-tree-encoding.sh\n+++ b/t/t0028-working-tree-encoding.sh\n@@ -17,7 +17,7 @@ test_lazy_prereq NO_UTF32_BOM '\n write_utf16 () {\n \tif test_have_prereq NO_UTF16_BOM\n \tthen\n-\t\tprintf '\\xfe\\xff'\n+\t\tprintf '\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-16\n }\n@@ -25,7 +25,7 @@ write_utf16 () {\n write_utf32 () {\n \tif test_have_prereq NO_UTF32_BOM\n \tthen\n-\t\tprintf '\\x00\\x00\\xfe\\xff'\n+\t\tprintf '\\0\\0\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-32\n }\n-- \n2.24.0.rc1.3.g89530838a3.dirty\n\n"},{"id":"385207","messageId":"20191031092618.29073-3-congdanhqx@gmail.com","threadId":"52167","inReplyTo":"20191031092618.29073-1-congdanhqx@gmail.com","subject":"[PATCH 2/3] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-10-31T09:26:17Z","receivedAt":"2019-10-31T09:26:46Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"From commit 79444c9294, (\"utf8: handle systems that don't write BOM for\nUTF-16\", 2019-02-12), we're supporting those systems with iconv that\nomits BOM with:\n\n make ICONV_OMITS_BOM=Yes\n\nHowever, typing the flag all the time is cumbersome and error-prone.\n\nAdd a checking into configure script to detect this flag automatically.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n\nNotes:\n We deliberately fail for ac_cv_iconv_omits_bom on cross-compiling,\n in order to ask builder provide the value for the target.\n \n We're relied on this technik for ac_cv_fread_reads_directories and\n ac_cv_snprintf_returns_bogus.\n \n Adding one more failure for configuring on cross-compiling\n is not going to be a burden for distro.\n\n configure.ac | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/configure.ac b/configure.ac\nindex a43b476402..790b53bbdc 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -690,6 +690,28 @@ fi\n \n fi\n \n+#\n+# Define ICONV_OMITS_BOM if you are on a system which\n+# iconv omits bom for utf-{16,32}\n+if test -z \"$NO_ICONV\"; then\n+AC_CACHE_CHECK([whether iconv omits bom for utf-16 and utf-32],\n+ [ac_cv_iconv_omits_bom],\n+[\n+if test \"x$cross_compiling\" = xyes; then\n+\tAC_MSG_FAILURE([please provide ac_cv_iconv_omits_bom])\n+elif test `printf a | iconv -f utf-8 -t utf-16 | wc -c` = 2; then\n+\tac_cv_iconv_omits_bom=yes\n+else\n+\tac_cv_iconv_omits_bom=no\n+fi\n+])\n+if test \"x$ac_cv_iconv_omits_bom\" = xyes; then\n+\tICONV_OMITS_BOM=Yes\n+else\n+\tICONV_OMITS_BOM=\n+fi\n+GIT_CONF_SUBST([ICONV_OMITS_BOM])\n+fi\n #\n # Define NO_DEFLATE_BOUND if deflateBound is missing from zlib.\n \n-- \n2.24.0.rc1.3.g89530838a3.dirty\n\n"},{"id":"385208","messageId":"20191031092618.29073-4-congdanhqx@gmail.com","threadId":"52167","inReplyTo":"20191031092618.29073-1-congdanhqx@gmail.com","subject":"[PATCH 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-10-31T09:26:18Z","receivedAt":"2019-10-31T09:26:50Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On musl libc, ISO-2022-JP encoder is too eager to switch back to\n1 byte encoding, musl's iconv always switch back after every combining\ncharacter. Comparing glibc and musl's output for this command\n$ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 |\n\ticonv -f utf-8 -t ISO-2022-JP | xxd\n\nglibc:\n00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B\n00000010: 0a .\n\nmusl:\n00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B\n00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B\n00000020: 1b24 4224 551b 2842 0a .$B$U.(B.\n\nAlthough musl iconv's output isn't optimal, it's still correct.\n\nFrom commit 7d509878b8, (\"pretty.c: format string with truncate respects\nlogOutputEncoding\", 2014-05-21), we're encoding the message to utf-8\nfirst, then format it and convert the message to the actual output\nencoding on git commit --squash.\n\nThus, t3900 is failing on Linux with musl libc.\n\nReencode to utf-8 before arranging rebase's todo list.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n\nNotes:\n The todo list shown to user has already been reencoded by sequencer_make_script,\n without this patch it looks like this:\n \n $ head -3 .git/rebase-merge/git-rebase-todo | xxd\n 00000000: 7069 636b 2065 6633 3961 3033 201b 2442 pick ef39a03 .$B\n 00000010: 244f 1b28 421b 2442 246c 1b28 421b 2442 $O.(B.$B$l.(B.$B\n 00000020: 2452 1b28 421b 2442 245b 1b28 421b 2442 $R.(B.$B$[.(B.$B\n 00000030: 2455 1b28 420a 7069 636b 2062 3832 3931 $U.(B.pick b8291\n 00000040: 3336 2073 7175 6173 6821 201b 2442 244f 36 squash! .$B$O\n 00000050: 1b28 421b 2442 246c 1b28 421b 2442 2452 .(B.$B$l.(B.$B$R\n 00000060: 1b28 421b 2442 245b 1b28 421b 2442 2455 .(B.$B$[.(B.$B$U\n 00000070: 1b28 420a 7069 636b 2062 3532 3132 6437 .(B.pick b5212d7\n 00000080: 2069 6e74 6572 6d65 6469 6174 6520 636f intermediate co\n 00000090: 6d6d 6974 0a mmit.\n\n sequencer.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 9d5964fd81..69430fe23f 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -5169,7 +5169,7 @@ int todo_list_rearrange_squash(struct todo_list *todo_list)\n \t\t*commit_todo_item_at(&commit_todo, item->commit) = item;\n \n \t\tparse_commit(item->commit);\n-\t\tcommit_buffer = get_commit_buffer(item->commit, NULL);\n+\t\tcommit_buffer = logmsg_reencode(item->commit, NULL, \"UTF-8\");\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tformat_subject(&buf, subject, \" \");\n \t\tsubject = subjects[i] = strbuf_detach(&buf, &subject_len);\n-- \n2.24.0.rc1.3.g89530838a3.dirty\n\n"},{"id":"385213","messageId":"nycvar.QRO.7.76.6.1910311134011.46@tvgsbejvaqbjf.bet","threadId":"52167","inReplyTo":"20191031092618.29073-4-congdanhqx@gmail.com","subject":"Re: [PATCH 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-10-31T10:38:14Z","receivedAt":"2019-10-31T10:38:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\n\nOn Thu, 31 Oct 2019, Doan Tran Cong Danh wrote:\n\n> On musl libc, ISO-2022-JP encoder is too eager to switch back to\n> 1 byte encoding, musl's iconv always switch back after every combining\n> character. Comparing glibc and musl's output for this command\n> $ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 |\n> \ticonv -f utf-8 -t ISO-2022-JP | xxd\n>\n> glibc:\n> 00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B\n> 00000010: 0a .\n>\n> musl:\n> 00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B\n> 00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B\n> 00000020: 1b24 4224 551b 2842 0a .$B$U.(B.\n>\n> Although musl iconv's output isn't optimal, it's still correct.\n>\n> From commit 7d509878b8, (\"pretty.c: format string with truncate respects\n> logOutputEncoding\", 2014-05-21), we're encoding the message to utf-8\n> first, then format it and convert the message to the actual output\n> encoding on git commit --squash.\n>\n> Thus, t3900 is failing on Linux with musl libc.\n>\n> Reencode to utf-8 before arranging rebase's todo list.\n\nSince the re-encoded commit messages are only used for figuring out the\nrelationships between the `fixup!`/`squash!` commits and their targets,\nbut are not used in any of the lines that are written out, this change\nlooks good to me.\n\nIn fact, all three patches look good to me.\n\nThanks for unbreaking Git with musl!\nJohannes\n\n>\n> Signed-off-by: Doan Tran Cong Danh \n> ---\n>\n> Notes:\n> The todo list shown to user has already been reencoded by sequencer_make_script,\n> without this patch it looks like this:\n>\n> $ head -3 .git/rebase-merge/git-rebase-todo | xxd\n> 00000000: 7069 636b 2065 6633 3961 3033 201b 2442 pick ef39a03 .$B\n> 00000010: 244f 1b28 421b 2442 246c 1b28 421b 2442 $O.(B.$B$l.(B.$B\n> 00000020: 2452 1b28 421b 2442 245b 1b28 421b 2442 $R.(B.$B$[.(B.$B\n> 00000030: 2455 1b28 420a 7069 636b 2062 3832 3931 $U.(B.pick b8291\n> 00000040: 3336 2073 7175 6173 6821 201b 2442 244f 36 squash! .$B$O\n> 00000050: 1b28 421b 2442 246c 1b28 421b 2442 2452 .(B.$B$l.(B.$B$R\n> 00000060: 1b28 421b 2442 245b 1b28 421b 2442 2455 .(B.$B$[.(B.$B$U\n> 00000070: 1b28 420a 7069 636b 2062 3532 3132 6437 .(B.pick b5212d7\n> 00000080: 2069 6e74 6572 6d65 6469 6174 6520 636f intermediate co\n> 00000090: 6d6d 6974 0a mmit.\n>\n> sequencer.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/sequencer.c b/sequencer.c\n> index 9d5964fd81..69430fe23f 100644\n> --- a/sequencer.c\n> +++ b/sequencer.c\n> @@ -5169,7 +5169,7 @@ int todo_list_rearrange_squash(struct todo_list *todo_list)\n> \t\t*commit_todo_item_at(&commit_todo, item->commit) = item;\n>\n> \t\tparse_commit(item->commit);\n> -\t\tcommit_buffer = get_commit_buffer(item->commit, NULL);\n> +\t\tcommit_buffer = logmsg_reencode(item->commit, NULL, \"UTF-8\");\n> \t\tfind_commit_subject(commit_buffer, &subject);\n> \t\tformat_subject(&buf, subject, \" \");\n> \t\tsubject = subjects[i] = strbuf_detach(&buf, &subject_len);\n> --\n> 2.24.0.rc1.3.g89530838a3.dirty\n>\n>\n"},{"id":"385227","messageId":"20191031174118.GA2133@sigill.intra.peff.net","threadId":"52167","inReplyTo":"20191031092618.29073-2-congdanhqx@gmail.com","subject":"Re: [PATCH 1/3] t0028: eliminate non-standard usage of printf","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-31T17:41:18Z","receivedAt":"2019-10-31T17:41:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 31, 2019 at 04:26:16PM +0700, Doan Tran Cong Danh wrote:\n\n> man 1p printf:\n> In addition to the escape sequences shown in the Base Definitions\n> volume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\\\',\n> '\\a', '\\b', '\\f', '\\n', '\\r', '\\t', '\\v'), \"\\ddd\", where ddd is a\n> one, two, or three-digit octal number, shall be written as a byte\n> with the numeric value specified by the octal number.\n> \n> printf '\\xfe\\xff' in an extension of some libc.\n\nI don't think it's libc here at all, but rather the shell.\n\nAnd as you note, dash does not handle this:\n\n> With dash:\n> $ printf '\\xfe\\xff' | xxd\n> 00000000: 5c78 6665 5c78 6666 \\xfe\\xff\n\nWhich makes me wonder how these tests worked at all for people on say,\nDebian systems.\n\nI think the answer is that they don't trigger this prereq:\n\n> write_utf16 () {\n> \tif test_have_prereq NO_UTF16_BOM\n> \tthen\n> -\t\tprintf '\\xfe\\xff'\n> +\t\tprintf '\\376\\377'\n> \tfi &&\n> \ticonv -f UTF-8 -t UTF-16\n> }\n\nwhich comes from:\n\n test_lazy_prereq NO_UTF16_BOM '\n test $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6\n '\n\nPresumably on your system iconv is using musl's internal iconv, which\nbehaves differently than the glibc one.\n\nSo I think your patch is the right thing to do (hex escapes in shell\nprintf are definitely not portable), but we might want to note something\nlike:\n\n This wasn't caught by most people running the tests, even though\n common shells like dash don't handle hex escapes, because their\n systems don't trigger the NO_UTF16_BOM prereq. But systems with musl\n libc do; when combined with dash, the test fails.\n\n-Peff\n"},{"id":"385229","messageId":"20191031181116.GC2133@sigill.intra.peff.net","threadId":"52167","inReplyTo":"20191031092618.29073-3-congdanhqx@gmail.com","subject":"Re: [PATCH 2/3] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-31T18:11:16Z","receivedAt":"2019-10-31T18:11:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 31, 2019 at 04:26:17PM +0700, Doan Tran Cong Danh wrote:\n\n> From commit 79444c9294, (\"utf8: handle systems that don't write BOM for\n> UTF-16\", 2019-02-12), we're supporting those systems with iconv that\n> omits BOM with:\n> \n> make ICONV_OMITS_BOM=Yes\n> \n> However, typing the flag all the time is cumbersome and error-prone.\n> \n> Add a checking into configure script to detect this flag automatically.\n\nI think it's worth adding this to the configure script, but note that\nyou can also do:\n\n echo ICONV_OMITS_BOM=Yes >config.mak\n\nand then \"make\" by itself will do what you want (it's still annoying and\nerror-prone to realize you have to specify the flag in the first place).\n\n> diff --git a/configure.ac b/configure.ac\n> index a43b476402..790b53bbdc 100644\n> --- a/configure.ac\n> +++ b/configure.ac\n> @@ -690,6 +690,28 @@ fi\n> \n> fi\n> \n> +#\n> +# Define ICONV_OMITS_BOM if you are on a system which\n> +# iconv omits bom for utf-{16,32}\n> +if test -z \"$NO_ICONV\"; then\n> +AC_CACHE_CHECK([whether iconv omits bom for utf-16 and utf-32],\n> + [ac_cv_iconv_omits_bom],\n> +[\n> +if test \"x$cross_compiling\" = xyes; then\n> +\tAC_MSG_FAILURE([please provide ac_cv_iconv_omits_bom])\n> +elif test `printf a | iconv -f utf-8 -t utf-16 | wc -c` = 2; then\n\nThe ICONV_OMITS_BOM flag is about the libc iconv that Git will be linked\nagainst. But this is checking the iconv tool. For a system that is using\nmusl across the board, that would work. But it might not always be the\ncase (in particular, I don't know if people statically link some\nbinaries against musl; certainly I've seen people do it with dietlibc).\n\nI think we should be test-compiling a small program, similar to the way\nOLD_ICONV works (though I guess we may even need to run the result to\nsee what happens).\n\n-Peff\n"},{"id":"385231","messageId":"20191031192650.GA12834@sigill.intra.peff.net","threadId":"52167","inReplyTo":"nycvar.QRO.7.76.6.1910311134011.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-31T19:26:50Z","receivedAt":"2019-10-31T19:26:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 31, 2019 at 11:38:14AM +0100, Johannes Schindelin wrote:\n\n> On Thu, 31 Oct 2019, Doan Tran Cong Danh wrote:\n> \n> > On musl libc, ISO-2022-JP encoder is too eager to switch back to\n> > 1 byte encoding, musl's iconv always switch back after every combining\n> > character. Comparing glibc and musl's output for this command\n> > $ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 |\n> > \ticonv -f utf-8 -t ISO-2022-JP | xxd\n> >\n> > glibc:\n> > 00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B\n> > 00000010: 0a .\n> >\n> > musl:\n> > 00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B\n> > 00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B\n> > 00000020: 1b24 4224 551b 2842 0a .$B$U.(B.\n> >\n> > Although musl iconv's output isn't optimal, it's still correct.\n> >\n> > From commit 7d509878b8, (\"pretty.c: format string with truncate respects\n> > logOutputEncoding\", 2014-05-21), we're encoding the message to utf-8\n> > first, then format it and convert the message to the actual output\n> > encoding on git commit --squash.\n> >\n> > Thus, t3900 is failing on Linux with musl libc.\n> >\n> > Reencode to utf-8 before arranging rebase's todo list.\n> \n> Since the re-encoded commit messages are only used for figuring out the\n> relationships between the `fixup!`/`squash!` commits and their targets,\n> but are not used in any of the lines that are written out, this change\n> looks good to me.\n\nI'm confused about a few things here, though. I agree with you that the\nsubjects here are only used for finding the fixup/squash relationships.\nBut I don't understand the musl connection.\n\nWouldn't failure to reencode here always be a problem? E.g., if I do:\n\n for encoding in utf-8 iso-8859-1; do\n # commit using the encoding\n echo $encoding >file && git add file\n echo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n git -c i18n.commitEncoding=$encoding commit -F -\n # and then fixup without it\n echo \"$encoding fixed\" >file && git add file\n git commit --fixup HEAD\n done\n \n GIT_EDITOR='echo; grep -v ^#' git rebase -i --root --autosquash\n\nthen the resulting todo-list output (on my glibc system) is:\n\n pick 3a5bace éñcödèd with utf-8\n fixup aa9f09c fixup! éñcödèd with utf-8\n pick 6e85d32 éñcödèd with iso-8859-1\n pick 3ceac05 fixup! éñcödèd with iso-8859-1\n\nI.e., we don't actually match up the second pair, and I think we\nprobably ought to.\n\nI guess the test in t3900 is less exotic; it uses the same encoding for\nboth commits. And it's just that \"foo\" and \"!fixup foo\" can (and do in\nmusl) end up with different encodings (because of the specific language,\nand the vagaries of each iconv implementation).\n\nWould we have similar problems in all of the other functions which use\nget_commit_buffer() without reencoding? For instance if I do this:\n\n echo base >file && git add file && git commit -m base\n for encoding in utf-8 iso-8859-1; do\n echo $encoding >file && git add file\n echo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n git -c i18n.commitEncoding=$encoding commit -F -\n done\n git checkout -b side HEAD~2\n git cherry-pick master master^\n cat .git/sequencer/todo\n\nthen the resulting todo file has a mix of iso-8859-1 and utf-8.\n\nIt seems to me that we should always be working with the subjects in a\nsingle encoding internally, and likewise outputting in that format\n(which should probably be git_log_output_encoding(), for the instances\nwhere we show it to the user).\n\nI.e., we should always call logmsg_reencode() instead of\nget_commit_buffer().\n\n-Peff\n"},{"id":"385234","messageId":"20191031195058.prp6cpx6ivf57lz7@camp.crustytoothpaste.net","threadId":"52167","inReplyTo":"20191031092618.29073-2-congdanhqx@gmail.com","subject":"Re: [PATCH 1/3] t0028: eliminate non-standard usage of printf","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2019-10-31T19:50:58Z","receivedAt":"2019-10-31T19:51:06Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2019-10-31 at 09:26:16, Doan Tran Cong Danh wrote:\n> diff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh\n> index 7aa0945d8d..bfc4fb9af5 100755\n> --- a/t/t0028-working-tree-encoding.sh\n> +++ b/t/t0028-working-tree-encoding.sh\n> @@ -17,7 +17,7 @@ test_lazy_prereq NO_UTF32_BOM '\n> write_utf16 () {\n> \tif test_have_prereq NO_UTF16_BOM\n> \tthen\n> -\t\tprintf '\\xfe\\xff'\n> +\t\tprintf '\\376\\377'\n> \tfi &&\n> \ticonv -f UTF-8 -t UTF-16\n> }\n> @@ -25,7 +25,7 @@ write_utf16 () {\n> write_utf32 () {\n> \tif test_have_prereq NO_UTF32_BOM\n> \tthen\n> -\t\tprintf '\\x00\\x00\\xfe\\xff'\n> +\t\tprintf '\\0\\0\\376\\377'\n> \tfi &&\n> \ticonv -f UTF-8 -t UTF-32\n> }\n\nYeah, this patch looks obviously correct. For some reason, I knew that\nprintf(1) didn't accept hex sequences and yet I still wrote them.\n\nAs Peff pointed out, this is because we only trigger this case on musl;\nmusl is the only known implementation of iconv that produces big-endian\ndata without a BOM for UTF-16 and UTF-32.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"385236","messageId":"20191031200214.h63elcyhzknd65hs@camp.crustytoothpaste.net","threadId":"52167","inReplyTo":"20191031181116.GC2133@sigill.intra.peff.net","subject":"Re: [PATCH 2/3] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2019-10-31T20:02:14Z","receivedAt":"2019-10-31T20:02:21Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2019-10-31 at 18:11:16, Jeff King wrote:\n> On Thu, Oct 31, 2019 at 04:26:17PM +0700, Doan Tran Cong Danh wrote:\n> > diff --git a/configure.ac b/configure.ac\n> > index a43b476402..790b53bbdc 100644\n> > --- a/configure.ac\n> > +++ b/configure.ac\n> > @@ -690,6 +690,28 @@ fi\n> > \n> > fi\n> > \n> > +#\n> > +# Define ICONV_OMITS_BOM if you are on a system which\n> > +# iconv omits bom for utf-{16,32}\n> > +if test -z \"$NO_ICONV\"; then\n> > +AC_CACHE_CHECK([whether iconv omits bom for utf-16 and utf-32],\n> > + [ac_cv_iconv_omits_bom],\n> > +[\n> > +if test \"x$cross_compiling\" = xyes; then\n> > +\tAC_MSG_FAILURE([please provide ac_cv_iconv_omits_bom])\n> > +elif test `printf a | iconv -f utf-8 -t utf-16 | wc -c` = 2; then\n> \n> The ICONV_OMITS_BOM flag is about the libc iconv that Git will be linked\n> against. But this is checking the iconv tool. For a system that is using\n> musl across the board, that would work. But it might not always be the\n> case (in particular, I don't know if people statically link some\n> binaries against musl; certainly I've seen people do it with dietlibc).\n\nYeah, I think we need to do that, and not only for musl. There are\nfolks for whom the iconv in libc is missing or inadequate and they use\nan additional iconv(3) implementation which may not be the same as\niconv(1) (or vice versa).\n\nGranted, as far as we know this option is only needed for musl, but that\ndoesn't mean there aren't other environments where this is a problem,\nonly that we don't yet know about them.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"385252","messageId":"20191101013307.GD30350@danh.dev","threadId":"52167","inReplyTo":"20191031174118.GA2133@sigill.intra.peff.net","subject":"Re: [PATCH 1/3] t0028: eliminate non-standard usage of printf","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-01T01:33:07Z","receivedAt":"2019-11-01T01:33:13Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-10-31 13:41:18 -0400, Jeff King wrote:\n> So I think your patch is the right thing to do (hex escapes in shell\n> printf are definitely not portable), but we might want to note something\n> like:\n> \n> This wasn't caught by most people running the tests, even though\n> common shells like dash don't handle hex escapes, because their\n> systems don't trigger the NO_UTF16_BOM prereq. But systems with musl\n> libc do; when combined with dash, the test fails.\n\nMake sense. I will take this note in the reroll.\n\n-- \nDanh\n"},{"id":"385253","messageId":"20191101014006.GE30350@danh.dev","threadId":"52167","inReplyTo":"20191031181116.GC2133@sigill.intra.peff.net","subject":"Re: [PATCH 2/3] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-01T01:40:06Z","receivedAt":"2019-11-01T01:40:10Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-10-31 14:11:16 -0400, Jeff King wrote:\n> The ICONV_OMITS_BOM flag is about the libc iconv that Git will be linked\n> against. But this is checking the iconv tool. For a system that is using\n> musl across the board, that would work. But it might not always be the\n> case (in particular, I don't know if people statically link some\n> binaries against musl; certainly I've seen people do it with dietlibc).\n> \n> I think we should be test-compiling a small program, similar to the way\n> OLD_ICONV works (though I guess we may even need to run the result to\n> see what happens).\n\nOriginally, I wrote a C program for AC_RUN_IFELSE, but I found out the\ncomplicated of OLD_ICONV and NEEDS_LIBICONV, then I switch to this\napproach.\n\nBut, your reasoning makes sense. I think people could build a static-linked\ngit binary for a clean-room build system.\n\nI'll pursue the C program in the re-roll.\n\n-- \nDanh\n"},{"id":"385254","messageId":"20191101044949.GA26545@danh.dev","threadId":"52167","inReplyTo":"20191031192650.GA12834@sigill.intra.peff.net","subject":"Re: [PATCH 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-01T04:49:49Z","receivedAt":"2019-11-01T04:49:57Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-10-31 15:26:50 -0400, Jeff King wrote:\n> I'm confused about a few things here, though. I agree with you that the\n> subjects here are only used for finding the fixup/squash relationships.\n> But I don't understand the musl connection.\n\nYou're right.\n\nBecause of musl's iconv implementation, the problem is being shown up\nearlier.\n\n> Wouldn't failure to reencode here always be a problem? E.g., if I do:\n> \n> for encoding in utf-8 iso-8859-1; do\n> # commit using the encoding\n> echo $encoding >file && git add file\n> echo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n> git -c i18n.commitEncoding=$encoding commit -F -\n> # and then fixup without it\n> echo \"$encoding fixed\" >file && git add file\n> git commit --fixup HEAD\n> done\n> \n> GIT_EDITOR='echo; grep -v ^#' git rebase -i --root --autosquash\n> \n> then the resulting todo-list output (on my glibc system) is:\n> \n> pick 3a5bace éñcödèd with utf-8\n> fixup aa9f09c fixup! éñcödèd with utf-8\n> pick 6e85d32 éñcödèd with iso-8859-1\n> pick 3ceac05 fixup! éñcödèd with iso-8859-1\n> \n> I.e., we don't actually match up the second pair, and I think we\n> probably ought to.\n\nYes, we ought to match up the second pair, and after changing\nget_commit_buffer to logmsg_reencode, we do.\n\n> \n> I guess the test in t3900 is less exotic; it uses the same encoding for\n> both commits. And it's just that \"foo\" and \"!fixup foo\" can (and do in\n> musl) end up with different encodings (because of the specific language,\n> and the vagaries of each iconv implementation).\n> \n> Would we have similar problems in all of the other functions which use\n> get_commit_buffer() without reencoding? For instance if I do this:\n> \n> echo base >file && git add file && git commit -m base\n> for encoding in utf-8 iso-8859-1; do\n> echo $encoding >file && git add file\n> echo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n> git -c i18n.commitEncoding=$encoding commit -F -\n> done\n> git checkout -b side HEAD~2\n> git cherry-pick master master^\n> cat .git/sequencer/todo\n> \n> then the resulting todo file has a mix of iso-8859-1 and utf-8.\n> \n> It seems to me that we should always be working with the subjects in a\n> single encoding internally,\n\nI'm in favour of this idea.\n\n> and likewise outputting in that format\n> (which should probably be git_log_output_encoding(), for the instances\n> where we show it to the user).\n\nThis is git's current behaviour but it's get_log_output_encoding()\ninstead of git_log_output_encoding().\n\n> I.e., we should always call logmsg_reencode() instead of\n> get_commit_buffer().\n\n-- \nDanh\n"},{"id":"385255","messageId":"cover.1572596278.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"20191031092618.29073-1-congdanhqx@gmail.com","subject":"[PATCH v2 0/3] Linux with musl libc improvement","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-01T08:25:08Z","receivedAt":"2019-11-01T08:25:25Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"This series of patch tries to ease the life of musl libc user\nand clear those last failure on Linux with musl libc.\n\nThe second patch is a complete rewrite.\n\nDoan Tran Cong Danh (3):\n t0028: eliminate non-standard usage of printf\n configure.ac: define ICONV_OMITS_BOM if necessary\n sequencer: reencode to utf-8 before arrange rebase's todo list\n\n configure.ac | 49 ++++++++++++++++++++++++++++++++\n sequencer.c | 2 +-\n t/t0028-working-tree-encoding.sh | 4 +--\n 3 files changed, 52 insertions(+), 3 deletions(-)\n\nRange-diff against v1:\n1: 840c40cea1 ! 1: 8b30028425 t0028: eliminate non-standard usage of printf\n @@ Commit message\n one, two, or three-digit octal number, shall be written as a byte\n with the numeric value specified by the octal number.\n \n - printf '\\xfe\\xff' in an extension of some libc.\n + printf '\\xfe\\xff' in an extension of some shell.\n + Dash, a popular yet simple shell, do not implement this extension.\n \n - With dash:\n - $ printf '\\xfe\\xff' | xxd\n - 00000000: 5c78 6665 5c78 6666 \\xfe\\xff\n + This wasn't caught by most people running the tests, even though\n + common shells like dash don't handle hex escapes, because their\n + systems don't trigger the NO_UTF16_BOM prereq. But systems with musl\n + libc do; when combined with dash, the test fails.\n \n - Correct its usage.\n + Correct it.\n \n Signed-off-by: Doan Tran Cong Danh \n - Despite that dash's printf doesn't accept \\x escape sequence.\n -\n - My glibc box (with sh linked to dash) can run the test just fine.\n - But my musl box couldn't run the test, (because the header).\n \n ## t/t0028-working-tree-encoding.sh ##\n @@ t/t0028-working-tree-encoding.sh: test_lazy_prereq NO_UTF32_BOM '\n2: 4a28ee7ef6 < -: ---------- configure.ac: define ICONV_OMITS_BOM if necessary\n-: ---------- > 2: 7c2c6f0603 configure.ac: define ICONV_OMITS_BOM if necessary\n3: c8da3990e5 ! 3: b7927b2723 sequencer: reencode to utf-8 before arrange rebase's todo list\n @@ Commit message\n \n Thus, t3900 is failing on Linux with musl libc.\n \n + This problem wasn't specific to musl libc. On Linux with glibc, this\n + problem can be observed by:\n +\n + for encoding in utf-8 iso-8859-1; do\n + # commit using the encoding\n + echo $encoding >file && git add file\n + echo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n + git -c i18n.commitEncoding=$encoding commit -F -\n + # and then fixup without it\n + echo \"$encoding fixed\" >file && git add file\n + git commit --fixup HEAD\n + done\n + git rebase -i --autosquash --root\n +\n Reencode to utf-8 before arranging rebase's todo list.\n \n Signed-off-by: Doan Tran Cong Danh \n - The todo list shown to user has already been reencoded by sequencer_make_script,\n - without this patch it looks like this:\n -\n - $ head -3 .git/rebase-merge/git-rebase-todo | xxd\n - 00000000: 7069 636b 2065 6633 3961 3033 201b 2442 pick ef39a03 .$B\n - 00000010: 244f 1b28 421b 2442 246c 1b28 421b 2442 $O.(B.$B$l.(B.$B\n - 00000020: 2452 1b28 421b 2442 245b 1b28 421b 2442 $R.(B.$B$[.(B.$B\n - 00000030: 2455 1b28 420a 7069 636b 2062 3832 3931 $U.(B.pick b8291\n - 00000040: 3336 2073 7175 6173 6821 201b 2442 244f 36 squash! .$B$O\n - 00000050: 1b28 421b 2442 246c 1b28 421b 2442 2452 .(B.$B$l.(B.$B$R\n - 00000060: 1b28 421b 2442 245b 1b28 421b 2442 2455 .(B.$B$[.(B.$B$U\n - 00000070: 1b28 420a 7069 636b 2062 3532 3132 6437 .(B.pick b5212d7\n - 00000080: 2069 6e74 6572 6d65 6469 6174 6520 636f intermediate co\n - 00000090: 6d6d 6974 0a mmit.\n \n ## sequencer.c ##\n @@ sequencer.c: int todo_list_rearrange_squash(struct todo_list *todo_list)\n-- \n2.24.0.rc2.296.geaf699fcc3\n\n"},{"id":"385256","messageId":"8b3002842537355c2387cb55f52c10ad8a7dd530.1572596278.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1572596278.git.congdanhqx@gmail.com","subject":"[PATCH v2 1/3] t0028: eliminate non-standard usage of printf","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-01T08:25:09Z","receivedAt":"2019-11-01T08:25:28Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"man 1p printf:\n In addition to the escape sequences shown in the Base Definitions\n volume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\\\',\n '\\a', '\\b', '\\f', '\\n', '\\r', '\\t', '\\v'), \"\\ddd\", where ddd is a\n one, two, or three-digit octal number, shall be written as a byte\n with the numeric value specified by the octal number.\n\nprintf '\\xfe\\xff' in an extension of some shell.\nDash, a popular yet simple shell, do not implement this extension.\n\nThis wasn't caught by most people running the tests, even though\ncommon shells like dash don't handle hex escapes, because their\nsystems don't trigger the NO_UTF16_BOM prereq. But systems with musl\nlibc do; when combined with dash, the test fails.\n\nCorrect it.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n t/t0028-working-tree-encoding.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh\nindex 7aa0945d8d..bfc4fb9af5 100755\n--- a/t/t0028-working-tree-encoding.sh\n+++ b/t/t0028-working-tree-encoding.sh\n@@ -17,7 +17,7 @@ test_lazy_prereq NO_UTF32_BOM '\n write_utf16 () {\n \tif test_have_prereq NO_UTF16_BOM\n \tthen\n-\t\tprintf '\\xfe\\xff'\n+\t\tprintf '\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-16\n }\n@@ -25,7 +25,7 @@ write_utf16 () {\n write_utf32 () {\n \tif test_have_prereq NO_UTF32_BOM\n \tthen\n-\t\tprintf '\\x00\\x00\\xfe\\xff'\n+\t\tprintf '\\0\\0\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-32\n }\n-- \n2.24.0.rc2.296.geaf699fcc3\n\n"},{"id":"385257","messageId":"7c2c6f060393abf3ac831837e4bf8ed366c2fa69.1572596278.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1572596278.git.congdanhqx@gmail.com","subject":"[PATCH v2 2/3] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-01T08:25:10Z","receivedAt":"2019-11-01T08:25:34Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"From commit 79444c9294, (\"utf8: handle systems that don't write BOM for\nUTF-16\", 2019-02-12), we're supporting those systems with iconv that\nomits BOM with:\n\n make ICONV_OMITS_BOM=Yes\n\nHowever, typing the flag all the time is cumbersome and error-prone.\n\nAdd a checking into configure script to detect this flag automatically.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n configure.ac | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/configure.ac b/configure.ac\nindex a43b476402..ecba7e6e51 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -844,12 +844,61 @@ AC_MSG_CHECKING([for old iconv()])\n AC_COMPILE_IFELSE([OLDICONVTEST_SRC],\n \t[AC_MSG_RESULT([no])],\n \t[AC_MSG_RESULT([yes])\n+\tAC_DEFINE(HAVE_OLD_ICONV, 1)\n \tOLD_ICONV=UnfortunatelyYes])\n \n GIT_UNSTASH_FLAGS($ICONVDIR)\n \n GIT_CONF_SUBST([OLD_ICONV])\n \n+#\n+# Define ICONV_OMITS_BOM if you are on a system which\n+# iconv omits bom for utf-{16,32}\n+if test -z \"$NO_ICONV\"; then\n+AC_CACHE_CHECK([whether iconv omits bom for utf-16 and utf-32],\n+ [ac_cv_iconv_omits_bom],\n+[\n+old_LIBS=\"$LIBS\"\n+if test -n \"$NEEDS_LIBICONV\"; then\n+\tLIBS=\"$LIBS -liconv\"\n+fi\n+\n+AC_RUN_IFELSE(\n+\t[AC_LANG_PROGRAM([AC_INCLUDES_DEFAULT\n+\t#include \n+\t#ifdef HAVE_OLD_ICONV\n+\ttypedef const char *iconv_ibp;\n+\t#else\n+\ttypedef char *iconv_ibp;\n+\t#endif\n+\t],\n+\t[[\n+\tint v;\n+\ticonv_t conv;\n+\tchar in[] = \"a\"; iconv_ibp pin = in;\n+\tchar out[20] = \"\"; char *pout = out;\n+\tsize_t isz = sizeof in;\n+\tsize_t osz = sizeof out;\n+\n+\tconv = iconv_open(\"UTF-16\", \"UTF-8\");\n+\ticonv(conv, &pin, &isz, &pout, &osz);\n+\ticonv_close(conv);\n+\tv = (unsigned char)(out[0]) + (unsigned char)(out[1]);\n+\treturn v != 0xfe + 0xff;\n+\t]])],\n+\t[ac_cv_iconv_omits_bom=no],\n+\t[ac_cv_iconv_omits_bom=yes])\n+\n+LIBS=\"$old_LIBS\"\n+])\n+if test \"x$ac_cv_iconv_omits_bom\" = xyes; then\n+\tICONV_OMITS_BOM=Yes\n+else\n+\tICONV_OMITS_BOM=\n+fi\n+GIT_CONF_SUBST([ICONV_OMITS_BOM])\n+fi\n+\n ## Checks for typedefs, structures, and compiler characteristics.\n AC_MSG_NOTICE([CHECKS for typedefs, structures, and compiler characteristics])\n #\n-- \n2.24.0.rc2.296.geaf699fcc3\n\n"},{"id":"385258","messageId":"b7927b27235422ac53595cfaa63b4f1cbe009013.1572596278.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1572596278.git.congdanhqx@gmail.com","subject":"[PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-01T08:25:11Z","receivedAt":"2019-11-01T08:25:37Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On musl libc, ISO-2022-JP encoder is too eager to switch back to\n1 byte encoding, musl's iconv always switch back after every combining\ncharacter. Comparing glibc and musl's output for this command\n$ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 |\n\ticonv -f utf-8 -t ISO-2022-JP | xxd\n\nglibc:\n00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B\n00000010: 0a .\n\nmusl:\n00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B\n00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B\n00000020: 1b24 4224 551b 2842 0a .$B$U.(B.\n\nAlthough musl iconv's output isn't optimal, it's still correct.\n\nFrom commit 7d509878b8, (\"pretty.c: format string with truncate respects\nlogOutputEncoding\", 2014-05-21), we're encoding the message to utf-8\nfirst, then format it and convert the message to the actual output\nencoding on git commit --squash.\n\nThus, t3900 is failing on Linux with musl libc.\n\nThis problem wasn't specific to musl libc. On Linux with glibc, this\nproblem can be observed by:\n\nfor encoding in utf-8 iso-8859-1; do\n\t# commit using the encoding\n\techo $encoding >file && git add file\n\techo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n\t git -c i18n.commitEncoding=$encoding commit -F -\n\t# and then fixup without it\n\techo \"$encoding fixed\" >file && git add file\n\tgit commit --fixup HEAD\ndone\ngit rebase -i --autosquash --root\n\nReencode to utf-8 before arranging rebase's todo list.\n\nSigned-off-by: Doan Tran Cong Danh \n---\nThe code that demonstrate the problem on Linux with glibc is written by Jeff.\nBut I don't know how to attribute him properly.\n\n sequencer.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 9d5964fd81..69430fe23f 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -5169,7 +5169,7 @@ int todo_list_rearrange_squash(struct todo_list *todo_list)\n \t\t*commit_todo_item_at(&commit_todo, item->commit) = item;\n \n \t\tparse_commit(item->commit);\n-\t\tcommit_buffer = get_commit_buffer(item->commit, NULL);\n+\t\tcommit_buffer = logmsg_reencode(item->commit, NULL, \"UTF-8\");\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tformat_subject(&buf, subject, \" \");\n \t\tsubject = subjects[i] = strbuf_detach(&buf, &subject_len);\n-- \n2.24.0.rc2.296.geaf699fcc3\n\n"},{"id":"385269","messageId":"20191101165415.GB26219@sigill.intra.peff.net","threadId":"52167","inReplyTo":"8b3002842537355c2387cb55f52c10ad8a7dd530.1572596278.git.congdanhqx@gmail.com","subject":"Re: [PATCH v2 1/3] t0028: eliminate non-standard usage of printf","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-01T16:54:15Z","receivedAt":"2019-11-01T16:54:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 01, 2019 at 03:25:09PM +0700, Doan Tran Cong Danh wrote:\n\n> man 1p printf:\n> In addition to the escape sequences shown in the Base Definitions\n> volume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\\\',\n> '\\a', '\\b', '\\f', '\\n', '\\r', '\\t', '\\v'), \"\\ddd\", where ddd is a\n> one, two, or three-digit octal number, shall be written as a byte\n> with the numeric value specified by the octal number.\n> \n> printf '\\xfe\\xff' in an extension of some shell.\n\nTypo: s/in/is/\n\nOther than that, this looks great to me. Thanks.\n\n-Peff\n"},{"id":"385270","messageId":"20191101165657.GC26219@sigill.intra.peff.net","threadId":"52167","inReplyTo":"7c2c6f060393abf3ac831837e4bf8ed366c2fa69.1572596278.git.congdanhqx@gmail.com","subject":"Re: [PATCH v2 2/3] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-01T16:56:57Z","receivedAt":"2019-11-01T16:57:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 01, 2019 at 03:25:10PM +0700, Doan Tran Cong Danh wrote:\n\n> From commit 79444c9294, (\"utf8: handle systems that don't write BOM for\n> UTF-16\", 2019-02-12), we're supporting those systems with iconv that\n> omits BOM with:\n> \n> make ICONV_OMITS_BOM=Yes\n> \n> However, typing the flag all the time is cumbersome and error-prone.\n> \n> Add a checking into configure script to detect this flag automatically.\n\ns/checking/check/\n\nThe patch looks good to me, though my knowledge of both iconv and\nautoconf is relatively lacking. I'll assume it works on your system,\nthough. :)\n\nOne thing I noticed:\n\n> +\tv = (unsigned char)(out[0]) + (unsigned char)(out[1]);\n> +\treturn v != 0xfe + 0xff;\n\nI wondered at first why not just:\n\n return v[0] != 0xfe || v[1] != 0xff;\n\nbut the answer is that you are catching BOMs of either endianness. We\nmight at some point need to distinguish the two, but I think for the\npurposes of OMITS_BOM, we don't have to care. So we can worry about that\nlater (if ever).\n\n-Peff\n"},{"id":"385271","messageId":"20191101165921.GD26219@sigill.intra.peff.net","threadId":"52167","inReplyTo":"b7927b27235422ac53595cfaa63b4f1cbe009013.1572596278.git.congdanhqx@gmail.com","subject":"Re: [PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-01T16:59:21Z","receivedAt":"2019-11-01T16:59:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 01, 2019 at 03:25:11PM +0700, Doan Tran Cong Danh wrote:\n\n> This problem wasn't specific to musl libc. On Linux with glibc, this\n> problem can be observed by:\n> \n> for encoding in utf-8 iso-8859-1; do\n> \t# commit using the encoding\n> \techo $encoding >file && git add file\n> \techo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n> \t git -c i18n.commitEncoding=$encoding commit -F -\n> \t# and then fixup without it\n> \techo \"$encoding fixed\" >file && git add file\n> \tgit commit --fixup HEAD\n> done\n> git rebase -i --autosquash --root\n\nIs it worth adding this as a test in t3900?\n\n> ---\n> The code that demonstrate the problem on Linux with glibc is written by Jeff.\n> But I don't know how to attribute him properly.\n\nI'm OK with or without attribution, but people sometimes add a\n\"Helped-by\" trailer if they want to acknowledge someone.\n\n> diff --git a/sequencer.c b/sequencer.c\n> index 9d5964fd81..69430fe23f 100644\n> --- a/sequencer.c\n> +++ b/sequencer.c\n> @@ -5169,7 +5169,7 @@ int todo_list_rearrange_squash(struct todo_list *todo_list)\n> \t\t*commit_todo_item_at(&commit_todo, item->commit) = item;\n> \n> \t\tparse_commit(item->commit);\n> -\t\tcommit_buffer = get_commit_buffer(item->commit, NULL);\n> +\t\tcommit_buffer = logmsg_reencode(item->commit, NULL, \"UTF-8\");\n\nI think there are several other spots in this file that could use the\nsame treatment. But I can live with it if you want to just fix the one\nthat's bugging you and move on. It's still a strict improvement.\n\n-Peff\n"},{"id":"385285","messageId":"20191102004342.GA17624@danh.dev","threadId":"52167","inReplyTo":"20191101165657.GC26219@sigill.intra.peff.net","subject":"Re: [PATCH v2 2/3] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-02T00:43:42Z","receivedAt":"2019-11-02T00:43:54Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-11-01 12:56:57 -0400, Jeff King wrote:\n> On Fri, Nov 01, 2019 at 03:25:10PM +0700, Doan Tran Cong Danh wrote:\n> > +\tv = (unsigned char)(out[0]) + (unsigned char)(out[1]);\n> > +\treturn v != 0xfe + 0xff;\n> \n> I wondered at first why not just:\n> \n> return v[0] != 0xfe || v[1] != 0xff;\n> \n> but the answer is that you are catching BOMs of either endianness. We\n> might at some point need to distinguish the two,\n\nYes, it's about catching BOM of either endianess. To distinguish\nbetween LE and BE, it'll be better to use AC_C_BIGENDIAN instead (it's\nfriendlier for cross-compiling).\n\nAnyway, we couldn't compare out[0] and out[1] with 0xf{e,f} directly\nbecause char could be either signed (-2 and -1) or unsigned (0xfe, 0xff).\nWe must cast it to unsigned char (required to be 8 bits by POSIX),\nthen let the integer promotion promote it to int (at least 16 bits).\n\n-- \nDanh\n"},{"id":"385286","messageId":"20191102010215.GB17624@danh.dev","threadId":"52167","inReplyTo":"20191101165921.GD26219@sigill.intra.peff.net","subject":"Re: [PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-02T01:02:15Z","receivedAt":"2019-11-02T01:02:21Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-11-01 12:59:21 -0400, Jeff King wrote:\n> On Fri, Nov 01, 2019 at 03:25:11PM +0700, Doan Tran Cong Danh wrote:\n> \n> > for encoding in utf-8 iso-8859-1; do\n> > \t# commit using the encoding\n> > \techo $encoding >file && git add file\n> > \techo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n> > \t git -c i18n.commitEncoding=$encoding commit -F -\n> > \t# and then fixup without it\n> > \techo \"$encoding fixed\" >file && git add file\n> > \tgit commit --fixup HEAD\n> > done\n> > git rebase -i --autosquash --root\n> \n> Is it worth adding this as a test in t3900?\n\nI think yes, but with a little more work.\nI'll make it as a separated patch in a re-roll.\n\n> > \t\tparse_commit(item->commit);\n> > -\t\tcommit_buffer = get_commit_buffer(item->commit, NULL);\n> > +\t\tcommit_buffer = logmsg_reencode(item->commit, NULL, \"UTF-8\");\n> \n> I think there are several other spots in this file that could use the\n> same treatment. But I can live with it if you want to just fix the one\n> that's bugging you and move on. It's still a strict improvement.\n\nThere're 6 more occurence of get_commit_buffer in sequencer.c, and 13\noccurences in other C source files. I'll try to figure out if it's\nsafe to change.\n\nAnyway, if we're going to working with a single encoding internally,\ncan we take other extreme approach: reencode the commit message to\nutf-8 before writing the commit object? (Is there any codepoint in\nother encoding that can't be reencoded to utf-8?)\nSince git-log and friends are doing 2 steps conversion for commit\nmessage for now (reencode to utf-8 first, then reencode again to\nget_log_output_encoding()). With this new approach, first step is\nlikely a noop (but must be kept for backward compatible).\n\n-- \nDanh\n"},{"id":"385314","messageId":"20191102122023.GA7468@danh.dev","threadId":"52167","inReplyTo":"20191102010215.GB17624@danh.dev","subject":"Re: [PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-02T12:20:23Z","receivedAt":"2019-11-02T12:20:28Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-11-02 08:02:15 +0700, Danh Doan wrote:\n> Anyway, if we're going to working with a single encoding internally,\n> can we take other extreme approach: reencode the commit message to\n> utf-8 before writing the commit object? (Is there any codepoint in\n> other encoding that can't be reencoded to utf-8?)\n\nWith this test (added into t3900):\n\n git checkout -b fixup-ISO-2022-JP-ISO-8859-1 C0 &&\n git config i18n.commitencoding ISO-2022-JP &&\n echo ISO-2022-JP >>F &&\n git commit -a -F \"$TEST_DIRECTORY\"/t3900/ISO-2022-JP.txt &&\n test_tick &&\n echo intermediate stuff >>G &&\n git add G &&\n git commit -a -m \"intermediate commit\" &&\n test_tick &&\n git config i18n.commitencoding ISO-8859-1 &&\n echo ISO-8859-1-fixup >>F &&\n git commit -a --fixup HEAD^ &&\n git config --unset-all i18n.commitencoding &&\n git rebase --autosquash -i HEAD^^^ &&\n git rev-list HEAD >actual &&\n test_line_count = 3 actual\n\nreencode the commit message to utf-8 before writing the commit object\nis (likely) the most simple option to fix it.\n\nAt the very least, the commit message for fixup/squash-ing commit must\nbe encoded in either utf-8 or the target-commit's encode.\n\n-- \nDanh\n"},{"id":"385446","messageId":"20191105080010.GA7415@sigill.intra.peff.net","threadId":"52167","inReplyTo":"20191102010215.GB17624@danh.dev","subject":"Re: [PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-05T08:00:10Z","receivedAt":"2019-11-05T08:00:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Nov 02, 2019 at 08:02:15AM +0700, Danh Doan wrote:\n\n> Anyway, if we're going to working with a single encoding internally,\n> can we take other extreme approach: reencode the commit message to\n> utf-8 before writing the commit object? (Is there any codepoint in\n> other encoding that can't be reencoded to utf-8?)\n\nThat's normally what we do. The only cases we're covering here are when\nsomebody has explicitly asked that the commit object be stored in\nanother encoding. Presumably they'd also be using a matching\ni18n.logOutputEncoding in that case, in which case logmsg_reencode()\nwould be a noop. I think the only reasons to do that are:\n\n 1. You're stuck on some legacy encoding for your terminal. But in that\n case, I think you'd still be better off storing utf-8 and\n translating on the fly, since whatever encoding you do store is\n baked into your objects for all time (so accept some slowness now,\n but eventually move to utf-8).\n\n 2. Your preferred language is bigger in utf-8 than in some specific\n encoding, and you'd rather save some bytes. I'm not sure how big a\n deal this is, given that commit messages don't tend to be that big\n in the first place (compared to trees and blobs). And the zlib\n deflation on the result might help remove some of the redundancy,\n too.\n\nSo I'd actually expect very few people to be using this feature at all\nthese days (which is part of why I would not be all that broken up if we\njust fix the test and move on, if nobody is reporting real-world\nproblems).\n\n> Since git-log and friends are doing 2 steps conversion for commit\n> message for now (reencode to utf-8 first, then reencode again to\n> get_log_output_encoding()). With this new approach, first step is\n> likely a noop (but must be kept for backward compatible).\n\nInteresting. Traditionally we did a single step conversion to the output\nformat, and it looks like most output formats still do that (i.e.,\neverything in pretty_print_commit() except FMT_USERFORMAT, which is what\npowers \"--pretty=format:%s\", etc).\n\nThe two-part user-format thing goes back to 7e77df39bf (pretty: two\nphase conversion for non utf-8 commits, 2013-04-19). It does seem like\nit would be cheaper to convert the format string into the output\nencoding (it would need to be an ascii superset, but that's already the\ncase, since we expect to parse \"author\", etc out of the re-encoded\ncommit object). But again, I have trouble caring too much about the\nperformance of this case, as I consider it to be mostly legacy at this\npoint. But I also don't write in (say) Japanese, so maybe I'm being too\nnarrow-minded about whether people really want to avoid utf-8.\n\n-Peff\n"},{"id":"385503","messageId":"xmqqftj1yeo5.fsf@gitster-ct.c.googlers.com","threadId":"52167","inReplyTo":"20191105080010.GA7415@sigill.intra.peff.net","subject":"Re: [PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-11-06T01:30:34Z","receivedAt":"2019-11-06T01:30:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> That's normally what we do. The only cases we're covering here are when\n> somebody has explicitly asked that the commit object be stored in\n> another encoding. Presumably they'd also be using a matching\n> i18n.logOutputEncoding in that case, in which case logmsg_reencode()\n> would be a noop. I think the only reasons to do that are:\n>\n> 1. You're stuck on some legacy encoding for your terminal. But in that\n> case, I think you'd still be better off storing utf-8 and\n> translating on the fly, since whatever encoding you do store is\n> baked into your objects for all time (so accept some slowness now,\n> but eventually move to utf-8).\n>\n> 2. Your preferred language is bigger in utf-8 than in some specific\n> encoding, and you'd rather save some bytes. I'm not sure how big a\n> deal this is, given that commit messages don't tend to be that big\n> in the first place (compared to trees and blobs). And the zlib\n> deflation on the result might help remove some of the redundancy,\n> too.\n\nPerhaps add\n\n 3. You are dealing with a project originated on and migrated\n from a foreign SCM, and older parts of the history is stored\n in a non-utf-8, even though recent history is in utf-8\n\nto the mix?\n\n> The two-part user-format thing goes back to 7e77df39bf (pretty: two\n> phase conversion for non utf-8 commits, 2013-04-19). It does seem like\n> it would be cheaper to convert the format string into the output\n> encoding (it would need to be an ascii superset, but that's already the\n> case, since we expect to parse \"author\", etc out of the re-encoded\n> commit object). But again, I have trouble caring too much about the\n> performance of this case, as I consider it to be mostly legacy at this\n> point. But I also don't write in (say) Japanese, so maybe I'm being too\n> narrow-minded about whether people really want to avoid utf-8.\n\nI suspect even the heavy Windows/Mac users in Japan have migrated\nout of legacy (the suspicion comes from an anecdote that is offtopic\nhere).\n"},{"id":"385523","messageId":"20191106040314.GA4307@sigill.intra.peff.net","threadId":"52167","inReplyTo":"xmqqftj1yeo5.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-06T04:03:14Z","receivedAt":"2019-11-06T04:03:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 06, 2019 at 10:30:34AM +0900, Junio C Hamano wrote:\n\n> > That's normally what we do. The only cases we're covering here are when\n> > somebody has explicitly asked that the commit object be stored in\n> > another encoding. Presumably they'd also be using a matching\n> > i18n.logOutputEncoding in that case, in which case logmsg_reencode()\n> > would be a noop. I think the only reasons to do that are:\n> >\n> > 1. You're stuck on some legacy encoding for your terminal. But in that\n> > case, I think you'd still be better off storing utf-8 and\n> > translating on the fly, since whatever encoding you do store is\n> > baked into your objects for all time (so accept some slowness now,\n> > but eventually move to utf-8).\n> >\n> > 2. Your preferred language is bigger in utf-8 than in some specific\n> > encoding, and you'd rather save some bytes. I'm not sure how big a\n> > deal this is, given that commit messages don't tend to be that big\n> > in the first place (compared to trees and blobs). And the zlib\n> > deflation on the result might help remove some of the redundancy,\n> > too.\n> \n> Perhaps add\n> \n> 3. You are dealing with a project originated on and migrated\n> from a foreign SCM, and older parts of the history is stored\n> in a non-utf-8, even though recent history is in utf-8\n> \n> to the mix?\n\nI would think you'd want to convert to utf-8 as you do the migration in\nthat case, since you're writing new hashes anyway. But I think a similar\ncase would just be an old Git repository, where for some reason you\nthought i18n.commitEncoding was a good idea back then (perhaps because\nyou were in situation (1) then, but now you aren't).\n\nIn either case, though, I don't think it's a compelling motivation for\noptimization, if only because those old commits will be shown less and\nless (and even without modern optimizations like commit-graph, we'd\ngenerally avoid reencoding those old commits unless we're actually going\nto _show_ them).\n\n> I suspect even the heavy Windows/Mac users in Japan have migrated\n> out of legacy (the suspicion comes from an anecdote that is offtopic\n> here).\n\nThanks for the data point. All of this is very far from my personal\nexperience, so I mostly go on scraps of hearsay I pick up reading this\nor that. :)\n\n-Peff\n"},{"id":"385543","messageId":"cover.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"20191031092618.29073-1-congdanhqx@gmail.com","subject":"[PATCH v3 0/8] Correct internal working and output encoding","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:19:58Z","receivedAt":"2019-11-06T09:20:27Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"The series is shifting from fixing test that failed on musl based Linux to\ncorrect the internal working encoding and output encoding of git-am\ngit-cherry-pick git-rebase and git-revert.\n\nDoan Tran Cong Danh (8):\n t0028: eliminate non-standard usage of printf\n configure.ac: define ICONV_OMITS_BOM if necessary\n t3900: demonstrate git-rebase problem with multi encoding\n sequencer: reencode to utf-8 before arrange rebase's todo list\n sequencer: reencode revert/cherry-pick's todo list\n sequencer: reencode squashing commit's message\n sequencer: reencode old merge-commit message\n sequencer: reencode commit message for am/rebase --show-current-patch\n\n configure.ac | 49 ++++++++++++++++++++++++++++++++\n sequencer.c | 21 +++++++++-----\n t/t0028-working-tree-encoding.sh | 4 +--\n t/t3900-i18n-commit.sh | 42 +++++++++++++++++++++++++++\n 4 files changed, 107 insertions(+), 9 deletions(-)\n\nRange-diff against v2:\n1: 8b30028425 ! 1: b3d6c4e720 t0028: eliminate non-standard usage of printf\n @@ Commit message\n one, two, or three-digit octal number, shall be written as a byte\n with the numeric value specified by the octal number.\n \n - printf '\\xfe\\xff' in an extension of some shell.\n + printf '\\xfe\\xff' is an extension of some shell.\n Dash, a popular yet simple shell, do not implement this extension.\n \n This wasn't caught by most people running the tests, even though\n2: 7c2c6f0603 ! 2: f07566c60c configure.ac: define ICONV_OMITS_BOM if necessary\n @@ Commit message\n \n However, typing the flag all the time is cumbersome and error-prone.\n \n - Add a checking into configure script to detect this flag automatically.\n + Add a check into configure script to detect this flag automatically.\n \n Signed-off-by: Doan Tran Cong Danh \n \n-: ---------- > 3: 662e5bd545 t3900: demonstrate git-rebase problem with multi encoding\n3: b7927b2723 ! 4: 6a51fdd29c sequencer: reencode to utf-8 before arrange rebase's todo list\n @@ Commit message\n first, then format it and convert the message to the actual output\n encoding on git commit --squash.\n \n - Thus, t3900 is failing on Linux with musl libc.\n -\n - This problem wasn't specific to musl libc. On Linux with glibc, this\n - problem can be observed by:\n -\n - for encoding in utf-8 iso-8859-1; do\n - # commit using the encoding\n - echo $encoding >file && git add file\n - echo \"éñcödèd with $encoding\" | iconv -f utf-8 -t $encoding |\n - git -c i18n.commitEncoding=$encoding commit -F -\n - # and then fixup without it\n - echo \"$encoding fixed\" >file && git add file\n - git commit --fixup HEAD\n - done\n - git rebase -i --autosquash --root\n + Thus, t3900::test_commit_autosquash_flags is failing on musl libc.\n \n Reencode to utf-8 before arranging rebase's todo list.\n + By doing this, we also remove a breakage introduced in the previous\n + commit.\n \n Signed-off-by: Doan Tran Cong Danh \n \n @@ sequencer.c: int todo_list_rearrange_squash(struct todo_list *todo_list)\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tformat_subject(&buf, subject, \" \");\n \t\tsubject = subjects[i] = strbuf_detach(&buf, &subject_len);\n +\n + ## t/t3900-i18n-commit.sh ##\n +@@ t/t3900-i18n-commit.sh: test_commit_autosquash_multi_encoding () {\n + \told=$2\n + \tnew=$3\n + \tmsg=$4\n +-\ttest_expect_failure \"commit --$flag into $old from $new\" '\n ++\ttest_expect_success \"commit --$flag into $old from $new\" '\n + \t\tgit checkout -b '$flag-$old-$new' C0 &&\n + \t\tgit config i18n.commitencoding '$old' &&\n + \t\techo '$old' >>F &&\n-: ---------- > 5: d382e35e4e sequencer: reencode revert/cherry-pick's todo list\n-: ---------- > 6: 340902eb67 sequencer: reencode squashing commit's message\n-: ---------- > 7: 7f0df0f685 sequencer: reencode old merge-commit message\n-: ---------- > 8: 69ec40bb1d sequencer: reencode commit message for am/rebase --show-current-patch\n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385544","messageId":"b3d6c4e72020072eb495741d125a08ec07f33ba4.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573031848.git.congdanhqx@gmail.com","subject":"[PATCH v3 1/8] t0028: eliminate non-standard usage of printf","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:19:59Z","receivedAt":"2019-11-06T09:20:31Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"man 1p printf:\n In addition to the escape sequences shown in the Base Definitions\n volume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\\\',\n '\\a', '\\b', '\\f', '\\n', '\\r', '\\t', '\\v'), \"\\ddd\", where ddd is a\n one, two, or three-digit octal number, shall be written as a byte\n with the numeric value specified by the octal number.\n\nprintf '\\xfe\\xff' is an extension of some shell.\nDash, a popular yet simple shell, do not implement this extension.\n\nThis wasn't caught by most people running the tests, even though\ncommon shells like dash don't handle hex escapes, because their\nsystems don't trigger the NO_UTF16_BOM prereq. But systems with musl\nlibc do; when combined with dash, the test fails.\n\nCorrect it.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n t/t0028-working-tree-encoding.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh\nindex 7aa0945d8d..bfc4fb9af5 100755\n--- a/t/t0028-working-tree-encoding.sh\n+++ b/t/t0028-working-tree-encoding.sh\n@@ -17,7 +17,7 @@ test_lazy_prereq NO_UTF32_BOM '\n write_utf16 () {\n \tif test_have_prereq NO_UTF16_BOM\n \tthen\n-\t\tprintf '\\xfe\\xff'\n+\t\tprintf '\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-16\n }\n@@ -25,7 +25,7 @@ write_utf16 () {\n write_utf32 () {\n \tif test_have_prereq NO_UTF32_BOM\n \tthen\n-\t\tprintf '\\x00\\x00\\xfe\\xff'\n+\t\tprintf '\\0\\0\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-32\n }\n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385545","messageId":"f07566c60c4421a5b33ef5c74a9a0a58947e7750.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573031848.git.congdanhqx@gmail.com","subject":"[PATCH v3 2/8] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:20:00Z","receivedAt":"2019-11-06T09:20:35Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"From commit 79444c9294, (\"utf8: handle systems that don't write BOM for\nUTF-16\", 2019-02-12), we're supporting those systems with iconv that\nomits BOM with:\n\n make ICONV_OMITS_BOM=Yes\n\nHowever, typing the flag all the time is cumbersome and error-prone.\n\nAdd a check into configure script to detect this flag automatically.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n configure.ac | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/configure.ac b/configure.ac\nindex a43b476402..ecba7e6e51 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -844,12 +844,61 @@ AC_MSG_CHECKING([for old iconv()])\n AC_COMPILE_IFELSE([OLDICONVTEST_SRC],\n \t[AC_MSG_RESULT([no])],\n \t[AC_MSG_RESULT([yes])\n+\tAC_DEFINE(HAVE_OLD_ICONV, 1)\n \tOLD_ICONV=UnfortunatelyYes])\n \n GIT_UNSTASH_FLAGS($ICONVDIR)\n \n GIT_CONF_SUBST([OLD_ICONV])\n \n+#\n+# Define ICONV_OMITS_BOM if you are on a system which\n+# iconv omits bom for utf-{16,32}\n+if test -z \"$NO_ICONV\"; then\n+AC_CACHE_CHECK([whether iconv omits bom for utf-16 and utf-32],\n+ [ac_cv_iconv_omits_bom],\n+[\n+old_LIBS=\"$LIBS\"\n+if test -n \"$NEEDS_LIBICONV\"; then\n+\tLIBS=\"$LIBS -liconv\"\n+fi\n+\n+AC_RUN_IFELSE(\n+\t[AC_LANG_PROGRAM([AC_INCLUDES_DEFAULT\n+\t#include \n+\t#ifdef HAVE_OLD_ICONV\n+\ttypedef const char *iconv_ibp;\n+\t#else\n+\ttypedef char *iconv_ibp;\n+\t#endif\n+\t],\n+\t[[\n+\tint v;\n+\ticonv_t conv;\n+\tchar in[] = \"a\"; iconv_ibp pin = in;\n+\tchar out[20] = \"\"; char *pout = out;\n+\tsize_t isz = sizeof in;\n+\tsize_t osz = sizeof out;\n+\n+\tconv = iconv_open(\"UTF-16\", \"UTF-8\");\n+\ticonv(conv, &pin, &isz, &pout, &osz);\n+\ticonv_close(conv);\n+\tv = (unsigned char)(out[0]) + (unsigned char)(out[1]);\n+\treturn v != 0xfe + 0xff;\n+\t]])],\n+\t[ac_cv_iconv_omits_bom=no],\n+\t[ac_cv_iconv_omits_bom=yes])\n+\n+LIBS=\"$old_LIBS\"\n+])\n+if test \"x$ac_cv_iconv_omits_bom\" = xyes; then\n+\tICONV_OMITS_BOM=Yes\n+else\n+\tICONV_OMITS_BOM=\n+fi\n+GIT_CONF_SUBST([ICONV_OMITS_BOM])\n+fi\n+\n ## Checks for typedefs, structures, and compiler characteristics.\n AC_MSG_NOTICE([CHECKS for typedefs, structures, and compiler characteristics])\n #\n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385546","messageId":"662e5bd545135f9ed26547f47c457bf6aa74d3f3.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573031848.git.congdanhqx@gmail.com","subject":"[PATCH v3 3/8] t3900: demonstrate git-rebase problem with multi encoding","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:20:01Z","receivedAt":"2019-11-06T09:20:40Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"We're using fixup!/squash! to mark if current commit will be\nused to be fixed up or squashed to a previous commit.\n\nHowever, if we're changing i18n.commitencoding after making the\noriginal commit but before making the fixing up, we couldn't find the\noriginal commit to do the fixup/squash.\n\nAdd a test to demonstrate that problem.\n\nHelped-by: Jeff King \nSigned-off-by: Doan Tran Cong Danh \n---\n t/t3900-i18n-commit.sh | 30 ++++++++++++++++++++++++++++++\n 1 file changed, 30 insertions(+)\n\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex b92ff95977..85cac6e2dc 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -204,4 +204,34 @@ test_commit_autosquash_flags eucJP fixup\n \n test_commit_autosquash_flags ISO-2022-JP squash\n \n+test_commit_autosquash_multi_encoding () {\n+\tflag=$1\n+\told=$2\n+\tnew=$3\n+\tmsg=$4\n+\ttest_expect_failure \"commit --$flag into $old from $new\" '\n+\t\tgit checkout -b '$flag-$old-$new' C0 &&\n+\t\tgit config i18n.commitencoding '$old' &&\n+\t\techo '$old' >>F &&\n+\t\tgit commit -a -F \"$TEST_DIRECTORY/t3900/'$msg'\" &&\n+\t\ttest_tick &&\n+\t\techo intermediate stuff >>G &&\n+\t\tgit add G &&\n+\t\tgit commit -a -m \"intermediate commit\" &&\n+\t\ttest_tick &&\n+\t\tgit config i18n.commitencoding '$new' &&\n+\t\techo '$new-$flag' >>F &&\n+\t\tgit commit -a --'$flag' HEAD^ &&\n+\t\tgit config --unset-all i18n.commitencoding &&\n+\t\tgit rebase --autosquash -i HEAD^^^ &&\n+\t\tgit rev-list HEAD >actual &&\n+\t\ttest_line_count = 3 actual\n+\t'\n+}\n+\n+test_commit_autosquash_multi_encoding fixup UTF-8 ISO-8859-1 1-UTF-8.txt\n+test_commit_autosquash_multi_encoding squash ISO-8859-1 UTF-8 ISO8859-1.txt\n+test_commit_autosquash_multi_encoding squash eucJP ISO-2022-JP eucJP.txt\n+test_commit_autosquash_multi_encoding fixup ISO-2022-JP UTF-8 ISO-2022-JP.txt\n+\n test_done\n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385547","messageId":"6a51fdd29cc9b1132548726028e882a5a99acbd1.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573031848.git.congdanhqx@gmail.com","subject":"[PATCH v3 4/8] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:20:02Z","receivedAt":"2019-11-06T09:20:43Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On musl libc, ISO-2022-JP encoder is too eager to switch back to\n1 byte encoding, musl's iconv always switch back after every combining\ncharacter. Comparing glibc and musl's output for this command\n$ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 |\n\ticonv -f utf-8 -t ISO-2022-JP | xxd\n\nglibc:\n00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B\n00000010: 0a .\n\nmusl:\n00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B\n00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B\n00000020: 1b24 4224 551b 2842 0a .$B$U.(B.\n\nAlthough musl iconv's output isn't optimal, it's still correct.\n\nFrom commit 7d509878b8, (\"pretty.c: format string with truncate respects\nlogOutputEncoding\", 2014-05-21), we're encoding the message to utf-8\nfirst, then format it and convert the message to the actual output\nencoding on git commit --squash.\n\nThus, t3900::test_commit_autosquash_flags is failing on musl libc.\n\nReencode to utf-8 before arranging rebase's todo list.\nBy doing this, we also remove a breakage introduced in the previous\ncommit.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 2 +-\n t/t3900-i18n-commit.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 9d5964fd81..69430fe23f 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -5169,7 +5169,7 @@ int todo_list_rearrange_squash(struct todo_list *todo_list)\n \t\t*commit_todo_item_at(&commit_todo, item->commit) = item;\n \n \t\tparse_commit(item->commit);\n-\t\tcommit_buffer = get_commit_buffer(item->commit, NULL);\n+\t\tcommit_buffer = logmsg_reencode(item->commit, NULL, \"UTF-8\");\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tformat_subject(&buf, subject, \" \");\n \t\tsubject = subjects[i] = strbuf_detach(&buf, &subject_len);\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex 85cac6e2dc..7501555c52 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -209,7 +209,7 @@ test_commit_autosquash_multi_encoding () {\n \told=$2\n \tnew=$3\n \tmsg=$4\n-\ttest_expect_failure \"commit --$flag into $old from $new\" '\n+\ttest_expect_success \"commit --$flag into $old from $new\" '\n \t\tgit checkout -b '$flag-$old-$new' C0 &&\n \t\tgit config i18n.commitencoding '$old' &&\n \t\techo '$old' >>F &&\n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385548","messageId":"d382e35e4ef93a77209c0a6659371d25c8ed219e.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573031848.git.congdanhqx@gmail.com","subject":"[PATCH v3 5/8] sequencer: reencode revert/cherry-pick's todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:20:03Z","receivedAt":"2019-11-06T09:20:47Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"Keep revert/cherry-pick's todo list in line with rebase todo list.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 69430fe23f..a19954f2bf 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -2564,14 +2564,17 @@ static int walk_revs_populate_todo(struct todo_list *todo_list,\n \tenum todo_command command = opts->action == REPLAY_PICK ?\n \t\tTODO_PICK : TODO_REVERT;\n \tconst char *command_string = todo_command_info[command].str;\n+\tconst char *encoding;\n \tstruct commit *commit;\n \n \tif (prepare_revs(opts))\n \t\treturn -1;\n \n+\tencoding = get_log_output_encoding();\n+\n \twhile ((commit = get_revision(opts->revs))) {\n \t\tstruct todo_item *item = append_new_todo(todo_list);\n-\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n+\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n \t\tconst char *subject;\n \t\tint subject_len;\n \n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385549","messageId":"340902eb67df7e9a5ecb9e085f5e3fcaa1fd6751.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573031848.git.congdanhqx@gmail.com","subject":"[PATCH v3 6/8] sequencer: reencode squashing commit's message","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:20:04Z","receivedAt":"2019-11-06T09:20:51Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On fixup/squash-ing rebase, git will create new commit in\ni18n.commitencoding, reencode the commit message to that said encode.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 8 +++++---\n t/t3900-i18n-commit.sh | 14 +++++++++++++-\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex a19954f2bf..833a928929 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1576,6 +1576,7 @@ static int update_squash_messages(struct repository *r,\n \tstruct strbuf buf = STRBUF_INIT;\n \tint res;\n \tconst char *message, *body;\n+\tconst char *encoding = get_commit_output_encoding();\n \n \tif (opts->current_fixup_count > 0) {\n \t\tstruct strbuf header = STRBUF_INIT;\n@@ -1602,7 +1603,7 @@ static int update_squash_messages(struct repository *r,\n \t\t\treturn error(_(\"need a HEAD to fixup\"));\n \t\tif (!(head_commit = lookup_commit_reference(r, &head)))\n \t\t\treturn error(_(\"could not read HEAD\"));\n-\t\tif (!(head_message = get_commit_buffer(head_commit, NULL)))\n+\t\tif (!(head_message = logmsg_reencode(head_commit, NULL, encoding)))\n \t\t\treturn error(_(\"could not read HEAD's commit message\"));\n \n \t\tfind_commit_subject(head_message, &body);\n@@ -1623,7 +1624,7 @@ static int update_squash_messages(struct repository *r,\n \t\tunuse_commit_buffer(head_commit, head_message);\n \t}\n \n-\tif (!(message = get_commit_buffer(commit, NULL)))\n+\tif (!(message = logmsg_reencode(commit, NULL, encoding)))\n \t\treturn error(_(\"could not read commit message of %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \tfind_commit_subject(message, &body);\n@@ -4154,9 +4155,10 @@ static int commit_staged_changes(struct repository *r,\n \t\t\t\t */\n \t\t\t\tstruct commit *commit;\n \t\t\t\tconst char *path = rebase_path_squash_msg();\n+\t\t\t\tconst char *encoding = get_commit_output_encoding();\n \n \t\t\t\tif (parse_head(r, &commit) ||\n-\t\t\t\t !(p = get_commit_buffer(commit, NULL)) ||\n+\t\t\t\t !(p = logmsg_reencode(commit, NULL, encoding)) ||\n \t\t\t\t write_message(p, strlen(p), path, 0)) {\n \t\t\t\t\tunuse_commit_buffer(commit, p);\n \t\t\t\t\treturn error(_(\"could not write file: \"\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex 7501555c52..241bda1099 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -209,6 +209,13 @@ test_commit_autosquash_multi_encoding () {\n \told=$2\n \tnew=$3\n \tmsg=$4\n+\tsquash_msg=\n+\tif test $flag = squash; then\n+\t\tsquash_msg='\n+\t\tsubject=\"squash! $(head -1 expect)\" &&\n+\t\tprintf \"\\n%s\\n\" \"$subject\" >> expect &&\n+\t\t'\n+\tfi\n \ttest_expect_success \"commit --$flag into $old from $new\" '\n \t\tgit checkout -b '$flag-$old-$new' C0 &&\n \t\tgit config i18n.commitencoding '$old' &&\n@@ -225,7 +232,12 @@ test_commit_autosquash_multi_encoding () {\n \t\tgit config --unset-all i18n.commitencoding &&\n \t\tgit rebase --autosquash -i HEAD^^^ &&\n \t\tgit rev-list HEAD >actual &&\n-\t\ttest_line_count = 3 actual\n+\t\ttest_line_count = 3 actual &&\n+\t\ticonv -f '$old' -t utf-8 \"$TEST_DIRECTORY/t3900/'$msg'\" >expect &&\n+\t\t'\"$squash_msg\"'\n+\t\tgit cat-file commit HEAD^ >raw &&\n+\t\t(sed \"1,/^$/d\" raw | iconv -f '$new' -t utf-8) >actual &&\n+\t\ttest_cmp expect actual\n \t'\n }\n \n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385550","messageId":"7f0df0f685544ab828a7f6569d5e50a975c423d3.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573031848.git.congdanhqx@gmail.com","subject":"[PATCH v3 7/8] sequencer: reencode old merge-commit message","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:20:05Z","receivedAt":"2019-11-06T09:20:54Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"During rebasing, old merge's message (encoded in old encoding)\nwill be used as message for new merge commit (created by rebase).\n\nIn case of the value of i18n.commitencoding has been changed after the\nold merge time. We will receive an usable message for this new merge.\n\nCorrect it.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 833a928929..d735d09f98 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -3374,7 +3374,8 @@ static int do_merge(struct repository *r,\n \t}\n \n \tif (commit) {\n-\t\tconst char *message = get_commit_buffer(commit, NULL);\n+\t\tconst char *encoding = get_commit_output_encoding();\n+\t\tconst char *message = logmsg_reencode(commit, NULL, encoding);\n \t\tconst char *body;\n \t\tint len;\n \n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385551","messageId":"69ec40bb1dbdf7eda21d8fa918164a191ba41875.1573031848.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573031848.git.congdanhqx@gmail.com","subject":"[PATCH v3 8/8] sequencer: reencode commit message for am/rebase --show-current-patch","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T09:20:06Z","receivedAt":"2019-11-06T09:20:56Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"The message file will be used as commit message for the\ngit-{am,rebase} --continue.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex d735d09f98..4c1ffad0f1 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -2972,7 +2972,8 @@ static int make_patch(struct repository *r,\n \n \tstrbuf_addf(&buf, \"%s/message\", get_dir(opts));\n \tif (!file_exists(buf.buf)) {\n-\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n+\t\tconst char *encoding = get_commit_output_encoding()\n+\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tres |= write_message(subject, strlen(subject), buf.buf, 1);\n \t\tunuse_commit_buffer(commit, commit_buffer);\n-- \n2.24.0.4.g6a51fdd29c\n\n"},{"id":"385575","messageId":"20191106100322.GC6351@danh.dev","threadId":"52167","inReplyTo":"20191106040314.GA4307@sigill.intra.peff.net","subject":"Re: [PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-06T10:03:22Z","receivedAt":"2019-11-06T10:03:28Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-11-05 23:03:14 -0500, Jeff King wrote:\n> > 3. You are dealing with a project originated on and migrated\n> > from a foreign SCM, and older parts of the history is stored\n> > in a non-utf-8, even though recent history is in utf-8\n> > \n> > to the mix?\n> \n> I would think you'd want to convert to utf-8 as you do the migration in\n> that case, since you're writing new hashes anyway.\n\nSorry but I'm confused.\nIf we're migrating from foreign SCM, we could make our commit in\nutf-8 (convert their commit message to utf8).\nEven if we need to synchronise history between the foreign SCM in\nquestion with git, we could use i18n.logoutputencoding for the output\ncomestic.\n\n> But I think a similar\n> case would just be an old Git repository, where for some reason you\n> thought i18n.commitEncoding was a good idea back then (perhaps because\n> you were in situation (1) then, but now you aren't).\n> \n> In either case, though, I don't think it's a compelling motivation for\n> optimization, if only because those old commits will be shown less and\n> less (and even without modern optimizations like commit-graph, we'd\n> generally avoid reencoding those old commits unless we're actually going\n> to _show_ them).\n\nI'm not sure if we're misunderstood each other.\nI've only suggested to encode _new_ commit from now on in utf-8.\nReencoding old history in utf-8 is definitely not in that suggestion.\n\n-- \nDanh\n"},{"id":"385595","messageId":"CAPig+cRA04oOO_NpcBqabwL6F4Aa37Nst-xzC3xQ3aZA2GuL5Q@mail.gmail.com","threadId":"52167","inReplyTo":"7f0df0f685544ab828a7f6569d5e50a975c423d3.1573031848.git.congdanhqx@gmail.com","subject":"Re: [PATCH v3 7/8] sequencer: reencode old merge-commit message","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-11-06T15:39:18Z","receivedAt":"2019-11-06T15:39:31Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Nov 6, 2019 at 4:21 AM Doan Tran Cong Danh wrote:\n> During rebasing, old merge's message (encoded in old encoding)\n> will be used as message for new merge commit (created by rebase).\n>\n> In case of the value of i18n.commitencoding has been changed after the\n> old merge time. We will receive an usable message for this new merge.\n\nDid you mean s/usable/unusable/ ?\n"},{"id":"385638","messageId":"cover.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"20191031092618.29073-1-congdanhqx@gmail.com","subject":"[PATCH v4 0/8] Correct internal working and output encoding","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:11Z","receivedAt":"2019-11-07T02:56:30Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"The series is shifting from fixing test that failed on musl based Linux to\ncorrect the internal working encoding and output encoding of git-am\ngit-cherry-pick git-rebase and git-revert.\n\nChange from v3:\n- Fix grammar and spelling in commit message.\n- Add mising semicolon (I ran the test with dirty workspace right before\n making last change). Everything is OK this time, though.\n\nCc: Eric Sunshine \n\nApparantly, I couldn't spell unusable.\n\n\nDoan Tran Cong Danh (8):\n t0028: eliminate non-standard usage of printf\n configure.ac: define ICONV_OMITS_BOM if necessary\n t3900: demonstrate git-rebase problem with multi encoding\n sequencer: reencode to utf-8 before arrange rebase's todo list\n sequencer: reencode revert/cherry-pick's todo list\n sequencer: reencode squashing commit's message\n sequencer: reencode old merge-commit message\n sequencer: reencode commit message for am/rebase --show-current-patch\n\n configure.ac | 49 ++++++++++++++++++++++++++++++++\n sequencer.c | 21 +++++++++-----\n t/t0028-working-tree-encoding.sh | 4 +--\n t/t3900-i18n-commit.sh | 41 ++++++++++++++++++++++++++\n 4 files changed, 106 insertions(+), 9 deletions(-)\n\nRange-diff against v3:\n1: b3d6c4e720 = 1: b3d6c4e720 t0028: eliminate non-standard usage of printf\n2: f07566c60c = 2: f07566c60c configure.ac: define ICONV_OMITS_BOM if necessary\n3: 662e5bd545 ! 3: ca869cef57 t3900: demonstrate git-rebase problem with multi encoding\n @@ t/t3900-i18n-commit.sh: test_commit_autosquash_flags eucJP fixup\n +\t\tgit config i18n.commitencoding '$new' &&\n +\t\techo '$new-$flag' >>F &&\n +\t\tgit commit -a --'$flag' HEAD^ &&\n -+\t\tgit config --unset-all i18n.commitencoding &&\n +\t\tgit rebase --autosquash -i HEAD^^^ &&\n +\t\tgit rev-list HEAD >actual &&\n +\t\ttest_line_count = 3 actual\n4: 6a51fdd29c = 4: 15c33fc245 sequencer: reencode to utf-8 before arrange rebase's todo list\n5: d382e35e4e = 5: 304ac6c289 sequencer: reencode revert/cherry-pick's todo list\n6: 340902eb67 ! 6: 97ab88e5d8 sequencer: reencode squashing commit's message\n @@ t/t3900-i18n-commit.sh: test_commit_autosquash_multi_encoding () {\n \t\tgit checkout -b '$flag-$old-$new' C0 &&\n \t\tgit config i18n.commitencoding '$old' &&\n @@ t/t3900-i18n-commit.sh: test_commit_autosquash_multi_encoding () {\n - \t\tgit config --unset-all i18n.commitencoding &&\n + \t\tgit commit -a --'$flag' HEAD^ &&\n \t\tgit rebase --autosquash -i HEAD^^^ &&\n \t\tgit rev-list HEAD >actual &&\n -\t\ttest_line_count = 3 actual\n7: 7f0df0f685 ! 7: f295d32d7b sequencer: reencode old merge-commit message\n @@ Commit message\n will be used as message for new merge commit (created by rebase).\n \n In case of the value of i18n.commitencoding has been changed after the\n - old merge time. We will receive an usable message for this new merge.\n + old merge time. We will receive an unusable message for this new merge.\n \n Correct it.\n \n8: 69ec40bb1d ! 8: 36796e2b67 sequencer: reencode commit message for am/rebase --show-current-patch\n @@ sequencer.c: static int make_patch(struct repository *r,\n \tstrbuf_addf(&buf, \"%s/message\", get_dir(opts));\n \tif (!file_exists(buf.buf)) {\n -\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n -+\t\tconst char *encoding = get_commit_output_encoding()\n ++\t\tconst char *encoding = get_commit_output_encoding();\n +\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tres |= write_message(subject, strlen(subject), buf.buf, 1);\n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385639","messageId":"b3d6c4e72020072eb495741d125a08ec07f33ba4.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573094789.git.congdanhqx@gmail.com","subject":"[PATCH v4 1/8] t0028: eliminate non-standard usage of printf","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:12Z","receivedAt":"2019-11-07T02:56:32Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"man 1p printf:\n In addition to the escape sequences shown in the Base Definitions\n volume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\\\',\n '\\a', '\\b', '\\f', '\\n', '\\r', '\\t', '\\v'), \"\\ddd\", where ddd is a\n one, two, or three-digit octal number, shall be written as a byte\n with the numeric value specified by the octal number.\n\nprintf '\\xfe\\xff' is an extension of some shell.\nDash, a popular yet simple shell, do not implement this extension.\n\nThis wasn't caught by most people running the tests, even though\ncommon shells like dash don't handle hex escapes, because their\nsystems don't trigger the NO_UTF16_BOM prereq. But systems with musl\nlibc do; when combined with dash, the test fails.\n\nCorrect it.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n t/t0028-working-tree-encoding.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh\nindex 7aa0945d8d..bfc4fb9af5 100755\n--- a/t/t0028-working-tree-encoding.sh\n+++ b/t/t0028-working-tree-encoding.sh\n@@ -17,7 +17,7 @@ test_lazy_prereq NO_UTF32_BOM '\n write_utf16 () {\n \tif test_have_prereq NO_UTF16_BOM\n \tthen\n-\t\tprintf '\\xfe\\xff'\n+\t\tprintf '\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-16\n }\n@@ -25,7 +25,7 @@ write_utf16 () {\n write_utf32 () {\n \tif test_have_prereq NO_UTF32_BOM\n \tthen\n-\t\tprintf '\\x00\\x00\\xfe\\xff'\n+\t\tprintf '\\0\\0\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-32\n }\n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385640","messageId":"f07566c60c4421a5b33ef5c74a9a0a58947e7750.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573094789.git.congdanhqx@gmail.com","subject":"[PATCH v4 2/8] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:13Z","receivedAt":"2019-11-07T02:56:34Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"From commit 79444c9294, (\"utf8: handle systems that don't write BOM for\nUTF-16\", 2019-02-12), we're supporting those systems with iconv that\nomits BOM with:\n\n make ICONV_OMITS_BOM=Yes\n\nHowever, typing the flag all the time is cumbersome and error-prone.\n\nAdd a check into configure script to detect this flag automatically.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n configure.ac | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/configure.ac b/configure.ac\nindex a43b476402..ecba7e6e51 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -844,12 +844,61 @@ AC_MSG_CHECKING([for old iconv()])\n AC_COMPILE_IFELSE([OLDICONVTEST_SRC],\n \t[AC_MSG_RESULT([no])],\n \t[AC_MSG_RESULT([yes])\n+\tAC_DEFINE(HAVE_OLD_ICONV, 1)\n \tOLD_ICONV=UnfortunatelyYes])\n \n GIT_UNSTASH_FLAGS($ICONVDIR)\n \n GIT_CONF_SUBST([OLD_ICONV])\n \n+#\n+# Define ICONV_OMITS_BOM if you are on a system which\n+# iconv omits bom for utf-{16,32}\n+if test -z \"$NO_ICONV\"; then\n+AC_CACHE_CHECK([whether iconv omits bom for utf-16 and utf-32],\n+ [ac_cv_iconv_omits_bom],\n+[\n+old_LIBS=\"$LIBS\"\n+if test -n \"$NEEDS_LIBICONV\"; then\n+\tLIBS=\"$LIBS -liconv\"\n+fi\n+\n+AC_RUN_IFELSE(\n+\t[AC_LANG_PROGRAM([AC_INCLUDES_DEFAULT\n+\t#include \n+\t#ifdef HAVE_OLD_ICONV\n+\ttypedef const char *iconv_ibp;\n+\t#else\n+\ttypedef char *iconv_ibp;\n+\t#endif\n+\t],\n+\t[[\n+\tint v;\n+\ticonv_t conv;\n+\tchar in[] = \"a\"; iconv_ibp pin = in;\n+\tchar out[20] = \"\"; char *pout = out;\n+\tsize_t isz = sizeof in;\n+\tsize_t osz = sizeof out;\n+\n+\tconv = iconv_open(\"UTF-16\", \"UTF-8\");\n+\ticonv(conv, &pin, &isz, &pout, &osz);\n+\ticonv_close(conv);\n+\tv = (unsigned char)(out[0]) + (unsigned char)(out[1]);\n+\treturn v != 0xfe + 0xff;\n+\t]])],\n+\t[ac_cv_iconv_omits_bom=no],\n+\t[ac_cv_iconv_omits_bom=yes])\n+\n+LIBS=\"$old_LIBS\"\n+])\n+if test \"x$ac_cv_iconv_omits_bom\" = xyes; then\n+\tICONV_OMITS_BOM=Yes\n+else\n+\tICONV_OMITS_BOM=\n+fi\n+GIT_CONF_SUBST([ICONV_OMITS_BOM])\n+fi\n+\n ## Checks for typedefs, structures, and compiler characteristics.\n AC_MSG_NOTICE([CHECKS for typedefs, structures, and compiler characteristics])\n #\n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385641","messageId":"ca869cef57bcf620a7b5d0519d362dcd9a27eae6.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573094789.git.congdanhqx@gmail.com","subject":"[PATCH v4 3/8] t3900: demonstrate git-rebase problem with multi encoding","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:14Z","receivedAt":"2019-11-07T02:56:36Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"We're using fixup!/squash! to mark if current commit will be\nused to be fixed up or squashed to a previous commit.\n\nHowever, if we're changing i18n.commitencoding after making the\noriginal commit but before making the fixing up, we couldn't find the\noriginal commit to do the fixup/squash.\n\nAdd a test to demonstrate that problem.\n\nHelped-by: Jeff King \nSigned-off-by: Doan Tran Cong Danh \n---\n t/t3900-i18n-commit.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex b92ff95977..0f978bfde1 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -204,4 +204,33 @@ test_commit_autosquash_flags eucJP fixup\n \n test_commit_autosquash_flags ISO-2022-JP squash\n \n+test_commit_autosquash_multi_encoding () {\n+\tflag=$1\n+\told=$2\n+\tnew=$3\n+\tmsg=$4\n+\ttest_expect_failure \"commit --$flag into $old from $new\" '\n+\t\tgit checkout -b '$flag-$old-$new' C0 &&\n+\t\tgit config i18n.commitencoding '$old' &&\n+\t\techo '$old' >>F &&\n+\t\tgit commit -a -F \"$TEST_DIRECTORY/t3900/'$msg'\" &&\n+\t\ttest_tick &&\n+\t\techo intermediate stuff >>G &&\n+\t\tgit add G &&\n+\t\tgit commit -a -m \"intermediate commit\" &&\n+\t\ttest_tick &&\n+\t\tgit config i18n.commitencoding '$new' &&\n+\t\techo '$new-$flag' >>F &&\n+\t\tgit commit -a --'$flag' HEAD^ &&\n+\t\tgit rebase --autosquash -i HEAD^^^ &&\n+\t\tgit rev-list HEAD >actual &&\n+\t\ttest_line_count = 3 actual\n+\t'\n+}\n+\n+test_commit_autosquash_multi_encoding fixup UTF-8 ISO-8859-1 1-UTF-8.txt\n+test_commit_autosquash_multi_encoding squash ISO-8859-1 UTF-8 ISO8859-1.txt\n+test_commit_autosquash_multi_encoding squash eucJP ISO-2022-JP eucJP.txt\n+test_commit_autosquash_multi_encoding fixup ISO-2022-JP UTF-8 ISO-2022-JP.txt\n+\n test_done\n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385642","messageId":"15c33fc245401791fd2530d9d4dbde8e0e451939.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573094789.git.congdanhqx@gmail.com","subject":"[PATCH v4 4/8] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:15Z","receivedAt":"2019-11-07T02:56:38Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On musl libc, ISO-2022-JP encoder is too eager to switch back to\n1 byte encoding, musl's iconv always switch back after every combining\ncharacter. Comparing glibc and musl's output for this command\n$ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 |\n\ticonv -f utf-8 -t ISO-2022-JP | xxd\n\nglibc:\n00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B\n00000010: 0a .\n\nmusl:\n00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B\n00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B\n00000020: 1b24 4224 551b 2842 0a .$B$U.(B.\n\nAlthough musl iconv's output isn't optimal, it's still correct.\n\nFrom commit 7d509878b8, (\"pretty.c: format string with truncate respects\nlogOutputEncoding\", 2014-05-21), we're encoding the message to utf-8\nfirst, then format it and convert the message to the actual output\nencoding on git commit --squash.\n\nThus, t3900::test_commit_autosquash_flags is failing on musl libc.\n\nReencode to utf-8 before arranging rebase's todo list.\nBy doing this, we also remove a breakage introduced in the previous\ncommit.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 2 +-\n t/t3900-i18n-commit.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 9d5964fd81..69430fe23f 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -5169,7 +5169,7 @@ int todo_list_rearrange_squash(struct todo_list *todo_list)\n \t\t*commit_todo_item_at(&commit_todo, item->commit) = item;\n \n \t\tparse_commit(item->commit);\n-\t\tcommit_buffer = get_commit_buffer(item->commit, NULL);\n+\t\tcommit_buffer = logmsg_reencode(item->commit, NULL, \"UTF-8\");\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tformat_subject(&buf, subject, \" \");\n \t\tsubject = subjects[i] = strbuf_detach(&buf, &subject_len);\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex 0f978bfde1..e8ce5323ee 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -209,7 +209,7 @@ test_commit_autosquash_multi_encoding () {\n \told=$2\n \tnew=$3\n \tmsg=$4\n-\ttest_expect_failure \"commit --$flag into $old from $new\" '\n+\ttest_expect_success \"commit --$flag into $old from $new\" '\n \t\tgit checkout -b '$flag-$old-$new' C0 &&\n \t\tgit config i18n.commitencoding '$old' &&\n \t\techo '$old' >>F &&\n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385643","messageId":"97ab88e5d886b376b396b2fbe563e1a21be03e44.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573094789.git.congdanhqx@gmail.com","subject":"[PATCH v4 6/8] sequencer: reencode squashing commit's message","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:17Z","receivedAt":"2019-11-07T02:56:41Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On fixup/squash-ing rebase, git will create new commit in\ni18n.commitencoding, reencode the commit message to that said encode.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 8 +++++---\n t/t3900-i18n-commit.sh | 14 +++++++++++++-\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex a19954f2bf..833a928929 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1576,6 +1576,7 @@ static int update_squash_messages(struct repository *r,\n \tstruct strbuf buf = STRBUF_INIT;\n \tint res;\n \tconst char *message, *body;\n+\tconst char *encoding = get_commit_output_encoding();\n \n \tif (opts->current_fixup_count > 0) {\n \t\tstruct strbuf header = STRBUF_INIT;\n@@ -1602,7 +1603,7 @@ static int update_squash_messages(struct repository *r,\n \t\t\treturn error(_(\"need a HEAD to fixup\"));\n \t\tif (!(head_commit = lookup_commit_reference(r, &head)))\n \t\t\treturn error(_(\"could not read HEAD\"));\n-\t\tif (!(head_message = get_commit_buffer(head_commit, NULL)))\n+\t\tif (!(head_message = logmsg_reencode(head_commit, NULL, encoding)))\n \t\t\treturn error(_(\"could not read HEAD's commit message\"));\n \n \t\tfind_commit_subject(head_message, &body);\n@@ -1623,7 +1624,7 @@ static int update_squash_messages(struct repository *r,\n \t\tunuse_commit_buffer(head_commit, head_message);\n \t}\n \n-\tif (!(message = get_commit_buffer(commit, NULL)))\n+\tif (!(message = logmsg_reencode(commit, NULL, encoding)))\n \t\treturn error(_(\"could not read commit message of %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \tfind_commit_subject(message, &body);\n@@ -4154,9 +4155,10 @@ static int commit_staged_changes(struct repository *r,\n \t\t\t\t */\n \t\t\t\tstruct commit *commit;\n \t\t\t\tconst char *path = rebase_path_squash_msg();\n+\t\t\t\tconst char *encoding = get_commit_output_encoding();\n \n \t\t\t\tif (parse_head(r, &commit) ||\n-\t\t\t\t !(p = get_commit_buffer(commit, NULL)) ||\n+\t\t\t\t !(p = logmsg_reencode(commit, NULL, encoding)) ||\n \t\t\t\t write_message(p, strlen(p), path, 0)) {\n \t\t\t\t\tunuse_commit_buffer(commit, p);\n \t\t\t\t\treturn error(_(\"could not write file: \"\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex e8ce5323ee..521d7bb927 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -209,6 +209,13 @@ test_commit_autosquash_multi_encoding () {\n \told=$2\n \tnew=$3\n \tmsg=$4\n+\tsquash_msg=\n+\tif test $flag = squash; then\n+\t\tsquash_msg='\n+\t\tsubject=\"squash! $(head -1 expect)\" &&\n+\t\tprintf \"\\n%s\\n\" \"$subject\" >> expect &&\n+\t\t'\n+\tfi\n \ttest_expect_success \"commit --$flag into $old from $new\" '\n \t\tgit checkout -b '$flag-$old-$new' C0 &&\n \t\tgit config i18n.commitencoding '$old' &&\n@@ -224,7 +231,12 @@ test_commit_autosquash_multi_encoding () {\n \t\tgit commit -a --'$flag' HEAD^ &&\n \t\tgit rebase --autosquash -i HEAD^^^ &&\n \t\tgit rev-list HEAD >actual &&\n-\t\ttest_line_count = 3 actual\n+\t\ttest_line_count = 3 actual &&\n+\t\ticonv -f '$old' -t utf-8 \"$TEST_DIRECTORY/t3900/'$msg'\" >expect &&\n+\t\t'\"$squash_msg\"'\n+\t\tgit cat-file commit HEAD^ >raw &&\n+\t\t(sed \"1,/^$/d\" raw | iconv -f '$new' -t utf-8) >actual &&\n+\t\ttest_cmp expect actual\n \t'\n }\n \n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385644","messageId":"304ac6c28972a7bec9c5c7b5e2ae3c107161e84f.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573094789.git.congdanhqx@gmail.com","subject":"[PATCH v4 5/8] sequencer: reencode revert/cherry-pick's todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:16Z","receivedAt":"2019-11-07T02:56:41Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"Keep revert/cherry-pick's todo list in line with rebase todo list.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 69430fe23f..a19954f2bf 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -2564,14 +2564,17 @@ static int walk_revs_populate_todo(struct todo_list *todo_list,\n \tenum todo_command command = opts->action == REPLAY_PICK ?\n \t\tTODO_PICK : TODO_REVERT;\n \tconst char *command_string = todo_command_info[command].str;\n+\tconst char *encoding;\n \tstruct commit *commit;\n \n \tif (prepare_revs(opts))\n \t\treturn -1;\n \n+\tencoding = get_log_output_encoding();\n+\n \twhile ((commit = get_revision(opts->revs))) {\n \t\tstruct todo_item *item = append_new_todo(todo_list);\n-\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n+\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n \t\tconst char *subject;\n \t\tint subject_len;\n \n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385645","messageId":"f295d32d7b1c1090aafeb5bdae03473e35db170a.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573094789.git.congdanhqx@gmail.com","subject":"[PATCH v4 7/8] sequencer: reencode old merge-commit message","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:18Z","receivedAt":"2019-11-07T02:56:42Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"During rebasing, old merge's message (encoded in old encoding)\nwill be used as message for new merge commit (created by rebase).\n\nIn case of the value of i18n.commitencoding has been changed after the\nold merge time. We will receive an unusable message for this new merge.\n\nCorrect it.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 833a928929..d735d09f98 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -3374,7 +3374,8 @@ static int do_merge(struct repository *r,\n \t}\n \n \tif (commit) {\n-\t\tconst char *message = get_commit_buffer(commit, NULL);\n+\t\tconst char *encoding = get_commit_output_encoding();\n+\t\tconst char *message = logmsg_reencode(commit, NULL, encoding);\n \t\tconst char *body;\n \t\tint len;\n \n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385646","messageId":"36796e2b679cd8b2d341058e775db401f9abcef7.1573094789.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573094789.git.congdanhqx@gmail.com","subject":"[PATCH v4 8/8] sequencer: reencode commit message for am/rebase --show-current-patch","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T02:56:19Z","receivedAt":"2019-11-07T02:56:44Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"The message file will be used as commit message for the\ngit-{am,rebase} --continue.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex d735d09f98..4d12ad3cc6 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -2972,7 +2972,8 @@ static int make_patch(struct repository *r,\n \n \tstrbuf_addf(&buf, \"%s/message\", get_dir(opts));\n \tif (!file_exists(buf.buf)) {\n-\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n+\t\tconst char *encoding = get_commit_output_encoding();\n+\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tres |= write_message(subject, strlen(subject), buf.buf, 1);\n \t\tunuse_commit_buffer(commit, commit_buffer);\n-- \n2.24.0.8.g36796e2b67\n\n"},{"id":"385662","messageId":"20191107055644.GA6431@sigill.intra.peff.net","threadId":"52167","inReplyTo":"20191106100322.GC6351@danh.dev","subject":"Re: [PATCH v2 3/3] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-07T05:56:44Z","receivedAt":"2019-11-07T05:56:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 06, 2019 at 05:03:22PM +0700, Danh Doan wrote:\n\n> On 2019-11-05 23:03:14 -0500, Jeff King wrote:\n> > > 3. You are dealing with a project originated on and migrated\n> > > from a foreign SCM, and older parts of the history is stored\n> > > in a non-utf-8, even though recent history is in utf-8\n> > > \n> > > to the mix?\n> > \n> > I would think you'd want to convert to utf-8 as you do the migration in\n> > that case, since you're writing new hashes anyway.\n> \n> Sorry but I'm confused.\n> If we're migrating from foreign SCM, we could make our commit in\n> utf-8 (convert their commit message to utf8).\n> Even if we need to synchronise history between the foreign SCM in\n> question with git, we could use i18n.logoutputencoding for the output\n> comestic.\n\nRight, that's the same thing I'm suggesting.\n\n> > But I think a similar\n> > case would just be an old Git repository, where for some reason you\n> > thought i18n.commitEncoding was a good idea back then (perhaps because\n> > you were in situation (1) then, but now you aren't).\n> > \n> > In either case, though, I don't think it's a compelling motivation for\n> > optimization, if only because those old commits will be shown less and\n> > less (and even without modern optimizations like commit-graph, we'd\n> > generally avoid reencoding those old commits unless we're actually going\n> > to _show_ them).\n> \n> I'm not sure if we're misunderstood each other.\n> I've only suggested to encode _new_ commit from now on in utf-8.\n> Reencoding old history in utf-8 is definitely not in that suggestion.\n\nYes. My point was that's _already_ the default behavior, unless you\nexplicitly set some config asking for non-utf8 commit objects. And I\ndon't think there's any good reason to set that these days.\n\n-Peff\n"},{"id":"385663","messageId":"20191107060233.GB6431@sigill.intra.peff.net","threadId":"52167","inReplyTo":"ca869cef57bcf620a7b5d0519d362dcd9a27eae6.1573094789.git.congdanhqx@gmail.com","subject":"Re: [PATCH v4 3/8] t3900: demonstrate git-rebase problem with multi encoding","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-07T06:02:33Z","receivedAt":"2019-11-07T06:02:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 07, 2019 at 09:56:14AM +0700, Doan Tran Cong Danh wrote:\n\n> We're using fixup!/squash! to mark if current commit will be\n> used to be fixed up or squashed to a previous commit.\n> \n> However, if we're changing i18n.commitencoding after making the\n> original commit but before making the fixing up, we couldn't find the\n> original commit to do the fixup/squash.\n> \n> Add a test to demonstrate that problem.\n\nOK, this makes sense to do.\n\nI'm not sure if we need to test so many combinations, as the problem is\napparent even on some vanilla ones. But I guess this is just following\nthe lead of the rest of the script.\n\n> +test_commit_autosquash_multi_encoding () {\n> +\tflag=$1\n> +\told=$2\n> +\tnew=$3\n> +\tmsg=$4\n> +\ttest_expect_failure \"commit --$flag into $old from $new\" '\n> +\t\tgit checkout -b '$flag-$old-$new' C0 &&\n\nThese single quotes are funny; they close the test-snippet string, so\nthese variables are outside of any quoting (and thus subject to\nwhitespace splitting).\n\nThe test snippets are run as an eval, so they have access to the\nvariables you set above. I.e., just:\n\n git checkout -b $flag-$old-$new C0\n\nwould work. Or:\n\n git checkout -b \"$flag-$old-$new\" C0\n\nif you wanted to be more careful inside the snippet.\n\n> +\t\tgit config i18n.commitencoding '$old' &&\n> +\t\techo '$old' >>F &&\n> +\t\tgit commit -a -F \"$TEST_DIRECTORY/t3900/'$msg'\" &&\n\nLikewise for all these other bits of the script.\n\n-Peff\n"},{"id":"385664","messageId":"20191107060410.GC6431@sigill.intra.peff.net","threadId":"52167","inReplyTo":"15c33fc245401791fd2530d9d4dbde8e0e451939.1573094789.git.congdanhqx@gmail.com","subject":"Re: [PATCH v4 4/8] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-07T06:04:10Z","receivedAt":"2019-11-07T06:04:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 07, 2019 at 09:56:15AM +0700, Doan Tran Cong Danh wrote:\n\n> On musl libc, ISO-2022-JP encoder is too eager to switch back to\n> 1 byte encoding, musl's iconv always switch back after every combining\n> character. Comparing glibc and musl's output for this command\n> $ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 |\n> \ticonv -f utf-8 -t ISO-2022-JP | xxd\n> \n> glibc:\n> 00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B\n> 00000010: 0a .\n> \n> musl:\n> 00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B\n> 00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B\n> 00000020: 1b24 4224 551b 2842 0a .$B$U.(B.\n> \n> Although musl iconv's output isn't optimal, it's still correct.\n> \n> From commit 7d509878b8, (\"pretty.c: format string with truncate respects\n> logOutputEncoding\", 2014-05-21), we're encoding the message to utf-8\n> first, then format it and convert the message to the actual output\n> encoding on git commit --squash.\n> \n> Thus, t3900::test_commit_autosquash_flags is failing on musl libc.\n> \n> Reencode to utf-8 before arranging rebase's todo list.\n\nOK, that makes sense.\n\n> By doing this, we also remove a breakage introduced in the previous\n> commit.\n\nWe'd usually say that a commit \"introduced a breakage\" if it was the\nsource of the bug. Perhaps:\n\n ...we also remove a breakage noticed by a test added in the previous\n commit.\n\n-Peff\n"},{"id":"385666","messageId":"20191107060605.GD6431@sigill.intra.peff.net","threadId":"52167","inReplyTo":"304ac6c28972a7bec9c5c7b5e2ae3c107161e84f.1573094789.git.congdanhqx@gmail.com","subject":"Re: [PATCH v4 5/8] sequencer: reencode revert/cherry-pick's todo list","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-07T06:06:05Z","receivedAt":"2019-11-07T06:06:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 07, 2019 at 09:56:16AM +0700, Doan Tran Cong Danh wrote:\n\n> Keep revert/cherry-pick's todo list in line with rebase todo list.\n\nI think this is the right thing to do, but it might bear a little more\nexplanation. Maybe:\n\n The user may see this todo list in their editor, so we should output\n it in the encoding they generally expect to see, not whatever encoding\n the commit was originally in. This also matches the behavior of the\n rebase todo list.\n\n-Peff\n"},{"id":"385667","messageId":"20191107061557.GE6431@sigill.intra.peff.net","threadId":"52167","inReplyTo":"97ab88e5d886b376b396b2fbe563e1a21be03e44.1573094789.git.congdanhqx@gmail.com","subject":"Re: [PATCH v4 6/8] sequencer: reencode squashing commit's message","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-07T06:15:57Z","receivedAt":"2019-11-07T06:16:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 07, 2019 at 09:56:17AM +0700, Doan Tran Cong Danh wrote:\n\n> On fixup/squash-ing rebase, git will create new commit in\n> i18n.commitencoding, reencode the commit message to that said encode.\n\nThat makes sense (and I agree this is logically distinct from the\nprevious ones, which were about _showing_ commits, not generating them).\n\nI wondered who is responsible for setting the \"encoding\" header in the\nresulting object. It looks like we just call out to a separate \"git\ncommit\", feeding it some content we wrote out to a file. So before this\npatch, I think we probably are writing out \"encoding iso8859-1\" or\nwhatever in the commit object, but actually outputting whatever the\noriginal commit happened to have in it.\n\nSo your approach here is right: we just need to make sure what we write\nout for git-commit to read back in is in i18n.commitEncoding.\n\n> diff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\n> index e8ce5323ee..521d7bb927 100755\n> --- a/t/t3900-i18n-commit.sh\n> +++ b/t/t3900-i18n-commit.sh\n> @@ -209,6 +209,13 @@ test_commit_autosquash_multi_encoding () {\n> \told=$2\n> \tnew=$3\n> \tmsg=$4\n> +\tsquash_msg=\n> +\tif test $flag = squash; then\n> +\t\tsquash_msg='\n> +\t\tsubject=\"squash! $(head -1 expect)\" &&\n> +\t\tprintf \"\\n%s\\n\" \"$subject\" >> expect &&\n> +\t\t'\n> +\tfi\n\nNow what's going on here? This is a snippet of code we man to evaluate\nlater:\n\n> -\t\ttest_line_count = 3 actual\n> +\t\ttest_line_count = 3 actual &&\n> +\t\ticonv -f '$old' -t utf-8 \"$TEST_DIRECTORY/t3900/'$msg'\" >expect &&\n> +\t\t'\"$squash_msg\"'\n\nI assume this is part of the same confusion that caused the\nsingle-quotes in the earlier patch. You can just include those lines\ninline in the quoted test snippet.\n\n-Peff\n"},{"id":"385668","messageId":"xmqqbltoqkdv.fsf@gitster-ct.c.googlers.com","threadId":"52167","inReplyTo":"f07566c60c4421a5b33ef5c74a9a0a58947e7750.1573094789.git.congdanhqx@gmail.com","subject":"Re: [PATCH v4 2/8] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-11-07T06:18:52Z","receivedAt":"2019-11-07T06:18:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Doan Tran Cong Danh writes:\n\n> From commit 79444c9294, (\"utf8: handle systems that don't write BOM for\n> UTF-16\", 2019-02-12), we're supporting those systems with iconv that\n> omits BOM with:\n>\n> make ICONV_OMITS_BOM=Yes\n>\n> However, typing the flag all the time is cumbersome and error-prone.\n\nDrop the last sentence. Not having to give the flags is an already\nsolved problem and the solution is to use config.mak.\n\n> Add a check into configure script to detect this flag automatically.\n\nThis is good. \"Makefile supports this knob, configure script was\nnot aware of it. Teach configure to do so\" flows the thought\nperfectly well (and \"typing all the time\" does not have to come into\npicture).\n\nThanks.\n"},{"id":"385670","messageId":"20191107063223.GF6431@sigill.intra.peff.net","threadId":"52167","inReplyTo":"36796e2b679cd8b2d341058e775db401f9abcef7.1573094789.git.congdanhqx@gmail.com","subject":"Re: [PATCH v4 8/8] sequencer: reencode commit message for am/rebase --show-current-patch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-07T06:32:23Z","receivedAt":"2019-11-07T06:32:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 07, 2019 at 09:56:19AM +0700, Doan Tran Cong Danh wrote:\n\n> The message file will be used as commit message for the\n> git-{am,rebase} --continue.\n>\n> [...]\n> \tstrbuf_addf(&buf, \"%s/message\", get_dir(opts));\n> \tif (!file_exists(buf.buf)) {\n> -\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n> +\t\tconst char *encoding = get_commit_output_encoding();\n> +\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n\nThat makes sense, though it's hard to understand the flow of this data\nthrough multiple sequencer invocations. I _think_ this would be fixing a\ncase like this:\n\n-- >8 --\ngit init repo\ncd repo\n\n# some commits to build off of\necho base >file\ngit add file\ngit commit -m base\n\necho side >file\ngit add file\ngit commit -m side\n\n# now make a commit in iso8859-1\ngit checkout -b side HEAD^\necho iso8859-1 >file\ngit add file\niconv -f utf8 -t iso8859-1 <<-\\EOF |\nsúbject\n\nbödy\nEOF\ngit -c i18n.commitEncoding=iso8859-1 commit -F -\n\n# and rebase it with the merge strategy, which will fail;\n# now .git/rebase-merge/message has iso8859-1 in it\ngit rebase -m master\n\n# and if we resolve and commit, presumably we'd get a broken commit,\n# with iso8859-1 and no encoding header\necho resolved >file\ngit add file\nGIT_EDITOR=: git rebase --continue\n-- 8< --\n\nBut somehow it all seems to work. The resulting commit has real utf8 in\nit. I'm not sure if we pull it from the original commit via \"commit -c\",\nor if it's in one of the other files. But it's not clear to me how\nthis \"message\" file is being used.\n\n-Peff\n"},{"id":"385673","messageId":"20191107064854.GB8096@danh.dev","threadId":"52167","inReplyTo":"20191107060233.GB6431@sigill.intra.peff.net","subject":"Re: [PATCH v4 3/8] t3900: demonstrate git-rebase problem with multi encoding","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T06:48:54Z","receivedAt":"2019-11-07T06:49:01Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-11-07 01:02:33 -0500, Jeff King wrote:\n> On Thu, Nov 07, 2019 at 09:56:14AM +0700, Doan Tran Cong Danh wrote:\n> \n> > +test_commit_autosquash_multi_encoding () {\n> > +\tflag=$1\n> > +\told=$2\n> > +\tnew=$3\n> > +\tmsg=$4\n> > +\ttest_expect_failure \"commit --$flag into $old from $new\" '\n> > +\t\tgit checkout -b '$flag-$old-$new' C0 &&\n> \n> These single quotes are funny; they close the test-snippet string, so\n> these variables are outside of any quoting (and thus subject to\n> whitespace splitting).\n\nYes, those quotes are funny, and I'm also aware that\nthey will be subjected to whitespace spliting.\nBut those quotes were intentional, they're there in order to have\nbetter log with:\n\n ./t3900-i18n-commit.sh -v\n\nWith those funny quotes, we will see this (verbose) log:\n\n expecting success of 3900.38 'commit --fixup into ISO-2022-JP from UTF-8':\n git checkout -b fixup-ISO-2022-JP-UTF-8 C0 &&\n git config i18n.commitencoding ISO-2022-JP &&\n echo ISO-2022-JP >>F &&\n git commit -a -F \"$TEST_DIRECTORY/t3900/ISO-2022-JP.txt\" &&\n test_tick &&\n echo intermediate stuff >>G &&\n git add G &&\n git commit -a -m \"intermediate commit\" &&\n test_tick &&\n git config i18n.commitencoding UTF-8 &&\n echo UTF-8-fixup >>F &&\n git commit -a --fixup HEAD^ &&\n git rebase --autosquash -i HEAD^^^ &&\n git rev-list HEAD >actual &&\n test_line_count = 3 actual &&\n iconv -f ISO-2022-JP -t utf-8 \"$TEST_DIRECTORY/t3900/ISO-2022-JP.txt\" >expect &&\n git cat-file commit HEAD^ >raw &&\n (sed \"1,/^$/d\" raw | iconv -f UTF-8 -t utf-8) >actual &&\n test_cmp expect actual\n\nI think it's easier to manual run the test step with this log.\n\n-- \nDanh\n"},{"id":"385677","messageId":"20191107074858.GC8096@danh.dev","threadId":"52167","inReplyTo":"20191107063223.GF6431@sigill.intra.peff.net","subject":"Re: [PATCH v4 8/8] sequencer: reencode commit message for am/rebase --show-current-patch","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T07:48:58Z","receivedAt":"2019-11-07T07:49:04Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-11-07 01:32:23 -0500, Jeff King wrote:\n> On Thu, Nov 07, 2019 at 09:56:19AM +0700, Doan Tran Cong Danh wrote:\n> \n> > The message file will be used as commit message for the\n> > git-{am,rebase} --continue.\n> >\n> > [...]\n> > \tstrbuf_addf(&buf, \"%s/message\", get_dir(opts));\n> > \tif (!file_exists(buf.buf)) {\n> > -\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n> > +\t\tconst char *encoding = get_commit_output_encoding();\n> > +\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n> \n> That makes sense, though it's hard to understand the flow of this data\n> through multiple sequencer invocations. I _think_ this would be fixing a\n> case like this:\n> \n> -- >8 --\n> git init repo\n> cd repo\n> \n> # some commits to build off of\n> echo base >file\n> git add file\n> git commit -m base\n> \n> echo side >file\n> git add file\n> git commit -m side\n> \n> # now make a commit in iso8859-1\n> git checkout -b side HEAD^\n> echo iso8859-1 >file\n> git add file\n> iconv -f utf8 -t iso8859-1 <<-\\EOF |\n> súbject\n> \n> bödy\n> EOF\n> git -c i18n.commitEncoding=iso8859-1 commit -F -\n> \n> # and rebase it with the merge strategy, which will fail;\n> # now .git/rebase-merge/message has iso8859-1 in it\n> git rebase -m master\n> \n> # and if we resolve and commit, presumably we'd get a broken commit,\n> # with iso8859-1 and no encoding header\n> echo resolved >file\n> git add file\n> GIT_EDITOR=: git rebase --continue\n> -- 8< --\n> \n> But somehow it all seems to work. The resulting commit has real utf8 in\n> it. I'm not sure if we pull it from the original commit via \"commit -c\",\n\nYes, somehow it worked. But, without this patch, git also warns:\n\n % GIT_EDITOR=: git rebase --continue\n Warning: commit message did not conform to UTF-8.\n You may want to amend it after fixing the message, or set the config\n variable i18n.commitencoding to the encoding your project uses.\n\nChecking with strace (on glibc, musl strace can't trace execve):\n\n> [pid 12848] execve(\"/home/danh/workspace/git/git\", [\"/home/danh/workspace/git/git\", \"commit\", \"-n\", \"-F\", \".git/rebase-merge/message\", \"-e\", \"--allow-empty\"], 0x558fb02e8240 /* 51 vars */) = 0\n\nTurn out, it's because of: commit.c::verify_utf8\n\n /*\n * This verifies that the buffer is in proper utf8 format.\n *\n * If it isn't, it assumes any non-utf8 characters are Latin1,\n * and does the conversion.\n */\n static int verify_utf8(struct strbuf *buf)\n\nHence, your test is just pure luck (because it's in latin1).\n\n-- \nDanh\n"},{"id":"385678","messageId":"20191107080218.GA11245@sigill.intra.peff.net","threadId":"52167","inReplyTo":"20191107064854.GB8096@danh.dev","subject":"Re: [PATCH v4 3/8] t3900: demonstrate git-rebase problem with multi encoding","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-07T08:02:18Z","receivedAt":"2019-11-07T08:02:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 07, 2019 at 01:48:54PM +0700, Danh Doan wrote:\n\n> On 2019-11-07 01:02:33 -0500, Jeff King wrote:\n> > On Thu, Nov 07, 2019 at 09:56:14AM +0700, Doan Tran Cong Danh wrote:\n> > \n> > > +test_commit_autosquash_multi_encoding () {\n> > > +\tflag=$1\n> > > +\told=$2\n> > > +\tnew=$3\n> > > +\tmsg=$4\n> > > +\ttest_expect_failure \"commit --$flag into $old from $new\" '\n> > > +\t\tgit checkout -b '$flag-$old-$new' C0 &&\n> > \n> > These single quotes are funny; they close the test-snippet string, so\n> > these variables are outside of any quoting (and thus subject to\n> > whitespace splitting).\n> \n> Yes, those quotes are funny, and I'm also aware that\n> they will be subjected to whitespace spliting.\n> But those quotes were intentional, they're there in order to have\n> better log with:\n> \n> ./t3900-i18n-commit.sh -v\n> \n> With those funny quotes, we will see this (verbose) log:\n> \n> expecting success of 3900.38 'commit --fixup into ISO-2022-JP from UTF-8':\n> git checkout -b fixup-ISO-2022-JP-UTF-8 C0 &&\n\nYes, it's true you get the expanded version here, but...\n\n> git config i18n.commitencoding ISO-2022-JP &&\n> echo ISO-2022-JP >>F &&\n> git commit -a -F \"$TEST_DIRECTORY/t3900/ISO-2022-JP.txt\" &&\n\n...you still can't just run this manually because of other lines like\nthis one.\n\nIt's also weirdly unlike all of the other tests, which creates confusion\nfor people reading the code. IMHO the tradeoff isn't worth it.\n\n-Peff\n"},{"id":"385679","messageId":"20191107080307.GB11245@sigill.intra.peff.net","threadId":"52167","inReplyTo":"20191107074858.GC8096@danh.dev","subject":"Re: [PATCH v4 8/8] sequencer: reencode commit message for am/rebase --show-current-patch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-07T08:03:07Z","receivedAt":"2019-11-07T08:03:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 07, 2019 at 02:48:58PM +0700, Danh Doan wrote:\n\n> > # and if we resolve and commit, presumably we'd get a broken commit,\n> > # with iso8859-1 and no encoding header\n> > echo resolved >file\n> > git add file\n> > GIT_EDITOR=: git rebase --continue\n> > -- 8< --\n> > \n> > But somehow it all seems to work. The resulting commit has real utf8 in\n> > it. I'm not sure if we pull it from the original commit via \"commit -c\",\n> \n> Yes, somehow it worked. But, without this patch, git also warns:\n> \n> % GIT_EDITOR=: git rebase --continue\n> Warning: commit message did not conform to UTF-8.\n> You may want to amend it after fixing the message, or set the config\n> variable i18n.commitencoding to the encoding your project uses.\n> \n> Checking with strace (on glibc, musl strace can't trace execve):\n> \n> > [pid 12848] execve(\"/home/danh/workspace/git/git\", [\"/home/danh/workspace/git/git\", \"commit\", \"-n\", \"-F\", \".git/rebase-merge/message\", \"-e\", \"--allow-empty\"], 0x558fb02e8240 /* 51 vars */) = 0\n> \n> Turn out, it's because of: commit.c::verify_utf8\n> \n> /*\n> * This verifies that the buffer is in proper utf8 format.\n> *\n> * If it isn't, it assumes any non-utf8 characters are Latin1,\n> * and does the conversion.\n> */\n> static int verify_utf8(struct strbuf *buf)\n> \n> Hence, your test is just pure luck (because it's in latin1).\n\nAh, thanks for resolving that mystery. Is it worth turning the scenario\nabove into a test?\n\n-Peff\n"},{"id":"385695","messageId":"20191107105109.GD8096@danh.dev","threadId":"52167","inReplyTo":"20191107080218.GA11245@sigill.intra.peff.net","subject":"Re: [PATCH v4 3/8] t3900: demonstrate git-rebase problem with multi encoding","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T10:51:09Z","receivedAt":"2019-11-07T10:51:14Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-11-07 03:02:18 -0500, Jeff King wrote:\n> > git config i18n.commitencoding ISO-2022-JP &&\n> > echo ISO-2022-JP >>F &&\n> > git commit -a -F \"$TEST_DIRECTORY/t3900/ISO-2022-JP.txt\" &&\n> \n> ...you still can't just run this manually because of other lines like\n> this one.\n\nExcept we can with a little effort:\n\n export TEST_DIRECTORY=..\n\n> It's also weirdly unlike all of the other tests, which creates confusion\n> for people reading the code. IMHO the tradeoff isn't worth it.\n\nHm, I think it's the test_commit_autosquash_flag is the one that is weird\nin this file. Most of other sets of tests (line 89-176) use the same quote.\n\ntest_commit_autosquash_flag is the inconsistent one,\nfor the most part, it doesn't employ the funny quote,\nbut it uses the funny one in the `git cat-file` line.\n\n-- \nDanh\n"},{"id":"385709","messageId":"20191107163240.GF8096@danh.dev","threadId":"52167","inReplyTo":"20191107080307.GB11245@sigill.intra.peff.net","subject":"Re: [PATCH v4 8/8] sequencer: reencode commit message for am/rebase --show-current-patch","fromName":"Danh Doan","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-07T16:32:40Z","receivedAt":"2019-11-07T16:32:45Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2019-11-07 03:03:07 -0500, Jeff King wrote:\n> On Thu, Nov 07, 2019 at 02:48:58PM +0700, Danh Doan wrote:\n> \n> > > # and if we resolve and commit, presumably we'd get a broken commit,\n> > > # with iso8859-1 and no encoding header\n> > > echo resolved >file\n> > > git add file\n> > > GIT_EDITOR=: git rebase --continue\n> > > -- 8< --\n> > > \n> > > But somehow it all seems to work. The resulting commit has real utf8 in\n> > > it. I'm not sure if we pull it from the original commit via \"commit -c\",\n> > \n> > Yes, somehow it worked. But, without this patch, git also warns:\n> > \n> > % GIT_EDITOR=: git rebase --continue\n> > Warning: commit message did not conform to UTF-8.\n> > You may want to amend it after fixing the message, or set the config\n> > variable i18n.commitencoding to the encoding your project uses.\n> > \n> > Checking with strace (on glibc, musl strace can't trace execve):\n> > \n> > > [pid 12848] execve(\"/home/danh/workspace/git/git\", [\"/home/danh/workspace/git/git\", \"commit\", \"-n\", \"-F\", \".git/rebase-merge/message\", \"-e\", \"--allow-empty\"], 0x558fb02e8240 /* 51 vars */) = 0\n> > \n> > Turn out, it's because of: commit.c::verify_utf8\n> > \n> > /*\n> > * This verifies that the buffer is in proper utf8 format.\n> > *\n> > * If it isn't, it assumes any non-utf8 characters are Latin1,\n> > * and does the conversion.\n> > */\n> > static int verify_utf8(struct strbuf *buf)\n> > \n> > Hence, your test is just pure luck (because it's in latin1).\n> \n> Ah, thanks for resolving that mystery. Is it worth turning the scenario\n> above into a test?\n\nYes, it's worth to have a test.\nIn fact, I found another breakage (rebase with encoding) while writing\nthis test. I'll delay the re-roll a bit to include that breakage.\n\n-- \nDanh\n"},{"id":"385770","messageId":"cover.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"20191031092618.29073-1-congdanhqx@gmail.com","subject":"[PATCH v5 0/9] Improve odd encoding integration","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:42Z","receivedAt":"2019-11-08T09:44:03Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"The series is shifting from fixing test that failed on musl based Linux to\ncorrect the internal working encoding and output encoding of git-am\ngit-cherry-pick git-rebase and git-revert.\n\nChange from v4:\n- Update commit message per review\n- Add test for last 2 patches\n- Notice a breakage with git rebase --rebase-merges (see patch 7)\n\n\nDoan Tran Cong Danh (9):\n t0028: eliminate non-standard usage of printf\n configure.ac: define ICONV_OMITS_BOM if necessary\n t3900: demonstrate git-rebase problem with multi encoding\n sequencer: reencode to utf-8 before arrange rebase's todo list\n sequencer: reencode revert/cherry-pick's todo list\n sequencer: reencode squashing commit's message\n sequencer: reencode old merge-commit message\n sequencer: reencode commit message for am/rebase --show-current-patch\n sequencer: fallback to sane label in making rebase todo list\n\n configure.ac | 49 ++++++++++++++++++\n sequencer.c | 32 ++++++++----\n t/t0028-working-tree-encoding.sh | 4 +-\n t/t3433-rebase-i18n.sh | 84 +++++++++++++++++++++++++++++++\n t/t3433/ISO8859-1.txt | Bin 0 -> 15 bytes\n t/t3433/eucJP.txt | Bin 0 -> 68 bytes\n t/t3900-i18n-commit.sh | 37 ++++++++++++++\n 7 files changed, 195 insertions(+), 11 deletions(-)\n create mode 100755 t/t3433-rebase-i18n.sh\n create mode 100644 t/t3433/ISO8859-1.txt\n create mode 100644 t/t3433/eucJP.txt\n\nRange-diff against v4:\n 1: daa0c27d28 = 1: b3d6c4e720 t0028: eliminate non-standard usage of printf\n 2: c50964f413 ! 2: fe63a6bc44 configure.ac: define ICONV_OMITS_BOM if necessary\n @@ Commit message\n \n make ICONV_OMITS_BOM=Yes\n \n - However, typing the flag all the time is cumbersome and error-prone.\n + However, configure script wasn't taught to detect those systems.\n \n - Add a check into configure script to detect this flag automatically.\n + Teach configure to do so.\n \n Signed-off-by: Doan Tran Cong Danh \n \n 3: 47888f236c ! 3: 30f15075c4 t3900: demonstrate git-rebase problem with multi encoding\n @@ t/t3900-i18n-commit.sh: test_commit_autosquash_flags eucJP fixup\n +\tnew=$3\n +\tmsg=$4\n +\ttest_expect_failure \"commit --$flag into $old from $new\" '\n -+\t\tgit checkout -b '$flag-$old-$new' C0 &&\n -+\t\tgit config i18n.commitencoding '$old' &&\n -+\t\techo '$old' >>F &&\n -+\t\tgit commit -a -F \"$TEST_DIRECTORY/t3900/'$msg'\" &&\n ++\t\tgit checkout -b $flag-$old-$new C0 &&\n ++\t\tgit config i18n.commitencoding $old &&\n ++\t\techo $old >>F &&\n ++\t\tgit commit -a -F \"$TEST_DIRECTORY\"/t3900/$msg &&\n +\t\ttest_tick &&\n +\t\techo intermediate stuff >>G &&\n +\t\tgit add G &&\n +\t\tgit commit -a -m \"intermediate commit\" &&\n +\t\ttest_tick &&\n -+\t\tgit config i18n.commitencoding '$new' &&\n -+\t\techo '$new-$flag' >>F &&\n -+\t\tgit commit -a --'$flag' HEAD^ &&\n ++\t\tgit config i18n.commitencoding $new &&\n ++\t\techo $new-$flag >>F &&\n ++\t\tgit commit -a --$flag HEAD^ &&\n +\t\tgit rebase --autosquash -i HEAD^^^ &&\n +\t\tgit rev-list HEAD >actual &&\n +\t\ttest_line_count = 3 actual\n 4: 42115f1e33 ! 4: 17165b81e5 sequencer: reencode to utf-8 before arrange rebase's todo list\n @@ Commit message\n Thus, t3900::test_commit_autosquash_flags is failing on musl libc.\n \n Reencode to utf-8 before arranging rebase's todo list.\n - By doing this, we also remove a breakage introduced in the previous\n - commit.\n +\n + By doing this, we also remove a breakage noticed by a test added in the\n + previous commit.\n \n Signed-off-by: Doan Tran Cong Danh \n \n @@ t/t3900-i18n-commit.sh: test_commit_autosquash_multi_encoding () {\n \tmsg=$4\n -\ttest_expect_failure \"commit --$flag into $old from $new\" '\n +\ttest_expect_success \"commit --$flag into $old from $new\" '\n - \t\tgit checkout -b '$flag-$old-$new' C0 &&\n - \t\tgit config i18n.commitencoding '$old' &&\n - \t\techo '$old' >>F &&\n + \t\tgit checkout -b $flag-$old-$new C0 &&\n + \t\tgit config i18n.commitencoding $old &&\n + \t\techo $old >>F &&\n 5: 5a871d7226 = 5: 40fa759492 sequencer: reencode revert/cherry-pick's todo list\n 6: 1c6194a598 ! 6: ed6cfab5d2 sequencer: reencode squashing commit's message\n @@ sequencer.c: static int commit_staged_changes(struct repository *r,\n \n ## t/t3900-i18n-commit.sh ##\n @@ t/t3900-i18n-commit.sh: test_commit_autosquash_multi_encoding () {\n - \told=$2\n - \tnew=$3\n - \tmsg=$4\n -+\tsquash_msg=\n -+\tif test $flag = squash; then\n -+\t\tsquash_msg='\n -+\t\tsubject=\"squash! $(head -1 expect)\" &&\n -+\t\tprintf \"\\n%s\\n\" \"$subject\" >> expect &&\n -+\t\t'\n -+\tfi\n - \ttest_expect_success \"commit --$flag into $old from $new\" '\n - \t\tgit checkout -b '$flag-$old-$new' C0 &&\n - \t\tgit config i18n.commitencoding '$old' &&\n -@@ t/t3900-i18n-commit.sh: test_commit_autosquash_multi_encoding () {\n - \t\tgit commit -a --'$flag' HEAD^ &&\n + \t\tgit commit -a --$flag HEAD^ &&\n \t\tgit rebase --autosquash -i HEAD^^^ &&\n \t\tgit rev-list HEAD >actual &&\n -\t\ttest_line_count = 3 actual\n +\t\ttest_line_count = 3 actual &&\n -+\t\ticonv -f '$old' -t utf-8 \"$TEST_DIRECTORY/t3900/'$msg'\" >expect &&\n -+\t\t'\"$squash_msg\"'\n ++\t\ticonv -f $old -t UTF-8 \"$TEST_DIRECTORY\"/t3900/$msg >expect &&\n ++\t\tif test $flag = squash; then\n ++\t\t\tsubject=\"$(head -1 expect)\" &&\n ++\t\t\tprintf \"\\nsquash! %s\\n\" \"$subject\" >>expect\n ++\t\tfi &&\n +\t\tgit cat-file commit HEAD^ >raw &&\n -+\t\t(sed \"1,/^$/d\" raw | iconv -f '$new' -t utf-8) >actual &&\n ++\t\t(sed \"1,/^$/d\" raw | iconv -f $new -t utf-8) >actual &&\n +\t\ttest_cmp expect actual\n \t'\n }\n 7: 95df3cdadf < -: ---------- sequencer: reencode old merge-commit message\n 8: 0606b2408d < -: ---------- sequencer: reencode commit message for am/rebase --show-current-patch\n -: ---------- > 7: def9adf97e sequencer: reencode old merge-commit message\n -: ---------- > 8: 2e95ca57d2 sequencer: reencode commit message for am/rebase --show-current-patch\n -: ---------- > 9: 860dee65f4 sequencer: fallback to sane label in making rebase todo list\n-- \n2.24.0.8.g2e95ca57d2.dirty\n\n"},{"id":"385771","messageId":"b3d6c4e72020072eb495741d125a08ec07f33ba4.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573205699.git.congdanhqx@gmail.com","subject":"[PATCH v5 1/9] t0028: eliminate non-standard usage of printf","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:43Z","receivedAt":"2019-11-08T09:44:03Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"man 1p printf:\n In addition to the escape sequences shown in the Base Definitions\n volume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\\\',\n '\\a', '\\b', '\\f', '\\n', '\\r', '\\t', '\\v'), \"\\ddd\", where ddd is a\n one, two, or three-digit octal number, shall be written as a byte\n with the numeric value specified by the octal number.\n\nprintf '\\xfe\\xff' is an extension of some shell.\nDash, a popular yet simple shell, do not implement this extension.\n\nThis wasn't caught by most people running the tests, even though\ncommon shells like dash don't handle hex escapes, because their\nsystems don't trigger the NO_UTF16_BOM prereq. But systems with musl\nlibc do; when combined with dash, the test fails.\n\nCorrect it.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n t/t0028-working-tree-encoding.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh\nindex 7aa0945d8d..bfc4fb9af5 100755\n--- a/t/t0028-working-tree-encoding.sh\n+++ b/t/t0028-working-tree-encoding.sh\n@@ -17,7 +17,7 @@ test_lazy_prereq NO_UTF32_BOM '\n write_utf16 () {\n \tif test_have_prereq NO_UTF16_BOM\n \tthen\n-\t\tprintf '\\xfe\\xff'\n+\t\tprintf '\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-16\n }\n@@ -25,7 +25,7 @@ write_utf16 () {\n write_utf32 () {\n \tif test_have_prereq NO_UTF32_BOM\n \tthen\n-\t\tprintf '\\x00\\x00\\xfe\\xff'\n+\t\tprintf '\\0\\0\\376\\377'\n \tfi &&\n \ticonv -f UTF-8 -t UTF-32\n }\n-- \n2.24.0.8.g2e95ca57d2.dirty\n\n"},{"id":"385772","messageId":"fe63a6bc4483606048acdf19247574200b4ae8bf.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573205699.git.congdanhqx@gmail.com","subject":"[PATCH v5 2/9] configure.ac: define ICONV_OMITS_BOM if necessary","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:44Z","receivedAt":"2019-11-08T09:44:05Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"From commit 79444c9294, (\"utf8: handle systems that don't write BOM for\nUTF-16\", 2019-02-12), we're supporting those systems with iconv that\nomits BOM with:\n\n make ICONV_OMITS_BOM=Yes\n\nHowever, configure script wasn't taught to detect those systems.\n\nTeach configure to do so.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n configure.ac | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/configure.ac b/configure.ac\nindex a43b476402..ecba7e6e51 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -844,12 +844,61 @@ AC_MSG_CHECKING([for old iconv()])\n AC_COMPILE_IFELSE([OLDICONVTEST_SRC],\n \t[AC_MSG_RESULT([no])],\n \t[AC_MSG_RESULT([yes])\n+\tAC_DEFINE(HAVE_OLD_ICONV, 1)\n \tOLD_ICONV=UnfortunatelyYes])\n \n GIT_UNSTASH_FLAGS($ICONVDIR)\n \n GIT_CONF_SUBST([OLD_ICONV])\n \n+#\n+# Define ICONV_OMITS_BOM if you are on a system which\n+# iconv omits bom for utf-{16,32}\n+if test -z \"$NO_ICONV\"; then\n+AC_CACHE_CHECK([whether iconv omits bom for utf-16 and utf-32],\n+ [ac_cv_iconv_omits_bom],\n+[\n+old_LIBS=\"$LIBS\"\n+if test -n \"$NEEDS_LIBICONV\"; then\n+\tLIBS=\"$LIBS -liconv\"\n+fi\n+\n+AC_RUN_IFELSE(\n+\t[AC_LANG_PROGRAM([AC_INCLUDES_DEFAULT\n+\t#include \n+\t#ifdef HAVE_OLD_ICONV\n+\ttypedef const char *iconv_ibp;\n+\t#else\n+\ttypedef char *iconv_ibp;\n+\t#endif\n+\t],\n+\t[[\n+\tint v;\n+\ticonv_t conv;\n+\tchar in[] = \"a\"; iconv_ibp pin = in;\n+\tchar out[20] = \"\"; char *pout = out;\n+\tsize_t isz = sizeof in;\n+\tsize_t osz = sizeof out;\n+\n+\tconv = iconv_open(\"UTF-16\", \"UTF-8\");\n+\ticonv(conv, &pin, &isz, &pout, &osz);\n+\ticonv_close(conv);\n+\tv = (unsigned char)(out[0]) + (unsigned char)(out[1]);\n+\treturn v != 0xfe + 0xff;\n+\t]])],\n+\t[ac_cv_iconv_omits_bom=no],\n+\t[ac_cv_iconv_omits_bom=yes])\n+\n+LIBS=\"$old_LIBS\"\n+])\n+if test \"x$ac_cv_iconv_omits_bom\" = xyes; then\n+\tICONV_OMITS_BOM=Yes\n+else\n+\tICONV_OMITS_BOM=\n+fi\n+GIT_CONF_SUBST([ICONV_OMITS_BOM])\n+fi\n+\n ## Checks for typedefs, structures, and compiler characteristics.\n AC_MSG_NOTICE([CHECKS for typedefs, structures, and compiler characteristics])\n #\n-- \n2.24.0.8.g2e95ca57d2.dirty\n\n"},{"id":"385774","messageId":"17165b81e53e45b0abe9d5a03d7f214a28043cc8.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573205699.git.congdanhqx@gmail.com","subject":"[PATCH v5 4/9] sequencer: reencode to utf-8 before arrange rebase's todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:46Z","receivedAt":"2019-11-08T09:44:09Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On musl libc, ISO-2022-JP encoder is too eager to switch back to\n1 byte encoding, musl's iconv always switch back after every combining\ncharacter. Comparing glibc and musl's output for this command\n$ sed q t/t3900/ISO-2022-JP.txt| iconv -f ISO-2022-JP -t utf-8 |\n\ticonv -f utf-8 -t ISO-2022-JP | xxd\n\nglibc:\n00000000: 1b24 4224 4f24 6c24 5224 5b24 551b 2842 .$B$O$l$R$[$U.(B\n00000010: 0a .\n\nmusl:\n00000000: 1b24 4224 4f1b 2842 1b24 4224 6c1b 2842 .$B$O.(B.$B$l.(B\n00000010: 1b24 4224 521b 2842 1b24 4224 5b1b 2842 .$B$R.(B.$B$[.(B\n00000020: 1b24 4224 551b 2842 0a .$B$U.(B.\n\nAlthough musl iconv's output isn't optimal, it's still correct.\n\nFrom commit 7d509878b8, (\"pretty.c: format string with truncate respects\nlogOutputEncoding\", 2014-05-21), we're encoding the message to utf-8\nfirst, then format it and convert the message to the actual output\nencoding on git commit --squash.\n\nThus, t3900::test_commit_autosquash_flags is failing on musl libc.\n\nReencode to utf-8 before arranging rebase's todo list.\n\nBy doing this, we also remove a breakage noticed by a test added in the\nprevious commit.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 2 +-\n t/t3900-i18n-commit.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 9d5964fd81..69430fe23f 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -5169,7 +5169,7 @@ int todo_list_rearrange_squash(struct todo_list *todo_list)\n \t\t*commit_todo_item_at(&commit_todo, item->commit) = item;\n \n \t\tparse_commit(item->commit);\n-\t\tcommit_buffer = get_commit_buffer(item->commit, NULL);\n+\t\tcommit_buffer = logmsg_reencode(item->commit, NULL, \"UTF-8\");\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tformat_subject(&buf, subject, \" \");\n \t\tsubject = subjects[i] = strbuf_detach(&buf, &subject_len);\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex dd56384b93..a518281b04 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -209,7 +209,7 @@ test_commit_autosquash_multi_encoding () {\n \told=$2\n \tnew=$3\n \tmsg=$4\n-\ttest_expect_failure \"commit --$flag into $old from $new\" '\n+\ttest_expect_success \"commit --$flag into $old from $new\" '\n \t\tgit checkout -b $flag-$old-$new C0 &&\n \t\tgit config i18n.commitencoding $old &&\n \t\techo $old >>F &&\n-- \n2.24.0.8.g2e95ca57d2.dirty\n\n"},{"id":"385773","messageId":"30f15075c405b9049790518b64f66cb1740184f1.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573205699.git.congdanhqx@gmail.com","subject":"[PATCH v5 3/9] t3900: demonstrate git-rebase problem with multi encoding","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:45Z","receivedAt":"2019-11-08T09:44:10Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"We're using fixup!/squash! to mark if current commit will be\nused to be fixed up or squashed to a previous commit.\n\nHowever, if we're changing i18n.commitencoding after making the\noriginal commit but before making the fixing up, we couldn't find the\noriginal commit to do the fixup/squash.\n\nAdd a test to demonstrate that problem.\n\nHelped-by: Jeff King \nSigned-off-by: Doan Tran Cong Danh \n---\n t/t3900-i18n-commit.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex b92ff95977..dd56384b93 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -204,4 +204,33 @@ test_commit_autosquash_flags eucJP fixup\n \n test_commit_autosquash_flags ISO-2022-JP squash\n \n+test_commit_autosquash_multi_encoding () {\n+\tflag=$1\n+\told=$2\n+\tnew=$3\n+\tmsg=$4\n+\ttest_expect_failure \"commit --$flag into $old from $new\" '\n+\t\tgit checkout -b $flag-$old-$new C0 &&\n+\t\tgit config i18n.commitencoding $old &&\n+\t\techo $old >>F &&\n+\t\tgit commit -a -F \"$TEST_DIRECTORY\"/t3900/$msg &&\n+\t\ttest_tick &&\n+\t\techo intermediate stuff >>G &&\n+\t\tgit add G &&\n+\t\tgit commit -a -m \"intermediate commit\" &&\n+\t\ttest_tick &&\n+\t\tgit config i18n.commitencoding $new &&\n+\t\techo $new-$flag >>F &&\n+\t\tgit commit -a --$flag HEAD^ &&\n+\t\tgit rebase --autosquash -i HEAD^^^ &&\n+\t\tgit rev-list HEAD >actual &&\n+\t\ttest_line_count = 3 actual\n+\t'\n+}\n+\n+test_commit_autosquash_multi_encoding fixup UTF-8 ISO-8859-1 1-UTF-8.txt\n+test_commit_autosquash_multi_encoding squash ISO-8859-1 UTF-8 ISO8859-1.txt\n+test_commit_autosquash_multi_encoding squash eucJP ISO-2022-JP eucJP.txt\n+test_commit_autosquash_multi_encoding fixup ISO-2022-JP UTF-8 ISO-2022-JP.txt\n+\n test_done\n-- \n2.24.0.8.g2e95ca57d2.dirty\n\n"},{"id":"385775","messageId":"40fa7594923d877c780f28ebc73f28cf16ab8568.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573205699.git.congdanhqx@gmail.com","subject":"[PATCH v5 5/9] sequencer: reencode revert/cherry-pick's todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:47Z","receivedAt":"2019-11-08T09:44:10Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"Keep revert/cherry-pick's todo list in line with rebase todo list.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 69430fe23f..a19954f2bf 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -2564,14 +2564,17 @@ static int walk_revs_populate_todo(struct todo_list *todo_list,\n \tenum todo_command command = opts->action == REPLAY_PICK ?\n \t\tTODO_PICK : TODO_REVERT;\n \tconst char *command_string = todo_command_info[command].str;\n+\tconst char *encoding;\n \tstruct commit *commit;\n \n \tif (prepare_revs(opts))\n \t\treturn -1;\n \n+\tencoding = get_log_output_encoding();\n+\n \twhile ((commit = get_revision(opts->revs))) {\n \t\tstruct todo_item *item = append_new_todo(todo_list);\n-\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n+\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n \t\tconst char *subject;\n \t\tint subject_len;\n \n-- \n2.24.0.8.g2e95ca57d2.dirty\n\n"},{"id":"385776","messageId":"ed6cfab5d2d304fa248fd024b0f4ce67a59d297d.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573205699.git.congdanhqx@gmail.com","subject":"[PATCH v5 6/9] sequencer: reencode squashing commit's message","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:48Z","receivedAt":"2019-11-08T09:44:12Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On fixup/squash-ing rebase, git will create new commit in\ni18n.commitencoding, reencode the commit message to that said encode.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n sequencer.c | 8 +++++---\n t/t3900-i18n-commit.sh | 10 +++++++++-\n 2 files changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex a19954f2bf..833a928929 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1576,6 +1576,7 @@ static int update_squash_messages(struct repository *r,\n \tstruct strbuf buf = STRBUF_INIT;\n \tint res;\n \tconst char *message, *body;\n+\tconst char *encoding = get_commit_output_encoding();\n \n \tif (opts->current_fixup_count > 0) {\n \t\tstruct strbuf header = STRBUF_INIT;\n@@ -1602,7 +1603,7 @@ static int update_squash_messages(struct repository *r,\n \t\t\treturn error(_(\"need a HEAD to fixup\"));\n \t\tif (!(head_commit = lookup_commit_reference(r, &head)))\n \t\t\treturn error(_(\"could not read HEAD\"));\n-\t\tif (!(head_message = get_commit_buffer(head_commit, NULL)))\n+\t\tif (!(head_message = logmsg_reencode(head_commit, NULL, encoding)))\n \t\t\treturn error(_(\"could not read HEAD's commit message\"));\n \n \t\tfind_commit_subject(head_message, &body);\n@@ -1623,7 +1624,7 @@ static int update_squash_messages(struct repository *r,\n \t\tunuse_commit_buffer(head_commit, head_message);\n \t}\n \n-\tif (!(message = get_commit_buffer(commit, NULL)))\n+\tif (!(message = logmsg_reencode(commit, NULL, encoding)))\n \t\treturn error(_(\"could not read commit message of %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \tfind_commit_subject(message, &body);\n@@ -4154,9 +4155,10 @@ static int commit_staged_changes(struct repository *r,\n \t\t\t\t */\n \t\t\t\tstruct commit *commit;\n \t\t\t\tconst char *path = rebase_path_squash_msg();\n+\t\t\t\tconst char *encoding = get_commit_output_encoding();\n \n \t\t\t\tif (parse_head(r, &commit) ||\n-\t\t\t\t !(p = get_commit_buffer(commit, NULL)) ||\n+\t\t\t\t !(p = logmsg_reencode(commit, NULL, encoding)) ||\n \t\t\t\t write_message(p, strlen(p), path, 0)) {\n \t\t\t\t\tunuse_commit_buffer(commit, p);\n \t\t\t\t\treturn error(_(\"could not write file: \"\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex a518281b04..d277a9f4b7 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -224,7 +224,15 @@ test_commit_autosquash_multi_encoding () {\n \t\tgit commit -a --$flag HEAD^ &&\n \t\tgit rebase --autosquash -i HEAD^^^ &&\n \t\tgit rev-list HEAD >actual &&\n-\t\ttest_line_count = 3 actual\n+\t\ttest_line_count = 3 actual &&\n+\t\ticonv -f $old -t UTF-8 \"$TEST_DIRECTORY\"/t3900/$msg >expect &&\n+\t\tif test $flag = squash; then\n+\t\t\tsubject=\"$(head -1 expect)\" &&\n+\t\t\tprintf \"\\nsquash! %s\\n\" \"$subject\" >>expect\n+\t\tfi &&\n+\t\tgit cat-file commit HEAD^ >raw &&\n+\t\t(sed \"1,/^$/d\" raw | iconv -f $new -t utf-8) >actual &&\n+\t\ttest_cmp expect actual\n \t'\n }\n \n-- \n2.24.0.8.g2e95ca57d2.dirty\n\n"},{"id":"385777","messageId":"def9adf97ecb946a2cbde94b3eed742c17eb46c2.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573205699.git.congdanhqx@gmail.com","subject":"[PATCH v5 7/9] sequencer: reencode old merge-commit message","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:49Z","receivedAt":"2019-11-08T09:44:13Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"During rebasing, old merge's message (encoded in old encoding)\nwill be used as message for new merge commit (created by rebase).\n\nIn case of the value of i18n.commitencoding has been changed after the\nold merge time. We will receive an unusable message for this new merge.\n\nCorrect it.\n\nThis change also notice a breakage with git-rebase label system.\n\nSigned-off-by: Doan Tran Cong Danh \n---\n\nNotes:\n\teucJP.txt is copied from t3900\n\n sequencer.c | 3 ++-\n t/t3433-rebase-i18n.sh | 57 +++++++++++++++++++++++++++++++++++++++++\n t/t3433/eucJP.txt | Bin 0 -> 68 bytes\n 3 files changed, 59 insertions(+), 1 deletion(-)\n create mode 100755 t/t3433-rebase-i18n.sh\n create mode 100644 t/t3433/eucJP.txt\n\ndiff --git a/sequencer.c b/sequencer.c\nindex 833a928929..d735d09f98 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -3374,7 +3374,8 @@ static int do_merge(struct repository *r,\n \t}\n \n \tif (commit) {\n-\t\tconst char *message = get_commit_buffer(commit, NULL);\n+\t\tconst char *encoding = get_commit_output_encoding();\n+\t\tconst char *message = logmsg_reencode(commit, NULL, encoding);\n \t\tconst char *body;\n \t\tint len;\n \ndiff --git a/t/t3433-rebase-i18n.sh b/t/t3433-rebase-i18n.sh\nnew file mode 100755\nindex 0000000000..bff5a68d25\n--- /dev/null\n+++ b/t/t3433-rebase-i18n.sh\n@@ -0,0 +1,57 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2019 Doan Tran Cong Danh\n+#\n+\n+test_description='rebase with changing encoding\n+\n+Initial setup:\n+\n+1 - 2 master\n+ \\\n+ 3 - 4 first\n+ \\\n+ 5 - 6 second\n+'\n+\n+. ./test-lib.sh\n+\n+compare_msg () {\n+\ticonv -f \"$2\" -t \"$3\" \"$TEST_DIRECTORY/t3433/$1\" >expect &&\n+\tgit cat-file commit HEAD >raw &&\n+\tsed \"1,/^$/d\" raw >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+test_expect_success setup '\n+\ttest_commit one &&\n+\tgit branch first &&\n+\ttest_commit two &&\n+\tgit switch first &&\n+\ttest_commit three &&\n+\tgit branch second &&\n+\ttest_commit four &&\n+\tgit switch second &&\n+\ttest_commit five &&\n+\ttest_commit six\n+'\n+\n+test_expect_success 'rebase --rebase-merges update encoding eucJP to UTF-8' '\n+\tgit switch -c merge-eucJP-UTF-8 first &&\n+\tgit config i18n.commitencoding eucJP &&\n+\tgit merge -F \"$TEST_DIRECTORY/t3433/eucJP.txt\" second &&\n+\tgit config i18n.commitencoding UTF-8 &&\n+\tgit rebase --rebase-merges master &&\n+\tcompare_msg eucJP.txt eucJP UTF-8\n+'\n+\n+test_expect_failure 'rebase --rebase-merges update encoding eucJP to ISO-2022-JP' '\n+\tgit switch -c merge-eucJP-ISO-2022-JP first &&\n+\tgit config i18n.commitencoding eucJP &&\n+\tgit merge -F \"$TEST_DIRECTORY/t3433/eucJP.txt\" second &&\n+\tgit config i18n.commitencoding ISO-2022-JP &&\n+\tgit rebase --rebase-merges master &&\n+\tcompare_msg eucJP.txt eucJP ISO-2022-JP\n+'\n+\n+test_done\ndiff --git a/t/t3433/eucJP.txt b/t/t3433/eucJP.txt\nnew file mode 100644\nindex 0000000000000000000000000000000000000000..546f2aac01b67e39d19de601f5586097b34a8325\nGIT binary patch\nliteral 68\nzcmZ2-e#x69mzLaKa+Ql~$@V43mMmHFddayZYZfkovW_oY%ys|3$+JKuZ\n---\n\nNotes:\n\tISO8859-1.txt is copied from t3900\n\n sequencer.c | 3 ++-\n t/t3433-rebase-i18n.sh | 27 +++++++++++++++++++++++++++\n t/t3433/ISO8859-1.txt | Bin 0 -> 15 bytes\n 3 files changed, 29 insertions(+), 1 deletion(-)\n create mode 100644 t/t3433/ISO8859-1.txt\n\ndiff --git a/sequencer.c b/sequencer.c\nindex d735d09f98..4d12ad3cc6 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -2972,7 +2972,8 @@ static int make_patch(struct repository *r,\n \n \tstrbuf_addf(&buf, \"%s/message\", get_dir(opts));\n \tif (!file_exists(buf.buf)) {\n-\t\tconst char *commit_buffer = get_commit_buffer(commit, NULL);\n+\t\tconst char *encoding = get_commit_output_encoding();\n+\t\tconst char *commit_buffer = logmsg_reencode(commit, NULL, encoding);\n \t\tfind_commit_subject(commit_buffer, &subject);\n \t\tres |= write_message(subject, strlen(subject), buf.buf, 1);\n \t\tunuse_commit_buffer(commit, commit_buffer);\ndiff --git a/t/t3433-rebase-i18n.sh b/t/t3433-rebase-i18n.sh\nindex bff5a68d25..537d18c330 100755\n--- a/t/t3433-rebase-i18n.sh\n+++ b/t/t3433-rebase-i18n.sh\n@@ -54,4 +54,31 @@ test_expect_failure 'rebase --rebase-merges update encoding eucJP to ISO-2022-JP\n \tcompare_msg eucJP.txt eucJP ISO-2022-JP\n '\n \n+test_rebase_continue_update_encode () {\n+\told=$1\n+\tnew=$2\n+\tmsgfile=$3\n+\ttest_expect_success \"rebase --continue update from $old to $new\" '\n+\t\t(git rebase --abort || : abort current git-rebase failure) &&\n+\t\tgit switch -c conflict-$old-$new one &&\n+\t\techo for-conflict >two.t &&\n+\t\tgit add two.t &&\n+\t\tgit config i18n.commitencoding $old &&\n+\t\tgit commit -F \"$TEST_DIRECTORY/t3433/$msgfile\" &&\n+\t\tgit config i18n.commitencoding $new &&\n+\t\ttest_must_fail git rebase -m master &&\n+\t\ttest -f .git/rebase-merge/message &&\n+\t\tgit stripspace <.git/rebase-merge/message >two.t &&\n+\t\tgit add two.t &&\n+\t\tgit rebase --continue &&\n+\t\tcompare_msg $msgfile $old $new &&\n+\t\t: git-commit assume invalid utf-8 is latin1 &&\n+\t\ttest_cmp expect two.t\n+\t'\n+}\n+\n+test_rebase_continue_update_encode ISO-8859-1 UTF-8 ISO8859-1.txt\n+test_rebase_continue_update_encode eucJP UTF-8 eucJP.txt\n+test_rebase_continue_update_encode eucJP ISO-2022-JP eucJP.txt\n+\n test_done\ndiff --git a/t/t3433/ISO8859-1.txt b/t/t3433/ISO8859-1.txt\nnew file mode 100644\nindex 0000000000000000000000000000000000000000..7cbef0ee6f430c611134a06a6dd6c12fbea589d5\nGIT binary patch\nliteral 15\nXcmX?d`r`R(TwDi}o~OJ>OXmUrN~8!!\n\nliteral 0\nHcmV?d00001\n\n-- \n2.24.0.8.g2e95ca57d2.dirty\n\n"},{"id":"385779","messageId":"860dee65f49ea7eacf5a0c7c8ffe59095a51b1ce.1573205699.git.congdanhqx@gmail.com","threadId":"52167","inReplyTo":"cover.1573205699.git.congdanhqx@gmail.com","subject":"[PATCH v5 9/9] sequencer: fallback to sane label in making rebase todo list","fromName":"Doan Tran Cong Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2019-11-08T09:43:51Z","receivedAt":"2019-11-08T09:44:17Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"In later stage of rebasing, we will make a ref in\nrefs/rewritten/