{"thread":{"id":"51017","subject":"[PATCH v2 0/5] Fix and extend encoding handling in fast export/import","startedAt":"2019-04-30T18:25:32Z","lastAt":"2019-05-16T18:15:12Z","messageCount":45,"participants":["Elijah Newren","Torsten Bögershausen","Junio C Hamano","Johannes Schindelin","Eric Sunshine"],"isPatch":true,"patchVersion":2,"patchTotal":5},"messages":[{"id":"374707","messageId":"20190430182523.3339-1-newren@gmail.com","threadId":"51017","inReplyTo":null,"subject":"[PATCH v2 0/5] Fix and extend encoding handling in fast export/import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-04-30T18:25:18Z","receivedAt":"2019-04-30T18:25:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"While stress testing `git filter-repo`, I noticed an issue with\nencoding; further digging led to the fixes and features in this series.\nSee the individual commit messages for details.\n\nChanges since v1 (full range-diff below):\n * Applied style fixes Eric pointed out in his review (thanks!)\n * Rebased on latest master (83232e38, \"The seventh batch\"), resolving\n a trivial merge conflict. Now merges cleanly with next and pu as\n well.\n\nI'm a bit under the weather so I may be slow to respond...\n\nElijah Newren (5):\n t9350: fix encoding test to actually test reencoding\n fast-import: support 'encoding' commit header\n fast-export: avoid stripping encoding header if we cannot reencode\n fast-export: differentiate between explicitly utf-8 and implicitly\n utf-8\n fast-export: do automatic reencoding of commit messages only if\n requested\n\n Documentation/git-fast-import.txt | 7 ++++\n builtin/fast-export.c | 44 +++++++++++++++++++++----\n fast-import.c | 11 +++++--\n t/t9300-fast-import.sh | 20 ++++++++++++\n t/t9350-fast-export.sh | 53 +++++++++++++++++++++++++------\n 5 files changed, 118 insertions(+), 17 deletions(-)\n\nRange-diff:\n1: d6efd05142 ! 1: 9cc04242bd t9350: fix encoding test to actually test reencoding\n @@ -26,8 +26,7 @@\n -\t# use author and committer name in ISO-8859-1 to match it.\n -\t. \"$TEST_DIRECTORY\"/t3901/8859-1.txt &&\n +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n -+\ttest_when_finished \"git config --unset i18n.commitencoding\" &&\n -+\tgit config i18n.commitencoding iso-8859-7 &&\n ++\ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n -\tgit commit -s -m den file &&\n2: 02f48c7559 ! 2: 0cd023ac7a fast-import: support 'encoding' commit header\n @@ -51,9 +51,8 @@\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n -+\tif (skip_prefix(command_buf.buf, \"encoding \", &encoding)) {\n ++\tif (skip_prefix(command_buf.buf, \"encoding \", &encoding))\n +\t\tread_next_command();\n -+\t}\n \tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n @@ -69,7 +68,7 @@\n +\t\tstrbuf_addf(&new_data,\n +\t\t\t\"encoding %s\\n\",\n +\t\t\tencoding);\n -+\tstrbuf_addf(&new_data, \"\\n\");\n ++\tstrbuf_addch(&new_data, '\\n');\n \tstrbuf_addbuf(&new_data, &msg);\n \tfree(author);\n \tfree(committer);\n @@ -78,14 +77,14 @@\n --- a/t/t9300-fast-import.sh\n +++ b/t/t9300-fast-import.sh\n @@\n - \tbackground_import_still_running\n + \tsed -e s/LFs/LLL/ W-input | tr L \"\\n\" | test_must_fail git fast-import\n '\n \n +###\n -+### series W (other new features)\n ++### series X (other new features)\n +###\n +\n -+test_expect_success 'W: handling encoding' '\n ++test_expect_success 'X: handling encoding' '\n +\ttest_tick &&\n +\tcat >input <<-INPUT_END &&\n +\tcommit refs/heads/encoding\n3: 86c348402d ! 3: 1fddf51402 fast-export: avoid stripping encoding header if we cannot reencode\n @@ -41,8 +41,7 @@\n +test_expect_success 'encoding preserved if reencoding fails' '\n +\n +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n -+\ttest_when_finished \"git config --unset i18n.commitencoding\" &&\n -+\tgit config i18n.commitencoding iso-8859-7 &&\n ++\ttest_config i18n.commitencoding iso-8859-7 &&\n +\techo rosten >file &&\n +\tgit commit -s -m \"$(printf \"Pi: \\360; Invalid: \\377\")\" file &&\n +\tgit fast-export wer^..wer >iso-8859-7.fi &&\n4: c09b23bc59 = 4: 4a2e04b3ae fast-export: differentiate between explicitly utf-8 and implicitly utf-8\n5: 24b69a0db9 ! 5: 44aacb1a0b fast-export: do automatic reencoding of commit messages only if requested\n @@ -92,8 +92,7 @@\n +test_expect_success 'reencoding iso-8859-7' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n - \ttest_when_finished \"git config --unset i18n.commitencoding\" &&\n -@@\n + \ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n \tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n @@ -109,8 +108,7 @@\n +test_expect_success 'aborting on iso-8859-7' '\n +\n +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n -+\ttest_when_finished \"git config --unset i18n.commitencoding\" &&\n -+\tgit config i18n.commitencoding iso-8859-7 &&\n ++\ttest_config i18n.commitencoding iso-8859-7 &&\n +\techo rosten >file &&\n +\tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n +\ttest_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi\n @@ -119,8 +117,7 @@\n +test_expect_success 'preserving iso-8859-7' '\n +\n +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n -+\ttest_when_finished \"git config --unset i18n.commitencoding\" &&\n -+\tgit config i18n.commitencoding iso-8859-7 &&\n ++\ttest_config i18n.commitencoding iso-8859-7 &&\n +\techo rosten >file &&\n +\tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n +\tgit fast-export --reencode=no wer^..wer >iso-8859-7.fi &&\n @@ -134,8 +131,7 @@\n test_expect_success 'encoding preserved if reencoding fails' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n -@@\n - \tgit config i18n.commitencoding iso-8859-7 &&\n + \ttest_config i18n.commitencoding iso-8859-7 &&\n \techo rosten >file &&\n \tgit commit -s -m \"$(printf \"Pi: \\360; Invalid: \\377\")\" file &&\n -\tgit fast-export wer^..wer >iso-8859-7.fi &&\n-- \n2.21.0.782.g44aacb1a0b\n\n"},{"id":"374708","messageId":"20190430182523.3339-2-newren@gmail.com","threadId":"51017","inReplyTo":"20190430182523.3339-1-newren@gmail.com","subject":"[PATCH v2 1/5] t9350: fix encoding test to actually test reencoding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-04-30T18:25:19Z","receivedAt":"2019-04-30T18:25:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This test used an author with non-ascii characters in the name, but\nno special commit message. It then grep'ed for those non-ascii\ncharacters, but those are guaranteed to exist regardless of the\nreencoding process since the reencoding only affects the commit message,\nnot the author or committer names. As such, the test would work even if\nthe re-encoding process simply stripped the commit message entirely.\nModify the test to actually check that the reencoding in utf-8 worked.\n\nSigned-off-by: Elijah Newren \n---\n t/t9350-fast-export.sh | 18 ++++++++----------\n 1 file changed, 8 insertions(+), 10 deletions(-)\n\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 5690fe2810..f55759651a 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,22 +94,21 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-1' '\n+test_expect_success 'iso-8859-7' '\n \n-\tgit config i18n.commitencoding ISO8859-1 &&\n-\t# use author and committer name in ISO-8859-1 to match it.\n-\t. \"$TEST_DIRECTORY\"/t3901/8859-1.txt &&\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n-\tgit commit -s -m den file &&\n-\tgit fast-export wer^..wer >iso8859-1.fi &&\n-\tsed \"s/wer/i18n/\" iso8859-1.fi |\n+\tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n \t\t git cat-file commit i18n >actual &&\n-\t\t grep \"Áéí óú\" actual)\n-\n+\t\t grep $(printf \"\\317\\200\") actual)\n '\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\n@@ -224,7 +223,6 @@ GIT_COMMITTER_NAME='C O Mitter'; export GIT_COMMITTER_NAME\n \n test_expect_success 'setup copies' '\n \n-\tgit config --unset i18n.commitencoding &&\n \tgit checkout -b copy rein &&\n \tgit mv file file3 &&\n \tgit commit -m move1 &&\n-- \n2.21.0.782.g44aacb1a0b\n\n"},{"id":"374709","messageId":"20190430182523.3339-3-newren@gmail.com","threadId":"51017","inReplyTo":"20190430182523.3339-1-newren@gmail.com","subject":"[PATCH v2 2/5] fast-import: support 'encoding' commit header","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-04-30T18:25:20Z","receivedAt":"2019-04-30T18:25:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Since git supports commit messages with an encoding other than utf-8,\nallow fast-import to import such commits. This may be useful for folks\nwho do not want to reencode commit messages from an external system, and\nmay also be useful to achieve reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions or subtree work) with git repositories that\nhave used specialized encodings in their commit history.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-import.txt | 7 +++++++\n fast-import.c | 11 +++++++++--\n t/t9300-fast-import.sh | 20 ++++++++++++++++++++\n 3 files changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex d65cdb3d08..7baf9e47b5 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -388,6 +388,7 @@ change to the project.\n \toriginal-oid?\n \t('author' (SP )? SP LT GT SP LF)?\n \t'committer' (SP )? SP LT GT SP LF\n+\t('encoding' SP )?\n \tdata\n \t('from' SP LF)?\n \t('merge' SP LF)?\n@@ -455,6 +456,12 @@ that was selected by the --date-format= command-line option.\n See ``Date Formats'' above for the set of supported formats, and\n their syntax.\n \n+`encoding`\n+^^^^^^^^^^\n+The optional `encoding` command indicates the encoding of the commit\n+message. Most commits are UTF-8 and the encoding is omitted, but this\n+allows importing commit messages into git without first reencoding them.\n+\n `from`\n ^^^^^^\n The `from` command is used to specify the commit to initialize\ndiff --git a/fast-import.c b/fast-import.c\nindex f38d04fa58..76a7bd3699 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2585,6 +2585,7 @@ static void parse_new_commit(const char *arg)\n \tstruct branch *b;\n \tchar *author = NULL;\n \tchar *committer = NULL;\n+\tconst char *encoding = NULL;\n \tstruct hash_list *merge_list = NULL;\n \tunsigned int merge_count;\n \tunsigned char prev_fanout, new_fanout;\n@@ -2607,6 +2608,8 @@ static void parse_new_commit(const char *arg)\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n+\tif (skip_prefix(command_buf.buf, \"encoding \", &encoding))\n+\t\tread_next_command();\n \tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n@@ -2670,9 +2673,13 @@ static void parse_new_commit(const char *arg)\n \t}\n \tstrbuf_addf(&new_data,\n \t\t\"author %s\\n\"\n-\t\t\"committer %s\\n\"\n-\t\t\"\\n\",\n+\t\t\"committer %s\\n\",\n \t\tauthor ? author : committer, committer);\n+\tif (encoding)\n+\t\tstrbuf_addf(&new_data,\n+\t\t\t\"encoding %s\\n\",\n+\t\t\tencoding);\n+\tstrbuf_addch(&new_data, '\\n');\n \tstrbuf_addbuf(&new_data, &msg);\n \tfree(author);\n \tfree(committer);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 3668263c40..141b7fa35e 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3299,4 +3299,24 @@ test_expect_success !MINGW 'W: get-mark & empty orphan commit with erroneous thi\n \tsed -e s/LFs/LLL/ W-input | tr L \"\\n\" | test_must_fail git fast-import\n '\n \n+###\n+### series X (other new features)\n+###\n+\n+test_expect_success 'X: handling encoding' '\n+\ttest_tick &&\n+\tcat >input <<-INPUT_END &&\n+\tcommit refs/heads/encoding\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tencoding iso-8859-7\n+\tdata <>input &&\n+\n+\tgit fast-import \n---\n builtin/fast-export.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 7734a9f5a5..66331fa401 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -453,7 +453,7 @@ static const char *find_encoding(const char *begin, const char *end)\n \tbol = memmem(begin, end ? end - begin : strlen(begin),\n \t\t needle, strlen(needle));\n \tif (!bol)\n-\t\treturn git_commit_encoding;\n+\t\treturn NULL;\n \tbol += strlen(needle);\n \teol = strchrnul(bol, '\\n');\n \t*eol = '\\0';\n-- \n2.21.0.782.g44aacb1a0b\n\n"},{"id":"374711","messageId":"20190430182523.3339-4-newren@gmail.com","threadId":"51017","inReplyTo":"20190430182523.3339-1-newren@gmail.com","subject":"[PATCH v2 3/5] fast-export: avoid stripping encoding header if we cannot reencode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-04-30T18:25:21Z","receivedAt":"2019-04-30T18:25:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"When fast-export encounters a commit with an 'encoding' header, it tries\nto reencode in utf-8 and then drops the encoding header. However, if it\nfails to reencode in utf-8 because e.g. one of the characters in the\ncommit message was invalid in the old encoding, then we need to retain\nthe original encoding or otherwise we lose information needed to\nunderstand all the other (valid) characters in the original commit\nmessage.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 7 +++++--\n t/t9350-fast-export.sh | 14 ++++++++++++++\n 2 files changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 9e283482ef..7734a9f5a5 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -642,9 +642,12 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n \tif (show_original_ids)\n \t\tprintf(\"original-oid %s\\n\", oid_to_hex(&commit->object.oid));\n-\tprintf(\"%.*s\\n%.*s\\ndata %u\\n%s\",\n+\tprintf(\"%.*s\\n%.*s\\n\",\n \t (int)(author_end - author), author,\n-\t (int)(committer_end - committer), committer,\n+\t (int)(committer_end - committer), committer);\n+\tif (!reencoded && encoding)\n+\t\tprintf(\"encoding %s\\n\", encoding);\n+\tprintf(\"data %u\\n%s\",\n \t (unsigned)(reencoded\n \t\t\t ? strlen(reencoded) : message\n \t\t\t ? strlen(message) : 0),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex f55759651a..67dd7ac7f4 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -109,6 +109,20 @@ test_expect_success 'iso-8859-7' '\n \t\t grep $(printf \"\\317\\200\") actual)\n '\n \n+test_expect_success 'encoding preserved if reencoding fails' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -m \"$(printf \"Pi: \\360; Invalid: \\377\")\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t git cat-file commit i18n-invalid >actual &&\n+\t\t grep ^encoding actual)\n+'\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\n-- \n2.21.0.782.g44aacb1a0b\n\n"},{"id":"374712","messageId":"20190430182523.3339-6-newren@gmail.com","threadId":"51017","inReplyTo":"20190430182523.3339-1-newren@gmail.com","subject":"[PATCH v2 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-04-30T18:25:23Z","receivedAt":"2019-04-30T18:25:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Automatic re-encoding of commit messages (and dropping of the encoding\nheader) hurts attempts to do reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions, some subtree rewrites), and seems\ninconsistent with the general principle followed elsewhere in\nfast-export of requiring explicit user requests to modify the output\n(e.g. --signed-tags=strip, --tag-of-filtered-object=rewrite). Add a\n--reencode flag that the user can use to specify, and like other\nfast-export flags, default it to 'abort'.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 35 ++++++++++++++++++++++++++++++++---\n t/t9350-fast-export.sh | 29 ++++++++++++++++++++++++++---\n 2 files changed, 58 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 66331fa401..43cc52331c 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -33,6 +33,7 @@ static const char *fast_export_usage[] = {\n static int progress;\n static enum { SIGNED_TAG_ABORT, VERBATIM, WARN, WARN_STRIP, STRIP } signed_tag_mode = SIGNED_TAG_ABORT;\n static enum { TAG_FILTERING_ABORT, DROP, REWRITE } tag_of_filtered_mode = TAG_FILTERING_ABORT;\n+static enum { REENCODE_ABORT, REENCODE_PLEASE, REENCODE_NEVER } reencode_mode = REENCODE_ABORT;\n static int fake_missing_tagger;\n static int use_done_feature;\n static int no_data;\n@@ -77,6 +78,20 @@ static int parse_opt_tag_of_filtered_mode(const struct option *opt,\n \treturn 0;\n }\n \n+static int parse_opt_reencode_mode(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tif (unset || !strcmp(arg, \"abort\"))\n+\t\treencode_mode = REENCODE_ABORT;\n+\telse if (!strcmp(arg, \"yes\"))\n+\t\treencode_mode = REENCODE_PLEASE;\n+\telse if (!strcmp(arg, \"no\"))\n+\t\treencode_mode = REENCODE_NEVER;\n+\telse\n+\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n+\treturn 0;\n+}\n+\n static struct decoration idnums;\n static uint32_t last_idnum;\n \n@@ -633,10 +648,21 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \t}\n \n \tmark_next_object(&commit->object);\n-\tif (anonymize)\n+\tif (anonymize) {\n \t\treencoded = anonymize_commit_message(message);\n-\telse if (!is_encoding_utf8(encoding))\n-\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t} else if (encoding) {\n+\t\tswitch(reencode_mode) {\n+\t\tcase REENCODE_PLEASE:\n+\t\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t\t\tbreak;\n+\t\tcase REENCODE_NEVER:\n+\t\t\tbreak;\n+\t\tcase REENCODE_ABORT:\n+\t\t\tdie(\"Encountered commit-specific encoding %s in commit \"\n+\t\t\t \"%s; use --reencode= to handle it\",\n+\t\t\t encoding, oid_to_hex(&commit->object.oid));\n+\t\t}\n+\t}\n \tif (!commit->parents)\n \t\tprintf(\"reset %s\\n\", refname);\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n@@ -1091,6 +1117,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK(0, \"tag-of-filtered-object\", &tag_of_filtered_mode, N_(\"mode\"),\n \t\t\t N_(\"select handling of tags that tag filtered objects\"),\n \t\t\t parse_opt_tag_of_filtered_mode),\n+\t\tOPT_CALLBACK(0, \"reencode\", &reencode_mode, N_(\"mode\"),\n+\t\t\t N_(\"select handling of commit messages in an alternate encoding\"),\n+\t\t\t parse_opt_reencode_mode),\n \t\tOPT_STRING(0, \"export-marks\", &export_filename, N_(\"file\"),\n \t\t\t N_(\"Dump marks to this file\")),\n \t\tOPT_STRING(0, \"import-marks\", &import_filename, N_(\"file\"),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 67dd7ac7f4..92cfeb6cfc 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,14 +94,14 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-7' '\n+test_expect_success 'reencoding iso-8859-7' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n \tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n@@ -109,13 +109,36 @@ test_expect_success 'iso-8859-7' '\n \t\t grep $(printf \"\\317\\200\") actual)\n '\n \n+test_expect_success 'aborting on iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n+\ttest_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi\n+'\n+\n+test_expect_success 'preserving iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n+\tgit fast-export --reencode=no wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-no-recoding/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t git cat-file commit i18n-no-recoding >actual &&\n+\t\t grep $(printf \"\\360\") actual)\n+'\n+\n test_expect_success 'encoding preserved if reencoding fails' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \techo rosten >file &&\n \tgit commit -s -m \"$(printf \"Pi: \\360; Invalid: \\377\")\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n-- \n2.21.0.782.g44aacb1a0b\n\n"},{"id":"375317","messageId":"20190510205335.19968-1-newren@gmail.com","threadId":"51017","inReplyTo":"20190430182523.3339-1-newren@gmail.com","subject":"[PATCH v3 0/5] Fix and extend encoding handling in fast export/import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-10T20:53:30Z","receivedAt":"2019-05-10T20:53:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"While stress testing `git filter-repo`, I noticed an issue with\nencoding; further digging led to the fixes and features in this series.\nSee the individual commit messages for details.\n\nChanges since v2 (full range-diff below):\n * Modified the testcases to pass on Windows[1], as verified via\n gitgitgadget pull request[2]. Required adding a couple new files\n (which store the desired bytes) and checking the size of the output\n instead of checking for particular bytes (but the lengths of the\n expected byte sequences differ so this works fine...).\n\n[1] Failures of previous patchset on Windows noticed and reported by Dscho;\n explanation from Hannes is that Windows munges users' command lines to\n force them to be characters instead of bytes.\n[2] https://github.com/gitgitgadget/git/pull/187\n\nElijah Newren (5):\nt9350: fix encoding test to actually test reencoding\nfast-import: support 'encoding' commit header\nfast-export: avoid stripping encoding header if we cannot reencode\nfast-export: differentiate between explicitly utf-8 and implicitly\nutf-8\nfast-export: do automatic reencoding of commit messages only if\nrequested\n\nDocumentation/git-fast-import.txt | 7 ++\nbuiltin/fast-export.c | 44 ++++++++++--\nfast-import.c | 11 ++-\nt/t9300-fast-import.sh | 20 ++++++\nt/t9350-fast-export.sh | 75 +++++++++++++++++---\nt/t9350/broken-iso-8859-7-commit-message.txt | 1 +\nt/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n7 files changed, 142 insertions(+), 17 deletions(-)\ncreate mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\ncreate mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n\nRange-diff:\n1: 9cc04242bd ! 1: 2d7bb64acf t9350: fix encoding test to actually test reencoding\n @@ -32,15 +32,26 @@\n -\tgit commit -s -m den file &&\n -\tgit fast-export wer^..wer >iso8859-1.fi &&\n -\tsed \"s/wer/i18n/\" iso8859-1.fi |\n -+\tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n ++\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n +\tgit fast-export wer^..wer >iso-8859-7.fi &&\n +\tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n ++\t\t # The commit object, if not re-encoded, would be 240 bytes.\n ++\t\t # Removing the \"encoding iso-8859-7\\n\" header drops 20 bytes.\n ++\t\t # Re-encoding the Pi character from \\xF0 in iso-8859-7 to\n ++\t\t # \\xCF\\x80 in utf-8 adds a byte. Grepping for specific bytes\n ++\t\t # would be nice, but Windows apparently munges user data\n ++\t\t # in the form of bytes on the command line to force them to\n ++\t\t # be characters instead, so we are limited for portability\n ++\t\t # reasons in subsequent similar tests in this file to check\n ++\t\t # for size rather than what bytes are present.\n ++\t\t test 221 -eq \"$(git cat-file -s i18n)\" &&\n ++\t\t # Also make sure the commit does not have the \"encoding\" header\n \t\t git cat-file commit i18n >actual &&\n -\t\t grep \"Áéí óú\" actual)\n -\n -+\t\t grep $(printf \"\\317\\200\") actual)\n ++\t\t ! grep ^encoding actual)\n '\n +\n test_expect_success 'import/export-marks' '\n @@ -54,3 +65,11 @@\n \tgit checkout -b copy rein &&\n \tgit mv file file3 &&\n \tgit commit -m move1 &&\n +\n + diff --git a/t/t9350/simple-iso-8859-7-commit-message.txt b/t/t9350/simple-iso-8859-7-commit-message.txt\n + new file mode 100644\n + --- /dev/null\n + +++ b/t/t9350/simple-iso-8859-7-commit-message.txt\n +@@\n ++Pi: �\n + \\ No newline at end of file\n2: 0cd023ac7a = 2: 9fa5695017 fast-import: support 'encoding' commit header\n3: 1fddf51402 ! 3: dfc76573e9 fast-export: avoid stripping encoding header if we cannot reencode\n @@ -35,7 +35,7 @@\n --- a/t/t9350-fast-export.sh\n +++ b/t/t9350-fast-export.sh\n @@\n - \t\t grep $(printf \"\\317\\200\") actual)\n + \t\t ! grep ^encoding actual)\n '\n \n +test_expect_success 'encoding preserved if reencoding fails' '\n @@ -43,15 +43,26 @@\n +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n +\ttest_config i18n.commitencoding iso-8859-7 &&\n +\techo rosten >file &&\n -+\tgit commit -s -m \"$(printf \"Pi: \\360; Invalid: \\377\")\" file &&\n ++\tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n +\tgit fast-export wer^..wer >iso-8859-7.fi &&\n +\tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n +\t\t(cd new &&\n +\t\t git fast-import &&\n +\t\t git cat-file commit i18n-invalid >actual &&\n -+\t\t grep ^encoding actual)\n ++\t\t grep ^encoding actual &&\n ++\t\t # Also verify that the commit has the expected size; i.e.\n ++\t\t # that no bytes were re-encoded to a different encoding.\n ++\t\t test 252 -eq \"$(git cat-file -s i18n-invalid)\")\n +'\n +\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\n +\n + diff --git a/t/t9350/broken-iso-8859-7-commit-message.txt b/t/t9350/broken-iso-8859-7-commit-message.txt\n + new file mode 100644\n + --- /dev/null\n + +++ b/t/t9350/broken-iso-8859-7-commit-message.txt\n +@@\n ++Pi: �; Invalid: �\n + \\ No newline at end of file\n4: 4a2e04b3ae = 4: 83b3656b76 fast-export: differentiate between explicitly utf-8 and implicitly utf-8\n5: 44aacb1a0b ! 5: 2063122293 fast-export: do automatic reencoding of commit messages only if requested\n @@ -95,14 +95,14 @@\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n - \tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n + \tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n -\tgit fast-export wer^..wer >iso-8859-7.fi &&\n +\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n @@\n - \t\t grep $(printf \"\\317\\200\") actual)\n + \t\t ! grep ^encoding actual)\n '\n \n +test_expect_success 'aborting on iso-8859-7' '\n @@ -110,7 +110,7 @@\n +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n +\ttest_config i18n.commitencoding iso-8859-7 &&\n +\techo rosten >file &&\n -+\tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n ++\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n +\ttest_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi\n +'\n +\n @@ -119,13 +119,21 @@\n +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n +\ttest_config i18n.commitencoding iso-8859-7 &&\n +\techo rosten >file &&\n -+\tgit commit -s -m \"$(printf \"Pi: \\360\")\" file &&\n ++\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n +\tgit fast-export --reencode=no wer^..wer >iso-8859-7.fi &&\n +\tsed \"s/wer/i18n-no-recoding/\" iso-8859-7.fi |\n +\t\t(cd new &&\n +\t\t git fast-import &&\n ++\t\t # The commit object, if not re-encoded, is 240 bytes.\n ++\t\t # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n ++\t\t # bytes. Re-encoding the Pi character from \\xF0 in\n ++\t\t # iso-8859-7 to \\xCF\\x80 in utf-8 would add a byte. I would\n ++\t\t # grep for the # specific bytes, but Windows lamely does not\n ++\t\t # allow that, so just search for the expected size.\n ++\t\t test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n ++\t\t # Also make sure the commit has the \"encoding\" header\n +\t\t git cat-file commit i18n-no-recoding >actual &&\n -+\t\t grep $(printf \"\\360\") actual)\n ++\t\t grep ^encoding actual)\n +'\n +\n test_expect_success 'encoding preserved if reencoding fails' '\n @@ -133,7 +141,7 @@\n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \techo rosten >file &&\n - \tgit commit -s -m \"$(printf \"Pi: \\360; Invalid: \\377\")\" file &&\n + \tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n -\tgit fast-export wer^..wer >iso-8859-7.fi &&\n +\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n\n-- \n2.21.0.782.g2063122293\n\n"},{"id":"375318","messageId":"20190510205335.19968-2-newren@gmail.com","threadId":"51017","inReplyTo":"20190510205335.19968-1-newren@gmail.com","subject":"[PATCH v3 1/5] t9350: fix encoding test to actually test reencoding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-10T20:53:31Z","receivedAt":"2019-05-10T20:53:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This test used an author with non-ascii characters in the name, but\nno special commit message. It then grep'ed for those non-ascii\ncharacters, but those are guaranteed to exist regardless of the\nreencoding process since the reencoding only affects the commit message,\nnot the author or committer names. As such, the test would work even if\nthe re-encoding process simply stripped the commit message entirely.\nModify the test to actually check that the reencoding in utf-8 worked.\n\nSigned-off-by: Elijah Newren \n---\n t/t9350-fast-export.sh | 29 +++++++++++++-------\n t/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n 2 files changed, 20 insertions(+), 10 deletions(-)\n create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 5690fe2810..ef9b1aa20b 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,22 +94,32 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-1' '\n+test_expect_success 'iso-8859-7' '\n \n-\tgit config i18n.commitencoding ISO8859-1 &&\n-\t# use author and committer name in ISO-8859-1 to match it.\n-\t. \"$TEST_DIRECTORY\"/t3901/8859-1.txt &&\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n-\tgit commit -s -m den file &&\n-\tgit fast-export wer^..wer >iso8859-1.fi &&\n-\tsed \"s/wer/i18n/\" iso8859-1.fi |\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n+\t\t # The commit object, if not re-encoded, would be 240 bytes.\n+\t\t # Removing the \"encoding iso-8859-7\\n\" header drops 20 bytes.\n+\t\t # Re-encoding the Pi character from \\xF0 in iso-8859-7 to\n+\t\t # \\xCF\\x80 in utf-8 adds a byte. Grepping for specific bytes\n+\t\t # would be nice, but Windows apparently munges user data\n+\t\t # in the form of bytes on the command line to force them to\n+\t\t # be characters instead, so we are limited for portability\n+\t\t # reasons in subsequent similar tests in this file to check\n+\t\t # for size rather than what bytes are present.\n+\t\t test 221 -eq \"$(git cat-file -s i18n)\" &&\n+\t\t # Also make sure the commit does not have the \"encoding\" header\n \t\t git cat-file commit i18n >actual &&\n-\t\t grep \"Áéí óú\" actual)\n-\n+\t\t ! grep ^encoding actual)\n '\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\n@@ -224,7 +234,6 @@ GIT_COMMITTER_NAME='C O Mitter'; export GIT_COMMITTER_NAME\n \n test_expect_success 'setup copies' '\n \n-\tgit config --unset i18n.commitencoding &&\n \tgit checkout -b copy rein &&\n \tgit mv file file3 &&\n \tgit commit -m move1 &&\ndiff --git a/t/t9350/simple-iso-8859-7-commit-message.txt b/t/t9350/simple-iso-8859-7-commit-message.txt\nnew file mode 100644\nindex 0000000000..8b3f0c3dba\n--- /dev/null\n+++ b/t/t9350/simple-iso-8859-7-commit-message.txt\n@@ -0,0 +1 @@\n+Pi: �\n\\ No newline at end of file\n-- \n2.21.0.782.g2063122293\n\n"},{"id":"375319","messageId":"20190510205335.19968-3-newren@gmail.com","threadId":"51017","inReplyTo":"20190510205335.19968-1-newren@gmail.com","subject":"[PATCH v3 2/5] fast-import: support 'encoding' commit header","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-10T20:53:32Z","receivedAt":"2019-05-10T20:53:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Since git supports commit messages with an encoding other than utf-8,\nallow fast-import to import such commits. This may be useful for folks\nwho do not want to reencode commit messages from an external system, and\nmay also be useful to achieve reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions or subtree work) with git repositories that\nhave used specialized encodings in their commit history.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-import.txt | 7 +++++++\n fast-import.c | 11 +++++++++--\n t/t9300-fast-import.sh | 20 ++++++++++++++++++++\n 3 files changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex d65cdb3d08..7baf9e47b5 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -388,6 +388,7 @@ change to the project.\n \toriginal-oid?\n \t('author' (SP )? SP LT GT SP LF)?\n \t'committer' (SP )? SP LT GT SP LF\n+\t('encoding' SP )?\n \tdata\n \t('from' SP LF)?\n \t('merge' SP LF)?\n@@ -455,6 +456,12 @@ that was selected by the --date-format= command-line option.\n See ``Date Formats'' above for the set of supported formats, and\n their syntax.\n \n+`encoding`\n+^^^^^^^^^^\n+The optional `encoding` command indicates the encoding of the commit\n+message. Most commits are UTF-8 and the encoding is omitted, but this\n+allows importing commit messages into git without first reencoding them.\n+\n `from`\n ^^^^^^\n The `from` command is used to specify the commit to initialize\ndiff --git a/fast-import.c b/fast-import.c\nindex f38d04fa58..76a7bd3699 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2585,6 +2585,7 @@ static void parse_new_commit(const char *arg)\n \tstruct branch *b;\n \tchar *author = NULL;\n \tchar *committer = NULL;\n+\tconst char *encoding = NULL;\n \tstruct hash_list *merge_list = NULL;\n \tunsigned int merge_count;\n \tunsigned char prev_fanout, new_fanout;\n@@ -2607,6 +2608,8 @@ static void parse_new_commit(const char *arg)\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n+\tif (skip_prefix(command_buf.buf, \"encoding \", &encoding))\n+\t\tread_next_command();\n \tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n@@ -2670,9 +2673,13 @@ static void parse_new_commit(const char *arg)\n \t}\n \tstrbuf_addf(&new_data,\n \t\t\"author %s\\n\"\n-\t\t\"committer %s\\n\"\n-\t\t\"\\n\",\n+\t\t\"committer %s\\n\",\n \t\tauthor ? author : committer, committer);\n+\tif (encoding)\n+\t\tstrbuf_addf(&new_data,\n+\t\t\t\"encoding %s\\n\",\n+\t\t\tencoding);\n+\tstrbuf_addch(&new_data, '\\n');\n \tstrbuf_addbuf(&new_data, &msg);\n \tfree(author);\n \tfree(committer);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 3668263c40..141b7fa35e 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3299,4 +3299,24 @@ test_expect_success !MINGW 'W: get-mark & empty orphan commit with erroneous thi\n \tsed -e s/LFs/LLL/ W-input | tr L \"\\n\" | test_must_fail git fast-import\n '\n \n+###\n+### series X (other new features)\n+###\n+\n+test_expect_success 'X: handling encoding' '\n+\ttest_tick &&\n+\tcat >input <<-INPUT_END &&\n+\tcommit refs/heads/encoding\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tencoding iso-8859-7\n+\tdata <>input &&\n+\n+\tgit fast-import \n---\n builtin/fast-export.c | 7 +++++--\n t/t9350-fast-export.sh | 17 +++++++++++++++++\n t/t9350/broken-iso-8859-7-commit-message.txt | 1 +\n 3 files changed, 23 insertions(+), 2 deletions(-)\n create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 9e283482ef..7734a9f5a5 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -642,9 +642,12 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n \tif (show_original_ids)\n \t\tprintf(\"original-oid %s\\n\", oid_to_hex(&commit->object.oid));\n-\tprintf(\"%.*s\\n%.*s\\ndata %u\\n%s\",\n+\tprintf(\"%.*s\\n%.*s\\n\",\n \t (int)(author_end - author), author,\n-\t (int)(committer_end - committer), committer,\n+\t (int)(committer_end - committer), committer);\n+\tif (!reencoded && encoding)\n+\t\tprintf(\"encoding %s\\n\", encoding);\n+\tprintf(\"data %u\\n%s\",\n \t (unsigned)(reencoded\n \t\t\t ? strlen(reencoded) : message\n \t\t\t ? strlen(message) : 0),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex ef9b1aa20b..fa124f4842 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -120,6 +120,23 @@ test_expect_success 'iso-8859-7' '\n \t\t ! grep ^encoding actual)\n '\n \n+test_expect_success 'encoding preserved if reencoding fails' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t git cat-file commit i18n-invalid >actual &&\n+\t\t grep ^encoding actual &&\n+\t\t # Also verify that the commit has the expected size; i.e.\n+\t\t # that no bytes were re-encoded to a different encoding.\n+\t\t test 252 -eq \"$(git cat-file -s i18n-invalid)\")\n+'\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\ndiff --git a/t/t9350/broken-iso-8859-7-commit-message.txt b/t/t9350/broken-iso-8859-7-commit-message.txt\nnew file mode 100644\nindex 0000000000..d06ad75b44\n--- /dev/null\n+++ b/t/t9350/broken-iso-8859-7-commit-message.txt\n@@ -0,0 +1 @@\n+Pi: �; Invalid: �\n\\ No newline at end of file\n-- \n2.21.0.782.g2063122293\n\n"},{"id":"375321","messageId":"20190510205335.19968-6-newren@gmail.com","threadId":"51017","inReplyTo":"20190510205335.19968-1-newren@gmail.com","subject":"[PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-10T20:53:35Z","receivedAt":"2019-05-10T20:53:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Automatic re-encoding of commit messages (and dropping of the encoding\nheader) hurts attempts to do reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions, some subtree rewrites), and seems\ninconsistent with the general principle followed elsewhere in\nfast-export of requiring explicit user requests to modify the output\n(e.g. --signed-tags=strip, --tag-of-filtered-object=rewrite). Add a\n--reencode flag that the user can use to specify, and like other\nfast-export flags, default it to 'abort'.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 35 ++++++++++++++++++++++++++++++++---\n t/t9350-fast-export.sh | 37 ++++++++++++++++++++++++++++++++++---\n 2 files changed, 66 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 66331fa401..43cc52331c 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -33,6 +33,7 @@ static const char *fast_export_usage[] = {\n static int progress;\n static enum { SIGNED_TAG_ABORT, VERBATIM, WARN, WARN_STRIP, STRIP } signed_tag_mode = SIGNED_TAG_ABORT;\n static enum { TAG_FILTERING_ABORT, DROP, REWRITE } tag_of_filtered_mode = TAG_FILTERING_ABORT;\n+static enum { REENCODE_ABORT, REENCODE_PLEASE, REENCODE_NEVER } reencode_mode = REENCODE_ABORT;\n static int fake_missing_tagger;\n static int use_done_feature;\n static int no_data;\n@@ -77,6 +78,20 @@ static int parse_opt_tag_of_filtered_mode(const struct option *opt,\n \treturn 0;\n }\n \n+static int parse_opt_reencode_mode(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tif (unset || !strcmp(arg, \"abort\"))\n+\t\treencode_mode = REENCODE_ABORT;\n+\telse if (!strcmp(arg, \"yes\"))\n+\t\treencode_mode = REENCODE_PLEASE;\n+\telse if (!strcmp(arg, \"no\"))\n+\t\treencode_mode = REENCODE_NEVER;\n+\telse\n+\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n+\treturn 0;\n+}\n+\n static struct decoration idnums;\n static uint32_t last_idnum;\n \n@@ -633,10 +648,21 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \t}\n \n \tmark_next_object(&commit->object);\n-\tif (anonymize)\n+\tif (anonymize) {\n \t\treencoded = anonymize_commit_message(message);\n-\telse if (!is_encoding_utf8(encoding))\n-\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t} else if (encoding) {\n+\t\tswitch(reencode_mode) {\n+\t\tcase REENCODE_PLEASE:\n+\t\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t\t\tbreak;\n+\t\tcase REENCODE_NEVER:\n+\t\t\tbreak;\n+\t\tcase REENCODE_ABORT:\n+\t\t\tdie(\"Encountered commit-specific encoding %s in commit \"\n+\t\t\t \"%s; use --reencode= to handle it\",\n+\t\t\t encoding, oid_to_hex(&commit->object.oid));\n+\t\t}\n+\t}\n \tif (!commit->parents)\n \t\tprintf(\"reset %s\\n\", refname);\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n@@ -1091,6 +1117,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK(0, \"tag-of-filtered-object\", &tag_of_filtered_mode, N_(\"mode\"),\n \t\t\t N_(\"select handling of tags that tag filtered objects\"),\n \t\t\t parse_opt_tag_of_filtered_mode),\n+\t\tOPT_CALLBACK(0, \"reencode\", &reencode_mode, N_(\"mode\"),\n+\t\t\t N_(\"select handling of commit messages in an alternate encoding\"),\n+\t\t\t parse_opt_reencode_mode),\n \t\tOPT_STRING(0, \"export-marks\", &export_filename, N_(\"file\"),\n \t\t\t N_(\"Dump marks to this file\")),\n \t\tOPT_STRING(0, \"import-marks\", &import_filename, N_(\"file\"),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex fa124f4842..3cf4c7fc0c 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,14 +94,14 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-7' '\n+test_expect_success 'reencoding iso-8859-7' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n \tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n@@ -120,13 +120,44 @@ test_expect_success 'iso-8859-7' '\n \t\t ! grep ^encoding actual)\n '\n \n+test_expect_success 'aborting on iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\ttest_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi\n+'\n+\n+test_expect_success 'preserving iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export --reencode=no wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-no-recoding/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t # The commit object, if not re-encoded, is 240 bytes.\n+\t\t # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n+\t\t # bytes. Re-encoding the Pi character from \\xF0 in\n+\t\t # iso-8859-7 to \\xCF\\x80 in utf-8 would add a byte. I would\n+\t\t # grep for the # specific bytes, but Windows lamely does not\n+\t\t # allow that, so just search for the expected size.\n+\t\t test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n+\t\t # Also make sure the commit has the \"encoding\" header\n+\t\t git cat-file commit i18n-no-recoding >actual &&\n+\t\t grep ^encoding actual)\n+'\n+\n test_expect_success 'encoding preserved if reencoding fails' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \techo rosten >file &&\n \tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n-- \n2.21.0.782.g2063122293\n\n"},{"id":"375322","messageId":"20190510205335.19968-5-newren@gmail.com","threadId":"51017","inReplyTo":"20190510205335.19968-1-newren@gmail.com","subject":"[PATCH v3 4/5] fast-export: differentiate between explicitly utf-8 and implicitly utf-8","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-10T20:53:34Z","receivedAt":"2019-05-10T20:53:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"The find_encoding() function returned the encoding used by a commit\nmessage, returning a default of git_commit_encoding (usually utf-8).\nAlthough the current code does not differentiate between a commit which\nexplicitly requested utf-8 and one where we just assume utf-8 because no\nencoding is set, it will become important when we try to preserve the\nencoding header. Since is_encoding_utf8() returns true when passed\nNULL, we can just return NULL from find_encoding() instead of returning\ngit_commit_encoding.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 7734a9f5a5..66331fa401 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -453,7 +453,7 @@ static const char *find_encoding(const char *begin, const char *end)\n \tbol = memmem(begin, end ? end - begin : strlen(begin),\n \t\t needle, strlen(needle));\n \tif (!bol)\n-\t\treturn git_commit_encoding;\n+\t\treturn NULL;\n \tbol += strlen(needle);\n \teol = strchrnul(bol, '\\n');\n \t*eol = '\\0';\n-- \n2.21.0.782.g2063122293\n\n"},{"id":"375341","messageId":"20190511210704.w2mxw3jv2ra2dr7w@tb-raspi4","threadId":"51017","inReplyTo":"20190510205335.19968-6-newren@gmail.com","subject":"Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-05-11T21:07:04Z","receivedAt":"2019-05-11T21:07:17Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, May 10, 2019 at 01:53:35PM -0700, Elijah Newren wrote:\n> Automatic re-encoding of commit messages (and dropping of the encoding\n> header) hurts attempts to do reversible history rewrites (e.g. sha1sum\n> <-> sha256sum transitions, some subtree rewrites), and seems\n> inconsistent with the general principle followed elsewhere in\n> fast-export of requiring explicit user requests to modify the output\n> (e.g. --signed-tags=strip, --tag-of-filtered-object=rewrite). Add a\n> --reencode flag that the user can use to specify, and like other\n> fast-export flags, default it to 'abort'.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> builtin/fast-export.c | 35 ++++++++++++++++++++++++++++++++---\n> t/t9350-fast-export.sh | 37 ++++++++++++++++++++++++++++++++++---\n> 2 files changed, 66 insertions(+), 6 deletions(-)\n>\n> diff --git a/builtin/fast-export.c b/builtin/fast-export.c\n> index 66331fa401..43cc52331c 100644\n> --- a/builtin/fast-export.c\n> +++ b/builtin/fast-export.c\n> @@ -33,6 +33,7 @@ static const char *fast_export_usage[] = {\n> static int progress;\n> static enum { SIGNED_TAG_ABORT, VERBATIM, WARN, WARN_STRIP, STRIP } signed_tag_mode = SIGNED_TAG_ABORT;\n> static enum { TAG_FILTERING_ABORT, DROP, REWRITE } tag_of_filtered_mode = TAG_FILTERING_ABORT;\n> +static enum { REENCODE_ABORT, REENCODE_PLEASE, REENCODE_NEVER } reencode_mode = REENCODE_ABORT;\n> static int fake_missing_tagger;\n> static int use_done_feature;\n> static int no_data;\n> @@ -77,6 +78,20 @@ static int parse_opt_tag_of_filtered_mode(const struct option *opt,\n> \treturn 0;\n> }\n>\n> +static int parse_opt_reencode_mode(const struct option *opt,\n> +\t\t\t\t const char *arg, int unset)\n> +{\n\nThis one is good:\n> +\tif (unset || !strcmp(arg, \"abort\"))\n> +\t\treencode_mode = REENCODE_ABORT;\n\nBut here: does it make sense to use REENCODE_YES/NO to be more consistant ?\n> +\telse if (!strcmp(arg, \"yes\"))\n> +\t\treencode_mode = REENCODE_PLEASE;\n> +\telse if (!strcmp(arg, \"no\"))\n> +\t\treencode_mode = REENCODE_NEVER;\n> +\telse\n> +\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n> +\treturn 0;\n> +}\n> +\n> static struct decoration idnums;\n> static uint32_t last_idnum;\n>\n> @@ -633,10 +648,21 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n> \t}\n>\n> \tmark_next_object(&commit->object);\n> -\tif (anonymize)\n> +\tif (anonymize) {\n> \t\treencoded = anonymize_commit_message(message);\n> -\telse if (!is_encoding_utf8(encoding))\n> -\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n> +\t} else if (encoding) {\n> +\t\tswitch(reencode_mode) {\n> +\t\tcase REENCODE_PLEASE:\n> +\t\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n> +\t\t\tbreak;\n> +\t\tcase REENCODE_NEVER:\n> +\t\t\tbreak;\n> +\t\tcase REENCODE_ABORT:\n> +\t\t\tdie(\"Encountered commit-specific encoding %s in commit \"\n> +\t\t\t \"%s; use --reencode= to handle it\",\nShould we be more helpfull and say !use --reencode=[yes|no] to handle it ?\n\n> +\t\t\t encoding, oid_to_hex(&commit->object.oid));\n> +\t\t}\n> +\t}\n> \tif (!commit->parents)\n> \t\tprintf(\"reset %s\\n\", refname);\n> \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n> @@ -1091,6 +1117,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n> \t\tOPT_CALLBACK(0, \"tag-of-filtered-object\", &tag_of_filtered_mode, N_(\"mode\"),\n> \t\t\t N_(\"select handling of tags that tag filtered objects\"),\n> \t\t\t parse_opt_tag_of_filtered_mode),\n> +\t\tOPT_CALLBACK(0, \"reencode\", &reencode_mode, N_(\"mode\"),\n> +\t\t\t N_(\"select handling of commit messages in an alternate encoding\"),\n> +\t\t\t parse_opt_reencode_mode),\n> \t\tOPT_STRING(0, \"export-marks\", &export_filename, N_(\"file\"),\n> \t\t\t N_(\"Dump marks to this file\")),\n> \t\tOPT_STRING(0, \"import-marks\", &import_filename, N_(\"file\"),\n> diff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\n> index fa124f4842..3cf4c7fc0c 100755\n> --- a/t/t9350-fast-export.sh\n> +++ b/t/t9350-fast-export.sh\n> @@ -94,14 +94,14 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n> \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n> '\n>\n> -test_expect_success 'iso-8859-7' '\n> +test_expect_success 'reencoding iso-8859-7' '\n>\n> \ttest_when_finished \"git reset --hard HEAD~1\" &&\n> \ttest_config i18n.commitencoding iso-8859-7 &&\n> \ttest_tick &&\n> \techo rosten >file &&\n> \tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n> -\tgit fast-export wer^..wer >iso-8859-7.fi &&\n> +\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n> \tsed \"s/wer/i18n/\" iso-8859-7.fi |\n> \t\t(cd new &&\n> \t\t git fast-import &&\n> @@ -120,13 +120,44 @@ test_expect_success 'iso-8859-7' '\n> \t\t ! grep ^encoding actual)\n> '\n>\n> +test_expect_success 'aborting on iso-8859-7' '\n> +\n> +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n> +\ttest_config i18n.commitencoding iso-8859-7 &&\n> +\techo rosten >file &&\n> +\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n> +\ttest_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi\n> +'\n> +\n> +test_expect_success 'preserving iso-8859-7' '\n> +\n> +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n> +\ttest_config i18n.commitencoding iso-8859-7 &&\n> +\techo rosten >file &&\n> +\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n> +\tgit fast-export --reencode=no wer^..wer >iso-8859-7.fi &&\n> +\tsed \"s/wer/i18n-no-recoding/\" iso-8859-7.fi |\n> +\t\t(cd new &&\n> +\t\t git fast-import &&\n> +\t\t # The commit object, if not re-encoded, is 240 bytes.\n> +\t\t # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n> +\t\t # bytes. Re-encoding the Pi character from \\xF0 in\n> +\t\t # iso-8859-7 to \\xCF\\x80 in utf-8 would add a byte. I would\n> +\t\t # grep for the # specific bytes, but Windows lamely does not\nThis is somewhat unclear to me. What does Windows not allow ?\n\n> +\t\t # allow that, so just search for the expected size.\n> +\t\t test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n> +\t\t # Also make sure the commit has the \"encoding\" header\n> +\t\t git cat-file commit i18n-no-recoding >actual &&\n> +\t\t grep ^encoding actual)\n> +'\n> +\n> test_expect_success 'encoding preserved if reencoding fails' '\n>\n> \ttest_when_finished \"git reset --hard HEAD~1\" &&\n> \ttest_config i18n.commitencoding iso-8859-7 &&\n> \techo rosten >file &&\n> \tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n> -\tgit fast-export wer^..wer >iso-8859-7.fi &&\n> +\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n> \tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n> \t\t(cd new &&\n> \t\t git fast-import &&\n> --\n> 2.21.0.782.g2063122293\n>\n"},{"id":"375343","messageId":"CABPp-BEzaSW_eY1yTpLr8tXHei0WV54PFGXmjCLAJNN03Zi3eQ@mail.gmail.com","threadId":"51017","inReplyTo":"20190511210704.w2mxw3jv2ra2dr7w@tb-raspi4","subject":"Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-11T21:42:17Z","receivedAt":"2019-05-11T21:42:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, May 11, 2019 at 2:07 PM Torsten Bögershausen wrote:\n> On Fri, May 10, 2019 at 01:53:35PM -0700, Elijah Newren wrote:\n\n> This one is good:\n> > + if (unset || !strcmp(arg, \"abort\"))\n> > + reencode_mode = REENCODE_ABORT;\n>\n> But here: does it make sense to use REENCODE_YES/NO to be more consistant ?\n> > + else if (!strcmp(arg, \"yes\"))\n> > + reencode_mode = REENCODE_PLEASE;\n> > + else if (!strcmp(arg, \"no\"))\n> > + reencode_mode = REENCODE_NEVER;\n\nDidn't realize there was any such convention, and even have difficulty\nfinding it with grep (CONTAINS_{YES,NO} appears to be the only example\nI can find), but the alternate wording seems fine; I'm happy to adopt\nit.\n\n> > + case REENCODE_ABORT:\n> > + die(\"Encountered commit-specific encoding %s in commit \"\n> > + \"%s; use --reencode= to handle it\",\n> Should we be more helpfull and say !use --reencode=[yes|no] to handle it ?\n\nSounds like a good idea; I'll adjust it.\n\n\n> > + sed \"s/wer/i18n-no-recoding/\" iso-8859-7.fi |\n> > + (cd new &&\n> > + git fast-import &&\n> > + # The commit object, if not re-encoded, is 240 bytes.\n> > + # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n> > + # bytes. Re-encoding the Pi character from \\xF0 in\n> > + # iso-8859-7 to \\xCF\\x80 in utf-8 would add a byte. I would\n> > + # grep for the specific bytes, but Windows lamely does not\n> This is somewhat unclear to me. What does Windows not allow ?\n> > + # allow that, so just search for the expected size.\n> > + test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n> > + # Also make sure the commit has the \"encoding\" header\n> > + git cat-file commit i18n-no-recoding >actual &&\n> > + grep ^encoding actual)\n> > +'\n\nWindows does not allow specifying the bytes I want to grep for on the\ncommand line; it'll munge the command line, resulting in it searching\nfor something other than what I wanted to be searched for, and return\nthe wrong answer based on searching for the wrong thing. See\nhttps://public-inbox.org/git/f8eb246f-a936-e9df-4bb4-068b86a62baf@kdbg.org/\nand https://public-inbox.org/git/nycvar.QRO.7.76.6.1905101551110.44@tvgsbejvaqbjf.bet/.\n\nMy comment was already pretty long because it looks like a crazy way\nto run the test and thus it feels like I need to explain it. And the\ncraziness is based on how Windows behaves; it seems insane to me that\nWindows decides to munge user data (in the form of the command line\nprovided), so much so that it makes me wonder if I really understood\nHannes' and Dscho's explanations of what it is doing. (How could\nanyone have thought munging user data was a good idea?) Anyway, long\nstory short, I'm not sure how to explain it correctly and succinctly.\nAny suggestions?\n\n\nElijah\n"},{"id":"375367","messageId":"xmqq1s12dcci.fsf@gitster-ct.c.googlers.com","threadId":"51017","inReplyTo":"CABPp-BEzaSW_eY1yTpLr8tXHei0WV54PFGXmjCLAJNN03Zi3eQ@mail.gmail.com","subject":"Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-05-13T07:48:13Z","receivedAt":"2019-05-13T07:48:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Sat, May 11, 2019 at 2:07 PM Torsten Bögershausen wrote:\n>> On Fri, May 10, 2019 at 01:53:35PM -0700, Elijah Newren wrote:\n>\n>> This one is good:\n>> > + if (unset || !strcmp(arg, \"abort\"))\n>> > + reencode_mode = REENCODE_ABORT;\n>>\n>> But here: does it make sense to use REENCODE_YES/NO to be more consistant ?\n>> > + else if (!strcmp(arg, \"yes\"))\n>> > + reencode_mode = REENCODE_PLEASE;\n>> > + else if (!strcmp(arg, \"no\"))\n>> > + reencode_mode = REENCODE_NEVER;\n>\n> Didn't realize there was any such convention, and even have difficulty\n> finding it with grep (CONTAINS_{YES,NO} appears to be the only example\n> I can find), but the alternate wording seems fine; I'm happy to adopt\n> it.\n\nI am OK with Yes/No. \n\nDon't we want to treat this as \"bool or literal 'abort'\", though?\nOther options that are \"bool or something else\" tend to accept\n\"true\" as a synonym for \"yes\", and I am wondering if we want to\nfollow suit here, too.\n\nThanks.\n"},{"id":"375373","messageId":"nycvar.QRO.7.76.6.1905131214260.44@tvgsbejvaqbjf.bet","threadId":"51017","inReplyTo":"20190510205335.19968-1-newren@gmail.com","subject":"Re: [PATCH v3 0/5] Fix and extend encoding handling in fast export/import","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-05-13T10:14:59Z","receivedAt":"2019-05-13T10:15:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Fri, 10 May 2019, Elijah Newren wrote:\n\n> * Modified the testcases to pass on Windows[1], as verified via\n> gitgitgadget pull request[2]. Required adding a couple new files\n> (which store the desired bytes) and checking the size of the output\n> instead of checking for particular bytes (but the lengths of the\n> expected byte sequences differ so this works fine...).\n>\n> [1] Failures of previous patchset on Windows noticed and reported by Dscho;\n> explanation from Hannes is that Windows munges users' command lines to\n> force them to be characters instead of bytes.\n> [2] https://github.com/gitgitgadget/git/pull/187\n\nExcellent! Thank you!\n\nCiao,\nDscho\n"},{"id":"375374","messageId":"nycvar.QRO.7.76.6.1905131219490.44@tvgsbejvaqbjf.bet","threadId":"51017","inReplyTo":"CABPp-BEzaSW_eY1yTpLr8tXHei0WV54PFGXmjCLAJNN03Zi3eQ@mail.gmail.com","subject":"Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-05-13T10:23:29Z","receivedAt":"2019-05-13T10:23:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Sat, 11 May 2019, Elijah Newren wrote:\n\n> [...] the craziness is based on how Windows behaves; it seems insane to\n> me that Windows decides to munge user data (in the form of the command\n> line provided), so much so that it makes me wonder if I really\n> understood Hannes' and Dscho's explanations of what it is doing.\n\nIt is not the user data that is munged by *Windows*, but by *Git for\nWindows*. The user data on Windows is encoded in UTF-16 (or some slight\nvariant thereof). Git *cannot* handle UTF-16. Git's test suite *cannot*\nhandle UTF-16. So we convert. That's all there is to it.\n\nCiao,\nDscho\n\nP.S.: Of course it is not *all* there is to it. There is also a current\ncode page which depends on the current user's current locale. We can\ndefinitely not rely on that, as Git has no idea about this and would quite\npositively produce incorrect output because of it. So we really just use\nthe `*W()` functions of the Win32 API (i.e. the ones accepting wide\nUnicode characters and strings, i.e. UTF-16). I don't think we can do\nbetter than that.\n"},{"id":"375384","messageId":"20190513125635.tntxhjbar2eold2i@tb-raspi4","threadId":"51017","inReplyTo":"nycvar.QRO.7.76.6.1905131219490.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-05-13T12:56:35Z","receivedAt":"2019-05-13T12:56:46Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, May 13, 2019 at 12:23:29PM +0200, Johannes Schindelin wrote:\n> Hi Elijah,\n>\n> On Sat, 11 May 2019, Elijah Newren wrote:\n>\n> > [...] the craziness is based on how Windows behaves; it seems insane to\n> > me that Windows decides to munge user data (in the form of the command\n> > line provided), so much so that it makes me wonder if I really\n> > understood Hannes' and Dscho's explanations of what it is doing.\n>\n> It is not the user data that is munged by *Windows*, but by *Git for\n> Windows*. The user data on Windows is encoded in UTF-16 (or some slight\n> variant thereof). Git *cannot* handle UTF-16. Git's test suite *cannot*\n> handle UTF-16. So we convert. That's all there is to it.\n>\n> Ciao,\n> Dscho\n>\n> P.S.: Of course it is not *all* there is to it. There is also a current\n> code page which depends on the current user's current locale. We can\n> definitely not rely on that, as Git has no idea about this and would quite\n> positively produce incorrect output because of it. So we really just use\n> the `*W()` functions of the Win32 API (i.e. the ones accepting wide\n> Unicode characters and strings, i.e. UTF-16). I don't think we can do\n> better than that.\n\nWe can actuall feed valid UTF-8 into a test case.\n(Remember that shell scripts need this octal numbering, see\nt/t0050)\n\n\nSee the \"ä\" code point:\n$ auml=$(printf '\\303\\244')\n$ printf $auml\nä\n\n\nNow we can feed those 2 bytes (wich are valid UTF) into\nGit and say \"convert them from ISO-8859-1 into UTF-8,\nresulting in 4 bytes.\nIs my explanation clear enough ?\nIf not, plese tell me.\n\n\n"},{"id":"375385","messageId":"CABPp-BGmbvpjWep8F0XGczKyKWVHwQNaJQ4zS=Y1zGrHT7O7DQ@mail.gmail.com","threadId":"51017","inReplyTo":"xmqq1s12dcci.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T13:24:40Z","receivedAt":"2019-05-13T13:24:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, May 13, 2019 at 12:48 AM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > On Sat, May 11, 2019 at 2:07 PM Torsten Bögershausen wrote:\n> >> On Fri, May 10, 2019 at 01:53:35PM -0700, Elijah Newren wrote:\n> >\n> >> This one is good:\n> >> > + if (unset || !strcmp(arg, \"abort\"))\n> >> > + reencode_mode = REENCODE_ABORT;\n> >>\n> >> But here: does it make sense to use REENCODE_YES/NO to be more consistant ?\n> >> > + else if (!strcmp(arg, \"yes\"))\n> >> > + reencode_mode = REENCODE_PLEASE;\n> >> > + else if (!strcmp(arg, \"no\"))\n> >> > + reencode_mode = REENCODE_NEVER;\n> >\n> > Didn't realize there was any such convention, and even have difficulty\n> > finding it with grep (CONTAINS_{YES,NO} appears to be the only example\n> > I can find), but the alternate wording seems fine; I'm happy to adopt\n> > it.\n>\n> I am OK with Yes/No.\n>\n> Don't we want to treat this as \"bool or literal 'abort'\", though?\n> Other options that are \"bool or something else\" tend to accept\n> \"true\" as a synonym for \"yes\", and I am wondering if we want to\n> follow suit here, too.\n\nMakes sense; will do.\n"},{"id":"375387","messageId":"CABPp-BGbq3EJfpw-NykHM5TJdkpmqmttHYWonBP5kydV9PwTBg@mail.gmail.com","threadId":"51017","inReplyTo":"20190513125635.tntxhjbar2eold2i@tb-raspi4","subject":"Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T13:29:28Z","receivedAt":"2019-05-13T13:29:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, May 13, 2019 at 5:56 AM Torsten Bögershausen wrote:\n>\n> On Mon, May 13, 2019 at 12:23:29PM +0200, Johannes Schindelin wrote:\n> > Hi Elijah,\n> >\n> > On Sat, 11 May 2019, Elijah Newren wrote:\n> >\n> > > [...] the craziness is based on how Windows behaves; it seems insane to\n> > > me that Windows decides to munge user data (in the form of the command\n> > > line provided), so much so that it makes me wonder if I really\n> > > understood Hannes' and Dscho's explanations of what it is doing.\n> >\n> > It is not the user data that is munged by *Windows*, but by *Git for\n> > Windows*. The user data on Windows is encoded in UTF-16 (or some slight\n> > variant thereof). Git *cannot* handle UTF-16. Git's test suite *cannot*\n> > handle UTF-16. So we convert. That's all there is to it.\n> >\n> > Ciao,\n> > Dscho\n> >\n> > P.S.: Of course it is not *all* there is to it. There is also a current\n> > code page which depends on the current user's current locale. We can\n> > definitely not rely on that, as Git has no idea about this and would quite\n> > positively produce incorrect output because of it. So we really just use\n> > the `*W()` functions of the Win32 API (i.e. the ones accepting wide\n> > Unicode characters and strings, i.e. UTF-16). I don't think we can do\n> > better than that.\n>\n> We can actuall feed valid UTF-8 into a test case.\n> (Remember that shell scripts need this octal numbering, see\n> t/t0050)\n\nSure, but that's not useful here. I need to feed both valid and\ninvalid ISO-8859-7 (or anything *other* than UTF-8) into a test case,\nin order to verify how git handles reencoding from something other\nthan utf-8. I did something like what you proposed originally, but\nsince it wasn't utf-8 it caused test failures on Windows.\n\nElijah\n"},{"id":"375394","messageId":"CABPp-BHfqgWVvf67JkPvE=hgzhZ1Jzy=D+vTc=v2j9+hgJzs9A@mail.gmail.com","threadId":"51017","inReplyTo":"nycvar.QRO.7.76.6.1905131219490.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T16:41:06Z","receivedAt":"2019-05-13T16:41:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, May 13, 2019 at 3:23 AM Johannes Schindelin\n wrote:\n>\n> Hi Elijah,\n>\n> On Sat, 11 May 2019, Elijah Newren wrote:\n>\n> > [...] the craziness is based on how Windows behaves; it seems insane to\n> > me that Windows decides to munge user data (in the form of the command\n> > line provided), so much so that it makes me wonder if I really\n> > understood Hannes' and Dscho's explanations of what it is doing.\n>\n> It is not the user data that is munged by *Windows*, but by *Git for\n> Windows*. The user data on Windows is encoded in UTF-16 (or some slight\n> variant thereof). Git *cannot* handle UTF-16. Git's test suite *cannot*\n> handle UTF-16. So we convert. That's all there is to it.\n\nOoh, it's Git for Windows doing the conversion? That means I can test\nfor the expected bytes with printf and grep, I only need to feed\nspecial bytes to git via file instead of command line. That's better.\n:-)\n\n> Ciao,\n> Dscho\n>\n> P.S.: Of course it is not *all* there is to it. There is also a current\n> code page which depends on the current user's current locale. We can\n> definitely not rely on that, as Git has no idea about this and would quite\n> positively produce incorrect output because of it. So we really just use\n> the `*W()` functions of the Win32 API (i.e. the ones accepting wide\n> Unicode characters and strings, i.e. UTF-16). I don't think we can do\n> better than that.\n\nGoing off on a tangent for a minute...okay, so you need to do some\nkind of conversion for Windows, but why is it automatically UTF-8 ->\nUTF-16? In particular, if i18n.commitencoding configuration option is\nset (to something other than UTF-8), then couldn't you instead convert\nthe commit message specified on the command line from $(git config\ni18n.commitencoding) to UTF-16? Or, perhaps convert from $(git config\ni18n.commitencoding) to UTF-8 before the automatic UTF-8 -> UTF-16\nconversion? It doesn't matter for this series anymore since I've\nworked around it (by passing the bytes via an external file as\nsuggested by Hannes), but it seems like Git For Windows might be able\nto still do better here. Or maybe I'm still not understanding the\nfull picture. Anyway, thanks for all the explanations and the help\ngetting this fixed up.\n\nElijah\n"},{"id":"375395","messageId":"20190513164722.31534-1-newren@gmail.com","threadId":"51017","inReplyTo":"20190510205335.19968-1-newren@gmail.com","subject":"[PATCH v4 0/5] Fix and extend encoding handling in fast export/import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T16:47:17Z","receivedAt":"2019-05-13T16:47:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"While stress testing `git filter-repo`, I noticed an issue with\nencoding; further digging led to the fixes and features in this series.\nSee the individual commit messages for details.\n\nChanges since v3 (full range-diff below):\n * YES/NO changes suggested by Torsten\n * more boolean synonyms as suggested by Junio\n * check for the exact expected special bytes, in addition to the size\n (Dscho pointed out that it was GitForWindows that munged bytes, not\n Windows, so while I need to be careful in what I pass to git, printf\n and grep can work directly with the special bytes)\n * also checked on gitgitgadget that it passes on the major platforms\n\n[1] https://github.com/gitgitgadget/git/pull/191\n\nElijah Newren (5):\n t9350: fix encoding test to actually test reencoding\n fast-import: support 'encoding' commit header\n fast-export: avoid stripping encoding header if we cannot reencode\n fast-export: differentiate between explicitly utf-8 and implicitly\n utf-8\n fast-export: do automatic reencoding of commit messages only if\n requested\n\n Documentation/git-fast-import.txt | 7 ++\n builtin/fast-export.c | 44 +++++++++--\n fast-import.c | 11 ++-\n t/t9300-fast-import.sh | 20 +++++\n t/t9350-fast-export.sh | 78 +++++++++++++++++---\n t/t9350/broken-iso-8859-7-commit-message.txt | 1 +\n t/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n 7 files changed, 145 insertions(+), 17 deletions(-)\n create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\n create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n\nRange-diff:\n1: 2d7bb64acf ! 1: 37a68a0ffd t9350: fix encoding test to actually test reencoding\n @@ -39,18 +39,16 @@\n \t\t git fast-import &&\n +\t\t # The commit object, if not re-encoded, would be 240 bytes.\n +\t\t # Removing the \"encoding iso-8859-7\\n\" header drops 20 bytes.\n -+\t\t # Re-encoding the Pi character from \\xF0 in iso-8859-7 to\n -+\t\t # \\xCF\\x80 in utf-8 adds a byte. Grepping for specific bytes\n -+\t\t # would be nice, but Windows apparently munges user data\n -+\t\t # in the form of bytes on the command line to force them to\n -+\t\t # be characters instead, so we are limited for portability\n -+\t\t # reasons in subsequent similar tests in this file to check\n -+\t\t # for size rather than what bytes are present.\n ++\t\t # Re-encoding the Pi character from \\xF0 (\\360) in iso-8859-7\n ++\t\t # to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte. Check for\n ++\t\t # the expected size.\n +\t\t test 221 -eq \"$(git cat-file -s i18n)\" &&\n -+\t\t # Also make sure the commit does not have the \"encoding\" header\n ++\t\t # ...and for the expected translation of bytes.\n \t\t git cat-file commit i18n >actual &&\n -\t\t grep \"Áéí óú\" actual)\n -\n ++\t\t grep $(printf \"\\317\\200\") actual &&\n ++\t\t # Also make sure the commit does not have the \"encoding\" header\n +\t\t ! grep ^encoding actual)\n '\n +\n2: 9fa5695017 = 2: 3d84f4613d fast-import: support 'encoding' commit header\n3: dfc76573e9 ! 3: baa8394a3a fast-export: avoid stripping encoding header if we cannot reencode\n @@ -49,10 +49,14 @@\n +\t\t(cd new &&\n +\t\t git fast-import &&\n +\t\t git cat-file commit i18n-invalid >actual &&\n ++\t\t # Make sure the commit still has the encoding header\n +\t\t grep ^encoding actual &&\n -+\t\t # Also verify that the commit has the expected size; i.e.\n ++\t\t # Verify that the commit has the expected size; i.e.\n +\t\t # that no bytes were re-encoded to a different encoding.\n -+\t\t test 252 -eq \"$(git cat-file -s i18n-invalid)\")\n ++\t\t test 252 -eq \"$(git cat-file -s i18n-invalid)\" &&\n ++\t\t # ...and check for the original special bytes\n ++\t\t grep $(printf \"\\360\") actual &&\n ++\t\t grep $(printf \"\\377\") actual)\n +'\n +\n test_expect_success 'import/export-marks' '\n4: 83b3656b76 = 4: 49960164c6 fast-export: differentiate between explicitly utf-8 and implicitly utf-8\n5: 2063122293 ! 5: 571613a09e fast-export: do automatic reencoding of commit messages only if requested\n @@ -20,7 +20,7 @@\n static int progress;\n static enum { SIGNED_TAG_ABORT, VERBATIM, WARN, WARN_STRIP, STRIP } signed_tag_mode = SIGNED_TAG_ABORT;\n static enum { TAG_FILTERING_ABORT, DROP, REWRITE } tag_of_filtered_mode = TAG_FILTERING_ABORT;\n -+static enum { REENCODE_ABORT, REENCODE_PLEASE, REENCODE_NEVER } reencode_mode = REENCODE_ABORT;\n ++static enum { REENCODE_ABORT, REENCODE_YES, REENCODE_NO } reencode_mode = REENCODE_ABORT;\n static int fake_missing_tagger;\n static int use_done_feature;\n static int no_data;\n @@ -33,10 +33,10 @@\n +{\n +\tif (unset || !strcmp(arg, \"abort\"))\n +\t\treencode_mode = REENCODE_ABORT;\n -+\telse if (!strcmp(arg, \"yes\"))\n -+\t\treencode_mode = REENCODE_PLEASE;\n -+\telse if (!strcmp(arg, \"no\"))\n -+\t\treencode_mode = REENCODE_NEVER;\n ++\telse if (!strcmp(arg, \"yes\") || !strcmp(arg, \"true\") || !strcmp(arg, \"on\"))\n ++\t\treencode_mode = REENCODE_YES;\n ++\telse if (!strcmp(arg, \"no\") || !strcmp(arg, \"false\") || !strcmp(arg, \"off\"))\n ++\t\treencode_mode = REENCODE_NO;\n +\telse\n +\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n +\treturn 0;\n @@ -56,14 +56,14 @@\n -\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n +\t} else if (encoding) {\n +\t\tswitch(reencode_mode) {\n -+\t\tcase REENCODE_PLEASE:\n ++\t\tcase REENCODE_YES:\n +\t\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n +\t\t\tbreak;\n -+\t\tcase REENCODE_NEVER:\n ++\t\tcase REENCODE_NO:\n +\t\t\tbreak;\n +\t\tcase REENCODE_ABORT:\n +\t\t\tdie(\"Encountered commit-specific encoding %s in commit \"\n -+\t\t\t \"%s; use --reencode= to handle it\",\n ++\t\t\t \"%s; use --reencode=[yes|no] to handle it\",\n +\t\t\t encoding, oid_to_hex(&commit->object.oid));\n +\t\t}\n +\t}\n @@ -126,13 +126,14 @@\n +\t\t git fast-import &&\n +\t\t # The commit object, if not re-encoded, is 240 bytes.\n +\t\t # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n -+\t\t # bytes. Re-encoding the Pi character from \\xF0 in\n -+\t\t # iso-8859-7 to \\xCF\\x80 in utf-8 would add a byte. I would\n -+\t\t # grep for the # specific bytes, but Windows lamely does not\n -+\t\t # allow that, so just search for the expected size.\n ++\t\t # bytes. Re-encoding the Pi character from \\xF0 (\\360) in\n ++\t\t # iso-8859-7 to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte.\n ++\t\t # Check for the expected size...\n +\t\t test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n -+\t\t # Also make sure the commit has the \"encoding\" header\n ++\t\t # ...as well as the expected byte.\n +\t\t git cat-file commit i18n-no-recoding >actual &&\n ++\t\t grep $(printf \"\\360\") actual &&\n ++\t\t # Also make sure the commit has the \"encoding\" header\n +\t\t grep ^encoding actual)\n +'\n +\n\n-- \n2.21.0.782.g571613a09e\n"},{"id":"375396","messageId":"20190513164722.31534-2-newren@gmail.com","threadId":"51017","inReplyTo":"20190513164722.31534-1-newren@gmail.com","subject":"[PATCH v4 1/5] t9350: fix encoding test to actually test reencoding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T16:47:18Z","receivedAt":"2019-05-13T16:47:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This test used an author with non-ascii characters in the name, but\nno special commit message. It then grep'ed for those non-ascii\ncharacters, but those are guaranteed to exist regardless of the\nreencoding process since the reencoding only affects the commit message,\nnot the author or committer names. As such, the test would work even if\nthe re-encoding process simply stripped the commit message entirely.\nModify the test to actually check that the reencoding in utf-8 worked.\n\nSigned-off-by: Elijah Newren \n---\n t/t9350-fast-export.sh | 27 ++++++++++++--------\n t/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n 2 files changed, 18 insertions(+), 10 deletions(-)\n create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 5690fe2810..c721026260 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,22 +94,30 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-1' '\n+test_expect_success 'iso-8859-7' '\n \n-\tgit config i18n.commitencoding ISO8859-1 &&\n-\t# use author and committer name in ISO-8859-1 to match it.\n-\t. \"$TEST_DIRECTORY\"/t3901/8859-1.txt &&\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n-\tgit commit -s -m den file &&\n-\tgit fast-export wer^..wer >iso8859-1.fi &&\n-\tsed \"s/wer/i18n/\" iso8859-1.fi |\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n+\t\t # The commit object, if not re-encoded, would be 240 bytes.\n+\t\t # Removing the \"encoding iso-8859-7\\n\" header drops 20 bytes.\n+\t\t # Re-encoding the Pi character from \\xF0 (\\360) in iso-8859-7\n+\t\t # to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte. Check for\n+\t\t # the expected size.\n+\t\t test 221 -eq \"$(git cat-file -s i18n)\" &&\n+\t\t # ...and for the expected translation of bytes.\n \t\t git cat-file commit i18n >actual &&\n-\t\t grep \"Áéí óú\" actual)\n-\n+\t\t grep $(printf \"\\317\\200\") actual &&\n+\t\t # Also make sure the commit does not have the \"encoding\" header\n+\t\t ! grep ^encoding actual)\n '\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\n@@ -224,7 +232,6 @@ GIT_COMMITTER_NAME='C O Mitter'; export GIT_COMMITTER_NAME\n \n test_expect_success 'setup copies' '\n \n-\tgit config --unset i18n.commitencoding &&\n \tgit checkout -b copy rein &&\n \tgit mv file file3 &&\n \tgit commit -m move1 &&\ndiff --git a/t/t9350/simple-iso-8859-7-commit-message.txt b/t/t9350/simple-iso-8859-7-commit-message.txt\nnew file mode 100644\nindex 0000000000..8b3f0c3dba\n--- /dev/null\n+++ b/t/t9350/simple-iso-8859-7-commit-message.txt\n@@ -0,0 +1 @@\n+Pi: �\n\\ No newline at end of file\n-- \n2.21.0.782.g571613a09e\n\n"},{"id":"375397","messageId":"20190513164722.31534-4-newren@gmail.com","threadId":"51017","inReplyTo":"20190513164722.31534-1-newren@gmail.com","subject":"[PATCH v4 3/5] fast-export: avoid stripping encoding header if we cannot reencode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T16:47:20Z","receivedAt":"2019-05-13T16:47:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"When fast-export encounters a commit with an 'encoding' header, it tries\nto reencode in utf-8 and then drops the encoding header. However, if it\nfails to reencode in utf-8 because e.g. one of the characters in the\ncommit message was invalid in the old encoding, then we need to retain\nthe original encoding or otherwise we lose information needed to\nunderstand all the other (valid) characters in the original commit\nmessage.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 7 +++++--\n t/t9350-fast-export.sh | 21 ++++++++++++++++++++\n t/t9350/broken-iso-8859-7-commit-message.txt | 1 +\n 3 files changed, 27 insertions(+), 2 deletions(-)\n create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 9e283482ef..7734a9f5a5 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -642,9 +642,12 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n \tif (show_original_ids)\n \t\tprintf(\"original-oid %s\\n\", oid_to_hex(&commit->object.oid));\n-\tprintf(\"%.*s\\n%.*s\\ndata %u\\n%s\",\n+\tprintf(\"%.*s\\n%.*s\\n\",\n \t (int)(author_end - author), author,\n-\t (int)(committer_end - committer), committer,\n+\t (int)(committer_end - committer), committer);\n+\tif (!reencoded && encoding)\n+\t\tprintf(\"encoding %s\\n\", encoding);\n+\tprintf(\"data %u\\n%s\",\n \t (unsigned)(reencoded\n \t\t\t ? strlen(reencoded) : message\n \t\t\t ? strlen(message) : 0),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex c721026260..4fd637312a 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -118,6 +118,27 @@ test_expect_success 'iso-8859-7' '\n \t\t ! grep ^encoding actual)\n '\n \n+test_expect_success 'encoding preserved if reencoding fails' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t git cat-file commit i18n-invalid >actual &&\n+\t\t # Make sure the commit still has the encoding header\n+\t\t grep ^encoding actual &&\n+\t\t # Verify that the commit has the expected size; i.e.\n+\t\t # that no bytes were re-encoded to a different encoding.\n+\t\t test 252 -eq \"$(git cat-file -s i18n-invalid)\" &&\n+\t\t # ...and check for the original special bytes\n+\t\t grep $(printf \"\\360\") actual &&\n+\t\t grep $(printf \"\\377\") actual)\n+'\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\ndiff --git a/t/t9350/broken-iso-8859-7-commit-message.txt b/t/t9350/broken-iso-8859-7-commit-message.txt\nnew file mode 100644\nindex 0000000000..d06ad75b44\n--- /dev/null\n+++ b/t/t9350/broken-iso-8859-7-commit-message.txt\n@@ -0,0 +1 @@\n+Pi: �; Invalid: �\n\\ No newline at end of file\n-- \n2.21.0.782.g571613a09e\n\n"},{"id":"375398","messageId":"20190513164722.31534-5-newren@gmail.com","threadId":"51017","inReplyTo":"20190513164722.31534-1-newren@gmail.com","subject":"[PATCH v4 4/5] fast-export: differentiate between explicitly utf-8 and implicitly utf-8","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T16:47:21Z","receivedAt":"2019-05-13T16:47:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"The find_encoding() function returned the encoding used by a commit\nmessage, returning a default of git_commit_encoding (usually utf-8).\nAlthough the current code does not differentiate between a commit which\nexplicitly requested utf-8 and one where we just assume utf-8 because no\nencoding is set, it will become important when we try to preserve the\nencoding header. Since is_encoding_utf8() returns true when passed\nNULL, we can just return NULL from find_encoding() instead of returning\ngit_commit_encoding.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 7734a9f5a5..66331fa401 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -453,7 +453,7 @@ static const char *find_encoding(const char *begin, const char *end)\n \tbol = memmem(begin, end ? end - begin : strlen(begin),\n \t\t needle, strlen(needle));\n \tif (!bol)\n-\t\treturn git_commit_encoding;\n+\t\treturn NULL;\n \tbol += strlen(needle);\n \teol = strchrnul(bol, '\\n');\n \t*eol = '\\0';\n-- \n2.21.0.782.g571613a09e\n\n"},{"id":"375400","messageId":"20190513164722.31534-6-newren@gmail.com","threadId":"51017","inReplyTo":"20190513164722.31534-1-newren@gmail.com","subject":"[PATCH v4 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T16:47:22Z","receivedAt":"2019-05-13T16:47:40Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Automatic re-encoding of commit messages (and dropping of the encoding\nheader) hurts attempts to do reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions, some subtree rewrites), and seems\ninconsistent with the general principle followed elsewhere in\nfast-export of requiring explicit user requests to modify the output\n(e.g. --signed-tags=strip, --tag-of-filtered-object=rewrite). Add a\n--reencode flag that the user can use to specify, and like other\nfast-export flags, default it to 'abort'.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 35 ++++++++++++++++++++++++++++++++---\n t/t9350-fast-export.sh | 38 +++++++++++++++++++++++++++++++++++---\n 2 files changed, 67 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 66331fa401..4906b23248 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -33,6 +33,7 @@ static const char *fast_export_usage[] = {\n static int progress;\n static enum { SIGNED_TAG_ABORT, VERBATIM, WARN, WARN_STRIP, STRIP } signed_tag_mode = SIGNED_TAG_ABORT;\n static enum { TAG_FILTERING_ABORT, DROP, REWRITE } tag_of_filtered_mode = TAG_FILTERING_ABORT;\n+static enum { REENCODE_ABORT, REENCODE_YES, REENCODE_NO } reencode_mode = REENCODE_ABORT;\n static int fake_missing_tagger;\n static int use_done_feature;\n static int no_data;\n@@ -77,6 +78,20 @@ static int parse_opt_tag_of_filtered_mode(const struct option *opt,\n \treturn 0;\n }\n \n+static int parse_opt_reencode_mode(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tif (unset || !strcmp(arg, \"abort\"))\n+\t\treencode_mode = REENCODE_ABORT;\n+\telse if (!strcmp(arg, \"yes\") || !strcmp(arg, \"true\") || !strcmp(arg, \"on\"))\n+\t\treencode_mode = REENCODE_YES;\n+\telse if (!strcmp(arg, \"no\") || !strcmp(arg, \"false\") || !strcmp(arg, \"off\"))\n+\t\treencode_mode = REENCODE_NO;\n+\telse\n+\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n+\treturn 0;\n+}\n+\n static struct decoration idnums;\n static uint32_t last_idnum;\n \n@@ -633,10 +648,21 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \t}\n \n \tmark_next_object(&commit->object);\n-\tif (anonymize)\n+\tif (anonymize) {\n \t\treencoded = anonymize_commit_message(message);\n-\telse if (!is_encoding_utf8(encoding))\n-\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t} else if (encoding) {\n+\t\tswitch(reencode_mode) {\n+\t\tcase REENCODE_YES:\n+\t\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t\t\tbreak;\n+\t\tcase REENCODE_NO:\n+\t\t\tbreak;\n+\t\tcase REENCODE_ABORT:\n+\t\t\tdie(\"Encountered commit-specific encoding %s in commit \"\n+\t\t\t \"%s; use --reencode=[yes|no] to handle it\",\n+\t\t\t encoding, oid_to_hex(&commit->object.oid));\n+\t\t}\n+\t}\n \tif (!commit->parents)\n \t\tprintf(\"reset %s\\n\", refname);\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n@@ -1091,6 +1117,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK(0, \"tag-of-filtered-object\", &tag_of_filtered_mode, N_(\"mode\"),\n \t\t\t N_(\"select handling of tags that tag filtered objects\"),\n \t\t\t parse_opt_tag_of_filtered_mode),\n+\t\tOPT_CALLBACK(0, \"reencode\", &reencode_mode, N_(\"mode\"),\n+\t\t\t N_(\"select handling of commit messages in an alternate encoding\"),\n+\t\t\t parse_opt_reencode_mode),\n \t\tOPT_STRING(0, \"export-marks\", &export_filename, N_(\"file\"),\n \t\t\t N_(\"Dump marks to this file\")),\n \t\tOPT_STRING(0, \"import-marks\", &import_filename, N_(\"file\"),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 4fd637312a..d21d7bf9a9 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,14 +94,14 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-7' '\n+test_expect_success 'reencoding iso-8859-7' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n \tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n@@ -118,13 +118,45 @@ test_expect_success 'iso-8859-7' '\n \t\t ! grep ^encoding actual)\n '\n \n+test_expect_success 'aborting on iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\ttest_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi\n+'\n+\n+test_expect_success 'preserving iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export --reencode=no wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-no-recoding/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t # The commit object, if not re-encoded, is 240 bytes.\n+\t\t # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n+\t\t # bytes. Re-encoding the Pi character from \\xF0 (\\360) in\n+\t\t # iso-8859-7 to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte.\n+\t\t # Check for the expected size...\n+\t\t test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n+\t\t # ...as well as the expected byte.\n+\t\t git cat-file commit i18n-no-recoding >actual &&\n+\t\t grep $(printf \"\\360\") actual &&\n+\t\t # Also make sure the commit has the \"encoding\" header\n+\t\t grep ^encoding actual)\n+'\n+\n test_expect_success 'encoding preserved if reencoding fails' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \techo rosten >file &&\n \tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n-- \n2.21.0.782.g571613a09e\n\n"},{"id":"375399","messageId":"20190513164722.31534-3-newren@gmail.com","threadId":"51017","inReplyTo":"20190513164722.31534-1-newren@gmail.com","subject":"[PATCH v4 2/5] fast-import: support 'encoding' commit header","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T16:47:19Z","receivedAt":"2019-05-13T16:47:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Since git supports commit messages with an encoding other than utf-8,\nallow fast-import to import such commits. This may be useful for folks\nwho do not want to reencode commit messages from an external system, and\nmay also be useful to achieve reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions or subtree work) with git repositories that\nhave used specialized encodings in their commit history.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-import.txt | 7 +++++++\n fast-import.c | 11 +++++++++--\n t/t9300-fast-import.sh | 20 ++++++++++++++++++++\n 3 files changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex d65cdb3d08..7baf9e47b5 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -388,6 +388,7 @@ change to the project.\n \toriginal-oid?\n \t('author' (SP )? SP LT GT SP LF)?\n \t'committer' (SP )? SP LT GT SP LF\n+\t('encoding' SP )?\n \tdata\n \t('from' SP LF)?\n \t('merge' SP LF)?\n@@ -455,6 +456,12 @@ that was selected by the --date-format= command-line option.\n See ``Date Formats'' above for the set of supported formats, and\n their syntax.\n \n+`encoding`\n+^^^^^^^^^^\n+The optional `encoding` command indicates the encoding of the commit\n+message. Most commits are UTF-8 and the encoding is omitted, but this\n+allows importing commit messages into git without first reencoding them.\n+\n `from`\n ^^^^^^\n The `from` command is used to specify the commit to initialize\ndiff --git a/fast-import.c b/fast-import.c\nindex f38d04fa58..76a7bd3699 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2585,6 +2585,7 @@ static void parse_new_commit(const char *arg)\n \tstruct branch *b;\n \tchar *author = NULL;\n \tchar *committer = NULL;\n+\tconst char *encoding = NULL;\n \tstruct hash_list *merge_list = NULL;\n \tunsigned int merge_count;\n \tunsigned char prev_fanout, new_fanout;\n@@ -2607,6 +2608,8 @@ static void parse_new_commit(const char *arg)\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n+\tif (skip_prefix(command_buf.buf, \"encoding \", &encoding))\n+\t\tread_next_command();\n \tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n@@ -2670,9 +2673,13 @@ static void parse_new_commit(const char *arg)\n \t}\n \tstrbuf_addf(&new_data,\n \t\t\"author %s\\n\"\n-\t\t\"committer %s\\n\"\n-\t\t\"\\n\",\n+\t\t\"committer %s\\n\",\n \t\tauthor ? author : committer, committer);\n+\tif (encoding)\n+\t\tstrbuf_addf(&new_data,\n+\t\t\t\"encoding %s\\n\",\n+\t\t\tencoding);\n+\tstrbuf_addch(&new_data, '\\n');\n \tstrbuf_addbuf(&new_data, &msg);\n \tfree(author);\n \tfree(committer);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 3668263c40..141b7fa35e 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3299,4 +3299,24 @@ test_expect_success !MINGW 'W: get-mark & empty orphan commit with erroneous thi\n \tsed -e s/LFs/LLL/ W-input | tr L \"\\n\" | test_must_fail git fast-import\n '\n \n+###\n+### series X (other new features)\n+###\n+\n+test_expect_success 'X: handling encoding' '\n+\ttest_tick &&\n+\tcat >input <<-INPUT_END &&\n+\tcommit refs/heads/encoding\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tencoding iso-8859-7\n+\tdata <>input &&\n+\n+\tgit fast-import writes:\n\n> +static int parse_opt_reencode_mode(const struct option *opt,\n> +\t\t\t\t const char *arg, int unset)\n> +{\n> +\tif (unset || !strcmp(arg, \"abort\"))\n> +\t\treencode_mode = REENCODE_ABORT;\n> +\telse if (!strcmp(arg, \"yes\") || !strcmp(arg, \"true\") || !strcmp(arg, \"on\"))\n> +\t\treencode_mode = REENCODE_YES;\n> +\telse if (!strcmp(arg, \"no\") || !strcmp(arg, \"false\") || !strcmp(arg, \"off\"))\n> +\t\treencode_mode = REENCODE_NO;\n\nDon't duplicate the logic; find existing callers of git_parse_maybe_bool(),\nlearn what they are using the helper for and mimic them, perhaps?\n"},{"id":"375433","messageId":"20190513231726.16218-1-newren@gmail.com","threadId":"51017","inReplyTo":"20190513164722.31534-1-newren@gmail.com","subject":"[PATCH v5 0/5] Fix and extend encoding handling in fast export/import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T23:17:21Z","receivedAt":"2019-05-13T23:17:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"While stress testing `git filter-repo`, I noticed an issue with\nencoding; further digging led to the fixes and features in this series.\nSee the individual commit messages for details.\n\nChanges since v4 (full range-diff below):\n * Used git_parse_maybe_bool()\n * Updated Documentation/git-fast-export.txt to document the new option\n\nElijah Newren (5):\n t9350: fix encoding test to actually test reencoding\n fast-import: support 'encoding' commit header\n fast-export: avoid stripping encoding header if we cannot reencode\n fast-export: differentiate between explicitly utf-8 and implicitly\n utf-8\n fast-export: do automatic reencoding of commit messages only if\n requested\n\n Documentation/git-fast-export.txt | 7 ++\n Documentation/git-fast-import.txt | 7 ++\n builtin/fast-export.c | 55 ++++++++++++--\n fast-import.c | 11 ++-\n t/t9300-fast-import.sh | 20 +++++\n t/t9350-fast-export.sh | 78 +++++++++++++++++---\n t/t9350/broken-iso-8859-7-commit-message.txt | 1 +\n t/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n 8 files changed, 163 insertions(+), 17 deletions(-)\n create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\n create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n\nRange-diff:\n1: 37a68a0ffd = 1: 37a68a0ffd t9350: fix encoding test to actually test reencoding\n2: 3d84f4613d = 2: 3d84f4613d fast-import: support 'encoding' commit header\n3: baa8394a3a = 3: baa8394a3a fast-export: avoid stripping encoding header if we cannot reencode\n4: 49960164c6 = 4: 49960164c6 fast-export: differentiate between explicitly utf-8 and implicitly utf-8\n5: 571613a09e ! 5: d8be4ee826 fast-export: do automatic reencoding of commit messages only if requested\n @@ -13,6 +13,24 @@\n \n Signed-off-by: Elijah Newren \n \n + diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\n + --- a/Documentation/git-fast-export.txt\n + +++ b/Documentation/git-fast-export.txt\n +@@\n + \tfor intermediary filters (e.g. for rewriting commit messages\n + \twhich refer to older commits, or for stripping blobs by id).\n + \n ++--reencode=(yes|no|abort)::\n ++\tSpecify how to handle `encoding` header in commit objects. When\n ++\tasking to 'abort' (which is the default), this program will die\n ++\twhen encountering such a commit object. With 'yes', the commit\n ++\tmessage will be reencoded into UTF-8. With 'no', the original\n ++\tencoding will be preserved.\n ++\n + --refspec::\n + \tApply the specified refspec to each ref exported. Multiple of them can\n + \tbe specified.\n +\n diff --git a/builtin/fast-export.c b/builtin/fast-export.c\n --- a/builtin/fast-export.c\n +++ b/builtin/fast-export.c\n @@ -31,14 +49,25 @@\n +static int parse_opt_reencode_mode(const struct option *opt,\n +\t\t\t\t const char *arg, int unset)\n +{\n -+\tif (unset || !strcmp(arg, \"abort\"))\n ++\tif (unset) {\n +\t\treencode_mode = REENCODE_ABORT;\n -+\telse if (!strcmp(arg, \"yes\") || !strcmp(arg, \"true\") || !strcmp(arg, \"on\"))\n -+\t\treencode_mode = REENCODE_YES;\n -+\telse if (!strcmp(arg, \"no\") || !strcmp(arg, \"false\") || !strcmp(arg, \"off\"))\n ++\t\treturn 0;\n ++\t}\n ++\n ++\tswitch (git_parse_maybe_bool(arg)) {\n ++\tcase 0:\n +\t\treencode_mode = REENCODE_NO;\n -+\telse\n -+\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n ++\t\tbreak;\n ++\tcase 1:\n ++\t\treencode_mode = REENCODE_YES;\n ++\t\tbreak;\n ++\tdefault:\n ++\t\tif (arg && !strcasecmp(arg, \"abort\"))\n ++\t\t\treencode_mode = REENCODE_ABORT;\n ++\t\telse\n ++\t\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n ++\t}\n ++\n +\treturn 0;\n +}\n +\n-- \n2.21.0.782.gd8be4ee826\n\n"},{"id":"375434","messageId":"20190513231726.16218-2-newren@gmail.com","threadId":"51017","inReplyTo":"20190513231726.16218-1-newren@gmail.com","subject":"[PATCH v5 1/5] t9350: fix encoding test to actually test reencoding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T23:17:22Z","receivedAt":"2019-05-13T23:17:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This test used an author with non-ascii characters in the name, but\nno special commit message. It then grep'ed for those non-ascii\ncharacters, but those are guaranteed to exist regardless of the\nreencoding process since the reencoding only affects the commit message,\nnot the author or committer names. As such, the test would work even if\nthe re-encoding process simply stripped the commit message entirely.\nModify the test to actually check that the reencoding in utf-8 worked.\n\nSigned-off-by: Elijah Newren \n---\n t/t9350-fast-export.sh | 27 ++++++++++++--------\n t/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n 2 files changed, 18 insertions(+), 10 deletions(-)\n create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 5690fe2810..c721026260 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,22 +94,30 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-1' '\n+test_expect_success 'iso-8859-7' '\n \n-\tgit config i18n.commitencoding ISO8859-1 &&\n-\t# use author and committer name in ISO-8859-1 to match it.\n-\t. \"$TEST_DIRECTORY\"/t3901/8859-1.txt &&\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n-\tgit commit -s -m den file &&\n-\tgit fast-export wer^..wer >iso8859-1.fi &&\n-\tsed \"s/wer/i18n/\" iso8859-1.fi |\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n+\t\t # The commit object, if not re-encoded, would be 240 bytes.\n+\t\t # Removing the \"encoding iso-8859-7\\n\" header drops 20 bytes.\n+\t\t # Re-encoding the Pi character from \\xF0 (\\360) in iso-8859-7\n+\t\t # to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte. Check for\n+\t\t # the expected size.\n+\t\t test 221 -eq \"$(git cat-file -s i18n)\" &&\n+\t\t # ...and for the expected translation of bytes.\n \t\t git cat-file commit i18n >actual &&\n-\t\t grep \"Áéí óú\" actual)\n-\n+\t\t grep $(printf \"\\317\\200\") actual &&\n+\t\t # Also make sure the commit does not have the \"encoding\" header\n+\t\t ! grep ^encoding actual)\n '\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\n@@ -224,7 +232,6 @@ GIT_COMMITTER_NAME='C O Mitter'; export GIT_COMMITTER_NAME\n \n test_expect_success 'setup copies' '\n \n-\tgit config --unset i18n.commitencoding &&\n \tgit checkout -b copy rein &&\n \tgit mv file file3 &&\n \tgit commit -m move1 &&\ndiff --git a/t/t9350/simple-iso-8859-7-commit-message.txt b/t/t9350/simple-iso-8859-7-commit-message.txt\nnew file mode 100644\nindex 0000000000..8b3f0c3dba\n--- /dev/null\n+++ b/t/t9350/simple-iso-8859-7-commit-message.txt\n@@ -0,0 +1 @@\n+Pi: �\n\\ No newline at end of file\n-- \n2.21.0.782.gd8be4ee826\n\n"},{"id":"375435","messageId":"20190513231726.16218-3-newren@gmail.com","threadId":"51017","inReplyTo":"20190513231726.16218-1-newren@gmail.com","subject":"[PATCH v5 2/5] fast-import: support 'encoding' commit header","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T23:17:23Z","receivedAt":"2019-05-13T23:17:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Since git supports commit messages with an encoding other than utf-8,\nallow fast-import to import such commits. This may be useful for folks\nwho do not want to reencode commit messages from an external system, and\nmay also be useful to achieve reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions or subtree work) with git repositories that\nhave used specialized encodings in their commit history.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-import.txt | 7 +++++++\n fast-import.c | 11 +++++++++--\n t/t9300-fast-import.sh | 20 ++++++++++++++++++++\n 3 files changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex d65cdb3d08..7baf9e47b5 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -388,6 +388,7 @@ change to the project.\n \toriginal-oid?\n \t('author' (SP )? SP LT GT SP LF)?\n \t'committer' (SP )? SP LT GT SP LF\n+\t('encoding' SP )?\n \tdata\n \t('from' SP LF)?\n \t('merge' SP LF)?\n@@ -455,6 +456,12 @@ that was selected by the --date-format= command-line option.\n See ``Date Formats'' above for the set of supported formats, and\n their syntax.\n \n+`encoding`\n+^^^^^^^^^^\n+The optional `encoding` command indicates the encoding of the commit\n+message. Most commits are UTF-8 and the encoding is omitted, but this\n+allows importing commit messages into git without first reencoding them.\n+\n `from`\n ^^^^^^\n The `from` command is used to specify the commit to initialize\ndiff --git a/fast-import.c b/fast-import.c\nindex f38d04fa58..76a7bd3699 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2585,6 +2585,7 @@ static void parse_new_commit(const char *arg)\n \tstruct branch *b;\n \tchar *author = NULL;\n \tchar *committer = NULL;\n+\tconst char *encoding = NULL;\n \tstruct hash_list *merge_list = NULL;\n \tunsigned int merge_count;\n \tunsigned char prev_fanout, new_fanout;\n@@ -2607,6 +2608,8 @@ static void parse_new_commit(const char *arg)\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n+\tif (skip_prefix(command_buf.buf, \"encoding \", &encoding))\n+\t\tread_next_command();\n \tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n@@ -2670,9 +2673,13 @@ static void parse_new_commit(const char *arg)\n \t}\n \tstrbuf_addf(&new_data,\n \t\t\"author %s\\n\"\n-\t\t\"committer %s\\n\"\n-\t\t\"\\n\",\n+\t\t\"committer %s\\n\",\n \t\tauthor ? author : committer, committer);\n+\tif (encoding)\n+\t\tstrbuf_addf(&new_data,\n+\t\t\t\"encoding %s\\n\",\n+\t\t\tencoding);\n+\tstrbuf_addch(&new_data, '\\n');\n \tstrbuf_addbuf(&new_data, &msg);\n \tfree(author);\n \tfree(committer);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 3668263c40..141b7fa35e 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3299,4 +3299,24 @@ test_expect_success !MINGW 'W: get-mark & empty orphan commit with erroneous thi\n \tsed -e s/LFs/LLL/ W-input | tr L \"\\n\" | test_must_fail git fast-import\n '\n \n+###\n+### series X (other new features)\n+###\n+\n+test_expect_success 'X: handling encoding' '\n+\ttest_tick &&\n+\tcat >input <<-INPUT_END &&\n+\tcommit refs/heads/encoding\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tencoding iso-8859-7\n+\tdata <>input &&\n+\n+\tgit fast-import \n---\n builtin/fast-export.c | 7 +++++--\n t/t9350-fast-export.sh | 21 ++++++++++++++++++++\n t/t9350/broken-iso-8859-7-commit-message.txt | 1 +\n 3 files changed, 27 insertions(+), 2 deletions(-)\n create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 9e283482ef..7734a9f5a5 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -642,9 +642,12 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n \tif (show_original_ids)\n \t\tprintf(\"original-oid %s\\n\", oid_to_hex(&commit->object.oid));\n-\tprintf(\"%.*s\\n%.*s\\ndata %u\\n%s\",\n+\tprintf(\"%.*s\\n%.*s\\n\",\n \t (int)(author_end - author), author,\n-\t (int)(committer_end - committer), committer,\n+\t (int)(committer_end - committer), committer);\n+\tif (!reencoded && encoding)\n+\t\tprintf(\"encoding %s\\n\", encoding);\n+\tprintf(\"data %u\\n%s\",\n \t (unsigned)(reencoded\n \t\t\t ? strlen(reencoded) : message\n \t\t\t ? strlen(message) : 0),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex c721026260..4fd637312a 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -118,6 +118,27 @@ test_expect_success 'iso-8859-7' '\n \t\t ! grep ^encoding actual)\n '\n \n+test_expect_success 'encoding preserved if reencoding fails' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t git cat-file commit i18n-invalid >actual &&\n+\t\t # Make sure the commit still has the encoding header\n+\t\t grep ^encoding actual &&\n+\t\t # Verify that the commit has the expected size; i.e.\n+\t\t # that no bytes were re-encoded to a different encoding.\n+\t\t test 252 -eq \"$(git cat-file -s i18n-invalid)\" &&\n+\t\t # ...and check for the original special bytes\n+\t\t grep $(printf \"\\360\") actual &&\n+\t\t grep $(printf \"\\377\") actual)\n+'\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\ndiff --git a/t/t9350/broken-iso-8859-7-commit-message.txt b/t/t9350/broken-iso-8859-7-commit-message.txt\nnew file mode 100644\nindex 0000000000..d06ad75b44\n--- /dev/null\n+++ b/t/t9350/broken-iso-8859-7-commit-message.txt\n@@ -0,0 +1 @@\n+Pi: �; Invalid: �\n\\ No newline at end of file\n-- \n2.21.0.782.gd8be4ee826\n\n"},{"id":"375437","messageId":"20190513231726.16218-5-newren@gmail.com","threadId":"51017","inReplyTo":"20190513231726.16218-1-newren@gmail.com","subject":"[PATCH v5 4/5] fast-export: differentiate between explicitly utf-8 and implicitly utf-8","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T23:17:25Z","receivedAt":"2019-05-13T23:17:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"The find_encoding() function returned the encoding used by a commit\nmessage, returning a default of git_commit_encoding (usually utf-8).\nAlthough the current code does not differentiate between a commit which\nexplicitly requested utf-8 and one where we just assume utf-8 because no\nencoding is set, it will become important when we try to preserve the\nencoding header. Since is_encoding_utf8() returns true when passed\nNULL, we can just return NULL from find_encoding() instead of returning\ngit_commit_encoding.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 7734a9f5a5..66331fa401 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -453,7 +453,7 @@ static const char *find_encoding(const char *begin, const char *end)\n \tbol = memmem(begin, end ? end - begin : strlen(begin),\n \t\t needle, strlen(needle));\n \tif (!bol)\n-\t\treturn git_commit_encoding;\n+\t\treturn NULL;\n \tbol += strlen(needle);\n \teol = strchrnul(bol, '\\n');\n \t*eol = '\\0';\n-- \n2.21.0.782.gd8be4ee826\n\n"},{"id":"375438","messageId":"20190513231726.16218-6-newren@gmail.com","threadId":"51017","inReplyTo":"20190513231726.16218-1-newren@gmail.com","subject":"[PATCH v5 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-13T23:17:26Z","receivedAt":"2019-05-13T23:17:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Automatic re-encoding of commit messages (and dropping of the encoding\nheader) hurts attempts to do reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions, some subtree rewrites), and seems\ninconsistent with the general principle followed elsewhere in\nfast-export of requiring explicit user requests to modify the output\n(e.g. --signed-tags=strip, --tag-of-filtered-object=rewrite). Add a\n--reencode flag that the user can use to specify, and like other\nfast-export flags, default it to 'abort'.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 7 +++++\n builtin/fast-export.c | 46 +++++++++++++++++++++++++++++--\n t/t9350-fast-export.sh | 38 +++++++++++++++++++++++--\n 3 files changed, 85 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex 64c01ba918..11427acdde 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -129,6 +129,13 @@ marks the same across runs.\n \tfor intermediary filters (e.g. for rewriting commit messages\n \twhich refer to older commits, or for stripping blobs by id).\n \n+--reencode=(yes|no|abort)::\n+\tSpecify how to handle `encoding` header in commit objects. When\n+\tasking to 'abort' (which is the default), this program will die\n+\twhen encountering such a commit object. With 'yes', the commit\n+\tmessage will be reencoded into UTF-8. With 'no', the original\n+\tencoding will be preserved.\n+\n --refspec::\n \tApply the specified refspec to each ref exported. Multiple of them can\n \tbe specified.\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 66331fa401..0bb65b3886 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -33,6 +33,7 @@ static const char *fast_export_usage[] = {\n static int progress;\n static enum { SIGNED_TAG_ABORT, VERBATIM, WARN, WARN_STRIP, STRIP } signed_tag_mode = SIGNED_TAG_ABORT;\n static enum { TAG_FILTERING_ABORT, DROP, REWRITE } tag_of_filtered_mode = TAG_FILTERING_ABORT;\n+static enum { REENCODE_ABORT, REENCODE_YES, REENCODE_NO } reencode_mode = REENCODE_ABORT;\n static int fake_missing_tagger;\n static int use_done_feature;\n static int no_data;\n@@ -77,6 +78,31 @@ static int parse_opt_tag_of_filtered_mode(const struct option *opt,\n \treturn 0;\n }\n \n+static int parse_opt_reencode_mode(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tif (unset) {\n+\t\treencode_mode = REENCODE_ABORT;\n+\t\treturn 0;\n+\t}\n+\n+\tswitch (git_parse_maybe_bool(arg)) {\n+\tcase 0:\n+\t\treencode_mode = REENCODE_NO;\n+\t\tbreak;\n+\tcase 1:\n+\t\treencode_mode = REENCODE_YES;\n+\t\tbreak;\n+\tdefault:\n+\t\tif (arg && !strcasecmp(arg, \"abort\"))\n+\t\t\treencode_mode = REENCODE_ABORT;\n+\t\telse\n+\t\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static struct decoration idnums;\n static uint32_t last_idnum;\n \n@@ -633,10 +659,21 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \t}\n \n \tmark_next_object(&commit->object);\n-\tif (anonymize)\n+\tif (anonymize) {\n \t\treencoded = anonymize_commit_message(message);\n-\telse if (!is_encoding_utf8(encoding))\n-\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t} else if (encoding) {\n+\t\tswitch(reencode_mode) {\n+\t\tcase REENCODE_YES:\n+\t\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t\t\tbreak;\n+\t\tcase REENCODE_NO:\n+\t\t\tbreak;\n+\t\tcase REENCODE_ABORT:\n+\t\t\tdie(\"Encountered commit-specific encoding %s in commit \"\n+\t\t\t \"%s; use --reencode=[yes|no] to handle it\",\n+\t\t\t encoding, oid_to_hex(&commit->object.oid));\n+\t\t}\n+\t}\n \tif (!commit->parents)\n \t\tprintf(\"reset %s\\n\", refname);\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n@@ -1091,6 +1128,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK(0, \"tag-of-filtered-object\", &tag_of_filtered_mode, N_(\"mode\"),\n \t\t\t N_(\"select handling of tags that tag filtered objects\"),\n \t\t\t parse_opt_tag_of_filtered_mode),\n+\t\tOPT_CALLBACK(0, \"reencode\", &reencode_mode, N_(\"mode\"),\n+\t\t\t N_(\"select handling of commit messages in an alternate encoding\"),\n+\t\t\t parse_opt_reencode_mode),\n \t\tOPT_STRING(0, \"export-marks\", &export_filename, N_(\"file\"),\n \t\t\t N_(\"Dump marks to this file\")),\n \t\tOPT_STRING(0, \"import-marks\", &import_filename, N_(\"file\"),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 4fd637312a..d21d7bf9a9 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,14 +94,14 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-7' '\n+test_expect_success 'reencoding iso-8859-7' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n \tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n@@ -118,13 +118,45 @@ test_expect_success 'iso-8859-7' '\n \t\t ! grep ^encoding actual)\n '\n \n+test_expect_success 'aborting on iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\ttest_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi\n+'\n+\n+test_expect_success 'preserving iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export --reencode=no wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-no-recoding/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t # The commit object, if not re-encoded, is 240 bytes.\n+\t\t # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n+\t\t # bytes. Re-encoding the Pi character from \\xF0 (\\360) in\n+\t\t # iso-8859-7 to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte.\n+\t\t # Check for the expected size...\n+\t\t test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n+\t\t # ...as well as the expected byte.\n+\t\t git cat-file commit i18n-no-recoding >actual &&\n+\t\t grep $(printf \"\\360\") actual &&\n+\t\t # Also make sure the commit has the \"encoding\" header\n+\t\t grep ^encoding actual)\n+'\n+\n test_expect_success 'encoding preserved if reencoding fails' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \techo rosten >file &&\n \tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n-- \n2.21.0.782.gd8be4ee826\n\n"},{"id":"375443","messageId":"CAPig+cTmgPnFud0NVcP0qcQ08t3h-QanyhSTtkx05fQVo3qV9Q@mail.gmail.com","threadId":"51017","inReplyTo":"20190513231726.16218-6-newren@gmail.com","subject":"Re: [PATCH v5 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-05-14T00:19:23Z","receivedAt":"2019-05-14T00:19:37Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, May 13, 2019 at 7:17 PM Elijah Newren wrote:\n> Automatic re-encoding of commit messages (and dropping of the encoding\n> header) hurts attempts to do reversible history rewrites (e.g. sha1sum\n> <-> sha256sum transitions, some subtree rewrites), and seems\n> inconsistent with the general principle followed elsewhere in\n> fast-export of requiring explicit user requests to modify the output\n> (e.g. --signed-tags=strip, --tag-of-filtered-object=rewrite). Add a\n> --reencode flag that the user can use to specify, and like other\n> fast-export flags, default it to 'abort'.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> diff --git a/builtin/fast-export.c b/builtin/fast-export.c\n> @@ -77,6 +78,31 @@ static int parse_opt_tag_of_filtered_mode(const struct option *opt,\n> +static int parse_opt_reencode_mode(const struct option *opt,\n> + const char *arg, int unset)\n> +{\n> + if (unset) {\n> + reencode_mode = REENCODE_ABORT;\n> + return 0;\n> + }\n> +\n> + switch (git_parse_maybe_bool(arg)) {\n> + case 0:\n> + reencode_mode = REENCODE_NO;\n> + break;\n> + case 1:\n> + reencode_mode = REENCODE_YES;\n> + break;\n> + default:\n> + if (arg && !strcasecmp(arg, \"abort\"))\n\nIf 'arg' is NULL, git_parse_maybe_bool() will have returned 1, which\nis already handled above, so no need to check it here. So, dropping\nthe \"arg &&\" from the conditional...\n\n> + reencode_mode = REENCODE_ABORT;\n> + else\n> + return error(\"Unknown reencoding mode: %s\", arg);\n\n...makes it clear that you won't ever be passing NULL to \"%s\", which\nis undefined behavior.\n\n> + }\n> +\n> + return 0;\n> +}\n"},{"id":"375463","messageId":"20190514025013.jg7zya7w5qypzxca@tb-raspi4","threadId":"51017","inReplyTo":"20190513231726.16218-2-newren@gmail.com","subject":"Re: [PATCH v5 1/5] t9350: fix encoding test to actually test reencoding","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-05-14T02:50:13Z","receivedAt":"2019-05-14T02:51:59Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, May 13, 2019 at 04:17:22PM -0700, Elijah Newren wrote:\n> This test used an author with non-ascii characters in the name, but\n> no special commit message. It then grep'ed for those non-ascii\n> characters, but those are guaranteed to exist regardless of the\n> reencoding process since the reencoding only affects the commit message,\n> not the author or committer names. As such, the test would work even if\n> the re-encoding process simply stripped the commit message entirely.\n> Modify the test to actually check that the reencoding in utf-8 worked.\n\n(Not a native english speaker here): Should that be\n\"...reencoding into utf-8 worked\" ?\n\n>\n> Signed-off-by: Elijah Newren \n> ---\n> t/t9350-fast-export.sh | 27 ++++++++++++--------\n> t/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n> 2 files changed, 18 insertions(+), 10 deletions(-)\n> create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n>\n> diff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\n> index 5690fe2810..c721026260 100755\n> --- a/t/t9350-fast-export.sh\n> +++ b/t/t9350-fast-export.sh\n> @@ -94,22 +94,30 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n> \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n> '\n>\n> -test_expect_success 'iso-8859-1' '\n> +test_expect_success 'iso-8859-7' '\n>\n> -\tgit config i18n.commitencoding ISO8859-1 &&\n> -\t# use author and committer name in ISO-8859-1 to match it.\n> -\t. \"$TEST_DIRECTORY\"/t3901/8859-1.txt &&\n> +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n> +\ttest_config i18n.commitencoding iso-8859-7 &&\n> \ttest_tick &&\n> \techo rosten >file &&\n> -\tgit commit -s -m den file &&\n> -\tgit fast-export wer^..wer >iso8859-1.fi &&\n> -\tsed \"s/wer/i18n/\" iso8859-1.fi |\n> +\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n> +\tgit fast-export wer^..wer >iso-8859-7.fi &&\n> +\tsed \"s/wer/i18n/\" iso-8859-7.fi |\n> \t\t(cd new &&\n> \t\t git fast-import &&\n> +\t\t # The commit object, if not re-encoded, would be 240 bytes.\n> +\t\t # Removing the \"encoding iso-8859-7\\n\" header drops 20 bytes.\n> +\t\t # Re-encoding the Pi character from \\xF0 (\\360) in iso-8859-7\n> +\t\t # to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte. Check for\n> +\t\t # the expected size.\n> +\t\t test 221 -eq \"$(git cat-file -s i18n)\" &&\n> +\t\t # ...and for the expected translation of bytes.\n> \t\t git cat-file commit i18n >actual &&\n> -\t\t grep \"Áéí óú\" actual)\n> -\n> +\t\t grep $(printf \"\\317\\200\") actual &&\n> +\t\t # Also make sure the commit does not have the \"encoding\" header\n> +\t\t ! grep ^encoding actual)\n> '\n> +\n> test_expect_success 'import/export-marks' '\n>\n> \tgit checkout -b marks master &&\n> @@ -224,7 +232,6 @@ GIT_COMMITTER_NAME='C O Mitter'; export GIT_COMMITTER_NAME\n>\n> test_expect_success 'setup copies' '\n>\n> -\tgit config --unset i18n.commitencoding &&\n> \tgit checkout -b copy rein &&\n> \tgit mv file file3 &&\n> \tgit commit -m move1 &&\n> diff --git a/t/t9350/simple-iso-8859-7-commit-message.txt b/t/t9350/simple-iso-8859-7-commit-message.txt\n> new file mode 100644\n> index 0000000000..8b3f0c3dba\n> --- /dev/null\n> +++ b/t/t9350/simple-iso-8859-7-commit-message.txt\n> @@ -0,0 +1 @@\n> +Pi: ?\n> \\ No newline at end of file\n> --\n> 2.21.0.782.gd8be4ee826\n>\n"},{"id":"375464","messageId":"20190514025651.gjtvikhxcjoudkrj@tb-raspi4","threadId":"51017","inReplyTo":"20190513231726.16218-4-newren@gmail.com","subject":"Re: [PATCH v5 3/5] fast-export: avoid stripping encoding header if we cannot reencode","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-05-14T02:56:51Z","receivedAt":"2019-05-14T02:57:01Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, May 13, 2019 at 04:17:24PM -0700, Elijah Newren wrote:\n> When fast-export encounters a commit with an 'encoding' header, it tries\n> to reencode in utf-8 and then drops the encoding header. However, if it\n> fails to reencode in utf-8 because e.g. one of the characters in the\n> commit message was invalid in the old encoding, then we need to retain\n> the original encoding or otherwise we lose information needed to\n> understand all the other (valid) characters in the original commit\n> message.\n\nMinor question: \"utf-8\" or \"UTF-8\" ?\nMostly we use UTF-8 in Git.\n\n>\n> Signed-off-by: Elijah Newren \n> ---\n> builtin/fast-export.c | 7 +++++--\n> t/t9350-fast-export.sh | 21 ++++++++++++++++++++\n> t/t9350/broken-iso-8859-7-commit-message.txt | 1 +\n> 3 files changed, 27 insertions(+), 2 deletions(-)\n> create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\n>\n> diff --git a/builtin/fast-export.c b/builtin/fast-export.c\n> index 9e283482ef..7734a9f5a5 100644\n> --- a/builtin/fast-export.c\n> +++ b/builtin/fast-export.c\n> @@ -642,9 +642,12 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n> \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n> \tif (show_original_ids)\n> \t\tprintf(\"original-oid %s\\n\", oid_to_hex(&commit->object.oid));\n> -\tprintf(\"%.*s\\n%.*s\\ndata %u\\n%s\",\n> +\tprintf(\"%.*s\\n%.*s\\n\",\n> \t (int)(author_end - author), author,\n> -\t (int)(committer_end - committer), committer,\n> +\t (int)(committer_end - committer), committer);\n> +\tif (!reencoded && encoding)\n> +\t\tprintf(\"encoding %s\\n\", encoding);\n> +\tprintf(\"data %u\\n%s\",\n> \t (unsigned)(reencoded\n> \t\t\t ? strlen(reencoded) : message\n> \t\t\t ? strlen(message) : 0),\n> diff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\n> index c721026260..4fd637312a 100755\n> --- a/t/t9350-fast-export.sh\n> +++ b/t/t9350-fast-export.sh\n> @@ -118,6 +118,27 @@ test_expect_success 'iso-8859-7' '\n> \t\t ! grep ^encoding actual)\n> '\n>\n> +test_expect_success 'encoding preserved if reencoding fails' '\n> +\n> +\ttest_when_finished \"git reset --hard HEAD~1\" &&\n> +\ttest_config i18n.commitencoding iso-8859-7 &&\n> +\techo rosten >file &&\n> +\tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n> +\tgit fast-export wer^..wer >iso-8859-7.fi &&\n> +\tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n> +\t\t(cd new &&\n> +\t\t git fast-import &&\n> +\t\t git cat-file commit i18n-invalid >actual &&\n> +\t\t # Make sure the commit still has the encoding header\n> +\t\t grep ^encoding actual &&\n> +\t\t # Verify that the commit has the expected size; i.e.\n> +\t\t # that no bytes were re-encoded to a different encoding.\n> +\t\t test 252 -eq \"$(git cat-file -s i18n-invalid)\" &&\n> +\t\t # ...and check for the original special bytes\n> +\t\t grep $(printf \"\\360\") actual &&\n> +\t\t grep $(printf \"\\377\") actual)\n> +'\n> +\n> test_expect_success 'import/export-marks' '\n>\n> \tgit checkout -b marks master &&\n> diff --git a/t/t9350/broken-iso-8859-7-commit-message.txt b/t/t9350/broken-iso-8859-7-commit-message.txt\n> new file mode 100644\n> index 0000000000..d06ad75b44\n> --- /dev/null\n> +++ b/t/t9350/broken-iso-8859-7-commit-message.txt\n> @@ -0,0 +1 @@\n> +Pi: ?; Invalid: ?\n> \\ No newline at end of file\n> --\n> 2.21.0.782.gd8be4ee826\n>\n"},{"id":"375465","messageId":"20190514030123.atqguzxh4v55ydty@tb-raspi4","threadId":"51017","inReplyTo":"20190513231726.16218-5-newren@gmail.com","subject":"Re: [PATCH v5 4/5] fast-export: differentiate between explicitly utf-8 and implicitly utf-8","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-05-14T03:01:23Z","receivedAt":"2019-05-14T03:01:34Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, May 13, 2019 at 04:17:25PM -0700, Elijah Newren wrote:\n> The find_encoding() function returned the encoding used by a commit\n> message, returning a default of git_commit_encoding (usually utf-8).\nI think \"UTF-8\" is preferred over \"utf-8\".\nUnless it is a function name like is_encoding_utf8()\n\n> Although the current code does not differentiate between a commit which\n> explicitly requested utf-8 and one where we just assume utf-8 because no\n> encoding is set, it will become important when we try to preserve the\n> encoding header. Since is_encoding_utf8() returns true when passed\n> NULL, we can just return NULL from find_encoding() instead of returning\n> git_commit_encoding.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> builtin/fast-export.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/fast-export.c b/builtin/fast-export.c\n> index 7734a9f5a5..66331fa401 100644\n> --- a/builtin/fast-export.c\n> +++ b/builtin/fast-export.c\n> @@ -453,7 +453,7 @@ static const char *find_encoding(const char *begin, const char *end)\n> \tbol = memmem(begin, end ? end - begin : strlen(begin),\n> \t\t needle, strlen(needle));\n> \tif (!bol)\n> -\t\treturn git_commit_encoding;\n> +\t\treturn NULL;\n> \tbol += strlen(needle);\n> \teol = strchrnul(bol, '\\n');\n> \t*eol = '\\0';\n> --\n> 2.21.0.782.gd8be4ee826\n>\n"},{"id":"375466","messageId":"20190514043102.31979-1-newren@gmail.com","threadId":"51017","inReplyTo":"20190513231726.16218-1-newren@gmail.com","subject":"[PATCH v6 0/5] Fix and extend encoding handling in fast export/import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-14T04:30:57Z","receivedAt":"2019-05-14T04:31:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"While stress testing `git filter-repo`, I noticed an issue with\nencoding; further digging led to the fixes and features in this series.\nSee the individual commit messages for details.\n\nChanges since v5 (full range-diff below):\n * s/utf-8/UTF-8/, as pointed out by Torsten (in commit messages and comments)\n * small code cleanup pointed out by Eric\n * rewrap the first commit message\n\nElijah Newren (5):\n t9350: fix encoding test to actually test reencoding\n fast-import: support 'encoding' commit header\n fast-export: avoid stripping encoding header if we cannot reencode\n fast-export: differentiate between explicitly UTF-8 and implicitly\n UTF-8\n fast-export: do automatic reencoding of commit messages only if\n requested\n\n Documentation/git-fast-export.txt | 7 ++\n Documentation/git-fast-import.txt | 7 ++\n builtin/fast-export.c | 55 ++++++++++++--\n fast-import.c | 11 ++-\n t/t9300-fast-import.sh | 20 +++++\n t/t9350-fast-export.sh | 78 +++++++++++++++++---\n t/t9350/broken-iso-8859-7-commit-message.txt | 1 +\n t/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n 8 files changed, 163 insertions(+), 17 deletions(-)\n create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\n create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n\nRange-diff:\n1: 37a68a0ffd ! 1: b5dcdab662 t9350: fix encoding test to actually test reencoding\n @@ -2,13 +2,13 @@\n \n t9350: fix encoding test to actually test reencoding\n \n - This test used an author with non-ascii characters in the name, but\n - no special commit message. It then grep'ed for those non-ascii\n - characters, but those are guaranteed to exist regardless of the\n - reencoding process since the reencoding only affects the commit message,\n - not the author or committer names. As such, the test would work even if\n - the re-encoding process simply stripped the commit message entirely.\n - Modify the test to actually check that the reencoding in utf-8 worked.\n + This test used an author with non-ascii characters in the name, but no\n + special commit message. It then grep'ed for those non-ascii characters,\n + but those are guaranteed to exist regardless of the reencoding process\n + since the reencoding only affects the commit message, not the author or\n + committer names. As such, the test would work even if the re-encoding\n + process simply stripped the commit message entirely. Modify the test to\n + actually check that the reencoding into UTF-8 worked.\n \n Signed-off-by: Elijah Newren \n \n @@ -40,7 +40,7 @@\n +\t\t # The commit object, if not re-encoded, would be 240 bytes.\n +\t\t # Removing the \"encoding iso-8859-7\\n\" header drops 20 bytes.\n +\t\t # Re-encoding the Pi character from \\xF0 (\\360) in iso-8859-7\n -+\t\t # to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte. Check for\n ++\t\t # to \\xCF\\x80 (\\317\\200) in UTF-8 adds a byte. Check for\n +\t\t # the expected size.\n +\t\t test 221 -eq \"$(git cat-file -s i18n)\" &&\n +\t\t # ...and for the expected translation of bytes.\n2: 3d84f4613d ! 2: af7d4e18fa fast-import: support 'encoding' commit header\n @@ -2,7 +2,7 @@\n \n fast-import: support 'encoding' commit header\n \n - Since git supports commit messages with an encoding other than utf-8,\n + Since git supports commit messages with an encoding other than UTF-8,\n allow fast-import to import such commits. This may be useful for folks\n who do not want to reencode commit messages from an external system, and\n may also be useful to achieve reversible history rewrites (e.g. sha1sum\n3: baa8394a3a ! 3: d5b300692a fast-export: avoid stripping encoding header if we cannot reencode\n @@ -3,8 +3,8 @@\n fast-export: avoid stripping encoding header if we cannot reencode\n \n When fast-export encounters a commit with an 'encoding' header, it tries\n - to reencode in utf-8 and then drops the encoding header. However, if it\n - fails to reencode in utf-8 because e.g. one of the characters in the\n + to reencode in UTF-8 and then drops the encoding header. However, if it\n + fails to reencode in UTF-8 because e.g. one of the characters in the\n commit message was invalid in the old encoding, then we need to retain\n the original encoding or otherwise we lose information needed to\n understand all the other (valid) characters in the original commit\n4: 49960164c6 ! 4: 2cef40c613 fast-export: differentiate between explicitly utf-8 and implicitly utf-8\n @@ -1,11 +1,11 @@\n Author: Elijah Newren \n \n - fast-export: differentiate between explicitly utf-8 and implicitly utf-8\n + fast-export: differentiate between explicitly UTF-8 and implicitly UTF-8\n \n The find_encoding() function returned the encoding used by a commit\n - message, returning a default of git_commit_encoding (usually utf-8).\n + message, returning a default of git_commit_encoding (usually UTF-8).\n Although the current code does not differentiate between a commit which\n - explicitly requested utf-8 and one where we just assume utf-8 because no\n + explicitly requested UTF-8 and one where we just assume UTF-8 because no\n encoding is set, it will become important when we try to preserve the\n encoding header. Since is_encoding_utf8() returns true when passed\n NULL, we can just return NULL from find_encoding() instead of returning\n5: d8be4ee826 ! 5: d18f03d1bf fast-export: do automatic reencoding of commit messages only if requested\n @@ -62,7 +62,7 @@\n +\t\treencode_mode = REENCODE_YES;\n +\t\tbreak;\n +\tdefault:\n -+\t\tif (arg && !strcasecmp(arg, \"abort\"))\n ++\t\tif (!strcasecmp(arg, \"abort\"))\n +\t\t\treencode_mode = REENCODE_ABORT;\n +\t\telse\n +\t\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n @@ -156,7 +156,7 @@\n +\t\t # The commit object, if not re-encoded, is 240 bytes.\n +\t\t # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n +\t\t # bytes. Re-encoding the Pi character from \\xF0 (\\360) in\n -+\t\t # iso-8859-7 to \\xCF\\x80 (\\317\\200) in utf-8 adds a byte.\n ++\t\t # iso-8859-7 to \\xCF\\x80 (\\317\\200) in UTF-8 adds a byte.\n +\t\t # Check for the expected size...\n +\t\t test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n +\t\t # ...as well as the expected byte.\n-- \n2.21.0.782.gd18f03d1bf\n\n"},{"id":"375467","messageId":"20190514043102.31979-2-newren@gmail.com","threadId":"51017","inReplyTo":"20190514043102.31979-1-newren@gmail.com","subject":"[PATCH v6 1/5] t9350: fix encoding test to actually test reencoding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-14T04:30:58Z","receivedAt":"2019-05-14T04:31:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This test used an author with non-ascii characters in the name, but no\nspecial commit message. It then grep'ed for those non-ascii characters,\nbut those are guaranteed to exist regardless of the reencoding process\nsince the reencoding only affects the commit message, not the author or\ncommitter names. As such, the test would work even if the re-encoding\nprocess simply stripped the commit message entirely. Modify the test to\nactually check that the reencoding into UTF-8 worked.\n\nSigned-off-by: Elijah Newren \n---\n t/t9350-fast-export.sh | 27 ++++++++++++--------\n t/t9350/simple-iso-8859-7-commit-message.txt | 1 +\n 2 files changed, 18 insertions(+), 10 deletions(-)\n create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt\n\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 5690fe2810..756d6a9905 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,22 +94,30 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-1' '\n+test_expect_success 'iso-8859-7' '\n \n-\tgit config i18n.commitencoding ISO8859-1 &&\n-\t# use author and committer name in ISO-8859-1 to match it.\n-\t. \"$TEST_DIRECTORY\"/t3901/8859-1.txt &&\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n-\tgit commit -s -m den file &&\n-\tgit fast-export wer^..wer >iso8859-1.fi &&\n-\tsed \"s/wer/i18n/\" iso8859-1.fi |\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n+\t\t # The commit object, if not re-encoded, would be 240 bytes.\n+\t\t # Removing the \"encoding iso-8859-7\\n\" header drops 20 bytes.\n+\t\t # Re-encoding the Pi character from \\xF0 (\\360) in iso-8859-7\n+\t\t # to \\xCF\\x80 (\\317\\200) in UTF-8 adds a byte. Check for\n+\t\t # the expected size.\n+\t\t test 221 -eq \"$(git cat-file -s i18n)\" &&\n+\t\t # ...and for the expected translation of bytes.\n \t\t git cat-file commit i18n >actual &&\n-\t\t grep \"Áéí óú\" actual)\n-\n+\t\t grep $(printf \"\\317\\200\") actual &&\n+\t\t # Also make sure the commit does not have the \"encoding\" header\n+\t\t ! grep ^encoding actual)\n '\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\n@@ -224,7 +232,6 @@ GIT_COMMITTER_NAME='C O Mitter'; export GIT_COMMITTER_NAME\n \n test_expect_success 'setup copies' '\n \n-\tgit config --unset i18n.commitencoding &&\n \tgit checkout -b copy rein &&\n \tgit mv file file3 &&\n \tgit commit -m move1 &&\ndiff --git a/t/t9350/simple-iso-8859-7-commit-message.txt b/t/t9350/simple-iso-8859-7-commit-message.txt\nnew file mode 100644\nindex 0000000000..8b3f0c3dba\n--- /dev/null\n+++ b/t/t9350/simple-iso-8859-7-commit-message.txt\n@@ -0,0 +1 @@\n+Pi: �\n\\ No newline at end of file\n-- \n2.21.0.782.gd18f03d1bf\n\n"},{"id":"375468","messageId":"20190514043102.31979-6-newren@gmail.com","threadId":"51017","inReplyTo":"20190514043102.31979-1-newren@gmail.com","subject":"[PATCH v6 5/5] fast-export: do automatic reencoding of commit messages only if requested","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-14T04:31:02Z","receivedAt":"2019-05-14T04:31:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Automatic re-encoding of commit messages (and dropping of the encoding\nheader) hurts attempts to do reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions, some subtree rewrites), and seems\ninconsistent with the general principle followed elsewhere in\nfast-export of requiring explicit user requests to modify the output\n(e.g. --signed-tags=strip, --tag-of-filtered-object=rewrite). Add a\n--reencode flag that the user can use to specify, and like other\nfast-export flags, default it to 'abort'.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 7 +++++\n builtin/fast-export.c | 46 +++++++++++++++++++++++++++++--\n t/t9350-fast-export.sh | 38 +++++++++++++++++++++++--\n 3 files changed, 85 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex 64c01ba918..11427acdde 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -129,6 +129,13 @@ marks the same across runs.\n \tfor intermediary filters (e.g. for rewriting commit messages\n \twhich refer to older commits, or for stripping blobs by id).\n \n+--reencode=(yes|no|abort)::\n+\tSpecify how to handle `encoding` header in commit objects. When\n+\tasking to 'abort' (which is the default), this program will die\n+\twhen encountering such a commit object. With 'yes', the commit\n+\tmessage will be reencoded into UTF-8. With 'no', the original\n+\tencoding will be preserved.\n+\n --refspec::\n \tApply the specified refspec to each ref exported. Multiple of them can\n \tbe specified.\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 66331fa401..c22cef3b2f 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -33,6 +33,7 @@ static const char *fast_export_usage[] = {\n static int progress;\n static enum { SIGNED_TAG_ABORT, VERBATIM, WARN, WARN_STRIP, STRIP } signed_tag_mode = SIGNED_TAG_ABORT;\n static enum { TAG_FILTERING_ABORT, DROP, REWRITE } tag_of_filtered_mode = TAG_FILTERING_ABORT;\n+static enum { REENCODE_ABORT, REENCODE_YES, REENCODE_NO } reencode_mode = REENCODE_ABORT;\n static int fake_missing_tagger;\n static int use_done_feature;\n static int no_data;\n@@ -77,6 +78,31 @@ static int parse_opt_tag_of_filtered_mode(const struct option *opt,\n \treturn 0;\n }\n \n+static int parse_opt_reencode_mode(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tif (unset) {\n+\t\treencode_mode = REENCODE_ABORT;\n+\t\treturn 0;\n+\t}\n+\n+\tswitch (git_parse_maybe_bool(arg)) {\n+\tcase 0:\n+\t\treencode_mode = REENCODE_NO;\n+\t\tbreak;\n+\tcase 1:\n+\t\treencode_mode = REENCODE_YES;\n+\t\tbreak;\n+\tdefault:\n+\t\tif (!strcasecmp(arg, \"abort\"))\n+\t\t\treencode_mode = REENCODE_ABORT;\n+\t\telse\n+\t\t\treturn error(\"Unknown reencoding mode: %s\", arg);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static struct decoration idnums;\n static uint32_t last_idnum;\n \n@@ -633,10 +659,21 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \t}\n \n \tmark_next_object(&commit->object);\n-\tif (anonymize)\n+\tif (anonymize) {\n \t\treencoded = anonymize_commit_message(message);\n-\telse if (!is_encoding_utf8(encoding))\n-\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t} else if (encoding) {\n+\t\tswitch(reencode_mode) {\n+\t\tcase REENCODE_YES:\n+\t\t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n+\t\t\tbreak;\n+\t\tcase REENCODE_NO:\n+\t\t\tbreak;\n+\t\tcase REENCODE_ABORT:\n+\t\t\tdie(\"Encountered commit-specific encoding %s in commit \"\n+\t\t\t \"%s; use --reencode=[yes|no] to handle it\",\n+\t\t\t encoding, oid_to_hex(&commit->object.oid));\n+\t\t}\n+\t}\n \tif (!commit->parents)\n \t\tprintf(\"reset %s\\n\", refname);\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n@@ -1091,6 +1128,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\tOPT_CALLBACK(0, \"tag-of-filtered-object\", &tag_of_filtered_mode, N_(\"mode\"),\n \t\t\t N_(\"select handling of tags that tag filtered objects\"),\n \t\t\t parse_opt_tag_of_filtered_mode),\n+\t\tOPT_CALLBACK(0, \"reencode\", &reencode_mode, N_(\"mode\"),\n+\t\t\t N_(\"select handling of commit messages in an alternate encoding\"),\n+\t\t\t parse_opt_reencode_mode),\n \t\tOPT_STRING(0, \"export-marks\", &export_filename, N_(\"file\"),\n \t\t\t N_(\"Dump marks to this file\")),\n \t\tOPT_STRING(0, \"import-marks\", &import_filename, N_(\"file\"),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex e2ab8eddc0..b4004e05c2 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -94,14 +94,14 @@ test_expect_success 'fast-export --show-original-ids | git fast-import' '\n \ttest $MUSS = $(git rev-parse --verify refs/tags/muss)\n '\n \n-test_expect_success 'iso-8859-7' '\n+test_expect_success 'reencoding iso-8859-7' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \ttest_tick &&\n \techo rosten >file &&\n \tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n@@ -118,13 +118,45 @@ test_expect_success 'iso-8859-7' '\n \t\t ! grep ^encoding actual)\n '\n \n+test_expect_success 'aborting on iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\ttest_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi\n+'\n+\n+test_expect_success 'preserving iso-8859-7' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export --reencode=no wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-no-recoding/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t # The commit object, if not re-encoded, is 240 bytes.\n+\t\t # Removing the \"encoding iso-8859-7\\n\" header would drops 20\n+\t\t # bytes. Re-encoding the Pi character from \\xF0 (\\360) in\n+\t\t # iso-8859-7 to \\xCF\\x80 (\\317\\200) in UTF-8 adds a byte.\n+\t\t # Check for the expected size...\n+\t\t test 240 -eq \"$(git cat-file -s i18n-no-recoding)\" &&\n+\t\t # ...as well as the expected byte.\n+\t\t git cat-file commit i18n-no-recoding >actual &&\n+\t\t grep $(printf \"\\360\") actual &&\n+\t\t # Also make sure the commit has the \"encoding\" header\n+\t\t grep ^encoding actual)\n+'\n+\n test_expect_success 'encoding preserved if reencoding fails' '\n \n \ttest_when_finished \"git reset --hard HEAD~1\" &&\n \ttest_config i18n.commitencoding iso-8859-7 &&\n \techo rosten >file &&\n \tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n-\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tgit fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&\n \tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n \t\t(cd new &&\n \t\t git fast-import &&\n-- \n2.21.0.782.gd18f03d1bf\n\n"},{"id":"375469","messageId":"20190514043102.31979-3-newren@gmail.com","threadId":"51017","inReplyTo":"20190514043102.31979-1-newren@gmail.com","subject":"[PATCH v6 2/5] fast-import: support 'encoding' commit header","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-14T04:30:59Z","receivedAt":"2019-05-14T04:31:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Since git supports commit messages with an encoding other than UTF-8,\nallow fast-import to import such commits. This may be useful for folks\nwho do not want to reencode commit messages from an external system, and\nmay also be useful to achieve reversible history rewrites (e.g. sha1sum\n<-> sha256sum transitions or subtree work) with git repositories that\nhave used specialized encodings in their commit history.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-import.txt | 7 +++++++\n fast-import.c | 11 +++++++++--\n t/t9300-fast-import.sh | 20 ++++++++++++++++++++\n 3 files changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex d65cdb3d08..7baf9e47b5 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -388,6 +388,7 @@ change to the project.\n \toriginal-oid?\n \t('author' (SP )? SP LT GT SP LF)?\n \t'committer' (SP )? SP LT GT SP LF\n+\t('encoding' SP )?\n \tdata\n \t('from' SP LF)?\n \t('merge' SP LF)?\n@@ -455,6 +456,12 @@ that was selected by the --date-format= command-line option.\n See ``Date Formats'' above for the set of supported formats, and\n their syntax.\n \n+`encoding`\n+^^^^^^^^^^\n+The optional `encoding` command indicates the encoding of the commit\n+message. Most commits are UTF-8 and the encoding is omitted, but this\n+allows importing commit messages into git without first reencoding them.\n+\n `from`\n ^^^^^^\n The `from` command is used to specify the commit to initialize\ndiff --git a/fast-import.c b/fast-import.c\nindex f38d04fa58..76a7bd3699 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2585,6 +2585,7 @@ static void parse_new_commit(const char *arg)\n \tstruct branch *b;\n \tchar *author = NULL;\n \tchar *committer = NULL;\n+\tconst char *encoding = NULL;\n \tstruct hash_list *merge_list = NULL;\n \tunsigned int merge_count;\n \tunsigned char prev_fanout, new_fanout;\n@@ -2607,6 +2608,8 @@ static void parse_new_commit(const char *arg)\n \t}\n \tif (!committer)\n \t\tdie(\"Expected committer but didn't get one\");\n+\tif (skip_prefix(command_buf.buf, \"encoding \", &encoding))\n+\t\tread_next_command();\n \tparse_data(&msg, 0, NULL);\n \tread_next_command();\n \tparse_from(b);\n@@ -2670,9 +2673,13 @@ static void parse_new_commit(const char *arg)\n \t}\n \tstrbuf_addf(&new_data,\n \t\t\"author %s\\n\"\n-\t\t\"committer %s\\n\"\n-\t\t\"\\n\",\n+\t\t\"committer %s\\n\",\n \t\tauthor ? author : committer, committer);\n+\tif (encoding)\n+\t\tstrbuf_addf(&new_data,\n+\t\t\t\"encoding %s\\n\",\n+\t\t\tencoding);\n+\tstrbuf_addch(&new_data, '\\n');\n \tstrbuf_addbuf(&new_data, &msg);\n \tfree(author);\n \tfree(committer);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 3668263c40..141b7fa35e 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3299,4 +3299,24 @@ test_expect_success !MINGW 'W: get-mark & empty orphan commit with erroneous thi\n \tsed -e s/LFs/LLL/ W-input | tr L \"\\n\" | test_must_fail git fast-import\n '\n \n+###\n+### series X (other new features)\n+###\n+\n+test_expect_success 'X: handling encoding' '\n+\ttest_tick &&\n+\tcat >input <<-INPUT_END &&\n+\tcommit refs/heads/encoding\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tencoding iso-8859-7\n+\tdata <>input &&\n+\n+\tgit fast-import \n---\n builtin/fast-export.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 7734a9f5a5..66331fa401 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -453,7 +453,7 @@ static const char *find_encoding(const char *begin, const char *end)\n \tbol = memmem(begin, end ? end - begin : strlen(begin),\n \t\t needle, strlen(needle));\n \tif (!bol)\n-\t\treturn git_commit_encoding;\n+\t\treturn NULL;\n \tbol += strlen(needle);\n \teol = strchrnul(bol, '\\n');\n \t*eol = '\\0';\n-- \n2.21.0.782.gd18f03d1bf\n\n"},{"id":"375471","messageId":"20190514043102.31979-4-newren@gmail.com","threadId":"51017","inReplyTo":"20190514043102.31979-1-newren@gmail.com","subject":"[PATCH v6 3/5] fast-export: avoid stripping encoding header if we cannot reencode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-05-14T04:31:00Z","receivedAt":"2019-05-14T04:31:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"When fast-export encounters a commit with an 'encoding' header, it tries\nto reencode in UTF-8 and then drops the encoding header. However, if it\nfails to reencode in UTF-8 because e.g. one of the characters in the\ncommit message was invalid in the old encoding, then we need to retain\nthe original encoding or otherwise we lose information needed to\nunderstand all the other (valid) characters in the original commit\nmessage.\n\nSigned-off-by: Elijah Newren \n---\n builtin/fast-export.c | 7 +++++--\n t/t9350-fast-export.sh | 21 ++++++++++++++++++++\n t/t9350/broken-iso-8859-7-commit-message.txt | 1 +\n 3 files changed, 27 insertions(+), 2 deletions(-)\n create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt\n\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 9e283482ef..7734a9f5a5 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -642,9 +642,12 @@ static void handle_commit(struct commit *commit, struct rev_info *rev,\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n\", refname, last_idnum);\n \tif (show_original_ids)\n \t\tprintf(\"original-oid %s\\n\", oid_to_hex(&commit->object.oid));\n-\tprintf(\"%.*s\\n%.*s\\ndata %u\\n%s\",\n+\tprintf(\"%.*s\\n%.*s\\n\",\n \t (int)(author_end - author), author,\n-\t (int)(committer_end - committer), committer,\n+\t (int)(committer_end - committer), committer);\n+\tif (!reencoded && encoding)\n+\t\tprintf(\"encoding %s\\n\", encoding);\n+\tprintf(\"data %u\\n%s\",\n \t (unsigned)(reencoded\n \t\t\t ? strlen(reencoded) : message\n \t\t\t ? strlen(message) : 0),\ndiff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh\nindex 756d6a9905..e2ab8eddc0 100755\n--- a/t/t9350-fast-export.sh\n+++ b/t/t9350-fast-export.sh\n@@ -118,6 +118,27 @@ test_expect_success 'iso-8859-7' '\n \t\t ! grep ^encoding actual)\n '\n \n+test_expect_success 'encoding preserved if reencoding fails' '\n+\n+\ttest_when_finished \"git reset --hard HEAD~1\" &&\n+\ttest_config i18n.commitencoding iso-8859-7 &&\n+\techo rosten >file &&\n+\tgit commit -s -F \"$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt\" file &&\n+\tgit fast-export wer^..wer >iso-8859-7.fi &&\n+\tsed \"s/wer/i18n-invalid/\" iso-8859-7.fi |\n+\t\t(cd new &&\n+\t\t git fast-import &&\n+\t\t git cat-file commit i18n-invalid >actual &&\n+\t\t # Make sure the commit still has the encoding header\n+\t\t grep ^encoding actual &&\n+\t\t # Verify that the commit has the expected size; i.e.\n+\t\t # that no bytes were re-encoded to a different encoding.\n+\t\t test 252 -eq \"$(git cat-file -s i18n-invalid)\" &&\n+\t\t # ...and check for the original special bytes\n+\t\t grep $(printf \"\\360\") actual &&\n+\t\t grep $(printf \"\\377\") actual)\n+'\n+\n test_expect_success 'import/export-marks' '\n \n \tgit checkout -b marks master &&\ndiff --git a/t/t9350/broken-iso-8859-7-commit-message.txt b/t/t9350/broken-iso-8859-7-commit-message.txt\nnew file mode 100644\nindex 0000000000..d06ad75b44\n--- /dev/null\n+++ b/t/t9350/broken-iso-8859-7-commit-message.txt\n@@ -0,0 +1 @@\n+Pi: �; Invalid: �\n\\ No newline at end of file\n-- \n2.21.0.782.gd18f03d1bf\n\n"},{"id":"375731","messageId":"20190516181502.zjucws2d5hw4sviv@tb-raspi4","threadId":"51017","inReplyTo":"20190514043102.31979-1-newren@gmail.com","subject":"Re: [PATCH v6 0/5] Fix and extend encoding handling in fast export/import","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-05-16T18:15:02Z","receivedAt":"2019-05-16T18:15:12Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, May 13, 2019 at 09:30:57PM -0700, Elijah Newren wrote:\n> While stress testing `git filter-repo`, I noticed an issue with\n> encoding; further digging led to the fixes and features in this series.\n> See the individual commit messages for details.\n>\n\nNo more comments from my side, thanks for the patience.\n"}]}