git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v5 2/5] fast-import: support 'encoding' commit header

From
Elijah Newren <newren@gmail.com>
Date
May 13, 2019, 23:17 UTC
Message-ID
<20190513231726.16218-3-newren@gmail.com>
In-Reply-To
<20190513231726.16218-1-newren@gmail.com>

Since git supports commit messages with an encoding other than utf-8, allow fast-import to import such commits. This may be useful for folks who do not want to reencode commit messages from an external system, and may also be useful to achieve reversible history rewrites (e.g. sha1sum <-> sha256sum transitions or subtree work) with git repositories that have used specialized encodings in their commit history.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 Documentation/git-fast-import.txt |  7 +++++++
 fast-import.c                     | 11 +++++++++--
 t/t9300-fast-import.sh            | 20 ++++++++++++++++++++
 3 files changed, 36 insertions(+), 2 deletions(-)
diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index d65cdb3d08..7baf9e47b5 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -388,6 +388,7 @@ change to the project.
 	original-oid?
 	('author' (SP <name>)? SP LT <email> GT SP <when> LF)?
 	'committer' (SP <name>)? SP LT <email> GT SP <when> LF
+	('encoding' SP <encoding>)?
 	data
 	('from' SP <commit-ish> LF)?
 	('merge' SP <commit-ish> LF)?
@@ -455,6 +456,12 @@ that was selected by the --date-format=<fmt> command-line option.
 See ``Date Formats'' above for the set of supported formats, and
 their syntax.
 
+`encoding`
+^^^^^^^^^^
+The optional `encoding` command indicates the encoding of the commit
+message.  Most commits are UTF-8 and the encoding is omitted, but this
+allows importing commit messages into git without first reencoding them.
+
 `from`
 ^^^^^^
 The `from` command is used to specify the commit to initialize
diff --git a/fast-import.c b/fast-import.c
index f38d04fa58..76a7bd3699 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -2585,6 +2585,7 @@ static void parse_new_commit(const char *arg)
 	struct branch *b;
 	char *author = NULL;
 	char *committer = NULL;
+	const char *encoding = NULL;
 	struct hash_list *merge_list = NULL;
 	unsigned int merge_count;
 	unsigned char prev_fanout, new_fanout;
@@ -2607,6 +2608,8 @@ static void parse_new_commit(const char *arg)
 	}
 	if (!committer)
 		die("Expected committer but didn't get one");
+	if (skip_prefix(command_buf.buf, "encoding ", &encoding))
+		read_next_command();
 	parse_data(&msg, 0, NULL);
 	read_next_command();
 	parse_from(b);
@@ -2670,9 +2673,13 @@ static void parse_new_commit(const char *arg)
 	}
 	strbuf_addf(&new_data,
 		"author %s\n"
-		"committer %s\n"
-		"\n",
+		"committer %s\n",
 		author ? author : committer, committer);
+	if (encoding)
+		strbuf_addf(&new_data,
+			"encoding %s\n",
+			encoding);
+	strbuf_addch(&new_data, '\n');
 	strbuf_addbuf(&new_data, &msg);
 	free(author);
 	free(committer);
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 3668263c40..141b7fa35e 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -3299,4 +3299,24 @@ test_expect_success !MINGW 'W: get-mark & empty orphan commit with erroneous thi
 	sed -e s/LFs/LLL/ W-input | tr L "\n" | test_must_fail git fast-import
 '
 
+###
+### series X (other new features)
+###
+
+test_expect_success 'X: handling encoding' '
+	test_tick &&
+	cat >input <<-INPUT_END &&
+	commit refs/heads/encoding
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	encoding iso-8859-7
+	data <<COMMIT
+	INPUT_END
+
+	printf "Pi: \360\nCOMMIT\n" >>input &&
+
+	git fast-import <input &&
+	git cat-file -p encoding | grep $(printf "\360") &&
+	git log -1 --format=%B encoding | grep $(printf "\317\200")
+'
+
 test_done

-- 
2.21.0.782.gd8be4ee826
Previous: Torsten BögershausenNext: Elijah Newren
Message 32 of 45 in “Fix and extend encoding handling in fast export/import”
  1. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, Apr 30, 2019
  2. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, Apr 30, 2019
  3. 2/5 fast-import: support 'encoding' commit headerElijah Newren, Apr 30, 2019
  4. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, Apr 30, 2019
  5. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, Apr 30, 2019
  6. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, Apr 30, 2019
  7. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 10, 2019
  8. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 10, 2019
  9. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 10, 2019
  10. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 10, 2019
  11. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 10, 2019
  12. Torsten BögershausenMay 11, 2019
  13. Elijah NewrenMay 11, 2019
  14. Junio C HamanoMay 13, 2019
  15. Elijah NewrenMay 13, 2019
  16. Johannes SchindelinMay 13, 2019
  17. Torsten BögershausenMay 13, 2019
  18. Elijah NewrenMay 13, 2019
  19. Elijah NewrenMay 13, 2019
  20. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 10, 2019
  21. Johannes SchindelinMay 13, 2019
  22. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 13, 2019
  23. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 13, 2019
  24. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 13, 2019
  25. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 13, 2019
  26. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 13, 2019
  27. Junio C HamanoMay 13, 2019
  28. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 13, 2019
  29. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 13, 2019
  30. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 13, 2019
  31. Torsten BögershausenMay 14, 2019
  32. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 13, 2019
  33. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 13, 2019
  34. Torsten BögershausenMay 14, 2019
  35. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 13, 2019
  36. Torsten BögershausenMay 14, 2019
  37. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 13, 2019
  38. Eric SunshineMay 14, 2019
  39. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 14, 2019
  40. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 14, 2019
  41. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 14, 2019
  42. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 14, 2019
  43. 4/5 fast-export: differentiate between explicitly UTF-8 and implicitly UTF-8Elijah Newren, May 14, 2019
  44. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 14, 2019
  45. Torsten BögershausenMay 16, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.