git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 0/5] Fix and extend encoding handling in fast export/import

From
Elijah Newren <newren@gmail.com>
Date
May 10, 2019, 20:53 UTC
Message-ID
<20190510205335.19968-1-newren@gmail.com>
In-Reply-To
<20190430182523.3339-1-newren@gmail.com>

While stress testing `git filter-repo`, I noticed an issue with encoding; further digging led to the fixes and features in this series. See the individual commit messages for details.

Changes since v2 (full range-diff below):
  * Modified the testcases to pass on Windows[1], as verified via
    gitgitgadget pull request[2].  Required adding a couple new files
    (which store the desired bytes) and checking the size of the output
    instead of checking for particular bytes (but the lengths of the
    expected byte sequences differ so this works fine...).
[1] Failures of previous patchset on Windows noticed and reported by Dscho;
    explanation from Hannes is that Windows munges users' command lines to
    force them to be characters instead of bytes.
[2] https://github.com/gitgitgadget/git/pull/187

Elijah Newren (5): t9350: fix encoding test to actually test reencoding fast-import: support 'encoding' commit header fast-export: avoid stripping encoding header if we cannot reencode fast-export: differentiate between explicitly utf-8 and implicitly utf-8 fast-export: do automatic reencoding of commit messages only if requested

Documentation/git-fast-import.txt | 7 ++ builtin/fast-export.c | 44 ++++++++++-- fast-import.c | 11 ++- t/t9300-fast-import.sh | 20 ++++++ t/t9350-fast-export.sh | 75 +++++++++++++++++--- t/t9350/broken-iso-8859-7-commit-message.txt | 1 + t/t9350/simple-iso-8859-7-commit-message.txt | 1 + 7 files changed, 142 insertions(+), 17 deletions(-) create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt

Range-diff:
1:  9cc04242bd ! 1:  2d7bb64acf t9350: fix encoding test to actually test reencoding
    @@ -32,15 +32,26 @@
     -	git commit -s -m den file &&
     -	git fast-export wer^..wer >iso8859-1.fi &&
     -	sed "s/wer/i18n/" iso8859-1.fi |
    -+	git commit -s -m "$(printf "Pi: \360")" file &&
    ++	git commit -s -F "$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt" file &&
     +	git fast-export wer^..wer >iso-8859-7.fi &&
     +	sed "s/wer/i18n/" iso-8859-7.fi |
      		(cd new &&
      		 git fast-import &&
    ++		 # The commit object, if not re-encoded, would be 240 bytes.
    ++		 # Removing the "encoding iso-8859-7\n" header drops 20 bytes.
    ++		 # Re-encoding the Pi character from \xF0 in iso-8859-7 to
    ++		 # \xCF\x80 in utf-8 adds a byte.  Grepping for specific bytes
    ++		 # would be nice, but Windows apparently munges user data
    ++		 # in the form of bytes on the command line to force them to
    ++		 # be characters instead, so we are limited for portability
    ++		 # reasons in subsequent similar tests in this file to check
    ++		 # for size rather than what bytes are present.
    ++		 test 221 -eq "$(git cat-file -s i18n)" &&
    ++		 # Also make sure the commit does not have the "encoding" header
      		 git cat-file commit i18n >actual &&
     -		 grep "Áéí óú" actual)
     -
    -+		 grep $(printf "\317\200") actual)
    ++		 ! grep ^encoding actual)
      '
     +
      test_expect_success 'import/export-marks' '
    @@ -54,3 +65,11 @@
      	git checkout -b copy rein &&
      	git mv file file3 &&
      	git commit -m move1 &&
    +
    + diff --git a/t/t9350/simple-iso-8859-7-commit-message.txt b/t/t9350/simple-iso-8859-7-commit-message.txt
    + new file mode 100644
    + --- /dev/null
    + +++ b/t/t9350/simple-iso-8859-7-commit-message.txt
    +@@
    ++Pi: �
    + \ No newline at end of file
2:  0cd023ac7a = 2:  9fa5695017 fast-import: support 'encoding' commit header
3:  1fddf51402 ! 3:  dfc76573e9 fast-export: avoid stripping encoding header if we cannot reencode
    @@ -35,7 +35,7 @@
      --- a/t/t9350-fast-export.sh
      +++ b/t/t9350-fast-export.sh
     @@
    - 		 grep $(printf "\317\200") actual)
    + 		 ! grep ^encoding actual)
      '
      
     +test_expect_success 'encoding preserved if reencoding fails' '
    @@ -43,15 +43,26 @@
     +	test_when_finished "git reset --hard HEAD~1" &&
     +	test_config i18n.commitencoding iso-8859-7 &&
     +	echo rosten >file &&
    -+	git commit -s -m "$(printf "Pi: \360; Invalid: \377")" file &&
    ++	git commit -s -F "$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt" file &&
     +	git fast-export wer^..wer >iso-8859-7.fi &&
     +	sed "s/wer/i18n-invalid/" iso-8859-7.fi |
     +		(cd new &&
     +		 git fast-import &&
     +		 git cat-file commit i18n-invalid >actual &&
    -+		 grep ^encoding actual)
    ++		 grep ^encoding actual &&
    ++		 # Also verify that the commit has the expected size; i.e.
    ++		 # that no bytes were re-encoded to a different encoding.
    ++		 test 252 -eq "$(git cat-file -s i18n-invalid)")
     +'
     +
      test_expect_success 'import/export-marks' '
      
      	git checkout -b marks master &&
    +
    + diff --git a/t/t9350/broken-iso-8859-7-commit-message.txt b/t/t9350/broken-iso-8859-7-commit-message.txt
    + new file mode 100644
    + --- /dev/null
    + +++ b/t/t9350/broken-iso-8859-7-commit-message.txt
    +@@
    ++Pi: �; Invalid: �
    + \ No newline at end of file
4:  4a2e04b3ae = 4:  83b3656b76 fast-export: differentiate between explicitly utf-8 and implicitly utf-8
5:  44aacb1a0b ! 5:  2063122293 fast-export: do automatic reencoding of commit messages only if requested
    @@ -95,14 +95,14 @@
      	test_config i18n.commitencoding iso-8859-7 &&
      	test_tick &&
      	echo rosten >file &&
    - 	git commit -s -m "$(printf "Pi: \360")" file &&
    + 	git commit -s -F "$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt" file &&
     -	git fast-export wer^..wer >iso-8859-7.fi &&
     +	git fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&
      	sed "s/wer/i18n/" iso-8859-7.fi |
      		(cd new &&
      		 git fast-import &&
     @@
    - 		 grep $(printf "\317\200") actual)
    + 		 ! grep ^encoding actual)
      '
      
     +test_expect_success 'aborting on iso-8859-7' '
    @@ -110,7 +110,7 @@
     +	test_when_finished "git reset --hard HEAD~1" &&
     +	test_config i18n.commitencoding iso-8859-7 &&
     +	echo rosten >file &&
    -+	git commit -s -m "$(printf "Pi: \360")" file &&
    ++	git commit -s -F "$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt" file &&
     +	test_must_fail git fast-export --reencode=abort wer^..wer >iso-8859-7.fi
     +'
     +
    @@ -119,13 +119,21 @@
     +	test_when_finished "git reset --hard HEAD~1" &&
     +	test_config i18n.commitencoding iso-8859-7 &&
     +	echo rosten >file &&
    -+	git commit -s -m "$(printf "Pi: \360")" file &&
    ++	git commit -s -F "$TEST_DIRECTORY/t9350/simple-iso-8859-7-commit-message.txt" file &&
     +	git fast-export --reencode=no wer^..wer >iso-8859-7.fi &&
     +	sed "s/wer/i18n-no-recoding/" iso-8859-7.fi |
     +		(cd new &&
     +		 git fast-import &&
    ++		 # The commit object, if not re-encoded, is 240 bytes.
    ++		 # Removing the "encoding iso-8859-7\n" header would drops 20
    ++		 # bytes.  Re-encoding the Pi character from \xF0 in
    ++		 # iso-8859-7 to \xCF\x80 in utf-8 would add a byte.  I would
    ++		 # grep for the # specific bytes, but Windows lamely does not
    ++		 # allow that, so just search for the expected size.
    ++		 test 240 -eq "$(git cat-file -s i18n-no-recoding)" &&
    ++		 # Also make sure the commit has the "encoding" header
     +		 git cat-file commit i18n-no-recoding >actual &&
    -+		 grep $(printf "\360") actual)
    ++		 grep ^encoding actual)
     +'
     +
      test_expect_success 'encoding preserved if reencoding fails' '
    @@ -133,7 +141,7 @@
      	test_when_finished "git reset --hard HEAD~1" &&
      	test_config i18n.commitencoding iso-8859-7 &&
      	echo rosten >file &&
    - 	git commit -s -m "$(printf "Pi: \360; Invalid: \377")" file &&
    + 	git commit -s -F "$TEST_DIRECTORY/t9350/broken-iso-8859-7-commit-message.txt" file &&
     -	git fast-export wer^..wer >iso-8859-7.fi &&
     +	git fast-export --reencode=yes wer^..wer >iso-8859-7.fi &&
      	sed "s/wer/i18n-invalid/" iso-8859-7.fi |
-- 
2.21.0.782.g2063122293
Previous: Elijah NewrenNext: Elijah Newren
Message 7 of 45 in “Fix and extend encoding handling in fast export/import”
  1. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, Apr 30, 2019
  2. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, Apr 30, 2019
  3. 2/5 fast-import: support 'encoding' commit headerElijah Newren, Apr 30, 2019
  4. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, Apr 30, 2019
  5. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, Apr 30, 2019
  6. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, Apr 30, 2019
  7. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 10, 2019
  8. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 10, 2019
  9. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 10, 2019
  10. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 10, 2019
  11. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 10, 2019
  12. Torsten BögershausenMay 11, 2019
  13. Elijah NewrenMay 11, 2019
  14. Junio C HamanoMay 13, 2019
  15. Elijah NewrenMay 13, 2019
  16. Johannes SchindelinMay 13, 2019
  17. Torsten BögershausenMay 13, 2019
  18. Elijah NewrenMay 13, 2019
  19. Elijah NewrenMay 13, 2019
  20. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 10, 2019
  21. Johannes SchindelinMay 13, 2019
  22. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 13, 2019
  23. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 13, 2019
  24. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 13, 2019
  25. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 13, 2019
  26. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 13, 2019
  27. Junio C HamanoMay 13, 2019
  28. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 13, 2019
  29. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 13, 2019
  30. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 13, 2019
  31. Torsten BögershausenMay 14, 2019
  32. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 13, 2019
  33. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 13, 2019
  34. Torsten BögershausenMay 14, 2019
  35. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 13, 2019
  36. Torsten BögershausenMay 14, 2019
  37. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 13, 2019
  38. Eric SunshineMay 14, 2019
  39. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 14, 2019
  40. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 14, 2019
  41. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 14, 2019
  42. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 14, 2019
  43. 4/5 fast-export: differentiate between explicitly UTF-8 and implicitly UTF-8Elijah Newren, May 14, 2019
  44. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 14, 2019
  45. Torsten BögershausenMay 16, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.