{"thread":{"id":"57655","subject":"[PATCH] git-p4: preserve utf8 BOM when importing from p4 to git","startedAt":"2022-04-04T05:50:54Z","lastAt":"2022-04-04T05:50:54Z","messageCount":1,"participants":["Tao Klerks via GitGitGadget"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"453019","messageId":"pull.1203.git.1649051436934.gitgitgadget@gmail.com","threadId":"57655","inReplyTo":null,"subject":"[PATCH] git-p4: preserve utf8 BOM when importing from p4 to git","fromName":"Tao Klerks via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-04T05:50:36Z","receivedAt":"2022-04-04T05:50:54Z","isPatch":true,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"From: Tao Klerks <tao@klerks.biz>\n\nPerforce has a file type \"utf8\" which represents a text file with\nexplicit BOM. utf8-encoded files *without* BOM are stored as\nregular file type \"text\". The \"utf8\" file type behaves like text\nin all but one important way: it is stored, internally, without\nthe leading 3 BOM bytes.\n\ngit-p4 has historically imported utf8-with-BOM files (files stored,\nin Perforce, as type \"utf8\") the same way as regular text files -\nlosing the BOM in the process.\n\nUnder most circumstances this issue has little functional impact,\nas most systems consider the BOM to be optional and redundant, but\nthis *is* a correctness failure, and can have lead to practical\nissues for example when BOMs are explicitly included in test files,\nfor example in a file encoding test suite.\n\nFix the handling of utf8-with-BOM files when importing changes from\np4 to git, and introduce a test that checks it is working correctly.\n\nSigned-off-by: Tao Klerks <tao@klerks.biz>\n---\n    git-p4: preserve utf8 BOM when importing from p4 to git\n    \n    I manually tested these changes with python2 and python3 - I don't know\n    whether there is a more rigorous approach possible than changing the\n    system default python and rerunning the \"t98xx\" tests, but that did work\n    (and initially highlighted an issue, now fixed).\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1203%2FTaoK%2Fgit-p4-utf8-bom-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1203/TaoK/git-p4-utf8-bom-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1203\n\n git-p4.py                  | 10 ++++++++++\n t/t9802-git-p4-filetype.sh | 34 ++++++++++++++++++++++++++++++++++\n 2 files changed, 44 insertions(+)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex a9b1f904410..6d932e7ed76 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2885,6 +2885,16 @@ class P4Sync(Command, P4UserMap):\n             print(\"\\nIgnoring apple filetype file %s\" % file['depotFile'])\n             return\n \n+        if type_base == \"utf8\":\n+            # The type utf8 explicitly means utf8 *with BOM*. These are\n+            # streamed just like regular text files, however, without\n+            # the BOM in the stream.\n+            # Therefore, to accurately import these files into git, we\n+            # need to explicitly re-add the BOM before writing.\n+            # 'contents' is a set of bytes in this case, so create the\n+            # BOM prefix as a b'' literal.\n+            contents = [b'\\xef\\xbb\\xbf' + contents[0]] + contents[1:]\n+\n         # Note that we do not try to de-mangle keywords on utf16 files,\n         # even though in theory somebody may want that.\n         regexp = p4_keywords_regexp_for_type(type_base, type_mods)\ndiff --git a/t/t9802-git-p4-filetype.sh b/t/t9802-git-p4-filetype.sh\nindex 19073c6e9f8..2a6ee2a4678 100755\n--- a/t/t9802-git-p4-filetype.sh\n+++ b/t/t9802-git-p4-filetype.sh\n@@ -333,4 +333,38 @@ test_expect_success SYMLINKS 'empty symlink target' '\n \t)\n '\n \n+test_expect_success SYMLINKS 'utf-8 with and without BOM in text file' '\n+\t(\n+\t\tcd \"$cli\" &&\n+\n+\t\t# some utf8 content\n+\t\techo some tǣxt >utf8-nobom-test &&\n+\n+\t\t# same utf8 content as before but with bom\n+\t\techo some tǣxt | sed '\\''s/^/\\xef\\xbb\\xbf/'\\'' >utf8-bom-test &&\n+\n+\t\t# bom only\n+\t\tdd bs=1 count=3 if=utf8-bom-test of=utf8-bom-empty-test &&\n+\n+\t\tp4 add utf8-nobom-test utf8-bom-test utf8-bom-empty-test &&\n+\t\tp4 submit -d \"add utf8 test files\"\n+\t) &&\n+\ttest_when_finished cleanup_git &&\n+\n+\tgit p4 clone --dest=\"$git\" //depot@all &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit checkout refs/remotes/p4/master &&\n+\n+\t\techo some tǣxt >utf8-nobom-check &&\n+\t\ttest_cmp utf8-nobom-check utf8-nobom-test &&\n+\n+\t\techo some tǣxt | sed '\\''s/^/\\xef\\xbb\\xbf/'\\'' >utf8-bom-check &&\n+\t\ttest_cmp utf8-bom-check utf8-bom-test &&\n+\n+\t\tdd bs=1 count=3 if=utf8-bom-check of=utf8-bom-empty-check &&\n+\t\ttest_cmp utf8-bom-empty-check utf8-bom-empty-test\n+\t)\n+'\n+\n test_done\n\nbase-commit: 4b6846d9dcd391164b72bd70e8a0c0e09776afe3\n-- \ngitgitgadget\n"}]}