{"thread":{"id":"28170","subject":"[PATCH] git-p4: don't convert utf16 files.","startedAt":"2011-08-19T22:50:20Z","lastAt":"2011-08-21T19:09:15Z","messageCount":3,"participants":["Chris Li","Pete Wyckoff"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"173896","messageId":"CANeU7QmGJTr4V-tWzuPnP2P=9eQbGDgQxv2phAVbGXjU5tqYPQ@mail.gmail.com","threadId":"28170","inReplyTo":null,"subject":"[PATCH] git-p4: don't convert utf16 files.","fromName":"Chris Li","fromEmail":"christ.li@gmail.com","sentAt":"2011-08-19T22:50:20Z","receivedAt":"2011-08-19T22:50:20Z","isPatch":true,"sender":{"key":"christ.li@gmail.com","avatar":null},"body":"Some repository has some utf16 files git-p4 don't know\nhow to convert. For those files, git-p4 just write the utf8\nfiles. That is wrong, because git get different file than\nperforce does, causing some windows resource file fail\nto compile.\n\nUsing the \"p4 print -o tmpfile depotfile\" can avoid this\nconvertion (and possible failure) all together.\n\nSigned-off-by: Chris Li <git@chrisli.org>\n\n---\n git-p4 |   11 +++++------\n 1 files changed, 5 insertions(+), 6 deletions(-)\n\ndiff --git a/git-p4 b/git-p4\nindex 672b0c2..0c6a5cc 100755\n--- a/git-p4\n+++ b/git-p4\n@@ -755,12 +755,11 @@ class P4FileReader:\n                     break\n\n             if header['type'].startswith('utf16'):\n-                try:\n-                    text = textBuffer.getvalue().encode('utf_16')\n-                except UnicodeDecodeError:\n-                    # File checked in to Perforce has an error. Try\nwithout encoding\n-                    print \"Corrupt UTF-16 file in Perforce: %s\" %\nheader['depotFile']\n-                    text = textBuffer.getvalue()\n+                # Don't even try to convert utf16. Ask p4 to write\nthe file directly.\n+                tmpFile = tempfile.NamedTemporaryFile()\n+                P4Helper().p4_system(\"print -o %s %s\"%(tmpFile.name,\nheader['depotFile']))\n+                text = open(tmpFile.name).read()\n+      \t        tmpFile.close()\n             else:\n                 text = textBuffer.getvalue()\n             textBuffer.close()\n-- \n1.7.6\n"},{"id":"173960","messageId":"20110821152156.GA30299@arf.padd.com","threadId":"28170","inReplyTo":"CANeU7QmGJTr4V-tWzuPnP2P=9eQbGDgQxv2phAVbGXjU5tqYPQ@mail.gmail.com","subject":"Re: [PATCH] git-p4: don't convert utf16 files.","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2011-08-21T15:21:56Z","receivedAt":"2011-08-21T15:21:56Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"christ.li@gmail.com wrote on Fri, 19 Aug 2011 15:50 -0700:\n> Some repository has some utf16 files git-p4 don't know\n> how to convert. For those files, git-p4 just write the utf8\n> files. That is wrong, because git get different file than\n> perforce does, causing some windows resource file fail\n> to compile.\n> \n> Using the \"p4 print -o tmpfile depotfile\" can avoid this\n> convertion (and possible failure) all together.\n\nThis isn't contrib/fast-import/git-p4.  Searching around, I\ndiscovered a 2009 fork of git-p4 that is fairly active.  CC-ing\nsome of the names I found on github.\n\nHere's one such repo:\n\n    http://github.com/ermshiperete/git-p4\n\nGit's git-p4 doesn't try to do anything special with utf-16.  It\ndoes \\r\\n mangling, but not $Keyword$ removal, then just streams\nit to disk however p4 sends it.  That's close to what you're\ntrying to do here.\n\n\t\t-- Pete\n\n> Signed-off-by: Chris Li <git@chrisli.org>\n> \n> ---\n>  git-p4 |   11 +++++------\n>  1 files changed, 5 insertions(+), 6 deletions(-)\n> \n> diff --git a/git-p4 b/git-p4\n> index 672b0c2..0c6a5cc 100755\n> --- a/git-p4\n> +++ b/git-p4\n> @@ -755,12 +755,11 @@ class P4FileReader:\n>                      break\n> \n>              if header['type'].startswith('utf16'):\n> -                try:\n> -                    text = textBuffer.getvalue().encode('utf_16')\n> -                except UnicodeDecodeError:\n> -                    # File checked in to Perforce has an error. Try\n> without encoding\n> -                    print \"Corrupt UTF-16 file in Perforce: %s\" %\n> header['depotFile']\n> -                    text = textBuffer.getvalue()\n> +                # Don't even try to convert utf16. Ask p4 to write\n> the file directly.\n> +                tmpFile = tempfile.NamedTemporaryFile()\n> +                P4Helper().p4_system(\"print -o %s %s\"%(tmpFile.name,\n> header['depotFile']))\n> +                text = open(tmpFile.name).read()\n> +      \t        tmpFile.close()\n>              else:\n>                  text = textBuffer.getvalue()\n>              textBuffer.close()\n> -- \n> 1.7.6\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n"},{"id":"173964","messageId":"CANeU7QkwekmnCQKG3DqyhW54hk7boL+FQSOOviJ3gn5K7++cjg@mail.gmail.com","threadId":"28170","inReplyTo":"20110821152156.GA30299@arf.padd.com","subject":"Re: [PATCH] git-p4: don't convert utf16 files.","fromName":"Chris Li","fromEmail":"git@chrisli.org","sentAt":"2011-08-21T19:09:15Z","receivedAt":"2011-08-21T19:09:15Z","isPatch":true,"sender":{"key":"git@chrisli.org","avatar":null},"body":"On Sun, Aug 21, 2011 at 8:21 AM, Pete Wyckoff <pw@padd.com> wrote:\n\n>> Using the \"p4 print -o tmpfile depotfile\" can avoid this\n>> convertion (and possible failure) all together.\n>\n> This isn't contrib/fast-import/git-p4.  Searching around, I\n> discovered a 2009 fork of git-p4 that is fairly active.  CC-ing\n> some of the names I found on github.\n\nOops. I forget I am trying some other git-p4 and this patch is\nnot for git-core. The git-p4 in git-core repository has the same\nproblem. I was original using the git-core one, hit the bug. Then\nI try some other branch and found the same problem, I just forget\nwhich git repository I am at. Silly me.\n\nLet me redo a patch for git.\n\n>\n> Here's one such repo:\n>\n>    http://github.com/ermshiperete/git-p4\n>\n> Git's git-p4 doesn't try to do anything special with utf-16.  It\n> does \\r\\n mangling, but not $Keyword$ removal, then just streams\n> it to disk however p4 sends it.  That's close to what you're\n> trying to do here.\n\nNo, it is not the same. Here is what I find out. If you ask use\n\"p4 print\" to fetch a perforce utf16 file, perforce convert that file into\nutf8 before it send out to stdout. I guess the perforce guys assume\np4 print is used for terminal console. If the git-p4 just stream to disk,\nthen you  those file are effective checkout as utf8, which is about\nhalf the size of the original utf16 file.\n\nI never like those utf16 files, I think those file should be check in\nas binary. But some one did check in the utf16 in the repository I\nuse at work, Now I have to deal with it. I guess the reason perforce\nwant to know about utf16 is to do the \\r\\n conversion properly on\nthose utf16 files. The file in question is the windows .mc file.\nIt contain different language translation of the error message.\n\nSo git-p4 in git-core effectively converted the perforce utf16 file to\nutf8 during import.\n\nHowever, if I do \"p4 print -o filename\". Perforce will write the file in\nthe original utf16 format. I don't need to do the utf8 to utf16 conversion,\nwhich can failed and it does in my case.\n\nThe git-p4 in git-core need the same treatment. I will work out a patch\nand resubmit that later.\n\nThanks\n\nChris\n"}]}