# [PATCH] git-p4: import utf16 file properly 6 messages from 2011-09-13 to 2011-09-18. Participants: Chris Li, Luke Diamand, Pete Wyckoff. Thread: https://gitlist.dev/t/28383 ## Chris Li, 2011-09-13 21:33 Subject: [PATCH] git-p4: import utf16 file properly Message-ID: URL: https://gitlist.dev/e/CANeU7QndA0yv1OzU3vta5B8r8nCRdBSqTy0Rboc_bbpst%2B1pcw%40mail.gmail.com ``` The current git-p4 does not handle utf16 files properly. The "p4 print" command, when output to stdout, converts the utf16 file into utf8. That effectively imported the utf16 file as utf8 for git. In other words, git-p4 import a different file compare to file check out by perforce. This breakes my windows build in the company project. The fix is simple, just ask perforce to print the depot file into a real file. This way perforce will not performe the utf16 to utf8 conversion. Git can import the exact same file as perforce checkout. --- contrib/fast-import/git-p4 | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4 index 6b9de9e..5fb1ac7 100755 --- a/contrib/fast-import/git-p4 +++ b/contrib/fast-import/git-p4 @@ -1239,6 +1239,11 @@ class P4Sync(Command, P4UserMap): contents = map(lambda text: re.sub(r'(?i)\$(Id|Header):[^$]*\$',r'$\1$', text), contents) elif file['type'] in ('text+k', 'ktext', 'kxtext', 'unicode+k', 'binary+k'): contents = map(lambda text: re.sub(r'\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$\n]*\$',r'$\1$', text), contents) + elif file['type'] == 'utf16': + tmpFile = tempfile.NamedTemporaryFile() + p4CmdList("print -o %s %s"%(tmpFile.name, file['depotFile'])) + contents = [ open(tmpFile.name).read() ] + tmpFile.close() self.gitStream.write("M %s inline %s\n" % (mode, relPath)) -- 1.7.6 ``` ## Luke Diamand, 2011-09-14 07:55 Subject: Re: [PATCH] git-p4: import utf16 file properly Message-ID: <4E705DF8.1040508@diamand.org> URL: https://gitlist.dev/e/4E705DF8.1040508%40diamand.org In-Reply-To: ``` On 13/09/11 22:33, Chris Li wrote: > The current git-p4 does not handle utf16 files properly. > The "p4 print" command, when output to stdout, converts the > utf16 file into utf8. That effectively imported the utf16 file > as utf8 for git. In other words, git-p4 import a different > file compare to file check out by perforce. This breakes my > windows build in the company project. > > The fix is simple, just ask perforce to print the depot > file into a real file. This way perforce will not performe > the utf16 to utf8 conversion. Git can import the exact same > file as perforce checkout. Does this change do the right thing with RCS keywords in UTF16 files? If p4CmdList() fails, e.g. due to running out of diskspace, will this just happily import a truncated/corrupt file? (And I could be wrong about this, but does you patch have newline damage? It didn't seem to apply for me). Regards! Luke > --- > contrib/fast-import/git-p4 | 5 +++++ > 1 files changed, 5 insertions(+), 0 deletions(-) > > diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4 > index 6b9de9e..5fb1ac7 100755 > --- a/contrib/fast-import/git-p4 > +++ b/contrib/fast-import/git-p4 > @@ -1239,6 +1239,11 @@ class P4Sync(Command, P4UserMap): > contents = map(lambda text: > re.sub(r'(?i)\$(Id|Header):[^$]*\$',r'$\1$', text), contents) > elif file['type'] in ('text+k', 'ktext', 'kxtext', > 'unicode+k', 'binary+k'): > contents = map(lambda text: > re.sub(r'\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$\n]*\$',r'$\1$', > text), contents) > + elif file['type'] == 'utf16': > + tmpFile = tempfile.NamedTemporaryFile() > + p4CmdList("print -o %s %s"%(tmpFile.name, file['depotFile'])) > + contents = [ open(tmpFile.name).read() ] > + tmpFile.close() > > self.gitStream.write("M %s inline %s\n" % (mode, relPath)) > ``` ## Chris Li, 2011-09-14 18:29 Subject: Re: [PATCH] git-p4: import utf16 file properly Message-ID: URL: https://gitlist.dev/e/CANeU7QnW5kSni0W9M9q-FTWv4p_qc67LG3mA6BQj_U-wxNuZeQ%40mail.gmail.com In-Reply-To: <4E705DF8.1040508@diamand.org> ``` On Wed, Sep 14, 2011 at 12:55 AM, Luke Diamand wrote: > On 13/09/11 22:33, Chris Li wrote: >> The fix is simple, just ask perforce to print the depot >> file into a real file. This way perforce will not performe >> the utf16 to utf8 conversion. Git can import the exact same >> file as perforce checkout. > > Does this change do the right thing with RCS keywords in UTF16 files? I don't know what is the rules about the RCS keyword in UTF16 files. I look at the current git-p4, it does not do any keyword replacement in utf16 files. So this patch did not change that. It should be a separate issue. The way I see it, this patch is a straight enhancement compare to the current git-p4 because the current git-p4 *corrupts* the utf16 files. > > If p4CmdList() fails, e.g. due to running out of diskspace, will this just > happily import a truncated/corrupt file? Good point. I add the error check and attach the new patch. > (And I could be wrong about this, but does you patch have newline damage? It > didn't seem to apply for me). Gmail dmage the white space. I should always use the attachment. Does the attached patch work for you? Thanks Chris From 06de9cfdcd89e8bfb6575f40d36fdfcefe1a1985 Mon Sep 17 00:00:00 2001 From: Chris Li Date: Tue, 13 Sep 2011 13:57:31 -0700 Subject: [PATCH] git-p4: import utf16 file properly The current git-p4 does not handle utf16 files properly. The "p4 print" command, when output to stdout, convert the utf16 file into utf8. That effectively imported the utf16 file as utf8 for git. In other words, git-p4 import a different file compare to file check out by perforce. This breaks my windows build in the company project. The fix is simple, just ask perforce to print the depot file into a real file. This way perforce will not perform the utf16 to utf8 conversion. Git can import the exact same file as perforce checkout. --- contrib/fast-import/git-p4 | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4 index 6b9de9e..c111cad 100755 --- a/contrib/fast-import/git-p4 +++ b/contrib/fast-import/git-p4 @@ -1239,6 +1239,14 @@ class P4Sync(Command, P4UserMap): contents = map(lambda text: re.sub(r'(?i)\$(Id|Header):[^$]*\$',r'$\1$', text), contents) elif file['type'] in ('text+k', 'ktext', 'kxtext', 'unicode+k', 'binary+k'): contents = map(lambda text: re.sub(r'\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$\n]*\$',r'$\1$', text), contents) + elif file['type'] == 'utf16': + tmpFile = tempfile.NamedTemporaryFile() + cmd = "print -o %s %s"%(tmpFile.name, file['depotFile']) + result = p4Cmd(cmd) + if "p4ExitCode" in result: + die("Problems executing p4 %s"%cmd) + contents = [ open(tmpFile.name).read() ] + tmpFile.close() self.gitStream.write("M %s inline %s\n" % (mode, relPath)) -- 1.7.6 ``` ## Chris Li, 2011-09-14 18:39 Subject: Re: [PATCH] git-p4: import utf16 file properly Message-ID: URL: https://gitlist.dev/e/CANeU7QnPqJ%2BigcmS1JX_vasCXr%2BWpcx2b4Z-sy_%3D0qKEkG%2Bv_w%40mail.gmail.com In-Reply-To: ``` On Wed, Sep 14, 2011 at 11:29 AM, Chris Li wrote: >> Does this change do the right thing with RCS keywords in UTF16 files? > > I don't know what is the rules about the RCS keyword in UTF16 files. I did a little bit research and found this: http://www.perforce.com/perforce/doc.current/manuals/p4guide/ab_filetypes.html RCS keyword expand should only happen for "+k" or "+ko" modifiers. There for, "utf16" files without modifier should not be converted. In that regard, the patch is correct. But both the original and patched version did not handle "utf16+k" type of files. I still consider it as a separate issue. Chris ``` ## Luke Diamand, 2011-09-14 18:56 Subject: Re: [PATCH] git-p4: import utf16 file properly Message-ID: <4E70F8DB.8080008@diamand.org> URL: https://gitlist.dev/e/4E70F8DB.8080008%40diamand.org In-Reply-To: ``` On 14/09/11 19:29, Chris Li wrote: > On Wed, Sep 14, 2011 at 12:55 AM, Luke Diamand wrote: >> On 13/09/11 22:33, Chris Li wrote: >>> The fix is simple, just ask perforce to print the depot >>> file into a real file. This way perforce will not performe >>> the utf16 to utf8 conversion. Git can import the exact same >>> file as perforce checkout. >> >> Does this change do the right thing with RCS keywords in UTF16 files? > > I don't know what is the rules about the RCS keyword in UTF16 files. > I look at the current git-p4, it does not do any keyword replacement in > utf16 files. So this patch did not change that. It should be a separate issue. > > The way I see it, this patch is a straight enhancement compare to the > current git-p4 because the current git-p4 *corrupts* the utf16 files. > >> >> If p4CmdList() fails, e.g. due to running out of diskspace, will this just >> happily import a truncated/corrupt file? > > Good point. I add the error check and attach the new patch. > >> (And I could be wrong about this, but does you patch have newline damage? It >> didn't seem to apply for me). Looks good to me. I think you're right about the RCS keywords not being relevant here. > > Gmail dmage the white space. I should always use the attachment. > Does the attached patch work for you? > > Thanks > > Chris ``` ## Pete Wyckoff, 2011-09-18 01:19 Subject: Re: [PATCH] git-p4: import utf16 file properly Message-ID: <20110918011942.GA13702@arf.padd.com> URL: https://gitlist.dev/e/20110918011942.GA13702%40arf.padd.com In-Reply-To: ``` git@chrisli.org wrote on Wed, 14 Sep 2011 11:39 -0700: > On Wed, Sep 14, 2011 at 11:29 AM, Chris Li wrote: > >> Does this change do the right thing with RCS keywords in UTF16 files? > > > > I don't know what is the rules about the RCS keyword in UTF16 files. > > I did a little bit research and found this: > > http://www.perforce.com/perforce/doc.current/manuals/p4guide/ab_filetypes.html > > RCS keyword expand should only happen for "+k" or "+ko" modifiers. > There for, "utf16" files without modifier should not be converted. > In that regard, the patch is correct. But both the original and patched version > did not handle "utf16+k" type of files. I still consider it as a separate issue. Your patch looks good and this all makes sense. I redid it, adding a test case, and some more patches to clean up some of the filetype detection code. I'll send it out for review soon here. Luke: thanks for the comments; they prompted me to think about keywords and beyond. -- Pete ```