threads / patch / 28170

patchgit-p4: don't convert utf16 files.

Subject: [PATCH] git-p4: don't convert utf16 files.

## tl;dr

3 messages between Aug 19, 2011 and Aug 21, 2011. Diffs are folded; open one to read it.

replies: 2people: 3as markdown or json

Chris Li· Aug 19, 2011, 22:50 UTC · lore

Some repository has some utf16 files git-p4 don't know how to convert. For those files, git-p4 just write the utf8 files. That is wrong, because git get different file than perforce does, causing some windows resource file fail to compile.

Using the "p4 print -o tmpfile depotfile" can avoid this convertion (and possible failure) all together.

Signed-off-by: Chris Li <git@chrisli.org>
---
 git-p4 |   11 +++++------
 1 files changed, 5 insertions(+), 6 deletions(-)
Show changes to git-p4 +5 −6
diff --git a/git-p4 b/git-p4
index 672b0c2..0c6a5cc 100755
--- a/git-p4
+++ b/git-p4
@@ -755,12 +755,11 @@ class P4FileReader:
                     break

             if header['type'].startswith('utf16'):
-                try:
-                    text = textBuffer.getvalue().encode('utf_16')
-                except UnicodeDecodeError:
-                    # File checked in to Perforce has an error. Try
without encoding
-                    print "Corrupt UTF-16 file in Perforce: %s" %
header['depotFile']
-                    text = textBuffer.getvalue()
+                # Don't even try to convert utf16. Ask p4 to write
the file directly.
+                tmpFile = tempfile.NamedTemporaryFile()
+                P4Helper().p4_system("print -o %s %s"%(tmpFile.name,
header['depotFile']))
+                text = open(tmpFile.name).read()
+      	        tmpFile.close()
             else:
                 text = textBuffer.getvalue()
             textBuffer.close()

-- 
1.7.6
Pete Wyckoff· Aug 21, 2011, 15:21 UTC · re: Chris Li · lore

Re: [PATCH] git-p4: don't convert utf16 files.

christ.li@gmail.com wrote on Fri, 19 Aug 2011 15:50 -0700:
Show 8 quoted lines
> Some repository has some utf16 files git-p4 don't know
> how to convert. For those files, git-p4 just write the utf8
> files. That is wrong, because git get different file than
> perforce does, causing some windows resource file fail
> to compile.
> 
> Using the "p4 print -o tmpfile depotfile" can avoid this
> convertion (and possible failure) all together.

This isn't contrib/fast-import/git-p4. Searching around, I discovered a 2009 fork of git-p4 that is fairly active. CC-ing some of the names I found on github.

Here's one such repo:
    http://github.com/ermshiperete/git-p4

Git's git-p4 doesn't try to do anything special with utf-16. It does \r\n mangling, but not $Keyword$ removal, then just streams it to disk however p4 sends it. That's close to what you're trying to do here.

		-- Pete
Show 39 quoted lines
> Signed-off-by: Chris Li <git@chrisli.org>
> 
> ---
>  git-p4 |   11 +++++------
>  1 files changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/git-p4 b/git-p4
> index 672b0c2..0c6a5cc 100755
> --- a/git-p4
> +++ b/git-p4
> @@ -755,12 +755,11 @@ class P4FileReader:
>                      break
> 
>              if header['type'].startswith('utf16'):
> -                try:
> -                    text = textBuffer.getvalue().encode('utf_16')
> -                except UnicodeDecodeError:
> -                    # File checked in to Perforce has an error. Try
> without encoding
> -                    print "Corrupt UTF-16 file in Perforce: %s" %
> header['depotFile']
> -                    text = textBuffer.getvalue()
> +                # Don't even try to convert utf16. Ask p4 to write
> the file directly.
> +                tmpFile = tempfile.NamedTemporaryFile()
> +                P4Helper().p4_system("print -o %s %s"%(tmpFile.name,
> header['depotFile']))
> +                text = open(tmpFile.name).read()
> +      	        tmpFile.close()
>              else:
>                  text = textBuffer.getvalue()
>              textBuffer.close()
> -- 
> 1.7.6
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
Chris Li· Aug 21, 2011, 19:09 UTC · re: Pete Wyckoff · lore

Re: [PATCH] git-p4: don't convert utf16 files.

On Sun, Aug 21, 2011 at 8:21 AM, Pete Wyckoff <pw@padd.com> wrote:
Show 6 quoted lines
>> Using the "p4 print -o tmpfile depotfile" can avoid this
>> convertion (and possible failure) all together.
>
> This isn't contrib/fast-import/git-p4.  Searching around, I
> discovered a 2009 fork of git-p4 that is fairly active.  CC-ing
> some of the names I found on github.

Oops. I forget I am trying some other git-p4 and this patch is not for git-core. The git-p4 in git-core repository has the same problem. I was original using the git-core one, hit the bug. Then I try some other branch and found the same problem, I just forget which git repository I am at. Silly me.

Let me redo a patch for git.
Show 9 quoted lines
>
> Here's one such repo:
>
>    http://github.com/ermshiperete/git-p4
>
> Git's git-p4 doesn't try to do anything special with utf-16.  It
> does \r\n mangling, but not $Keyword$ removal, then just streams
> it to disk however p4 sends it.  That's close to what you're
> trying to do here.

No, it is not the same. Here is what I find out. If you ask use "p4 print" to fetch a perforce utf16 file, perforce convert that file into utf8 before it send out to stdout. I guess the perforce guys assume p4 print is used for terminal console. If the git-p4 just stream to disk, then you those file are effective checkout as utf8, which is about half the size of the original utf16 file.

I never like those utf16 files, I think those file should be check in as binary. But some one did check in the utf16 in the repository I use at work, Now I have to deal with it. I guess the reason perforce want to know about utf16 is to do the \r\n conversion properly on those utf16 files. The file in question is the windows .mc file. It contain different language translation of the error message.

So git-p4 in git-core effectively converted the perforce utf16 file to utf8 during import.

However, if I do "p4 print -o filename". Perforce will write the file in the original utf16 format. I don't need to do the utf8 to utf16 conversion, which can failed and it does in my case.

The git-p4 in git-core need the same treatment. I will work out a patch and resubmit that later.

Thanks
Chris

← back to recent threads