Re: [PATCH 2/2] git-p4: do not decode data from perforce by default

From: Andrew Oakley <andrew@adoakley.name>
Date: Apr 30, 2021, 08:53 UTC
Message-ID: <20210430095342.58134e4e@ado-tr>
In-Reply-To: <CAKu1iLXRrsB4mRsDfhBH5aahWzDjpfqLuWP9t47RMB=RdpL1iA@mail.gmail.com>

On Thu, 29 Apr 2021 03:00:06 -0700 Tzadik Vanderhoof <tzadik.vanderhoof@gmail.com> wrote:

Show 8 quoted lines

> However, on Windows, UTF-8 strings passed to "p4 submit -d" are
> somehow converted to the default Windows code page by the time they
> are stored in the Perforce database, probably as part of the process
> of passing the command line arguments to the Windows p4 executable.
> However, the "code page" data is *not* converted to UTF-8 on the way
> back from p4 to git-p4.py.  The only way to get it into UTF-8 is to
> call string.decode().  As a result, this patch, which takes out the
> call to string.decode() will not work on Windows.

Thanks for that explanation, the reencoding of the data on Windows is not something I was expecting. Given the behaviour you've described, I suspect that there might be two different problems that we are trying to solve.

The perforce depot I'm working with has a mixture of encodings, and commits are created from a variety of different environments. The majority of commits are ASCII or UTF-8, there are a small number that are in some other encoding. Any attempt to reencode the data is likely to make the problem worse in at least some cases.

I suspect that other perforce depots are used primarily from Windows machines, and have data that is encoded in a mostly consistent way but the encoding is not UTF-8. Re-encoding the data for git makes sense in that case. Is this the kind of repository you have?

If there are these two different cases then we probably need to come up with a patch that solves both issues.

For my cases where we've got a repository containing all sorts of junk, it sounds like it might be awkward to create a test case that works on Windows.