{"thread":{"id":"52365","subject":"[RFC PATCH 0/4] git-p4: python 3 compatability","startedAt":"2019-11-28T01:29:05Z","lastAt":"2019-11-28T12:55:05Z","messageCount":7,"participants":["Yang Zhao","Elijah Newren","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"387257","messageId":"20191128012807.3103-1-yang.zhao@skyboxlabs.com","threadId":"52365","inReplyTo":null,"subject":"[RFC PATCH 0/4] git-p4: python 3 compatability","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-11-28T01:28:03Z","receivedAt":"2019-11-28T01:29:05Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"This patchset the necessary changes to have git-p4 run under python 3.\n\nI'm submitting this as an RFC as the changes are not exhaustive. I do not have\np4d available, and therefore cannot run the tests locally. I have made\nbest-effort attempts to test the changes against a repo I do have read access to.\n\nYang Zhao (4):\n git-p4: decode response from p4 to str for python3\n git-p4: properly encode/decode communication with git for python 3\n git-p4: open .gitp4-usercache.txt in text mode\n git-p4: use utf-8 encoding for file paths throughout\n\n git-p4.py | 58 +++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 41 insertions(+), 17 deletions(-)\n\n-- \n2.24.0.windows.2\n\n"},{"id":"387258","messageId":"20191128012807.3103-3-yang.zhao@skyboxlabs.com","threadId":"52365","inReplyTo":"20191128012807.3103-1-yang.zhao@skyboxlabs.com","subject":"[RFC PATCH 2/4] git-p4: properly encode/decode communication with git for python 3","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-11-28T01:28:05Z","receivedAt":"2019-11-28T01:29:09Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Under python3, calls to write() on the stream to `git fast-import` must\nbe encoded. This patch wraps the IO object such that this encoding is\ndone transparently.\n\nConversely, any text data read from subprocesses must also be decoded\nbefore running through the rest of the pipeline.\n\nSigned-off-by: Yang Zhao \n---\n git-p4.py | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex ead9d816e1..d79cbf7005 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -237,6 +237,9 @@ def read_pipe_lines(c):\n if pipe.close() or p.wait():\n die('Command failed: %s' % str(c))\n \n+ if use_encoded_streams:\n+ val = [line.decode() for line in val]\n+\n return val\n \n def p4_read_pipe_lines(c):\n@@ -631,7 +634,7 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,\n stdin_file.write(stdin)\n else:\n for i in stdin:\n- stdin_file.write(i + '\\n')\n+ stdin_file.write((i + '\\n').encode())\n stdin_file.flush()\n stdin_file.seek(0)\n \n@@ -3547,6 +3550,15 @@ def openStreams(self):\n self.gitStream = self.importProcess.stdin\n self.gitError = self.importProcess.stderr\n \n+ if use_encoded_streams:\n+ # Wrap gitStream.write() so that it can be called using `str` arguments\n+ def make_encoded_write(write):\n+ def encoded_write(s):\n+ return write(s.encode() if isinstance(s, str) else s)\n+ return encoded_write\n+\n+ self.gitStream.write = make_encoded_write(self.gitStream.write)\n+\n def closeStreams(self):\n self.gitStream.close()\n if self.importProcess.wait() != 0:\n-- \n2.24.0.windows.2\n\n"},{"id":"387259","messageId":"20191128012807.3103-2-yang.zhao@skyboxlabs.com","threadId":"52365","inReplyTo":"20191128012807.3103-1-yang.zhao@skyboxlabs.com","subject":"[RFC PATCH 1/4] git-p4: decode response from p4 to str for python3","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-11-28T01:28:04Z","receivedAt":"2019-11-28T01:29:10Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"The marshalled dict in the response given on STDOUT by p4 uses `str` for\nkeys and string values. When run using python3, these values are\ndeserialized as `bytes`, leading to a whole host of problems as the rest\nof the code assumes `str` is used throughout.\n\nAn exception is made for the `data` field as it may contain arbitrary\nbinary data that is not text, as well as `depotFile` which may contain\ntext encoded with something other than ASCII or UTF-8.\n\nSigned-off-by: Yang Zhao \n---\n git-p4.py | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 60c73b6a37..ead9d816e1 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -36,6 +36,7 @@\n unicode = str\n bytes = bytes\n basestring = (str,bytes)\n+ use_encoded_streams = True\n else:\n # 'unicode' exists, must be Python 2\n str = str\n@@ -643,6 +644,15 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,\n try:\n while True:\n entry = marshal.load(p4.stdout)\n+ if use_encoded_streams:\n+ # Decode unmarshalled dict to use str keys and values, except for:\n+ # - `data` which may contain arbitrary binary data\n+ # - `depotFile` which may contain non-UTF8 encoded text\n+ decoded_entry = {}\n+ for key, value in entry.items():\n+ key = key.decode()\n+ decoded_entry[key] = value.decode() if not (key in ['data', 'depotFile'] or isinstance(value, str)) else value\n+ entry = decoded_entry\n if skip_info:\n if 'code' in entry and entry['code'] == 'info':\n continue\n-- \n2.24.0.windows.2\n\n"},{"id":"387260","messageId":"20191128012807.3103-4-yang.zhao@skyboxlabs.com","threadId":"52365","inReplyTo":"20191128012807.3103-1-yang.zhao@skyboxlabs.com","subject":"[RFC PATCH 3/4] git-p4: open .gitp4-usercache.txt in text mode","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-11-28T01:28:06Z","receivedAt":"2019-11-28T01:29:11Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Opening .gitp4-usercache.txt in text mode makes python 3 happy without\nexplicitly adding encoding and decoding.\n\nSigned-off-by: Yang Zhao \n---\n git-p4.py | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex d79cbf7005..6821d6aafd 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -1387,14 +1387,14 @@ def getUserMapFromPerforceServer(self):\n for (key, val) in self.users.items():\n s += \"%s\\t%s\\n\" % (key.expandtabs(1), val.expandtabs(1))\n \n- open(self.getUserCacheFilename(), \"wb\").write(s)\n+ open(self.getUserCacheFilename(), 'w', encoding='utf-8').write(s)\n self.userMapFromPerforceServer = True\n \n def loadUserMapFromCache(self):\n self.users = {}\n self.userMapFromPerforceServer = False\n try:\n- cache = open(self.getUserCacheFilename(), \"rb\")\n+ cache = open(self.getUserCacheFilename(), 'r', encoding='utf-8')\n lines = cache.readlines()\n cache.close()\n for line in lines:\n-- \n2.24.0.windows.2\n\n"},{"id":"387261","messageId":"20191128012807.3103-5-yang.zhao@skyboxlabs.com","threadId":"52365","inReplyTo":"20191128012807.3103-1-yang.zhao@skyboxlabs.com","subject":"[RFC PATCH 4/4] git-p4: use utf-8 encoding for file paths throughout","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-11-28T01:28:07Z","receivedAt":"2019-11-28T01:29:13Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Try to decode file paths in responses from p4 as soon as possible so\nthat we are working with unicode string throughout the rest of the flow.\nThis makes python 3 a lot happier.\n\nSigned-off-by: Yang Zhao \n---\n\nThis is probably the most risky patch out of the set. It's very likely\nthat I've neglected to consider certain corner cases with decoding of\npath data.\n\n git-p4.py | 34 ++++++++++++++++++----------------\n 1 file changed, 18 insertions(+), 16 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 6821d6aafd..bd693e1404 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -650,11 +650,27 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,\n if use_encoded_streams:\n # Decode unmarshalled dict to use str keys and values, except for:\n # - `data` which may contain arbitrary binary data\n- # - `depotFile` which may contain non-UTF8 encoded text\n+ # - `depotFile` which may contain non-UTF8 encoded text, and is decoded\n+ # according to git-p4.pathEncoding config\n decoded_entry = {}\n for key, value in entry.items():\n key = key.decode()\n- decoded_entry[key] = value.decode() if not (key in ['data', 'depotFile'] or isinstance(value, str)) else value\n+ if key == 'data':\n+ pass\n+ elif key == 'depotFile':\n+ try:\n+ value = value.decode('ascii')\n+ except:\n+ encoding = 'utf-8'\n+ if gitConfig('git-p4.pathEncoding'):\n+ encoding = gitConfig('git-p4.pathEncoding')\n+ path = path.decode(encoding, 'replace')\n+ if verbose:\n+ print('Path with non-ASCII characters detected. Used %s to decode: %s ' % (encoding, path))\n+ elif not isinstance(value, str):\n+ value = value.decode()\n+\n+ decoded_entry[key] = value\n entry = decoded_entry\n if skip_info:\n if 'code' in entry and entry['code'] == 'info':\n@@ -2758,24 +2774,11 @@ def writeToGitStream(self, gitMode, relPath, contents):\n self.gitStream.write(d)\n self.gitStream.write('\\n')\n \n- def encodeWithUTF8(self, path):\n- try:\n- path.decode('ascii')\n- except:\n- encoding = 'utf8'\n- if gitConfig('git-p4.pathEncoding'):\n- encoding = gitConfig('git-p4.pathEncoding')\n- path = path.decode(encoding, 'replace').encode('utf8', 'replace')\n- if self.verbose:\n- print('Path with non-ASCII characters detected. Used %s to encode: %s ' % (encoding, path))\n- return path\n-\n # output one file from the P4 stream\n # - helper for streamP4Files\n \n def streamOneP4File(self, file, contents):\n relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes)\n- relPath = self.encodeWithUTF8(relPath)\n if verbose:\n if 'fileSize' in self.stream_file:\n size = int(self.stream_file['fileSize'])\n@@ -2858,7 +2861,6 @@ def streamOneP4File(self, file, contents):\n \n def streamOneP4Deletion(self, file):\n relPath = self.stripRepoPath(file['path'], self.branchPrefixes)\n- relPath = self.encodeWithUTF8(relPath)\n if verbose:\n sys.stdout.write(\"delete %s\\n\" % relPath)\n sys.stdout.flush()\n-- \n2.24.0.windows.2\n\n"},{"id":"387264","messageId":"CABPp-BEa883U+m1emhZHU4s5EqsJHyRiPvjnsyw6_0guKLUsLg@mail.gmail.com","threadId":"52365","inReplyTo":"20191128012807.3103-5-yang.zhao@skyboxlabs.com","subject":"Re: [RFC PATCH 4/4] git-p4: use utf-8 encoding for file paths throughout","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-11-28T02:57:35Z","receivedAt":"2019-11-28T02:57:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Nov 27, 2019 at 5:32 PM Yang Zhao wrote:\n>\n> Try to decode file paths in responses from p4 as soon as possible so\n> that we are working with unicode string throughout the rest of the flow.\n> This makes python 3 a lot happier.\n>\n> Signed-off-by: Yang Zhao \n> ---\n>\n> This is probably the most risky patch out of the set. It's very likely\n> that I've neglected to consider certain corner cases with decoding of\n> path data.\n\nYes, this does seem somewhat risky to me. It may go well on platforms\nthat require all filenames to be unicode. And it may work for users\nwho happen to restrict their filenames to valid utf-8. But this\nabstraction doesn't fit the general problem, so some users may be left\nout in the cold.\n\nI tried multiple times while switching git-filter-repo from python2 to\npython3, at different levels of pervasiveness, to use unicode more\ngenerally. But I mostly gave up; everyone knows files won't\nnecessarily be unicode, but you just can't assume filenames or commit\nmessages or branch or tag names (and perhaps a few other things I'm\nforgetting) are either. I ended up using bytestrings everywhere\nexcept messages displayed to the user, and I only decode at that\npoint.\n\n\nOf course, if perforce happens to only work with unicode filenames\nthen you'll be fine. And perhaps you don't want or need to be as\nparanoid as I was about what people could do. So I don't know if my\nexperience applies in your case (I've never used perforce myself), but\nI just thought I'd mention it in case it's useful.\n"},{"id":"387284","messageId":"nycvar.QRO.7.76.6.1911281352130.31080@tvgsbejvaqbjf.bet","threadId":"52365","inReplyTo":"20191128012807.3103-1-yang.zhao@skyboxlabs.com","subject":"Re: [RFC PATCH 0/4] git-p4: python 3 compatability","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-11-28T12:54:47Z","receivedAt":"2019-11-28T12:55:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 27 Nov 2019, Yang Zhao wrote:\n\n> This patchset the necessary changes to have git-p4 run under python 3.\n>\n> I'm submitting this as an RFC as the changes are not exhaustive. I do not have\n> p4d available, and therefore cannot run the tests locally. I have made\n> best-effort attempts to test the changes against a repo I do have read access to.\n\nHow about opening a PR at https://github.com/git/git? This will run our\nTravis and Azure Pipelines builds, and I am sure that you can add a\ndebug-only patch to force-use Python3 on the Perforce tests (and if I were\nyou, I would also restrict the run to those tests by editing `.travis.yml`\nand `azure-pipelines.yml`, both of which files should be easy enough to\nunderstand without additional documentation).\n\nCiao,\nJohannes\n"}]}