{"thread":{"id":"52454","subject":"[PATCH v2 00/14] git-p4: python3 compatibility","startedAt":"2019-12-13T23:53:21Z","lastAt":"2020-02-04T01:36:12Z","messageCount":22,"participants":["Yang Zhao","Junio C Hamano","Luke Diamand"],"isPatch":true,"patchVersion":2,"patchTotal":14},"messages":[{"id":"388185","messageId":"20191213235247.23660-2-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 00/14] git-p4: python3 compatibility","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:34Z","receivedAt":"2019-12-13T23:53:21Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"\nThis patchset adds python3 compatibility to git-p4.\n\nWhile further clean-up refactoring would have been nice, I specifically avoided\nmaking any major changes to the internal API, aiming to have passing tests\nwith as few changes as possible.\n\nCI results can be seen from this GitHub PR: https://github.com/git/git/pull/673\n\nChanges since v1:\n - incorporate Ben's change on dropping usage of basestring\n - don't alias string types based on python verson\n - use feature detection instead of version checking to detect python3\n - CI patch now at end of series; needs further discussion\n - some general clean-up to commit messages\n\nBen Keene (1):\n git-p4: change the expansion test from basestring to list\n\nYang Zhao (13):\n git-p4: make python2.7 the oldest supported version\n git-p4: remove string type aliasing\n git-p4: encode/decode communication with p4 for python3\n git-p4: encode/decode communication with git for python 4\n git-p4: convert path to unicode before processing them\n git-p4: open .gitp4-usercache.txt in text mode\n git-p4: use marshal format version 2 when sending to p4\n git-p4: fix freezing while waiting for fast-import progress\n git-p4: use functools.reduce instead of reduce\n git-p4: use dict.items() iteration for python3 compatibility\n git-p4: simplify regex pattern generation for parsing diff-tree\n git-p4: use python3's input() everywhere\n ci: also run linux-gcc pipeline with python3.5 environment\n\n azure-pipelines.yml | 11 ++\n git-p4.py | 237 ++++++++++++++++++++++++++------------------\n 2 files changed, 152 insertions(+), 96 deletions(-)\n\n-- \n2.21.0.windows.1\n\n"},{"id":"388186","messageId":"20191213235247.23660-3-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 01/14] git-p4: make python2.7 the oldest supported version","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:35Z","receivedAt":"2019-12-13T23:53:23Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Python2.6 and earlier have been end-of-life'd for many years now, and\nwe actually already use 2.7-only features in the code. Make the version\ncheck reflect current realities.\n\nThis also removes the need to explicitly define CalledProcessError if\nit's not available.\n\nSigned-off-by: Yang Zhao \n---\n git-p4.py | 20 ++------------------\n 1 file changed, 2 insertions(+), 18 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 60c73b6a37..37777bb9fd 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -8,9 +8,8 @@\n # License: MIT \n #\n import sys\n-if sys.hexversion < 0x02040000:\n- # The limiter is the subprocess module\n- sys.stderr.write(\"git-p4: requires Python 2.4 or later.\\n\")\n+if sys.version_info.major < 3 and sys.version_info.minor < 7:\n+ sys.stderr.write(\"git-p4: requires Python 2.7 or later.\\n\")\n sys.exit(1)\n import os\n import optparse\n@@ -43,21 +42,6 @@\n bytes = str\n basestring = basestring\n \n-try:\n- from subprocess import CalledProcessError\n-except ImportError:\n- # from python2.7:subprocess.py\n- # Exception classes used by this module.\n- class CalledProcessError(Exception):\n- \"\"\"This exception is raised when a process run by check_call() returns\n- a non-zero exit status. The exit status will be stored in the\n- returncode attribute.\"\"\"\n- def __init__(self, returncode, cmd):\n- self.returncode = returncode\n- self.cmd = cmd\n- def __str__(self):\n- return \"Command '%s' returned non-zero exit status %d\" % (self.cmd, self.returncode)\n-\n verbose = False\n \n # Only labels/tags matching this will be imported/exported\n-- \n2.21.0.windows.1\n\n"},{"id":"388187","messageId":"20191213235247.23660-4-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 02/14] git-p4: change the expansion test from basestring to list","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:36Z","receivedAt":"2019-12-13T23:53:24Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"From: Ben Keene \n\nPython 3 handles strings differently than Python 2.7. Since Python 2\nis reaching it's end of life, a series of changes are being submitted to\nenable python 3.5 and following support. The current code fails basic\ntests under python 3.5.\n\nSome codepaths can represent a command line the program\ninternally prepares to execute either as a single string\n(i.e. each token properly quoted, concatenated with $IFS) or\nas a list of argv[] elements, and there are 9 places where\nwe say \"if X is isinstance(_, basestring), then do this\nthing to handle X as a command line in a single string; if\nnot, X is a command line in a list form\".\n\nThis does not work well with Python 3, as there is no\nbasestring (everything is Unicode now), and even with Python\n2, it was not an ideal way to tell the two cases apart,\nbecause an internally formed command line could have been in\na single Unicode string.\n\nFlip the check to say \"if X is not a list, then handle X as\na command line in a single string; otherwise treat it as a\ncommand line in a list form\".\n\nThis will get rid of references to 'basestring', to migrate\nthe code ready for Python 3.\n\nThanks-to: Junio C Hamano \nSigned-off-by: Ben Keene \nSigned-off-by: Yang Zhao \n---\n git-p4.py | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 37777bb9fd..2f177fb43b 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -89,7 +89,7 @@ def p4_build_cmd(cmd):\n # Provide a way to not pass this option by setting git-p4.retries to 0\n real_cmd += [\"-r\", str(retries)]\n \n- if isinstance(cmd,basestring):\n+ if not isinstance(cmd, list):\n real_cmd = ' '.join(real_cmd) + ' ' + cmd\n else:\n real_cmd += cmd\n@@ -155,7 +155,7 @@ def write_pipe(c, stdin):\n if verbose:\n sys.stderr.write('Writing pipe: %s\\n' % str(c))\n \n- expand = isinstance(c,basestring)\n+ expand = not isinstance(c, list)\n p = subprocess.Popen(c, stdin=subprocess.PIPE, shell=expand)\n pipe = p.stdin\n val = pipe.write(stdin)\n@@ -177,7 +177,7 @@ def read_pipe_full(c):\n if verbose:\n sys.stderr.write('Reading pipe: %s\\n' % str(c))\n \n- expand = isinstance(c,basestring)\n+ expand = not isinstance(c, list)\n p = subprocess.Popen(c, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=expand)\n (out, err) = p.communicate()\n return (p.returncode, out, err)\n@@ -213,7 +213,7 @@ def read_pipe_lines(c):\n if verbose:\n sys.stderr.write('Reading pipe: %s\\n' % str(c))\n \n- expand = isinstance(c, basestring)\n+ expand = not isinstance(c, list)\n p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand)\n pipe = p.stdout\n val = pipe.readlines()\n@@ -256,7 +256,7 @@ def p4_has_move_command():\n return True\n \n def system(cmd, ignore_error=False):\n- expand = isinstance(cmd,basestring)\n+ expand = not isinstance(cmd, list)\n if verbose:\n sys.stderr.write(\"executing %s\\n\" % str(cmd))\n retcode = subprocess.call(cmd, shell=expand)\n@@ -268,7 +268,7 @@ def system(cmd, ignore_error=False):\n def p4_system(cmd):\n \"\"\"Specifically invoke p4 as the system command. \"\"\"\n real_cmd = p4_build_cmd(cmd)\n- expand = isinstance(real_cmd, basestring)\n+ expand = not isinstance(real_cmd, list)\n retcode = subprocess.call(real_cmd, shell=expand)\n if retcode:\n raise CalledProcessError(retcode, real_cmd)\n@@ -506,7 +506,7 @@ def getP4OpenedType(file):\n # Return the set of all p4 labels\n def getP4Labels(depotPaths):\n labels = set()\n- if isinstance(depotPaths,basestring):\n+ if not isinstance(depotPaths, list):\n depotPaths = [depotPaths]\n \n for l in p4CmdList([\"labels\"] + [\"%s...\" % p for p in depotPaths]):\n@@ -593,7 +593,7 @@ def isModeExecChanged(src_mode, dst_mode):\n def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,\n errors_as_exceptions=False):\n \n- if isinstance(cmd,basestring):\n+ if not isinstance(cmd, list):\n cmd = \"-G \" + cmd\n expand = True\n else:\n@@ -610,7 +610,7 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,\n stdin_file = None\n if stdin is not None:\n stdin_file = tempfile.TemporaryFile(prefix='p4-stdin', mode=stdin_mode)\n- if isinstance(stdin,basestring):\n+ if not isinstance(stdin, list):\n stdin_file.write(stdin)\n else:\n for i in stdin:\n-- \n2.21.0.windows.1\n\n"},{"id":"388188","messageId":"20191213235247.23660-7-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 05/14] git-p4: encode/decode communication with git for python3","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:39Z","receivedAt":"2019-12-13T23:53:27Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Under python3, calls to write() on the stream to `git fast-import` must\nbe encoded. This patch wraps the IO object such that this encoding is\ndone transparently.\n\nConversely, any text data read from subprocesses must also be decoded\nbefore running through the rest of the pipeline.\n\nSigned-off-by: Yang Zhao \nReviewed-by: Ben Keene \n---\n git-p4.py | 21 +++++++++++++++++----\n 1 file changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex ca891e3d5d..d62fb05989 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -183,10 +183,12 @@ def read_pipe_full(c):\n (out, err) = p.communicate()\n return (p.returncode, out, decode_text_stream(err))\n \n-def read_pipe(c, ignore_error=False):\n+def read_pipe(c, ignore_error=False, raw=False):\n \"\"\" Read output from command. Returns the output text on\n success. On failure, terminates execution, unless\n ignore_error is True, when it returns an empty string.\n+\n+ If raw is True, do not attempt to decode output text.\n \"\"\"\n (retcode, out, err) = read_pipe_full(c)\n if retcode != 0:\n@@ -194,6 +196,8 @@ def read_pipe(c, ignore_error=False):\n out = \"\"\n else:\n die('Command failed: %s\\nError: %s' % (str(c), err))\n+ if not raw:\n+ out = decode_text_stream(out)\n return out\n \n def read_pipe_text(c):\n@@ -220,7 +224,6 @@ def read_pipe_lines(c):\n val = [decode_text_stream(line) for line in pipe.readlines()]\n if pipe.close() or p.wait():\n die('Command failed: %s' % str(c))\n-\n return val\n \n def p4_read_pipe_lines(c):\n@@ -616,7 +619,8 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,\n stdin_file.write(stdin)\n else:\n for i in stdin:\n- stdin_file.write(i + '\\n')\n+ stdin_file.write(encode_text_stream(i))\n+ stdin_file.write(b'\\n')\n stdin_file.flush()\n stdin_file.seek(0)\n \n@@ -1245,7 +1249,7 @@ def generatePointer(self, contentFile):\n ['git', 'lfs', 'pointer', '--file=' + contentFile],\n stdout=subprocess.PIPE\n )\n- pointerFile = pointerProcess.stdout.read()\n+ pointerFile = decode_text_stream(pointerProcess.stdout.read())\n if pointerProcess.wait():\n os.remove(contentFile)\n die('git-lfs pointer command failed. Did you install the extension?')\n@@ -3538,6 +3542,15 @@ def openStreams(self):\n self.gitStream = self.importProcess.stdin\n self.gitError = self.importProcess.stderr\n \n+ if bytes is not str:\n+ # Wrap gitStream.write() so that it can be called using `str` arguments\n+ def make_encoded_write(write):\n+ def encoded_write(s):\n+ return write(s.encode() if isinstance(s, str) else s)\n+ return encoded_write\n+\n+ self.gitStream.write = make_encoded_write(self.gitStream.write)\n+\n def closeStreams(self):\n self.gitStream.close()\n if self.importProcess.wait() != 0:\n-- \n2.21.0.windows.1\n\n"},{"id":"388189","messageId":"20191213235247.23660-5-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 03/14] git-p4: remove string type aliasing","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:37Z","receivedAt":"2019-12-13T23:53:27Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Now that python2.7 is the minimum required version and we no longer use\nthe basestring type, it is not necessary to use type aliasing to ensure\npython3 compatibility.\n\nSigned-off-by: Yang Zhao \n---\n git-p4.py | 16 ----------------\n 1 file changed, 16 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 2f177fb43b..153aff16f3 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -26,22 +26,6 @@\n import ctypes\n import errno\n \n-# support basestring in python3\n-try:\n- unicode = unicode\n-except NameError:\n- # 'unicode' is undefined, must be Python 3\n- str = str\n- unicode = str\n- bytes = bytes\n- basestring = (str,bytes)\n-else:\n- # 'unicode' exists, must be Python 2\n- str = str\n- unicode = unicode\n- bytes = str\n- basestring = basestring\n-\n verbose = False\n \n # Only labels/tags matching this will be imported/exported\n-- \n2.21.0.windows.1\n\n"},{"id":"388190","messageId":"20191213235247.23660-6-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 04/14] git-p4: encode/decode communication with p4 for python3","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:38Z","receivedAt":"2019-12-13T23:53:28Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"The marshalled dict in the response given on STDOUT by p4 uses `str` for\nkeys and string values. When run using python3, these values are\ndeserialized as `bytes`, leading to a whole host of problems as the rest\nof the code assumes `str` is used throughout.\n\nThis patch changes the deserialization behaviour such that, as much as\npossible, text output from p4 is decoded to native unicode strings.\nExceptions are made for the field `data` as it is usually arbitrary\nbinary data. `depotFile[0-9]*`, `path`, and `clientFile` are also exempt\nas they contain path strings not encoded with UTF-8, and must survive\nsurvive round-trip back to p4.\n\nConversely, text data being piped to p4 must always be encoded when\nrunning under python3.\n\nencode_text_stream() and decode_text_stream() were added to make these\ntransformations more convenient.\n\nSigned-off-by: Yang Zhao \n---\n git-p4.py | 59 +++++++++++++++++++++++++++++++++++++++++++------------\n 1 file changed, 46 insertions(+), 13 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 153aff16f3..ca891e3d5d 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -135,6 +135,21 @@ def die(msg):\n sys.stderr.write(msg + \"\\n\")\n sys.exit(1)\n \n+# We need different encoding/decoding strategies for text data being passed\n+# around in pipes depending on python version\n+if bytes is not str:\n+ # For python3, always encode and decode as appropriate\n+ def decode_text_stream(s):\n+ return s.decode() if isinstance(s, bytes) else s\n+ def encode_text_stream(s):\n+ return s.encode() if isinstance(s, str) else s\n+else:\n+ # For python2.7, pass read strings as-is, but also allow writing unicode\n+ def decode_text_stream(s):\n+ return s\n+ def encode_text_stream(s):\n+ return s.encode('utf_8') if isinstance(s, unicode) else s\n+\n def write_pipe(c, stdin):\n if verbose:\n sys.stderr.write('Writing pipe: %s\\n' % str(c))\n@@ -151,6 +166,8 @@ def write_pipe(c, stdin):\n \n def p4_write_pipe(c, stdin):\n real_cmd = p4_build_cmd(c)\n+ if bytes is not str and isinstance(stdin, str):\n+ stdin = encode_text_stream(stdin)\n return write_pipe(real_cmd, stdin)\n \n def read_pipe_full(c):\n@@ -164,7 +181,7 @@ def read_pipe_full(c):\n expand = not isinstance(c, list)\n p = subprocess.Popen(c, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=expand)\n (out, err) = p.communicate()\n- return (p.returncode, out, err)\n+ return (p.returncode, out, decode_text_stream(err))\n \n def read_pipe(c, ignore_error=False):\n \"\"\" Read output from command. Returns the output text on\n@@ -187,11 +204,11 @@ def read_pipe_text(c):\n if retcode != 0:\n return None\n else:\n- return out.rstrip()\n+ return decode_text_stream(out).rstrip()\n \n-def p4_read_pipe(c, ignore_error=False):\n+def p4_read_pipe(c, ignore_error=False, raw=False):\n real_cmd = p4_build_cmd(c)\n- return read_pipe(real_cmd, ignore_error)\n+ return read_pipe(real_cmd, ignore_error, raw=raw)\n \n def read_pipe_lines(c):\n if verbose:\n@@ -200,7 +217,7 @@ def read_pipe_lines(c):\n expand = not isinstance(c, list)\n p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand)\n pipe = p.stdout\n- val = pipe.readlines()\n+ val = [decode_text_stream(line) for line in pipe.readlines()]\n if pipe.close() or p.wait():\n die('Command failed: %s' % str(c))\n \n@@ -231,6 +248,7 @@ def p4_has_move_command():\n cmd = p4_build_cmd([\"move\", \"-k\", \"@from\", \"@to\"])\n p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n (out, err) = p.communicate()\n+ err = decode_text_stream(err)\n # return code will be 1 in either case\n if err.find(\"Invalid option\") >= 0:\n return False\n@@ -611,6 +629,20 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,\n try:\n while True:\n entry = marshal.load(p4.stdout)\n+ if bytes is not str:\n+ # Decode unmarshalled dict to use str keys and values, except for:\n+ # - `data` which may contain arbitrary binary data\n+ # - `depotFile[0-9]*`, `path`, or `clientFile` which may contain non-UTF8 encoded text\n+ decoded_entry = {}\n+ for key, value in entry.items():\n+ key = key.decode()\n+ if isinstance(value, bytes) and not (key in ('data', 'path', 'clientFile') or key.startswith('depotFile')):\n+ value = value.decode()\n+ decoded_entry[key] = value\n+ # Parse out data if it's an error response\n+ if decoded_entry.get('code') == 'error' and 'data' in decoded_entry:\n+ decoded_entry['data'] = decoded_entry['data'].decode()\n+ entry = decoded_entry\n if skip_info:\n if 'code' in entry and entry['code'] == 'info':\n continue\n@@ -828,6 +860,7 @@ def branch_exists(branch):\n cmd = [ \"git\", \"rev-parse\", \"--symbolic\", \"--verify\", branch ]\n p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n out, _ = p.communicate()\n+ out = decode_text_stream(out)\n if p.returncode:\n return False\n # expect exactly one line of output: the branch name\n@@ -1971,7 +2004,7 @@ def applyCommit(self, id):\n tmpFile = os.fdopen(handle, \"w+b\")\n if self.isWindows:\n submitTemplate = submitTemplate.replace(\"\\n\", \"\\r\\n\")\n- tmpFile.write(submitTemplate)\n+ tmpFile.write(encode_text_stream(submitTemplate))\n tmpFile.close()\n \n if self.prepare_p4_only:\n@@ -2018,7 +2051,7 @@ def applyCommit(self, id):\n if self.edit_template(fileName):\n # read the edited message and submit\n tmpFile = open(fileName, \"rb\")\n- message = tmpFile.read()\n+ message = decode_text_stream(tmpFile.read())\n tmpFile.close()\n if self.isWindows:\n message = message.replace(\"\\r\\n\", \"\\n\")\n@@ -2707,7 +2740,7 @@ def splitFilesIntoBranches(self, commit):\n return branches\n \n def writeToGitStream(self, gitMode, relPath, contents):\n- self.gitStream.write('M %s inline %s\\n' % (gitMode, relPath))\n+ self.gitStream.write(encode_text_stream(u'M {} inline {}\\n'.format(gitMode, relPath)))\n self.gitStream.write('data %d\\n' % sum(len(d) for d in contents))\n for d in contents:\n self.gitStream.write(d)\n@@ -2748,7 +2781,7 @@ def streamOneP4File(self, file, contents):\n git_mode = \"120000\"\n # p4 print on a symlink sometimes contains \"target\\n\";\n # if it does, remove the newline\n- data = ''.join(contents)\n+ data = ''.join(decode_text_stream(c) for c in contents)\n if not data:\n # Some version of p4 allowed creating a symlink that pointed\n # to nothing. This causes p4 errors when checking out such\n@@ -2802,7 +2835,7 @@ def streamOneP4File(self, file, contents):\n pattern = p4_keywords_regexp_for_type(type_base, type_mods)\n if pattern:\n regexp = re.compile(pattern, re.VERBOSE)\n- text = ''.join(contents)\n+ text = ''.join(decode_text_stream(c) for c in contents)\n text = regexp.sub(r'$\\1$', text)\n contents = [ text ]\n \n@@ -2817,7 +2850,7 @@ def streamOneP4Deletion(self, file):\n if verbose:\n sys.stdout.write(\"delete %s\\n\" % relPath)\n sys.stdout.flush()\n- self.gitStream.write(\"D %s\\n\" % relPath)\n+ self.gitStream.write(encode_text_stream(u'D {}\\n'.format(relPath)))\n \n if self.largeFileSystem and self.largeFileSystem.isLargeFile(relPath):\n self.largeFileSystem.removeLargeFile(relPath)\n@@ -2917,9 +2950,9 @@ def streamP4FilesCbSelf(entry):\n if 'shelved_cl' in f:\n # Handle shelved CLs using the \"p4 print file@=N\" syntax to print\n # the contents\n- fileArg = '%s@=%d' % (f['path'], f['shelved_cl'])\n+ fileArg = f['path'] + encode_text_stream('@={}'.format(f['shelved_cl']))\n else:\n- fileArg = '%s#%s' % (f['path'], f['rev'])\n+ fileArg = f['path'] + encode_text_stream('#{}'.format(f['rev']))\n \n fileArgs.append(fileArg)\n \n-- \n2.21.0.windows.1\n\n"},{"id":"388191","messageId":"20191213235247.23660-10-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 08/14] git-p4: use marshal format version 2 when sending to p4","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:42Z","receivedAt":"2019-12-13T23:53:30Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"p4 does not appear to understand marshal format version 3 and above.\nVersion 2 was the latest supported by python-2.7.\n\nSigned-off-by: Yang Zhao \nReviewed-by: Ben Keene \n---\n git-p4.py | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex abcda60eee..c7170c9ae6 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -1679,7 +1679,8 @@ def modifyChangelistUser(self, changelist, newUser):\n c = changes[0]\n if c['User'] == newUser: return # nothing to do\n c['User'] = newUser\n- input = marshal.dumps(c)\n+ # p4 does not understand format version 3 and above\n+ input = marshal.dumps(c, 2)\n \n result = p4CmdList(\"change -f -i\", stdin=input)\n for r in result:\n-- \n2.21.0.windows.1\n\n"},{"id":"388192","messageId":"20191213235247.23660-9-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 07/14] git-p4: open .gitp4-usercache.txt in text mode","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:41Z","receivedAt":"2019-12-13T23:53:31Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Opening .gitp4-usercache.txt in text mode makes python 3 happy without\nexplicitly adding encoding and decoding.\n\nSigned-off-by: Yang Zhao \nReviewed-by: Ben Keene \n---\n git-p4.py | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex d43f373b2d..abcda60eee 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -1395,14 +1395,14 @@ def getUserMapFromPerforceServer(self):\n for (key, val) in self.users.items():\n s += \"%s\\t%s\\n\" % (key.expandtabs(1), val.expandtabs(1))\n \n- open(self.getUserCacheFilename(), \"wb\").write(s)\n+ open(self.getUserCacheFilename(), 'w').write(s)\n self.userMapFromPerforceServer = True\n \n def loadUserMapFromCache(self):\n self.users = {}\n self.userMapFromPerforceServer = False\n try:\n- cache = open(self.getUserCacheFilename(), \"rb\")\n+ cache = open(self.getUserCacheFilename(), 'r')\n lines = cache.readlines()\n cache.close()\n for line in lines:\n-- \n2.21.0.windows.1\n\n"},{"id":"388193","messageId":"20191213235247.23660-8-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 06/14] git-p4: convert path to unicode before processing them","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:40Z","receivedAt":"2019-12-13T23:53:32Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"P4 allows essentially arbitrary encoding for path data while we would\nperfer to be dealing only with unicode strings. Since path data need to\nsurvive round-trip back to p4, this patch implements the general policy\nthat we store path data as-is, but decode them to unicode before doing\nany non-trivial processing.\n\nA new `decode_path()` method is provided that generally does the correct\nconversion, taking into account `git-p4.pathEncoding` configuration.\n\nFor python2.7, path strings will be left as-is if it only contains ASCII\ncharacters.\n\nFor python3, decoding is always done so that we have str objects.\n\nSigned-off-by: Yang Zhao \nReviewed-by: Ben Keene \n---\n git-p4.py | 69 +++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 44 insertions(+), 25 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex d62fb05989..d43f373b2d 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -150,6 +150,21 @@ def decode_text_stream(s):\n def encode_text_stream(s):\n return s.encode('utf_8') if isinstance(s, unicode) else s\n \n+def decode_path(path):\n+ \"\"\"Decode a given string (bytes or otherwise) using configured path encoding options\n+ \"\"\"\n+ encoding = gitConfig('git-p4.pathEncoding') or 'utf_8'\n+ if bytes is not str:\n+ return path.decode(encoding, errors='replace') if isinstance(path, bytes) else path\n+ else:\n+ try:\n+ path.decode('ascii')\n+ except:\n+ path = path.decode(encoding, errors='replace')\n+ if verbose:\n+ print('Path with non-ASCII characters detected. Used {} to decode: {}'.format(encoding, path))\n+ return path\n+\n def write_pipe(c, stdin):\n if verbose:\n sys.stderr.write('Writing pipe: %s\\n' % str(c))\n@@ -697,7 +712,8 @@ def p4Where(depotPath):\n if \"depotFile\" in entry:\n # Search for the base client side depot path, as long as it starts with the branch's P4 path.\n # The base path always ends with \"/...\".\n- if entry[\"depotFile\"].find(depotPath) == 0 and entry[\"depotFile\"][-4:] == \"/...\":\n+ entry_path = decode_path(entry['depotFile'])\n+ if entry_path.find(depotPath) == 0 and entry_path[-4:] == \"/...\":\n output = entry\n break\n elif \"data\" in entry:\n@@ -712,11 +728,11 @@ def p4Where(depotPath):\n return \"\"\n clientPath = \"\"\n if \"path\" in output:\n- clientPath = output.get(\"path\")\n+ clientPath = decode_path(output['path'])\n elif \"data\" in output:\n data = output.get(\"data\")\n- lastSpace = data.rfind(\" \")\n- clientPath = data[lastSpace + 1:]\n+ lastSpace = data.rfind(b\" \")\n+ clientPath = decode_path(data[lastSpace + 1:])\n \n if clientPath.endswith(\"...\"):\n clientPath = clientPath[:-3]\n@@ -2484,7 +2500,7 @@ def append(self, view_line):\n \n def convert_client_path(self, clientFile):\n # chop off //client/ part to make it relative\n- if not clientFile.startswith(self.client_prefix):\n+ if not decode_path(clientFile).startswith(self.client_prefix):\n die(\"No prefix '%s' on clientFile '%s'\" %\n (self.client_prefix, clientFile))\n return clientFile[len(self.client_prefix):]\n@@ -2493,7 +2509,7 @@ def update_client_spec_path_cache(self, files):\n \"\"\" Caching file paths by \"p4 where\" batch query \"\"\"\n \n # List depot file paths exclude that already cached\n- fileArgs = [f['path'] for f in files if f['path'] not in self.client_spec_path_cache]\n+ fileArgs = [f['path'] for f in files if decode_path(f['path']) not in self.client_spec_path_cache]\n \n if len(fileArgs) == 0:\n return # All files in cache\n@@ -2508,16 +2524,18 @@ def update_client_spec_path_cache(self, files):\n if \"unmap\" in res:\n # it will list all of them, but only one not unmap-ped\n continue\n+ depot_path = decode_path(res['depotFile'])\n if gitConfigBool(\"core.ignorecase\"):\n- res['depotFile'] = res['depotFile'].lower()\n- self.client_spec_path_cache[res['depotFile']] = self.convert_client_path(res[\"clientFile\"])\n+ depot_path = depot_path.lower()\n+ self.client_spec_path_cache[depot_path] = self.convert_client_path(res[\"clientFile\"])\n \n # not found files or unmap files set to \"\"\n for depotFile in fileArgs:\n+ depotFile = decode_path(depotFile)\n if gitConfigBool(\"core.ignorecase\"):\n depotFile = depotFile.lower()\n if depotFile not in self.client_spec_path_cache:\n- self.client_spec_path_cache[depotFile] = \"\"\n+ self.client_spec_path_cache[depotFile] = b''\n \n def map_in_client(self, depot_path):\n \"\"\"Return the relative location in the client where this\n@@ -2635,7 +2653,7 @@ def isPathWanted(self, path):\n elif path.lower() == p.lower():\n return False\n for p in self.depotPaths:\n- if p4PathStartsWith(path, p):\n+ if p4PathStartsWith(path, decode_path(p)):\n return True\n return False\n \n@@ -2644,7 +2662,7 @@ def extractFilesFromCommit(self, commit, shelved=False, shelved_cl = 0):\n fnum = 0\n while \"depotFile%s\" % fnum in commit:\n path = commit[\"depotFile%s\" % fnum]\n- found = self.isPathWanted(path)\n+ found = self.isPathWanted(decode_path(path))\n if not found:\n fnum = fnum + 1\n continue\n@@ -2678,7 +2696,7 @@ def stripRepoPath(self, path, prefixes):\n if self.useClientSpec:\n # branch detection moves files up a level (the branch name)\n # from what client spec interpretation gives\n- path = self.clientSpecDirs.map_in_client(path)\n+ path = decode_path(self.clientSpecDirs.map_in_client(path))\n if self.detectBranches:\n for b in self.knownBranches:\n if p4PathStartsWith(path, b + \"/\"):\n@@ -2712,14 +2730,15 @@ def splitFilesIntoBranches(self, commit):\n branches = {}\n fnum = 0\n while \"depotFile%s\" % fnum in commit:\n- path = commit[\"depotFile%s\" % fnum]\n+ raw_path = commit[\"depotFile%s\" % fnum]\n+ path = decode_path(raw_path)\n found = self.isPathWanted(path)\n if not found:\n fnum = fnum + 1\n continue\n \n file = {}\n- file[\"path\"] = path\n+ file[\"path\"] = raw_path\n file[\"rev\"] = commit[\"rev%s\" % fnum]\n file[\"action\"] = commit[\"action%s\" % fnum]\n file[\"type\"] = commit[\"type%s\" % fnum]\n@@ -2728,7 +2747,7 @@ def splitFilesIntoBranches(self, commit):\n # start with the full relative path where this file would\n # go in a p4 client\n if self.useClientSpec:\n- relPath = self.clientSpecDirs.map_in_client(path)\n+ relPath = decode_path(self.clientSpecDirs.map_in_client(path))\n else:\n relPath = self.stripRepoPath(path, self.depotPaths)\n \n@@ -2766,14 +2785,15 @@ def encodeWithUTF8(self, path):\n # - helper for streamP4Files\n \n def streamOneP4File(self, file, contents):\n- relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes)\n- relPath = self.encodeWithUTF8(relPath)\n+ file_path = file['depotFile']\n+ relPath = self.stripRepoPath(decode_path(file_path), self.branchPrefixes)\n+\n if verbose:\n if 'fileSize' in self.stream_file:\n size = int(self.stream_file['fileSize'])\n else:\n size = 0 # deleted files don't get a fileSize apparently\n- sys.stdout.write('\\r%s --> %s (%i MB)\\n' % (file['depotFile'], relPath, size/1024/1024))\n+ sys.stdout.write('\\r%s --> %s (%i MB)\\n' % (file_path, relPath, size/1024/1024))\n sys.stdout.flush()\n \n (type_base, type_mods) = split_p4_type(file[\"type\"])\n@@ -2791,7 +2811,7 @@ def streamOneP4File(self, file, contents):\n # to nothing. This causes p4 errors when checking out such\n # a change, and errors here too. Work around it by ignoring\n # the bad symlink; hopefully a future change fixes it.\n- print(\"\\nIgnoring empty symlink in %s\" % file['depotFile'])\n+ print(\"\\nIgnoring empty symlink in %s\" % file_path)\n return\n elif data[-1] == '\\n':\n contents = [data[:-1]]\n@@ -2810,7 +2830,7 @@ def streamOneP4File(self, file, contents):\n # just the native \"NT\" type.\n #\n try:\n- text = p4_read_pipe(['print', '-q', '-o', '-', '%s@%s' % (file['depotFile'], file['change'])])\n+ text = p4_read_pipe(['print', '-q', '-o', '-', '%s@%s' % (decode_path(file['depotFile']), file['change'])], raw=True)\n except Exception as e:\n if 'Translation of file content failed' in str(e):\n type_base = 'binary'\n@@ -2818,7 +2838,7 @@ def streamOneP4File(self, file, contents):\n raise e\n else:\n if p4_version_string().find('/NT') >= 0:\n- text = text.replace('\\r\\n', '\\n')\n+ text = text.replace(b'\\r\\n', b'\\n')\n contents = [ text ]\n \n if type_base == \"apple\":\n@@ -2849,8 +2869,7 @@ def streamOneP4File(self, file, contents):\n self.writeToGitStream(git_mode, relPath, contents)\n \n def streamOneP4Deletion(self, file):\n- relPath = self.stripRepoPath(file['path'], self.branchPrefixes)\n- relPath = self.encodeWithUTF8(relPath)\n+ relPath = self.stripRepoPath(decode_path(file['path']), self.branchPrefixes)\n if verbose:\n sys.stdout.write(\"delete %s\\n\" % relPath)\n sys.stdout.flush()\n@@ -3037,8 +3056,8 @@ def commit(self, details, files, branch, parent = \"\", allow_empty=False):\n if self.clientSpecDirs:\n self.clientSpecDirs.update_client_spec_path_cache(files)\n \n- files = [f for f in files\n- if self.inClientSpec(f['path']) and self.hasBranchPrefix(f['path'])]\n+ files = [f for (f, path) in ((f, decode_path(f['path'])) for f in files)\n+ if self.inClientSpec(path) and self.hasBranchPrefix(path)]\n \n if gitConfigBool('git-p4.keepEmptyCommits'):\n allow_empty = True\n-- \n2.21.0.windows.1\n\n"},{"id":"388194","messageId":"20191213235247.23660-11-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 09/14] git-p4: fix freezing while waiting for fast-import progress","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:43Z","receivedAt":"2019-12-13T23:53:33Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"As part of its importing process, git-p4 sends a `checkpoint` followed\nimmediately by `progress` to fast-import to force synchronization.\nDue to buffering, it is possible for the `progress` command to not be\nflushed before git-p4 begins to wait for the corresponding response.\nThis causes the script to freeze completely, and is consistently\nobservable at least on python-3.6.9.\n\nMake sure this command sequence is completely flushed before waiting.\n\nSigned-off-by: Yang Zhao \nReviewed-by: Ben Keene \n---\n git-p4.py | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex c7170c9ae6..68f9f4fdc6 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2641,6 +2641,7 @@ def __init__(self):\n def checkpoint(self):\n self.gitStream.write(\"checkpoint\\n\\n\")\n self.gitStream.write(\"progress checkpoint\\n\\n\")\n+ self.gitStream.flush()\n out = self.gitOutput.readline()\n if self.verbose:\n print(\"checkpoint finished: \" + out)\n-- \n2.21.0.windows.1\n\n"},{"id":"388195","messageId":"20191213235247.23660-12-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 10/14] git-p4: use functools.reduce instead of reduce","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:44Z","receivedAt":"2019-12-13T23:53:34Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"For python3, reduce() has been moved to functools.reduce(). This is\nalso available in python2.7.\n\nSigned-off-by: Yang Zhao \n---\n git-p4.py | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 68f9f4fdc6..7deee1b939 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -13,6 +13,7 @@\n sys.exit(1)\n import os\n import optparse\n+import functools\n import marshal\n import subprocess\n import tempfile\n@@ -1158,7 +1159,7 @@ def pushFile(self, localLargeFile):\n assert False, \"Method 'pushFile' required in \" + self.__class__.__name__\n \n def hasLargeFileExtension(self, relPath):\n- return reduce(\n+ return functools.reduce(\n lambda a, b: a or b,\n [relPath.endswith('.' + e) for e in gitConfigList('git-p4.largeFileExtensions')],\n False\n-- \n2.21.0.windows.1\n\n"},{"id":"388196","messageId":"20191213235247.23660-14-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 12/14] git-p4: simplify regex pattern generation for parsing diff-tree","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:46Z","receivedAt":"2019-12-13T23:53:35Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"It is not clear why a generator was used to create the regex used to\nparse git-diff-tree output; I assume an early implementation required\nit, but is not part of the mainline change.\n\nSimply use a lazily initialized global instead.\n\nSigned-off-by: Yang Zhao \nReviewed-by: Ben Keene \n---\n git-p4.py | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex b7e31d4738..3af8df9f83 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -544,12 +544,7 @@ def getGitTags():\n gitTags.add(tag)\n return gitTags\n \n-def diffTreePattern():\n- # This is a simple generator for the diff tree regex pattern. This could be\n- # a class variable if this and parseDiffTreeEntry were a part of a class.\n- pattern = re.compile(':(\\d+) (\\d+) (\\w+) (\\w+) ([A-Z])(\\d+)?\\t(.*?)((\\t(.*))|$)')\n- while True:\n- yield pattern\n+_diff_tree_pattern = None\n \n def parseDiffTreeEntry(entry):\n \"\"\"Parses a single diff tree entry into its component elements.\n@@ -570,7 +565,11 @@ def parseDiffTreeEntry(entry):\n \n If the pattern is not matched, None is returned.\"\"\"\n \n- match = diffTreePattern().next().match(entry)\n+ global _diff_tree_pattern\n+ if not _diff_tree_pattern:\n+ _diff_tree_pattern = re.compile(':(\\d+) (\\d+) (\\w+) (\\w+) ([A-Z])(\\d+)?\\t(.*?)((\\t(.*))|$)')\n+\n+ match = _diff_tree_pattern.match(entry)\n if match:\n return {\n 'src_mode': match.group(1),\n-- \n2.21.0.windows.1\n\n"},{"id":"388197","messageId":"20191213235247.23660-13-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 11/14] git-p4: use dict.items() iteration for python3 compatibility","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:45Z","receivedAt":"2019-12-13T23:53:36Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Python3 uses dict.items() instead of .iteritems() to provide\niteratoration over dict. Although items() is technically less efficient\nfor python2.7 (allocates a new list instead of simply iterating), the\namount of data involved is very small and the penalty negligible.\n\nSigned-off-by: Yang Zhao \nReviewed-by: Ben Keene \n---\n git-p4.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 7deee1b939..b7e31d4738 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -1745,7 +1745,7 @@ def prepareSubmitTemplate(self, changelist=None):\n break\n if not change_entry:\n die('Failed to decode output of p4 change -o')\n- for key, value in change_entry.iteritems():\n+ for key, value in change_entry.items():\n if key.startswith('File'):\n if 'depot-paths' in settings:\n if not [p for p in settings['depot-paths']\n-- \n2.21.0.windows.1\n\n"},{"id":"388198","messageId":"20191213235247.23660-16-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[RFC PATCH v2 14/14] ci: also run linux-gcc pipeline with python3.5 environment","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:48Z","receivedAt":"2019-12-13T23:53:37Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"git-p4.py includes support for python3, but this was not previously\nvalidated in CI. Lets actually do that.\n\nAs of writing, python-3.5 has reached end-of-life, but has been updated\nrecently enough that it's reasonable to attemp to support it. We do not\nhave a pressing need for features only available in 3.6 and later.\n\nUsage of python3 is limited to the linux-gcc pipeline on Azure. It is\nassumed that passing both python2 and python3 tests on one platform\ntranslates to doing the same on others.\n\nTravis-CI is unchanged, as no tests are run in those environments.\n\nSigned-off-by: Yang Zhao \n---\n\nThere has been some desire to make this more generally applicable instead\nof being Azure Pipelines specific. There may be some significant work\nrequired to make that work for all platforms where t98** tests are being run.\n\nI most likely won't have the bandwith to take on that task in the near future.\nIf this patch is deemed insufficient, I would perfer it be dropped form the\nseries rather than become a roadblock.\n\nPrevious discussion thread here:\nhttp://public-inbox.org/git/20191210103014.GF6527@szeder.dev/\n\n azure-pipelines.yml | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex af2a5ea484..c473365812 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -331,7 +331,18 @@ jobs:\n displayName: linux-gcc\n condition: succeeded()\n pool: Hosted Ubuntu 1604\n+ strategy:\n+ matrix:\n+ python27:\n+ python.version: '2.7'\n+ python35:\n+ python.version: '3.5'\n steps:\n+ - task: UsePythonVersion@0\n+ inputs:\n+ versionSpec: '$(python.version)'\n+ - bash: |\n+ echo \"##vso[task.setvariable variable=python_path]$(which python)\"\n - bash: |\n test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n \n-- \n2.21.0.windows.1\n\n"},{"id":"388199","messageId":"20191213235247.23660-15-yang.zhao@skyboxlabs.com","threadId":"52454","inReplyTo":"20191213235247.23660-1-yang.zhao@skyboxlabs.com","subject":"[PATCH v2 13/14] git-p4: use python3's input() everywhere","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2019-12-13T23:52:47Z","receivedAt":"2019-12-13T23:53:40Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"Python3 deprecates raw_input() from 2.7 and replaced it with input().\nSince we do not need 2.7's input() semantics, `raw_input()` is aliased\nto `input()` for easy forward compatability.\n\nSigned-off-by: Yang Zhao \n---\n git-p4.py | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 3af8df9f83..17f72f4309 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -27,6 +27,16 @@\n import ctypes\n import errno\n \n+# On python2.7 where raw_input() and input() are both availble,\n+# we want raw_input's semantics, but aliased to input for python3\n+# compatibility\n+# support basestring in python3\n+try:\n+ if raw_input and input:\n+ input = raw_input\n+except:\n+ pass\n+\n verbose = False\n \n # Only labels/tags matching this will be imported/exported\n@@ -1801,7 +1811,7 @@ def edit_template(self, template_file):\n return True\n \n while True:\n- response = raw_input(\"Submit template unchanged. Submit anyway? [y]es, [n]o (skip this patch) \")\n+ response = input(\"Submit template unchanged. Submit anyway? [y]es, [n]o (skip this patch) \")\n if response == 'y':\n return True\n if response == 'n':\n@@ -2372,7 +2382,7 @@ def run(self, args):\n # prompt for what to do, or use the option/variable\n if self.conflict_behavior == \"ask\":\n print(\"What do you want to do?\")\n- response = raw_input(\"[s]kip this commit but apply\"\n+ response = input(\"[s]kip this commit but apply\"\n \" the rest, or [q]uit? \")\n if not response:\n continue\n-- \n2.21.0.windows.1\n\n"},{"id":"388410","messageId":"xmqq1rt2ftv5.fsf@gitster-ct.c.googlers.com","threadId":"52454","inReplyTo":"20191213235247.23660-6-yang.zhao@skyboxlabs.com","subject":"Re: [PATCH v2 04/14] git-p4: encode/decode communication with p4 for python3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-12-17T22:51:58Z","receivedAt":"2019-12-17T22:52:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Yang Zhao writes:\n\n> The marshalled dict in the response given on STDOUT by p4 uses `str` for\n> keys and string values. When run using python3, these values are\n> deserialized as `bytes`, leading to a whole host of problems as the rest\n> of the code assumes `str` is used throughout.\n>\n> This patch changes the deserialization behaviour such that, as much as\n> possible, text output from p4 is decoded to native unicode strings.\n> Exceptions are made for the field `data` as it is usually arbitrary\n> binary data. `depotFile[0-9]*`, `path`, and `clientFile` are also exempt\n> as they contain path strings not encoded with UTF-8, and must survive\n> survive round-trip back to p4.\n\nDoubled \"survive\"; will drop one of them while queuing.\n"},{"id":"390027","messageId":"CABvFv3LH2tYbobn8xBykyC43FB+z16FFOSAX1XjTLFZ2BUqZ1Q@mail.gmail.com","threadId":"52454","inReplyTo":"20191213235247.23660-2-yang.zhao@skyboxlabs.com","subject":"Re: [PATCH v2 00/14] git-p4: python3 compatibility","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2020-01-17T22:00:40Z","receivedAt":"2020-01-17T22:00:54Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"On Fri, Dec 13, 2019 at 3:53 PM Yang Zhao wrote:\n> This patchset adds python3 compatibility to git-p4.\n\nPing?\n"},{"id":"390387","messageId":"CAE5ih7-LrK1XZgjhpx3qQqRMQ5MUf-e0q=45VY4cNhHV--nSuQ@mail.gmail.com","threadId":"52454","inReplyTo":"CABvFv3LH2tYbobn8xBykyC43FB+z16FFOSAX1XjTLFZ2BUqZ1Q@mail.gmail.com","subject":"Re: [PATCH v2 00/14] git-p4: python3 compatibility","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2020-01-24T20:14:53Z","receivedAt":"2020-01-24T20:15:09Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On Fri, 17 Jan 2020 at 22:00, Yang Zhao wrote:\n>\n> On Fri, Dec 13, 2019 at 3:53 PM Yang Zhao wrote:\n> > This patchset adds python3 compatibility to git-p4.\n>\n> Ping?\n\nHi!\n\nSorry, I've been a bit busy.\n\nI quickly tried this and it was failing for me with:\n\n> failure accessing depot: could not connect\n\nIt works fine with the interpreter set to /usr/bin/python, but\nchanging it to python3 caused this problem.\n\nI'm at 050387ce43179f1b0da364dd6eec10ce578d6583.\n\nThanks,\nLuke\n"},{"id":"390791","messageId":"CAE5ih7-ngXsn1+RYSJ9qk_Ajr9x9zLRWNCyUiBGjg=sx_cOuzA@mail.gmail.com","threadId":"52454","inReplyTo":"CAE5ih7-LrK1XZgjhpx3qQqRMQ5MUf-e0q=45VY4cNhHV--nSuQ@mail.gmail.com","subject":"Re: [PATCH v2 00/14] git-p4: python3 compatibility","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2020-01-30T13:35:49Z","receivedAt":"2020-01-30T13:36:04Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On Fri, 24 Jan 2020 at 20:14, Luke Diamand wrote:\n>\n> On Fri, 17 Jan 2020 at 22:00, Yang Zhao wrote:\n> >\n> > On Fri, Dec 13, 2019 at 3:53 PM Yang Zhao wrote:\n> > > This patchset adds python3 compatibility to git-p4.\n> >\n> > Ping?\n>\n> Hi!\n>\n> Sorry, I've been a bit busy.\n>\n> I quickly tried this and it was failing for me with:\n>\n> > failure accessing depot: could not connect\n>\n> It works fine with the interpreter set to /usr/bin/python, but\n> changing it to python3 caused this problem.\n>\n> I'm at 050387ce43179f1b0da364dd6eec10ce578d6583.\n\nI'm using it for day-to-day work, so far it's working well, thanks!\n"},{"id":"391040","messageId":"CAE5ih78dpmPhdsR73cwGJuDYQECDaCvdOuosOAhLyLun9DyL_A@mail.gmail.com","threadId":"52454","inReplyTo":"CAE5ih7-ngXsn1+RYSJ9qk_Ajr9x9zLRWNCyUiBGjg=sx_cOuzA@mail.gmail.com","subject":"Re: [PATCH v2 00/14] git-p4: python3 compatibility","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2020-02-03T12:54:46Z","receivedAt":"2020-02-03T12:55:00Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On Thu, 30 Jan 2020 at 13:35, Luke Diamand wrote:\n>\n> On Fri, 24 Jan 2020 at 20:14, Luke Diamand wrote:\n> >\n> > On Fri, 17 Jan 2020 at 22:00, Yang Zhao wrote:\n> > >\n> > > On Fri, Dec 13, 2019 at 3:53 PM Yang Zhao wrote:\n> > > > This patchset adds python3 compatibility to git-p4.\n> > >\n> > > Ping?\n\nOne very small bug report:\n\nWhen doing \"git p4 sync\" it prints out the percent complete. It looks\nlike it's no longer rounding it sensibly, so where before it would say\n77%, now it says 77.7777777%.\n\nIt's this line:\n\n sys.stdout.write(\"\\rImporting revision %s (%s%%)\" % (change, cnt *\n100 / len(changes)))\n\nI think / just needs replacing with //.\n\n\n> >\n> > Hi!\n> >\n> > Sorry, I've been a bit busy.\n> >\n> > I quickly tried this and it was failing for me with:\n> >\n> > > failure accessing depot: could not connect\n> >\n> > It works fine with the interpreter set to /usr/bin/python, but\n> > changing it to python3 caused this problem.\n> >\n> > I'm at 050387ce43179f1b0da364dd6eec10ce578d6583.\n>\n> I'm using it for day-to-day work, so far it's working well, thanks!\n"},{"id":"391050","messageId":"CABvFv3KcpmCXsM2hSaHTzrb0Lz9BAD23sVcNsjO4uf8xCLDh5g@mail.gmail.com","threadId":"52454","inReplyTo":"CAE5ih78dpmPhdsR73cwGJuDYQECDaCvdOuosOAhLyLun9DyL_A@mail.gmail.com","subject":"Re: [PATCH v2 00/14] git-p4: python3 compatibility","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2020-02-03T18:11:05Z","receivedAt":"2020-02-03T18:11:20Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"On Mon, Feb 3, 2020 at 4:54 AM Luke Diamand wrote:\n> One very small bug report:\n>\n> When doing \"git p4 sync\" it prints out the percent complete. It looks\n> like it's no longer rounding it sensibly, so where before it would say\n> 77%, now it says 77.7777777%.\n>\n> It's this line:\n>\n> sys.stdout.write(\"\\rImporting revision %s (%s%%)\" % (change, cnt *\n> 100 / len(changes)))\n>\n> I think / just needs replacing with //.\n\nGood catch.\n\nThe patch below should do the trick, and be more explicit about how\nwe're formatting things.\n\nI'll roll it into the GitHub branch.\n\ndiff --git a/git-p4.py b/git-p4.py\nindex ca0a874501..183959ec8c 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2960,7 +2960,7 @@ def streamP4FilesCb(self, marshalled):\n size = int(self.stream_file[\"fileSize\"])\n if size > 0:\n progress = 100*self.stream_file['streamContentSize']/size\n- sys.stdout.write('\\r%s %d%% (%i MB)' %\n(self.stream_file['depotFile'], progress, int(size/1024/1024)))\n+ sys.stdout.write('\\r{} {:.0%} ({}\nMB)'.format(self.stream_file['depotFile'], progress,\nint(size/1024/1024)))\n sys.stdout.flush()\n\n self.stream_have_file_info = True\n@@ -3435,7 +3435,7 @@ def importChanges(self, changes, origin_revision=0):\n self.updateOptionDict(description)\n\n if not self.silent:\n- sys.stdout.write(\"\\rImporting revision %s (%s%%)\" %\n(change, cnt * 100 / len(changes)))\n+ sys.stdout.write(\"\\rImporting revision {}\n({:.0%})\".format(change, cnt / len(changes)))\n sys.stdout.flush()\n cnt = cnt + 1\n"},{"id":"391069","messageId":"CABvFv3KPZ9ibx9+WDkgWJ9Zn1_VyuoX_KiOPxYSvL2AwRVsMxg@mail.gmail.com","threadId":"52454","inReplyTo":"CABvFv3KcpmCXsM2hSaHTzrb0Lz9BAD23sVcNsjO4uf8xCLDh5g@mail.gmail.com","subject":"Re: [PATCH v2 00/14] git-p4: python3 compatibility","fromName":"Yang Zhao","fromEmail":"yang.zhao@skyboxlabs.com","sentAt":"2020-02-04T01:35:58Z","receivedAt":"2020-02-04T01:36:12Z","isPatch":true,"sender":{"key":"yang.zhao@skyboxlabs.com","avatar":"https://avatars.githubusercontent.com/u/45857825?v=4"},"body":"On Mon, Feb 3, 2020 at 10:11 AM Yang Zhao wrote:\n>\n> On Mon, Feb 3, 2020 at 4:54 AM Luke Diamand wrote:\n> > One very small bug report:\n> >\n> > When doing \"git p4 sync\" it prints out the percent complete. It looks\n> > like it's no longer rounding it sensibly, so where before it would say\n> > 77%, now it says 77.7777777%.\n> >\n> > It's this line:\n> >\n> > sys.stdout.write(\"\\rImporting revision %s (%s%%)\" % (change, cnt *\n> > 100 / len(changes)))\n> >\n> > I think / just needs replacing with //.\n>\n> Good catch.\n>\n> The patch below should do the trick, and be more explicit about how\n> we're formatting things.\n\nOops, missed a line. Below is more correct (pardon the webmail patch clobbering)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex ca0a874501..da3a5aa684 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2959,8 +2959,8 @@ def streamP4FilesCb(self, marshalled):\n 'depotFile' in self.stream_file):\n size = int(self.stream_file[\"fileSize\"])\n if size > 0:\n- progress = 100*self.stream_file['streamContentSize']/size\n- sys.stdout.write('\\r%s %d%% (%i MB)' %\n(self.stream_file['depotFile'], progress, int(size/1024/1024)))\n+ progress = self.stream_file['streamContentSize']/size\n+ sys.stdout.write('\\r{} {:.0%} ({}\nMB)'.format(self.stream_file['depotFile'], progress,\nint(size/1024/1024)))\n sys.stdout.flush()\n\n self.stream_have_file_info = True\n@@ -3435,7 +3435,7 @@ def importChanges(self, changes, origin_revision=0):\n self.updateOptionDict(description)\n\n if not self.silent:\n- sys.stdout.write(\"\\rImporting revision %s (%s%%)\" %\n(change, cnt * 100 / len(changes)))\n+ sys.stdout.write(\"\\rImporting revision {}\n({:.0%})\".format(change, cnt / len(changes)))\n sys.stdout.flush()\n cnt = cnt + 1\n"}]}