{"thread":{"id":"32609","subject":"[PATCH 0/8] Initial support for Python 3","startedAt":"2013-01-12T19:23:38Z","lastAt":"2013-01-19T07:52:16Z","messageCount":53,"participants":["John Keeping","Pete Wyckoff","Michael Haggerty","Junio C Hamano","Sverre Rabbelier"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"206615","messageId":"cover.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":null,"subject":"[PATCH 0/8] Initial support for Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:38Z","receivedAt":"2013-01-12T19:23:38Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"I started having a look to see how much work would be needed to make Git\nwork with Python 3 and the answer is mostly not much.  The exception is\ngit-p4.py which is hit hard by the distinction between byte strings and\nunicode strings, particularly because the Python output mode of p4\ntargets Python 2.\n\nI don't know if it's worthwhile to actually apply these but here they\nare in case anyone's interested.\n\nHaving said that, the changes are minimal and involve either wrapping\nparentheses around arguments to print or being a bit more explicit about\nhow we expect byte strings to be decoded to unicode.\n\nWith these patches all tests pass with python3 except t98* (git-p4), but\nthere are a couple of topics in-flight which will affect that\n(fc/remote-testgit-feature-done and er/replace-cvsimport).\n\nJohn Keeping (8):\n  git_remote_helpers: Allow building with Python 3\n  git_remote_helpers: fix input when running under Python 3\n  git_remote_helpers: Force rebuild if python version changes\n  git_remote_helpers: Use 2to3 if building with Python 3\n  svn-fe: allow svnrdump_sim.py to run with Python 3\n  git-remote-testpy: hash bytes explicitly\n  git-remote-testpy: don't do unbuffered text I/O\n  git-remote-testpy: call print as a function\n\n contrib/svn-fe/svnrdump_sim.py     |  4 ++--\n git-remote-testpy.py               | 40 +++++++++++++++++++-------------------\n git_remote_helpers/.gitignore      |  1 +\n git_remote_helpers/Makefile        | 10 ++++++++--\n git_remote_helpers/git/importer.py |  2 +-\n git_remote_helpers/setup.py        | 10 ++++++++++\n 6 files changed, 42 insertions(+), 25 deletions(-)\n\n-- \n1.8.1\n"},{"id":"206616","messageId":"225a94d3d14770d46f48ac1a9309e660d24f4441.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH 1/8] git_remote_helpers: Allow building with Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:39Z","receivedAt":"2013-01-12T19:23:39Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Change inline Python to call \"print\" as a function not a statement.\n\nThis is harmless because Python 2 will see the parentheses as redundant\ngrouping but they are necessary to run this code with Python 3.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git_remote_helpers/Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\nindex 74b05dc..f65f064 100644\n--- a/git_remote_helpers/Makefile\n+++ b/git_remote_helpers/Makefile\n@@ -23,7 +23,7 @@ endif\n \n PYLIBDIR=$(shell $(PYTHON_PATH) -c \\\n \t \"import sys; \\\n-\t print 'lib/python%i.%i/site-packages' % sys.version_info[:2]\")\n+\t print('lib/python%i.%i/site-packages' % sys.version_info[:2])\")\n \n all: $(pysetupfile)\n \t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n-- \n1.8.1\n"},{"id":"206617","messageId":"a8c3aabfab64f49fa0cbb2d45bda79997a875ee8.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH 2/8] git_remote_helpers: fix input when running under Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:40Z","receivedAt":"2013-01-12T19:23:40Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Although 2to3 will fix most issues in Python 2 code to make it run under\nPython 3, it does not handle the new strict separation between byte\nstrings and unicode strings.  There is one instance in\ngit_remote_helpers where we are caught by this.\n\nFix it by explicitly decoding the incoming byte string into a unicode\nstring.  In this instance, use the locale under which the application is\nrunning.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git_remote_helpers/git/importer.py | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\nindex e28cc8f..6814003 100644\n--- a/git_remote_helpers/git/importer.py\n+++ b/git_remote_helpers/git/importer.py\n@@ -20,7 +20,7 @@ class GitImporter(object):\n         \"\"\"Returns a dictionary with refs.\n         \"\"\"\n         args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n-        lines = check_output(args).strip().split('\\n')\n+        lines = check_output(args).decode().strip().split('\\n')\n         refs = {}\n         for line in lines:\n             value, name = line.split(' ')\n-- \n1.8.1\n"},{"id":"206618","messageId":"89f55d20da9a4c0a8490f95107cbf5d04219d0fb.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH 3/8] git_remote_helpers: Force rebuild if python version changes","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:41Z","receivedAt":"2013-01-12T19:23:41Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"When different version of python are used to build via distutils, the\nbehaviour can change.  Detect changes in version and pass --force in\nthis case.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git_remote_helpers/.gitignore | 1 +\n git_remote_helpers/Makefile   | 8 +++++++-\n 2 files changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/git_remote_helpers/.gitignore b/git_remote_helpers/.gitignore\nindex 2247d5f..06c664f 100644\n--- a/git_remote_helpers/.gitignore\n+++ b/git_remote_helpers/.gitignore\n@@ -1,2 +1,3 @@\n+/GIT-PYTHON_VERSION\n /build\n /dist\ndiff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\nindex f65f064..91f458f 100644\n--- a/git_remote_helpers/Makefile\n+++ b/git_remote_helpers/Makefile\n@@ -25,8 +25,14 @@ PYLIBDIR=$(shell $(PYTHON_PATH) -c \\\n \t \"import sys; \\\n \t print('lib/python%i.%i/site-packages' % sys.version_info[:2])\")\n \n+py_version=$(shell $(PYTHON_PATH) -c \\\n+\t'import sys; print(\"%i.%i\" % sys.version_info[:2])')\n+\n all: $(pysetupfile)\n-\t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n+\t$(QUIET)test \"$$(cat GIT-PYTHON_VERSION 2>/dev/null)\" = \"$(py_version)\" || \\\n+\tflags=--force; \\\n+\t$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build $$flags\n+\t$(QUIET)echo \"$(py_version)\" >GIT-PYTHON_VERSION\n \n install: $(pysetupfile)\n \t$(PYTHON_PATH) $(pysetupfile) install --prefix $(DESTDIR_SQ)$(prefix)\n-- \n1.8.1\n"},{"id":"206619","messageId":"4a17f813970d0e6c4c7795bc2c0ac7842465e41d.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH 4/8] git_remote_helpers: Use 2to3 if building with Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:42Z","receivedAt":"2013-01-12T19:23:42Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Using the approach detailed on the Python wiki[1], run 2to3 on the code\nas part of the build if building with Python 3.\n\nThe code itself requires no changes to convert cleanly.\n\n[1] http://wiki.python.org/moin/PortingPythonToPy3k\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git_remote_helpers/setup.py | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/git_remote_helpers/setup.py b/git_remote_helpers/setup.py\nindex 4d434b6..6de41de 100644\n--- a/git_remote_helpers/setup.py\n+++ b/git_remote_helpers/setup.py\n@@ -4,6 +4,15 @@\n \n from distutils.core import setup\n \n+# If building under Python3 we need to run 2to3 on the code, do this by\n+# trying to import distutils' 2to3 builder, which is only available in\n+# Python3.\n+try:\n+    from distutils.command.build_py import build_py_2to3 as build_py\n+except ImportError:\n+    # 2.x\n+    from distutils.command.build_py import build_py\n+\n setup(\n     name = 'git_remote_helpers',\n     version = '0.1.0',\n@@ -14,4 +23,5 @@ setup(\n     url = 'http://www.git-scm.com/',\n     package_dir = {'git_remote_helpers': ''},\n     packages = ['git_remote_helpers', 'git_remote_helpers.git'],\n+    cmdclass = {'build_py': build_py},\n )\n-- \n1.8.1\n"},{"id":"206620","messageId":"fcc99192ded58bf017a56a3e2f3cfb2fce06bd5b.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH 5/8] svn-fe: allow svnrdump_sim.py to run with Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:43Z","receivedAt":"2013-01-12T19:23:43Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"The changes to allow this script to run with Python 3 are minimal and do\nnot affect its functionality on the versions of Python 2 that are\nalready supported (2.4 onwards).\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n contrib/svn-fe/svnrdump_sim.py | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/svn-fe/svnrdump_sim.py b/contrib/svn-fe/svnrdump_sim.py\nindex 17cf6f9..4e78a1c 100755\n--- a/contrib/svn-fe/svnrdump_sim.py\n+++ b/contrib/svn-fe/svnrdump_sim.py\n@@ -14,7 +14,7 @@ if sys.hexversion < 0x02040000:\n \n def getrevlimit():\n         var = 'SVNRMAX'\n-        if os.environ.has_key(var):\n+        if var in os.environ:\n                 return os.environ[var]\n         return None\n \n@@ -44,7 +44,7 @@ def writedump(url, lower, upper):\n \n if __name__ == \"__main__\":\n         if not (len(sys.argv) in (3, 4, 5)):\n-                print \"usage: %s dump URL -rLOWER:UPPER\"\n+                print(\"usage: %s dump URL -rLOWER:UPPER\")\n                 sys.exit(1)\n         if not sys.argv[1] == 'dump': raise NotImplementedError('only \"dump\" is suppported.')\n         url = sys.argv[2]\n-- \n1.8.1\n"},{"id":"206621","messageId":"111ef16a926ae37a075304634d3fb2976fe6dee7.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH 6/8] git-remote-testpy: hash bytes explicitly","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:44Z","receivedAt":"2013-01-12T19:23:44Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Under Python 3 'hasher.update(...)' must take a byte string and not a\nunicode string.  Explicitly encode the argument to this method as UTF-8\nso that this code works under Python 3.\n\nThis moves the required Python version forward to 2.0.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git-remote-testpy.py | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/git-remote-testpy.py b/git-remote-testpy.py\nindex e4533b1..58aa1ae 100644\n--- a/git-remote-testpy.py\n+++ b/git-remote-testpy.py\n@@ -31,9 +31,9 @@ from git_remote_helpers.git.exporter import GitExporter\n from git_remote_helpers.git.importer import GitImporter\n from git_remote_helpers.git.non_local import NonLocalGit\n \n-if sys.hexversion < 0x01050200:\n-    # os.makedirs() is the limiter\n-    sys.stderr.write(\"git-remote-testgit: requires Python 1.5.2 or later.\\n\")\n+if sys.hexversion < 0x02000000:\n+    # string.encode() is the limiter\n+    sys.stderr.write(\"git-remote-testgit: requires Python 2.0 or later.\\n\")\n     sys.exit(1)\n \n def get_repo(alias, url):\n@@ -45,7 +45,7 @@ def get_repo(alias, url):\n     repo.get_head()\n \n     hasher = _digest()\n-    hasher.update(repo.path)\n+    hasher.update(repo.path.encode('utf-8'))\n     repo.hash = hasher.hexdigest()\n \n     repo.get_base_path = lambda base: os.path.join(\n-- \n1.8.1\n"},{"id":"206622","messageId":"d6f040809ad7457ff5f97fb23a2531988938a96a.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH 7/8] git-remote-testpy: don't do unbuffered text I/O","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:45Z","receivedAt":"2013-01-12T19:23:45Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Python 3 forbids unbuffered I/O in text mode.  Change the reading of\nstdin in git-remote-testpy so that we read the lines as bytes and then\ndecode them a line at a time.\n\nThis allows us to keep the I/O unbuffered in order to avoid\nreintroducing the bug fixed by commit 7fb8e16 (git-remote-testgit: fix\nrace when spawning fast-import).\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git-remote-testpy.py | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/git-remote-testpy.py b/git-remote-testpy.py\nindex 58aa1ae..815222f 100644\n--- a/git-remote-testpy.py\n+++ b/git-remote-testpy.py\n@@ -154,7 +154,7 @@ def do_import(repo, args):\n     refs = [ref]\n \n     while True:\n-        line = sys.stdin.readline()\n+        line = sys.stdin.readline().decode()\n         if line == '\\n':\n             break\n         if not line.startswith('import '):\n@@ -217,7 +217,7 @@ def read_one_line(repo):\n \n     line = sys.stdin.readline()\n \n-    cmdline = line\n+    cmdline = line.decode()\n \n     if not cmdline:\n         warn(\"Unexpected EOF\")\n@@ -269,7 +269,7 @@ def main(args):\n \n     more = True\n \n-    sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n+    sys.stdin = os.fdopen(sys.stdin.fileno(), 'rb', 0)\n     while (more):\n         more = read_one_line(repo)\n \n-- \n1.8.1\n"},{"id":"206623","messageId":"96d803276c95af6bf909bc48e8e984cc7ba25ead.1358018078.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH 8/8] git-remote-testpy: call print as a function","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-12T19:23:46Z","receivedAt":"2013-01-12T19:23:46Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"This is harmless in Python 2, which sees the parentheses as redundant\ngrouping, but is required for Python 3.  Since this is the only change\nrequired to make this script just run under Python 3 without needing\n2to3 it seems worthwhile.\n\nThe case of an empty print must be handled specially because in that\ncase Python 2 will interpret '()' as an empty tuple and print it as\n'()'; inserting an empty string fixes this.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git-remote-testpy.py | 26 +++++++++++++-------------\n 1 file changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/git-remote-testpy.py b/git-remote-testpy.py\nindex 815222f..8ba5d28 100644\n--- a/git-remote-testpy.py\n+++ b/git-remote-testpy.py\n@@ -87,9 +87,9 @@ def do_capabilities(repo, args):\n     \"\"\"Prints the supported capabilities.\n     \"\"\"\n \n-    print \"import\"\n-    print \"export\"\n-    print \"refspec refs/heads/*:%s*\" % repo.prefix\n+    print(\"import\")\n+    print(\"export\")\n+    print(\"refspec refs/heads/*:%s*\" % repo.prefix)\n \n     dirname = repo.get_base_path(repo.gitdir)\n \n@@ -98,11 +98,11 @@ def do_capabilities(repo, args):\n \n     path = os.path.join(dirname, 'git.marks')\n \n-    print \"*export-marks %s\" % path\n+    print(\"*export-marks %s\" % path)\n     if os.path.exists(path):\n-        print \"*import-marks %s\" % path\n+        print(\"*import-marks %s\" % path)\n \n-    print # end capabilities\n+    print('') # end capabilities\n \n \n def do_list(repo, args):\n@@ -115,16 +115,16 @@ def do_list(repo, args):\n \n     for ref in repo.revs:\n         debug(\"? refs/heads/%s\", ref)\n-        print \"? refs/heads/%s\" % ref\n+        print(\"? refs/heads/%s\" % ref)\n \n     if repo.head:\n         debug(\"@refs/heads/%s HEAD\" % repo.head)\n-        print \"@refs/heads/%s HEAD\" % repo.head\n+        print(\"@refs/heads/%s HEAD\" % repo.head)\n     else:\n         debug(\"@refs/heads/master HEAD\")\n-        print \"@refs/heads/master HEAD\"\n+        print(\"@refs/heads/master HEAD\")\n \n-    print # end list\n+    print('') # end list\n \n \n def update_local_repo(repo):\n@@ -167,7 +167,7 @@ def do_import(repo, args):\n     repo = update_local_repo(repo)\n     repo.exporter.export_repo(repo.gitdir, refs)\n \n-    print \"done\"\n+    print(\"done\")\n \n \n def do_export(repo, args):\n@@ -184,8 +184,8 @@ def do_export(repo, args):\n         repo.non_local.push(repo.gitdir)\n \n     for ref in changed:\n-        print \"ok %s\" % ref\n-    print\n+        print(\"ok %s\" % ref)\n+    print('')\n \n \n COMMANDS = {\n-- \n1.8.1\n"},{"id":"206628","messageId":"20130112233044.GB23079@padd.com","threadId":"32609","inReplyTo":"89f55d20da9a4c0a8490f95107cbf5d04219d0fb.1358018078.git.john@keeping.me.uk","subject":"Re: [PATCH 3/8] git_remote_helpers: Force rebuild if python version changes","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-01-12T23:30:44Z","receivedAt":"2013-01-12T23:30:44Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"john@keeping.me.uk wrote on Sat, 12 Jan 2013 19:23 +0000:\n> When different version of python are used to build via distutils, the\n> behaviour can change.  Detect changes in version and pass --force in\n> this case.\n[..]\n> diff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\n[..]\n> +py_version=$(shell $(PYTHON_PATH) -c \\\n> +\t'import sys; print(\"%i.%i\" % sys.version_info[:2])')\n> +\n>  all: $(pysetupfile)\n> -\t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n> +\t$(QUIET)test \"$$(cat GIT-PYTHON_VERSION 2>/dev/null)\" = \"$(py_version)\" || \\\n> +\tflags=--force; \\\n> +\t$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build $$flags\n> +\t$(QUIET)echo \"$(py_version)\" >GIT-PYTHON_VERSION\n\nCan you depend on ../GIT-PYTHON-VARS instead?  It comes from\n96a4647 (Makefile: detect when PYTHON_PATH changes, 2012-12-18).\nIt doesn't check version, just path, but hopefully that's good\nenough.  I'm imagining a rule that would do \"clean\" if\n../GIT-PYTHON-VARS changed, then build without --force.\n\n\t\t-- Pete\n"},{"id":"206629","messageId":"20130112234304.GC23079@padd.com","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"Re: [PATCH 0/8] Initial support for Python 3","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-01-12T23:43:04Z","receivedAt":"2013-01-12T23:43:04Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"john@keeping.me.uk wrote on Sat, 12 Jan 2013 19:23 +0000:\n> I started having a look to see how much work would be needed to make Git\n> work with Python 3 and the answer is mostly not much.  The exception is\n> git-p4.py which is hit hard by the distinction between byte strings and\n> unicode strings, particularly because the Python output mode of p4\n> targets Python 2.\n> \n> I don't know if it's worthwhile to actually apply these but here they\n> are in case anyone's interested.\n> \n> Having said that, the changes are minimal and involve either wrapping\n> parentheses around arguments to print or being a bit more explicit about\n> how we expect byte strings to be decoded to unicode.\n> \n> With these patches all tests pass with python3 except t98* (git-p4), but\n> there are a couple of topics in-flight which will affect that\n> (fc/remote-testgit-feature-done and er/replace-cvsimport).\n> \n> John Keeping (8):\n>   git_remote_helpers: Allow building with Python 3\n>   git_remote_helpers: fix input when running under Python 3\n>   git_remote_helpers: Force rebuild if python version changes\n>   git_remote_helpers: Use 2to3 if building with Python 3\n>   svn-fe: allow svnrdump_sim.py to run with Python 3\n>   git-remote-testpy: hash bytes explicitly\n>   git-remote-testpy: don't do unbuffered text I/O\n>   git-remote-testpy: call print as a function\n> \n>  contrib/svn-fe/svnrdump_sim.py     |  4 ++--\n>  git-remote-testpy.py               | 40 +++++++++++++++++++-------------------\n>  git_remote_helpers/.gitignore      |  1 +\n>  git_remote_helpers/Makefile        | 10 ++++++++--\n>  git_remote_helpers/git/importer.py |  2 +-\n>  git_remote_helpers/setup.py        | 10 ++++++++++\n>  6 files changed, 42 insertions(+), 25 deletions(-)\n\nThese look good, in that there are relatively few changed needed.\n\nSebastian Morr tried a similar patch a year ago, in\n\n    http://thread.gmane.org/gmane.comp.version-control.git/187545\n\nHe made changes beyond yours, in particular \"print >>\" lines,\nthat you seem to handle with 2to3 during the build.  I'm not sure\nwhich approach is better in the long run.  He worked on the\nother .py in contrib/ too.\n\nCan you give me some hints about the byte/unicode string issues\nin git-p4.py?  There's really only one place that does:\n\n    p4 = subprocess.Popen(\"p4 -G ...\")\n    marshal.load(p4.stdout)\n\nIf that's the only issue, this might not be too paniful.\n\nI hesitated to take Sebastian's changes due to the huge number of\nprint() lines, but maybe a 2to3 approach would make that aspect\nof python3 support not too onerous.\n\n\t\t-- Pete\n"},{"id":"206630","messageId":"20130113004129.GH4574@serenity.lan","threadId":"32609","inReplyTo":"20130112234304.GC23079@padd.com","subject":"Re: [PATCH 0/8] Initial support for Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-13T00:41:30Z","receivedAt":"2013-01-13T00:41:30Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sat, Jan 12, 2013 at 06:43:04PM -0500, Pete Wyckoff wrote:\n> john@keeping.me.uk wrote on Sat, 12 Jan 2013 19:23 +0000:\n>> I started having a look to see how much work would be needed to make Git\n>> work with Python 3 and the answer is mostly not much.  The exception is\n>> git-p4.py which is hit hard by the distinction between byte strings and\n>> unicode strings, particularly because the Python output mode of p4\n>> targets Python 2.\n>> \n>> I don't know if it's worthwhile to actually apply these but here they\n>> are in case anyone's interested.\n>> \n>> Having said that, the changes are minimal and involve either wrapping\n>> parentheses around arguments to print or being a bit more explicit about\n>> how we expect byte strings to be decoded to unicode.\n>> \n>> With these patches all tests pass with python3 except t98* (git-p4), but\n>> there are a couple of topics in-flight which will affect that\n>> (fc/remote-testgit-feature-done and er/replace-cvsimport).\n>> \n>> John Keeping (8):\n>>   git_remote_helpers: Allow building with Python 3\n>>   git_remote_helpers: fix input when running under Python 3\n>>   git_remote_helpers: Force rebuild if python version changes\n>>   git_remote_helpers: Use 2to3 if building with Python 3\n>>   svn-fe: allow svnrdump_sim.py to run with Python 3\n>>   git-remote-testpy: hash bytes explicitly\n>>   git-remote-testpy: don't do unbuffered text I/O\n>>   git-remote-testpy: call print as a function\n>> \n>>  contrib/svn-fe/svnrdump_sim.py     |  4 ++--\n>>  git-remote-testpy.py               | 40 +++++++++++++++++++-------------------\n>>  git_remote_helpers/.gitignore      |  1 +\n>>  git_remote_helpers/Makefile        | 10 ++++++++--\n>>  git_remote_helpers/git/importer.py |  2 +-\n>>  git_remote_helpers/setup.py        | 10 ++++++++++\n>>  6 files changed, 42 insertions(+), 25 deletions(-)\n> \n> These look good, in that there are relatively few changed needed.\n> \n> Sebastian Morr tried a similar patch a year ago, in\n> \n>     http://thread.gmane.org/gmane.comp.version-control.git/187545\n> \n> He made changes beyond yours, in particular \"print >>\" lines,\n> that you seem to handle with 2to3 during the build.  I'm not sure\n> which approach is better in the long run.  He worked on the\n> other .py in contrib/ too.\n\nIn the long run I'd want to move away from \"print >>\" to use\n\"print(file=..., ...)\" but that's only available from Python 2.6 onwards\n(via a __future__ import) and I think we probably don't want to rule out\nPython 2.5 yet.\n\nWithout 2to3 the only way to do this for both Python 2 and 3 is as\n\"file.write('...\\n')\".\n\n> Can you give me some hints about the byte/unicode string issues\n> in git-p4.py?  There's really only one place that does:\n> \n>     p4 = subprocess.Popen(\"p4 -G ...\")\n>     marshal.load(p4.stdout)\n> \n> If that's the only issue, this might not be too paniful.\n\nThe problem is that what gets loaded there is a dictionary (encoded by\np4) that maps byte strings to byte strings, so all of the accesses to\nthat dictionary need to either:\n\n   1) explicitly call encode() on a string constant\nor 2) use a byte string constant with a \"b\" prefix\n\nOr we could re-write the dictionary once, which handles the keys... but\nsome of the values are also used as strings and we can't handle that as\na one-off conversion since in other places we really do want the byte\nstring (think content of binary files).\n\nBasically a thorough audit of all access to variables that come from p4\nwould be needed, with explicit decode()s for authors, dates, etc.\n\n> I hesitated to take Sebastian's changes due to the huge number of\n> print() lines, but maybe a 2to3 approach would make that aspect\n> of python3 support not too onerous.\n\nI think we'd want to change to print() eventually and having a single\ncodebase for 2 and 3 would be nicer for development, but I think we need\nto be able to say \"no one is using Python 2.5 or earlier\" before we can\ndo that and I'm not sure we're there yet.  From where we are at the\nmoment I think 2to3 is a good answer, particularly where we're already\nusing distutils to generate a release image.\n\n\nJohn\n"},{"id":"206642","messageId":"50F2296F.8030909@alum.mit.edu","threadId":"32609","inReplyTo":"a8c3aabfab64f49fa0cbb2d45bda79997a875ee8.1358018078.git.john@keeping.me.uk","subject":"Re: [PATCH 2/8] git_remote_helpers: fix input when running under Python 3","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-01-13T03:26:39Z","receivedAt":"2013-01-13T03:26:39Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/12/2013 08:23 PM, John Keeping wrote:\n> Although 2to3 will fix most issues in Python 2 code to make it run under\n> Python 3, it does not handle the new strict separation between byte\n> strings and unicode strings.  There is one instance in\n> git_remote_helpers where we are caught by this.\n> \n> Fix it by explicitly decoding the incoming byte string into a unicode\n> string.  In this instance, use the locale under which the application is\n> running.\n> \n> Signed-off-by: John Keeping <john@keeping.me.uk>\n> ---\n>  git_remote_helpers/git/importer.py | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\n> index e28cc8f..6814003 100644\n> --- a/git_remote_helpers/git/importer.py\n> +++ b/git_remote_helpers/git/importer.py\n> @@ -20,7 +20,7 @@ class GitImporter(object):\n>          \"\"\"Returns a dictionary with refs.\n>          \"\"\"\n>          args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n> -        lines = check_output(args).strip().split('\\n')\n> +        lines = check_output(args).decode().strip().split('\\n')\n>          refs = {}\n>          for line in lines:\n>              value, name = line.split(' ')\n> \n\nWon't this change cause an exception if the branch names are not all\nvalid strings in the current locale's encoding?  I don't see how this\nassumption is justified (e.g., see git-check-ref-format(1) for the rules\ngoverning reference names).\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"206651","messageId":"20130113123404.GJ4574@serenity.lan","threadId":"32609","inReplyTo":"20130113004129.GH4574@serenity.lan","subject":"Re: [PATCH 0/8] Initial support for Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-13T12:34:04Z","receivedAt":"2013-01-13T12:34:04Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Jan 13, 2013 at 12:41:30AM +0000, John Keeping wrote:\n> On Sat, Jan 12, 2013 at 06:43:04PM -0500, Pete Wyckoff wrote:\n>> Can you give me some hints about the byte/unicode string issues\n>> in git-p4.py?  There's really only one place that does:\n>> \n>>     p4 = subprocess.Popen(\"p4 -G ...\")\n>>     marshal.load(p4.stdout)\n>> \n>> If that's the only issue, this might not be too paniful.\n> \n> The problem is that what gets loaded there is a dictionary (encoded by\n> p4) that maps byte strings to byte strings, so all of the accesses to\n> that dictionary need to either:\n> \n>    1) explicitly call encode() on a string constant\n> or 2) use a byte string constant with a \"b\" prefix\n> \n> Or we could re-write the dictionary once, which handles the keys... but\n> some of the values are also used as strings and we can't handle that as\n> a one-off conversion since in other places we really do want the byte\n> string (think content of binary files).\n> \n> Basically a thorough audit of all access to variables that come from p4\n> would be needed, with explicit decode()s for authors, dates, etc.\n\nHaving thought about this a bit more, another possibility would be to\napply this transformation once using something like this (completely\nuntested, I haven't looked up the keys of interest):\n\n-- >8 --\n\ndef _noop(s):\n    return s\n\ndef _decode(s):\n    return s.decode('utf-8')\n\nCONVERSION_MAP = {\n    'user': _decode,\n    'data': _decode\n}\n\nd = marshal.load(p4.stdout)\nretval = {}\nfor k, v in d.items():\n    key = k.decode('utf-8')\n    retval[key] = CONVERSION_MAP.get(key, _noop)(v)\nreturn retval\n\n-- 8< --\n\nObviously this isn't ideal but without p4 gaining a Python 3 output mode\nI suspect this would be the best we could do.\n\n\nJohn\n"},{"id":"206696","messageId":"20130113161724.GK4574@serenity.lan","threadId":"32609","inReplyTo":"50F2296F.8030909@alum.mit.edu","subject":"Re: [PATCH 2/8] git_remote_helpers: fix input when running under Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-13T16:17:24Z","receivedAt":"2013-01-13T16:17:24Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Jan 13, 2013 at 04:26:39AM +0100, Michael Haggerty wrote:\n> On 01/12/2013 08:23 PM, John Keeping wrote:\n>> Although 2to3 will fix most issues in Python 2 code to make it run under\n>> Python 3, it does not handle the new strict separation between byte\n>> strings and unicode strings.  There is one instance in\n>> git_remote_helpers where we are caught by this.\n>> \n>> Fix it by explicitly decoding the incoming byte string into a unicode\n>> string.  In this instance, use the locale under which the application is\n>> running.\n>> \n>> Signed-off-by: John Keeping <john@keeping.me.uk>\n>> ---\n>>  git_remote_helpers/git/importer.py | 2 +-\n>>  1 file changed, 1 insertion(+), 1 deletion(-)\n>> \n>> diff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\n>> index e28cc8f..6814003 100644\n>> --- a/git_remote_helpers/git/importer.py\n>> +++ b/git_remote_helpers/git/importer.py\n>> @@ -20,7 +20,7 @@ class GitImporter(object):\n>>          \"\"\"Returns a dictionary with refs.\n>>          \"\"\"\n>>          args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n>> -        lines = check_output(args).strip().split('\\n')\n>> +        lines = check_output(args).decode().strip().split('\\n')\n>>          refs = {}\n>>          for line in lines:\n>>              value, name = line.split(' ')\n>> \n> \n> Won't this change cause an exception if the branch names are not all\n> valid strings in the current locale's encoding?  I don't see how this\n> assumption is justified (e.g., see git-check-ref-format(1) for the rules\n> governing reference names).\n\nYes it will.  The problem is that for Python 3 we need to decode the\nbyte string into a unicode string, which means we need to know what\nencoding it is.\n\nI don't think we can just say \"git-for-each-ref will print refs in\nUTF-8\" since AFAIK git doesn't care what encoding the refs are in - I\nsuspect that's determined by the filesystem which in the end probably\nmaps to whatever bytes the shell fed git when the ref was created.\n\nThat's why I chose the current locale in this case.  I'm hoping someone\nhere will correct me if we can do better, but I don't see any way of\navoiding choosing some encoding here if we want to support Python 3\n(which I think we will, even if we don't right now).\n\n\nJohn\n"},{"id":"206697","messageId":"20130113162605.GL4574@serenity.lan","threadId":"32609","inReplyTo":"20130112233044.GB23079@padd.com","subject":"Re: [PATCH 3/8] git_remote_helpers: Force rebuild if python version changes","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-13T16:26:05Z","receivedAt":"2013-01-13T16:26:05Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sat, Jan 12, 2013 at 06:30:44PM -0500, Pete Wyckoff wrote:\n> john@keeping.me.uk wrote on Sat, 12 Jan 2013 19:23 +0000:\n>> When different version of python are used to build via distutils, the\n>> behaviour can change.  Detect changes in version and pass --force in\n>> this case.\n>[..]\n>> diff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\n>[..]\n>> +py_version=$(shell $(PYTHON_PATH) -c \\\n>> +\t'import sys; print(\"%i.%i\" % sys.version_info[:2])')\n>> +\n>>  all: $(pysetupfile)\n>> -\t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n>> +\t$(QUIET)test \"$$(cat GIT-PYTHON_VERSION 2>/dev/null)\" = \"$(py_version)\" || \\\n>> +\tflags=--force; \\\n>> +\t$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build $$flags\n>> +\t$(QUIET)echo \"$(py_version)\" >GIT-PYTHON_VERSION\n> \n> Can you depend on ../GIT-PYTHON-VARS instead?  It comes from\n> 96a4647 (Makefile: detect when PYTHON_PATH changes, 2012-12-18).\n> It doesn't check version, just path, but hopefully that's good\n> enough.  I'm imagining a rule that would do \"clean\" if\n> ../GIT-PYTHON-VARS changed, then build without --force.\n\nI was trying to keep the git_remote_helpers directory self contained.  I\ncan't see how to depend on ../GIT-PYTHON-VARS in a way that is as simple\nas this and keeps \"make -C git_remote_helpers\" working in a clean tree.\n\nAm I missing something obvious here?\n\n\nJohn\n"},{"id":"206698","messageId":"20130113164045.GA30371@padd.com","threadId":"32609","inReplyTo":"20130113004129.GH4574@serenity.lan","subject":"Re: [PATCH 0/8] Initial support for Python 3","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-01-13T16:40:45Z","receivedAt":"2013-01-13T16:40:45Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"john@keeping.me.uk wrote on Sun, 13 Jan 2013 00:41 +0000:\n> On Sat, Jan 12, 2013 at 06:43:04PM -0500, Pete Wyckoff wrote:\n> > Can you give me some hints about the byte/unicode string issues\n> > in git-p4.py?  There's really only one place that does:\n> > \n> >     p4 = subprocess.Popen(\"p4 -G ...\")\n> >     marshal.load(p4.stdout)\n> > \n> > If that's the only issue, this might not be too paniful.\n> \n> The problem is that what gets loaded there is a dictionary (encoded by\n> p4) that maps byte strings to byte strings, so all of the accesses to\n> that dictionary need to either:\n> \n>    1) explicitly call encode() on a string constant\n> or 2) use a byte string constant with a \"b\" prefix\n> \n> Or we could re-write the dictionary once, which handles the keys... but\n> some of the values are also used as strings and we can't handle that as\n> a one-off conversion since in other places we really do want the byte\n> string (think content of binary files).\n> \n> Basically a thorough audit of all access to variables that come from p4\n> would be needed, with explicit decode()s for authors, dates, etc.\n\nYour auto-conversion snippet in the follow-up mail would work\nfine for most keys and values.  A few perforce docs and some\nplaying around convince me that it is mostly utf-8, except for\nfile data for particular types.\n\nI'd still rather handle each command separately, and think about\nthe conversions, to do it right in the long run.\n\n> > I hesitated to take Sebastian's changes due to the huge number of\n> > print() lines, but maybe a 2to3 approach would make that aspect\n> > of python3 support not too onerous.\n> \n> I think we'd want to change to print() eventually and having a single\n> codebase for 2 and 3 would be nicer for development, but I think we need\n> to be able to say \"no one is using Python 2.5 or earlier\" before we can\n> do that and I'm not sure we're there yet.  From where we are at the\n> moment I think 2to3 is a good answer, particularly where we're already\n> using distutils to generate a release image.\n\nAgreed.  The 2to3 diff is large but straightforward.  But these\np4 -G interface errors require a lot of thought and work.  I'm\nnot too eager to work on this yet.\n\nThanks.\n\n\t\t-- Pete\n"},{"id":"206700","messageId":"20130113171402.GA1307@padd.com","threadId":"32609","inReplyTo":"20130113162605.GL4574@serenity.lan","subject":"Re: [PATCH 3/8] git_remote_helpers: Force rebuild if python version changes","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-01-13T17:14:02Z","receivedAt":"2013-01-13T17:14:02Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"john@keeping.me.uk wrote on Sun, 13 Jan 2013 16:26 +0000:\n> On Sat, Jan 12, 2013 at 06:30:44PM -0500, Pete Wyckoff wrote:\n> > john@keeping.me.uk wrote on Sat, 12 Jan 2013 19:23 +0000:\n> >> When different version of python are used to build via distutils, the\n> >> behaviour can change.  Detect changes in version and pass --force in\n> >> this case.\n> >[..]\n> >> diff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\n> >[..]\n> >> +py_version=$(shell $(PYTHON_PATH) -c \\\n> >> +\t'import sys; print(\"%i.%i\" % sys.version_info[:2])')\n> >> +\n> >>  all: $(pysetupfile)\n> >> -\t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n> >> +\t$(QUIET)test \"$$(cat GIT-PYTHON_VERSION 2>/dev/null)\" = \"$(py_version)\" || \\\n> >> +\tflags=--force; \\\n> >> +\t$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build $$flags\n> >> +\t$(QUIET)echo \"$(py_version)\" >GIT-PYTHON_VERSION\n> > \n> > Can you depend on ../GIT-PYTHON-VARS instead?  It comes from\n> > 96a4647 (Makefile: detect when PYTHON_PATH changes, 2012-12-18).\n> > It doesn't check version, just path, but hopefully that's good\n> > enough.  I'm imagining a rule that would do \"clean\" if\n> > ../GIT-PYTHON-VARS changed, then build without --force.\n> \n> I was trying to keep the git_remote_helpers directory self contained.  I\n> can't see how to depend on ../GIT-PYTHON-VARS in a way that is as simple\n> as this and keeps \"make -C git_remote_helpers\" working in a clean tree.\n> \n> Am I missing something obvious here?\n\nNot if it wants to stay self-contained; you're right.\n\nI'm not thrilled with how git_remote_helpers/Makefile always\nruns setup.py, and always generates PYLIBDIR, and now always\ninvokes python a third time to see if its version changed.\n\n\t\t-- Pete\n"},{"id":"206703","messageId":"20130113173557.GN4574@serenity.lan","threadId":"32609","inReplyTo":"20130113164045.GA30371@padd.com","subject":"Re: [PATCH 0/8] Initial support for Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-13T17:35:57Z","receivedAt":"2013-01-13T17:35:57Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Jan 13, 2013 at 11:40:45AM -0500, Pete Wyckoff wrote:\n> john@keeping.me.uk wrote on Sun, 13 Jan 2013 00:41 +0000:\n>> On Sat, Jan 12, 2013 at 06:43:04PM -0500, Pete Wyckoff wrote:\n>> > Can you give me some hints about the byte/unicode string issues\n>> > in git-p4.py?  There's really only one place that does:\n>> > \n>> >     p4 = subprocess.Popen(\"p4 -G ...\")\n>> >     marshal.load(p4.stdout)\n>> > \n>> > If that's the only issue, this might not be too paniful.\n>> \n>> The problem is that what gets loaded there is a dictionary (encoded by\n>> p4) that maps byte strings to byte strings, so all of the accesses to\n>> that dictionary need to either:\n>> \n>>    1) explicitly call encode() on a string constant\n>> or 2) use a byte string constant with a \"b\" prefix\n>> \n>> Or we could re-write the dictionary once, which handles the keys... but\n>> some of the values are also used as strings and we can't handle that as\n>> a one-off conversion since in other places we really do want the byte\n>> string (think content of binary files).\n>> \n>> Basically a thorough audit of all access to variables that come from p4\n>> would be needed, with explicit decode()s for authors, dates, etc.\n> \n> Your auto-conversion snippet in the follow-up mail would work\n> fine for most keys and values.  A few perforce docs and some\n> playing around convince me that it is mostly utf-8, except for\n> file data for particular types.\n> \n> I'd still rather handle each command separately, and think about\n> the conversions, to do it right in the long run.\n\nI sent that on the assumption that the same key would have similar\nsemantics wherever its used, but I don't use git-p4 or know much about\nperforce.\n\nIt would be interesting to know whether there is any likelihood of p4\ngaining a Python 3 output mode (since the documentation currently say\nnot to use \"p4 -G\" with Python 3).  If it does then I would assume that\nit will make a sensible choice about unicode/bytes such that the\nexisting git-p4 would Just Work with only a small change to the\ninvocation of p4 to add the new argument.\n\n>> > I hesitated to take Sebastian's changes due to the huge number of\n>> > print() lines, but maybe a 2to3 approach would make that aspect\n>> > of python3 support not too onerous.\n>> \n>> I think we'd want to change to print() eventually and having a single\n>> codebase for 2 and 3 would be nicer for development, but I think we need\n>> to be able to say \"no one is using Python 2.5 or earlier\" before we can\n>> do that and I'm not sure we're there yet.  From where we are at the\n>> moment I think 2to3 is a good answer, particularly where we're already\n>> using distutils to generate a release image.\n> \n> Agreed.  The 2to3 diff is large but straightforward.  But these\n> p4 -G interface errors require a lot of thought and work.  I'm\n> not too eager to work on this yet.\n\nFair enough.  As I don't use git-p4, it's not something I intend to\ntackle either (given the scale of the changes involved).\n\nGiven the minimal scope of the changes needed for everything else, I\nsent this series wondering whether it's sensible to move forward on the\nbasis of \"Python scripts except git-p4 work with Python 3.  You must use\nPython 2 if you want to use git-p4\".\n\n\nJohn\n"},{"id":"206705","messageId":"20130113175238.GO4574@serenity.lan","threadId":"32609","inReplyTo":"20130113171402.GA1307@padd.com","subject":"Re: [PATCH 3/8] git_remote_helpers: Force rebuild if python version changes","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-13T17:52:38Z","receivedAt":"2013-01-13T17:52:38Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Jan 13, 2013 at 12:14:02PM -0500, Pete Wyckoff wrote:\n> john@keeping.me.uk wrote on Sun, 13 Jan 2013 16:26 +0000:\n>> On Sat, Jan 12, 2013 at 06:30:44PM -0500, Pete Wyckoff wrote:\n>> > john@keeping.me.uk wrote on Sat, 12 Jan 2013 19:23 +0000:\n>> >> When different version of python are used to build via distutils, the\n>> >> behaviour can change.  Detect changes in version and pass --force in\n>> >> this case.\n>> >[..]\n>> >> diff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\n>> >[..]\n>> >> +py_version=$(shell $(PYTHON_PATH) -c \\\n>> >> +\t'import sys; print(\"%i.%i\" % sys.version_info[:2])')\n>> >> +\n>> >>  all: $(pysetupfile)\n>> >> -\t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n>> >> +\t$(QUIET)test \"$$(cat GIT-PYTHON_VERSION 2>/dev/null)\" = \"$(py_version)\" || \\\n>> >> +\tflags=--force; \\\n>> >> +\t$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build $$flags\n>> >> +\t$(QUIET)echo \"$(py_version)\" >GIT-PYTHON_VERSION\n>> > \n>> > Can you depend on ../GIT-PYTHON-VARS instead?  It comes from\n>> > 96a4647 (Makefile: detect when PYTHON_PATH changes, 2012-12-18).\n>> > It doesn't check version, just path, but hopefully that's good\n>> > enough.  I'm imagining a rule that would do \"clean\" if\n>> > ../GIT-PYTHON-VARS changed, then build without --force.\n>> \n>> I was trying to keep the git_remote_helpers directory self contained.  I\n>> can't see how to depend on ../GIT-PYTHON-VARS in a way that is as simple\n>> as this and keeps \"make -C git_remote_helpers\" working in a clean tree.\n>> \n>> Am I missing something obvious here?\n> \n> Not if it wants to stay self-contained; you're right.\n> \n> I'm not thrilled with how git_remote_helpers/Makefile always\n> runs setup.py, and always generates PYLIBDIR, and now always\n> invokes python a third time to see if its version changed.\n\nI don't think PYLIBDIR will be calculated unless it's used ('=' not\n':=' means its a deferred variable).\n\nI wonder if the version check should move into setup.py - it would be\njust as easy to check the file there and massage sys.args, although\npossibly not as neat.\n\n\nJohn\n"},{"id":"206742","messageId":"50F38E12.6090207@alum.mit.edu","threadId":"32609","inReplyTo":"20130113161724.GK4574@serenity.lan","subject":"Re: [PATCH 2/8] git_remote_helpers: fix input when running under Python 3","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-01-14T04:48:18Z","receivedAt":"2013-01-14T04:48:18Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/13/2013 05:17 PM, John Keeping wrote:\n> On Sun, Jan 13, 2013 at 04:26:39AM +0100, Michael Haggerty wrote:\n>> On 01/12/2013 08:23 PM, John Keeping wrote:\n>>> Although 2to3 will fix most issues in Python 2 code to make it run under\n>>> Python 3, it does not handle the new strict separation between byte\n>>> strings and unicode strings.  There is one instance in\n>>> git_remote_helpers where we are caught by this.\n>>>\n>>> Fix it by explicitly decoding the incoming byte string into a unicode\n>>> string.  In this instance, use the locale under which the application is\n>>> running.\n>>>\n>>> Signed-off-by: John Keeping <john@keeping.me.uk>\n>>> ---\n>>>  git_remote_helpers/git/importer.py | 2 +-\n>>>  1 file changed, 1 insertion(+), 1 deletion(-)\n>>>\n>>> diff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\n>>> index e28cc8f..6814003 100644\n>>> --- a/git_remote_helpers/git/importer.py\n>>> +++ b/git_remote_helpers/git/importer.py\n>>> @@ -20,7 +20,7 @@ class GitImporter(object):\n>>>          \"\"\"Returns a dictionary with refs.\n>>>          \"\"\"\n>>>          args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n>>> -        lines = check_output(args).strip().split('\\n')\n>>> +        lines = check_output(args).decode().strip().split('\\n')\n>>>          refs = {}\n>>>          for line in lines:\n>>>              value, name = line.split(' ')\n>>>\n>>\n>> Won't this change cause an exception if the branch names are not all\n>> valid strings in the current locale's encoding?  I don't see how this\n>> assumption is justified (e.g., see git-check-ref-format(1) for the rules\n>> governing reference names).\n> \n> Yes it will.  The problem is that for Python 3 we need to decode the\n> byte string into a unicode string, which means we need to know what\n> encoding it is.\n> \n> I don't think we can just say \"git-for-each-ref will print refs in\n> UTF-8\" since AFAIK git doesn't care what encoding the refs are in - I\n> suspect that's determined by the filesystem which in the end probably\n> maps to whatever bytes the shell fed git when the ref was created.\n> \n> That's why I chose the current locale in this case.  I'm hoping someone\n> here will correct me if we can do better, but I don't see any way of\n> avoiding choosing some encoding here if we want to support Python 3\n> (which I think we will, even if we don't right now).\n\nI'm not just trying to be a nuisance here; I'm struggling myself to\nunderstand how a program that cares about strings-vs-bytes (e.g., a\nPython3 script) should coexist with a program that doesn't (e.g., git\n[1]).  I think this will become a big issue if my Python version of the\ncommit email script ever gets integrated and then made compatible with\nPython3.\n\nYou claim \"for Python 3 we need to decode the byte string into a unicode\nstring\".  I understand that Python 3 strings are Unicode, but why/when\nis it necessary to decode data into a Unicode string as opposed to\nleaving it as a byte sequence?\n\nIn this particular case (from a cursory look over the code) it seems to\nme that (1) decoding to Unicode will sometimes fail for data that git\nconsiders valid and (2) there is no obvious reason that the data cannot\nbe processed as byte sequences.\n\nMichael\n\n[1] And it doesn't just seem that \"git doesn't care about Unicode\n*yet*\".  It seems more likely that \"git will adamantly refuse to deal\nwith Unicode\".  For example, Linus is quite clearly in favor of treating\ndata as byte sequences in most situations:\nhttps://plus.google.com/111049168280159033135/posts/f3fngVm174f\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"206789","messageId":"20130114094721.GQ4574@serenity.lan","threadId":"32609","inReplyTo":"50F38E12.6090207@alum.mit.edu","subject":"Re: [PATCH 2/8] git_remote_helpers: fix input when running under Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-14T09:47:21Z","receivedAt":"2013-01-14T09:47:21Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Mon, Jan 14, 2013 at 05:48:18AM +0100, Michael Haggerty wrote:\n> On 01/13/2013 05:17 PM, John Keeping wrote:\n>> On Sun, Jan 13, 2013 at 04:26:39AM +0100, Michael Haggerty wrote:\n>>> On 01/12/2013 08:23 PM, John Keeping wrote:\n>>>> Although 2to3 will fix most issues in Python 2 code to make it run under\n>>>> Python 3, it does not handle the new strict separation between byte\n>>>> strings and unicode strings.  There is one instance in\n>>>> git_remote_helpers where we are caught by this.\n>>>>\n>>>> Fix it by explicitly decoding the incoming byte string into a unicode\n>>>> string.  In this instance, use the locale under which the application is\n>>>> running.\n>>>>\n>>>> Signed-off-by: John Keeping <john@keeping.me.uk>\n>>>> ---\n>>>>  git_remote_helpers/git/importer.py | 2 +-\n>>>>  1 file changed, 1 insertion(+), 1 deletion(-)\n>>>>\n>>>> diff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\n>>>> index e28cc8f..6814003 100644\n>>>> --- a/git_remote_helpers/git/importer.py\n>>>> +++ b/git_remote_helpers/git/importer.py\n>>>> @@ -20,7 +20,7 @@ class GitImporter(object):\n>>>>          \"\"\"Returns a dictionary with refs.\n>>>>          \"\"\"\n>>>>          args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n>>>> -        lines = check_output(args).strip().split('\\n')\n>>>> +        lines = check_output(args).decode().strip().split('\\n')\n>>>>          refs = {}\n>>>>          for line in lines:\n>>>>              value, name = line.split(' ')\n>>>>\n>>>\n>>> Won't this change cause an exception if the branch names are not all\n>>> valid strings in the current locale's encoding?  I don't see how this\n>>> assumption is justified (e.g., see git-check-ref-format(1) for the rules\n>>> governing reference names).\n>> \n>> Yes it will.  The problem is that for Python 3 we need to decode the\n>> byte string into a unicode string, which means we need to know what\n>> encoding it is.\n>> \n>> I don't think we can just say \"git-for-each-ref will print refs in\n>> UTF-8\" since AFAIK git doesn't care what encoding the refs are in - I\n>> suspect that's determined by the filesystem which in the end probably\n>> maps to whatever bytes the shell fed git when the ref was created.\n>> \n>> That's why I chose the current locale in this case.  I'm hoping someone\n>> here will correct me if we can do better, but I don't see any way of\n>> avoiding choosing some encoding here if we want to support Python 3\n>> (which I think we will, even if we don't right now).\n> \n> I'm not just trying to be a nuisance here;\n\nYou're not being - I think this is a difficult issue and I don't know\nmyself what the right answer is.\n\n>                                            I'm struggling myself to\n> understand how a program that cares about strings-vs-bytes (e.g., a\n> Python3 script) should coexist with a program that doesn't (e.g., git\n> [1]).  I think this will become a big issue if my Python version of the\n> commit email script ever gets integrated and then made compatible with\n> Python3.\n> \n> You claim \"for Python 3 we need to decode the byte string into a unicode\n> string\".  I understand that Python 3 strings are Unicode, but why/when\n> is it necessary to decode data into a Unicode string as opposed to\n> leaving it as a byte sequence?\n> \n> In this particular case (from a cursory look over the code) it seems to\n> me that (1) decoding to Unicode will sometimes fail for data that git\n> considers valid and (2) there is no obvious reason that the data cannot\n> be processed as byte sequences.\n\nI've been thinking about this overnight and I think you're right that\ntreating them as byte strings in Python is most correct.  Having said\nthat, when I'm programming in Python I would find it quite surprising\nthat I had to treat ref strings specially - and as soon as I want to use\none in a string context (e.g. printing it as part of a message to the\nuser) I'm back to the same problem.\n\nSo I think we should try to solve the problem once rather than forcing\neveryone who wants to use the library to solve it individually.  I just\nwish it was obvious what we should do!\n\n\nJohn\n"},{"id":"206967","messageId":"20130115194809.GU4574@serenity.lan","threadId":"32609","inReplyTo":"20130114094721.GQ4574@serenity.lan","subject":"[RFC/PATCH 2/8 v2] git_remote_helpers: fix input when running under Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-15T19:48:09Z","receivedAt":"2013-01-15T19:48:09Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Although 2to3 will fix most issues in Python 2 code to make it run under\nPython 3, it does not handle the new strict separation between byte\nstrings and unicode strings.  There is one instance in\ngit_remote_helpers where we are caught by this, which is when reading\nrefs from \"git for-each-ref\".\n\nWhile we could fix this by explicitly handling refs as byte strings,\nthis is merely punting the problem to users of the library since the\nsame problem will be encountered as soon you want to display the ref\nname to a user.\n\nInstead of doing this, explicit decode the incoming byte string into a\nunicode string.  Following the lead of pygit2 (the Python bindings for\nlibgit2 - see [1] and [2]), use the filesystem encoding by default,\nproviding a way for callers to override this if necessary.\n\n[1] https://github.com/libgit2/pygit2/blob/e34911b63e5d2266f9f72a4e3f32e27b13190feb/src/pygit2/reference.c#L261\n[2] https://github.com/libgit2/pygit2/blob/e34911b63e5d2266f9f72a4e3f32e27b13190feb/include/pygit2/utils.h#L55\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n\nI think this is in fact the best way to handle this, and I hope the\nabove description clarified why I don't think we want to treat refs as\nbyte strings in Python 3.\n\nMy only remaining question is whether it would be better to set the\nerror mode when decoding to \"replace\" instead of \"strict\" (the default).\n\"strict\" will cause a UnicodeError if the string cannot be decoded\nwhereas \"replace\" will use U+FFFD (the replacement character). [3]\n\nI think it's better to use \"strict\" and let the user know that\nsomething has gone wrong rather than silently change the string, but I'd\nwelcome other opinions.\n\n[3] http://docs.python.org/2/library/codecs.html#codec-base-classes\n\n git_remote_helpers/git/importer.py | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\nindex e28cc8f..5bc16a4 100644\n--- a/git_remote_helpers/git/importer.py\n+++ b/git_remote_helpers/git/importer.py\n@@ -1,5 +1,6 @@\n import os\n import subprocess\n+import sys\n \n from git_remote_helpers.util import check_call, check_output\n \n@@ -10,17 +11,26 @@ class GitImporter(object):\n     This importer simply delegates to git fast-import.\n     \"\"\"\n \n-    def __init__(self, repo):\n+    def __init__(self, repo, ref_encoding=None):\n         \"\"\"Creates a new importer for the specified repo.\n+\n+        If ref_encoding is specified that refs are decoded using that\n+        encoding.  Otherwise the system filesystem encoding is used.\n         \"\"\"\n \n         self.repo = repo\n+        self.ref_encoding = ref_encoding\n \n     def get_refs(self, gitdir):\n         \"\"\"Returns a dictionary with refs.\n         \"\"\"\n         args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n-        lines = check_output(args).strip().split('\\n')\n+        encoding = self.ref_encoding\n+        if encoding is None:\n+            encoding = sys.getfilesystemencoding()\n+            if encoding is None:\n+                encoding = sys.getdefaultencoding()\n+        lines = check_output(args).decode(encoding).strip().split('\\n')\n         refs = {}\n         for line in lines:\n             value, name = line.split(' ')\n-- \n1.8.1\n"},{"id":"206976","messageId":"7vbocq2mri.fsf@alter.siamese.dyndns.org","threadId":"32609","inReplyTo":"20130115194809.GU4574@serenity.lan","subject":"Re: [RFC/PATCH 2/8 v2] git_remote_helpers: fix input when running under Python 3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-15T20:51:13Z","receivedAt":"2013-01-15T20:51:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Keeping <john@keeping.me.uk> writes:\n\n> Although 2to3 will fix most issues in Python 2 code to make it run under\n> Python 3, it does not handle the new strict separation between byte\n> strings and unicode strings.  There is one instance in\n> git_remote_helpers where we are caught by this, which is when reading\n> refs from \"git for-each-ref\".\n>\n> While we could fix this by explicitly handling refs as byte strings,\n> this is merely punting the problem to users of the library since the\n> same problem will be encountered as soon you want to display the ref\n> name to a user.\n>\n> Instead of doing this, explicit decode the incoming byte string into a\n> unicode string.\n\nThat really feels wrong.  Displaying is a separate issue and it is\nthe _right_ thing to punt the problem at the lower-level machinery\nlevel.\n\n> Following the lead of pygit2 (the Python bindings for\n> libgit2 - see [1] and [2]),...\n\nI do not think other people getting it wrong is not an excuse to\nrepeat the same mistake.\n\nIs it really so cumbersome to handle byte strings as byte strings in\nPython?\n"},{"id":"206979","messageId":"20130115215412.GX4574@serenity.lan","threadId":"32609","inReplyTo":"7vbocq2mri.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 2/8 v2] git_remote_helpers: fix input when running under Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-15T21:54:12Z","receivedAt":"2013-01-15T21:54:12Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Tue, Jan 15, 2013 at 12:51:13PM -0800, Junio C Hamano wrote:\n> John Keeping <john@keeping.me.uk> writes:\n>> Although 2to3 will fix most issues in Python 2 code to make it run under\n>> Python 3, it does not handle the new strict separation between byte\n>> strings and unicode strings.  There is one instance in\n>> git_remote_helpers where we are caught by this, which is when reading\n>> refs from \"git for-each-ref\".\n>>\n>> While we could fix this by explicitly handling refs as byte strings,\n>> this is merely punting the problem to users of the library since the\n>> same problem will be encountered as soon you want to display the ref\n>> name to a user.\n>>\n>> Instead of doing this, explicit decode the incoming byte string into a\n>> unicode string.\n> \n> That really feels wrong.  Displaying is a separate issue and it is\n> the _right_ thing to punt the problem at the lower-level machinery\n> level.\n\nBut the display will require decoding the ref name to a Unicode string,\nwhich depends on the encoding of the underlying ref name, so it feels\nlike it should be decoded where it's read (see [1]).\n\n>> Following the lead of pygit2 (the Python bindings for\n>> libgit2 - see [1] and [2]),...\n> \n> I do not think other people getting it wrong is not an excuse to\n> repeat the same mistake.\n>\n> Is it really so cumbersome to handle byte strings as byte strings in\n> Python?\n\nAs [1] says, there is a potential for bugs whenever people attempt to\ncombine Unicode and byte strings.  I think it also violates the\nprinciple of least surprise if a ref name (a string) doesn't behave like\na normal string.\n\n[1] http://docs.python.org/3.3/howto/unicode.html#tips-for-writing-unicode-aware-programs\n\n\nJohn\n"},{"id":"206980","messageId":"7vy5fu14sy.fsf@alter.siamese.dyndns.org","threadId":"32609","inReplyTo":"20130115215412.GX4574@serenity.lan","subject":"Re: [RFC/PATCH 2/8 v2] git_remote_helpers: fix input when running under Python 3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-15T22:04:29Z","receivedAt":"2013-01-15T22:04:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Keeping <john@keeping.me.uk> writes:\n\n>> That really feels wrong.  Displaying is a separate issue and it is\n>> the _right_ thing to punt the problem at the lower-level machinery\n>> level.\n>\n> But the display will require decoding the ref name to a Unicode string,\n> which depends on the encoding of the underlying ref name, so it feels\n> like it should be decoded where it's read (see [1]).\n\nIf you botch the decoding in a way you cannot recover the original\nbyte string, you cannot create a ref whose name is the original byte\nstring, no?  Keeping the original byte string internally (this\nincludes where you use it to create new refs or update existing\nrefs), and attempting to convert it to Unicode when you choose to\nshow that string as a part of a message to the user (and falling\nback to replacing some bytes to '?' if you cannot, but do so only in\nthe message), you won't have that problem.\n"},{"id":"206982","messageId":"20130115224049.GZ4574@serenity.lan","threadId":"32609","inReplyTo":"7vy5fu14sy.fsf@alter.siamese.dyndns.org","subject":"[RFC/PATCH 2/8 v3] git_remote_helpers: fix input when running under Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-15T22:40:49Z","receivedAt":"2013-01-15T22:40:49Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Although 2to3 will fix most issues in Python 2 code to make it run under\nPython 3, it does not handle the new strict separation between byte\nstrings and unicode strings.  There is one instance in\ngit_remote_helpers where we are caught by this, which is when reading\nrefs from \"git for-each-ref\".\n\nFix this by operating on the returned string as a byte string rather\nthan a unicode string.  As this method is currently only used internally\nby the class this does not affect code anywhere else.\n\nNote that we cannot use byte strings in the source as the 'b' prefix is\nnot supported before Python 2.7 so in order to maintain compatibility\nwith the maximum range of Python versions we use an explicit call to\nencode().\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n\nOn Tue, Jan 15, 2013 at 02:04:29PM -0800, Junio C Hamano wrote:\n> John Keeping <john@keeping.me.uk> writes:\n>>> That really feels wrong.  Displaying is a separate issue and it is\n>>> the _right_ thing to punt the problem at the lower-level machinery\n>>> level.\n>>\n>> But the display will require decoding the ref name to a Unicode string,\n>> which depends on the encoding of the underlying ref name, so it feels\n>> like it should be decoded where it's read (see [1]).\n> \n> If you botch the decoding in a way you cannot recover the original\n> byte string, you cannot create a ref whose name is the original byte\n> string, no?  Keeping the original byte string internally (this\n> includes where you use it to create new refs or update existing\n> refs), and attempting to convert it to Unicode when you choose to\n> show that string as a part of a message to the user (and falling\n> back to replacing some bytes to '?' if you cannot, but do so only in\n> the message), you won't have that problem.\n\nActually, this method is currently only used internally so I don't think\nmy argument holds.\n\nThis is what keeping the refs as byte strings looks like.\n\n\n git_remote_helpers/git/importer.py | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\nindex e28cc8f..c54846c 100644\n--- a/git_remote_helpers/git/importer.py\n+++ b/git_remote_helpers/git/importer.py\n@@ -18,13 +18,16 @@ class GitImporter(object):\n \n     def get_refs(self, gitdir):\n         \"\"\"Returns a dictionary with refs.\n+\n+        Note that the keys in the returned dictionary are byte strings as\n+        read from git.\n         \"\"\"\n         args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n-        lines = check_output(args).strip().split('\\n')\n+        lines = check_output(args).strip().split('\\n'.encode('utf-8'))\n         refs = {}\n         for line in lines:\n-            value, name = line.split(' ')\n-            name = name.strip('commit\\t')\n+            value, name = line.split(' '.encode('utf-8'))\n+            name = name.strip('commit\\t'.encode('utf-8'))\n             refs[name] = value\n         return refs\n \n-- \n1.8.1\n"},{"id":"206983","messageId":"20130115225805.GA4574@serenity.lan","threadId":"32609","inReplyTo":"20130113175238.GO4574@serenity.lan","subject":"Re: [PATCH 3/8] git_remote_helpers: Force rebuild if python version changes","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-15T22:58:05Z","receivedAt":"2013-01-15T22:58:05Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Jan 13, 2013 at 05:52:38PM +0000, John Keeping wrote:\n> On Sun, Jan 13, 2013 at 12:14:02PM -0500, Pete Wyckoff wrote:\n>> john@keeping.me.uk wrote on Sun, 13 Jan 2013 16:26 +0000:\n>>> On Sat, Jan 12, 2013 at 06:30:44PM -0500, Pete Wyckoff wrote:\n>>> > john@keeping.me.uk wrote on Sat, 12 Jan 2013 19:23 +0000:\n>>> >> When different version of python are used to build via distutils, the\n>>> >> behaviour can change.  Detect changes in version and pass --force in\n>>> >> this case.\n>>> >[..]\n>>> >> diff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\n>>> >[..]\n>>> >> +py_version=$(shell $(PYTHON_PATH) -c \\\n>>> >> +\t'import sys; print(\"%i.%i\" % sys.version_info[:2])')\n>>> >> +\n>>> >>  all: $(pysetupfile)\n>>> >> -\t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n>>> >> +\t$(QUIET)test \"$$(cat GIT-PYTHON_VERSION 2>/dev/null)\" = \"$(py_version)\" || \\\n>>> >> +\tflags=--force; \\\n>>> >> +\t$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build $$flags\n>>> >> +\t$(QUIET)echo \"$(py_version)\" >GIT-PYTHON_VERSION\n>>> > \n>>> > Can you depend on ../GIT-PYTHON-VARS instead?  It comes from\n>>> > 96a4647 (Makefile: detect when PYTHON_PATH changes, 2012-12-18).\n>>> > It doesn't check version, just path, but hopefully that's good\n>>> > enough.  I'm imagining a rule that would do \"clean\" if\n>>> > ../GIT-PYTHON-VARS changed, then build without --force.\n>>> \n>>> I was trying to keep the git_remote_helpers directory self contained.  I\n>>> can't see how to depend on ../GIT-PYTHON-VARS in a way that is as simple\n>>> as this and keeps \"make -C git_remote_helpers\" working in a clean tree.\n>>> \n>>> Am I missing something obvious here?\n>> \n>> Not if it wants to stay self-contained; you're right.\n>> \n>> I'm not thrilled with how git_remote_helpers/Makefile always\n>> runs setup.py, and always generates PYLIBDIR, and now always\n>> invokes python a third time to see if its version changed.\n> \n> I don't think PYLIBDIR will be calculated unless it's used ('=' not\n> ':=' means its a deferred variable).\n> \n> I wonder if the version check should move into setup.py - it would be\n> just as easy to check the file there and massage sys.args, although\n> possibly not as neat.\n\nFor reference, putting the version check in setup.py looks like this:\n\n-- >8 --\n\ndiff --git a/git_remote_helpers/setup.py b/git_remote_helpers/setup.py\nindex 6de41de..2c21eb5 100644\n--- a/git_remote_helpers/setup.py\n+++ b/git_remote_helpers/setup.py\n@@ -3,6 +3,7 @@\n \"\"\"Distutils build/install script for the git_remote_helpers package.\"\"\"\n \n from distutils.core import setup\n+import sys\n \n # If building under Python3 we need to run 2to3 on the code, do this by\n # trying to import distutils' 2to3 builder, which is only available in\n@@ -13,6 +14,24 @@ except ImportError:\n     # 2.x\n     from distutils.command.build_py import build_py\n \n+\n+current_version = '%d.%d' % sys.version_info[:2]\n+try:\n+    f = open('GIT-PYTHON_VERSION', 'r')\n+    latest_version = f.read().strip()\n+    f.close()\n+\n+    if latest_version != current_version:\n+        if not '--force' in sys.argv:\n+            sys.argv.insert(0, '--force')\n+except IOError:\n+    pass\n+\n+f = open('GIT-PYTHON_VERSION', 'w')\n+f.write(current_version)\n+f.close()\n+\n+\n setup(\n     name = 'git_remote_helpers',\n     version = '0.1.0',\n"},{"id":"206994","messageId":"20130116000316.GA26999@padd.com","threadId":"32609","inReplyTo":"20130115224049.GZ4574@serenity.lan","subject":"Re: [RFC/PATCH 2/8 v3] git_remote_helpers: fix input when running under Python 3","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-01-16T00:03:16Z","receivedAt":"2013-01-16T00:03:16Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"john@keeping.me.uk wrote on Tue, 15 Jan 2013 22:40 +0000:\n> This is what keeping the refs as byte strings looks like.\n\nAs John knows, it is not possible to interpret text from a byte\nstring without talking about the character encoding.\n\nGit is (largely) a C program and uses the character set defined\nin the C standard, which is a subset of ASCII.  But git does\n\"math\" on strings, like this snippet that takes something from\nargv[] and prepends \"refs/heads/\":\n\n    strcpy(refname, \"refs/heads/\");\n    strcpy(refname + strlen(\"refs/heads/\"), ret->name);\n\nThe result doesn't talk about what character set it is using,\nbut because it combines a prefix from ASCII with its input,\ngit makes the assumption that the input is ASCII-compatible.\n\nIf you feed a UTF-16 string in argv, e.g.\n\n    $ echo master | iconv -f ascii -t utf16 | xargs git branch\n    xargs: Warning: a NUL character occurred in the input.  It cannot be passed through in the argument list.  Did you mean to use the --null option?\n    fatal: Not a valid object name: ''.\n\nyou get an error about NUL, and not the branch you hoped for.\nGit assumes that the input character set contains roughly ASCII\nin byte positions 0..127.\n\nThat's one small reason why the useful character encodings put\nASCII in the 0..127 range, including utf-8, big5 and shift-jis.\nASCII is indeed special due to its legacy, and both C and Python\nrecognize this.\n\n> diff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\n> @@ -18,13 +18,16 @@ class GitImporter(object):\n>  \n>      def get_refs(self, gitdir):\n>          \"\"\"Returns a dictionary with refs.\n> +\n> +        Note that the keys in the returned dictionary are byte strings as\n> +        read from git.\n>          \"\"\"\n>          args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n> -        lines = check_output(args).strip().split('\\n')\n> +        lines = check_output(args).strip().split('\\n'.encode('utf-8'))\n>          refs = {}\n>          for line in lines:\n> -            value, name = line.split(' ')\n> -            name = name.strip('commit\\t')\n> +            value, name = line.split(' '.encode('utf-8'))\n> +            name = name.strip('commit\\t'.encode('utf-8'))\n>              refs[name] = value\n>          return refs\n\nI'd suggest for this Python conundrum using byte-string literals, e.g.:\n\n        lines = check_output(args).strip().split(b'\\n')\n\tvalue, name = line.split(b' ')\n\tname = name.strip(b'commit\\t')\n\nEssentially identical to what you have, but avoids naming \"utf-8\" as\nthe encoding.  It instead relies on Python's interpretation of\nASCII characters in string context, which is exactly what C does.\n\n\t\t-- Pete\n"},{"id":"207040","messageId":"20130116094418.GA9089@river","threadId":"32609","inReplyTo":"20130116000316.GA26999@padd.com","subject":"Re: [RFC/PATCH 2/8 v3] git_remote_helpers: fix input when running under Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-16T09:45:34Z","receivedAt":"2013-01-16T09:45:34Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Tue, Jan 15, 2013 at 07:03:16PM -0500, Pete Wyckoff wrote:\n> john@keeping.me.uk wrote on Tue, 15 Jan 2013 22:40 +0000:\n>> This is what keeping the refs as byte strings looks like.\n> \n> As John knows, it is not possible to interpret text from a byte\n> string without talking about the character encoding.\n> \n> Git is (largely) a C program and uses the character set defined\n> in the C standard, which is a subset of ASCII.  But git does\n> \"math\" on strings, like this snippet that takes something from\n> argv[] and prepends \"refs/heads/\":\n> \n>     strcpy(refname, \"refs/heads/\");\n>     strcpy(refname + strlen(\"refs/heads/\"), ret->name);\n> \n> The result doesn't talk about what character set it is using,\n> but because it combines a prefix from ASCII with its input,\n> git makes the assumption that the input is ASCII-compatible.\n> \n> If you feed a UTF-16 string in argv, e.g.\n> \n>     $ echo master | iconv -f ascii -t utf16 | xargs git branch\n>     xargs: Warning: a NUL character occurred in the input.  It cannot be passed through in the argument list.  Did you mean to use the --null option?\n>     fatal: Not a valid object name: ''.\n> \n> you get an error about NUL, and not the branch you hoped for.\n> Git assumes that the input character set contains roughly ASCII\n> in byte positions 0..127.\n> \n> That's one small reason why the useful character encodings put\n> ASCII in the 0..127 range, including utf-8, big5 and shift-jis.\n> ASCII is indeed special due to its legacy, and both C and Python\n> recognize this.\n> \n>> diff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\n>> @@ -18,13 +18,16 @@ class GitImporter(object):\n>>  \n>>      def get_refs(self, gitdir):\n>>          \"\"\"Returns a dictionary with refs.\n>> +\n>> +        Note that the keys in the returned dictionary are byte strings as\n>> +        read from git.\n>>          \"\"\"\n>>          args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n>> -        lines = check_output(args).strip().split('\\n')\n>> +        lines = check_output(args).strip().split('\\n'.encode('utf-8'))\n>>          refs = {}\n>>          for line in lines:\n>> -            value, name = line.split(' ')\n>> -            name = name.strip('commit\\t')\n>> +            value, name = line.split(' '.encode('utf-8'))\n>> +            name = name.strip('commit\\t'.encode('utf-8'))\n>>              refs[name] = value\n>>          return refs\n> \n> I'd suggest for this Python conundrum using byte-string literals, e.g.:\n> \n>         lines = check_output(args).strip().split(b'\\n')\n> \tvalue, name = line.split(b' ')\n> \tname = name.strip(b'commit\\t')\n> \n> Essentially identical to what you have, but avoids naming \"utf-8\" as\n> the encoding.  It instead relies on Python's interpretation of\n> ASCII characters in string context, which is exactly what C does.\n\nThe problem is that AFAICT the byte-string prefix is only available in\nPython 2.7 and later (compare [1] and [2]).  I think we need this more\nconvoluted code if we want to keep supporting Python 2.6 (although\nperhaps 'ascii' would be a better choice than 'utf-8').\n\n[1] http://docs.python.org/2.6/reference/lexical_analysis.html#literals\n[2] http://docs.python.org/2.7/reference/lexical_analysis.html#literals\n\n\nJohn\n"},{"id":"207123","messageId":"20130117002708.GA15517@padd.com","threadId":"32609","inReplyTo":"20130115225805.GA4574@serenity.lan","subject":"Re: [PATCH 3/8] git_remote_helpers: Force rebuild if python version changes","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-01-17T00:27:08Z","receivedAt":"2013-01-17T00:27:08Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"john@keeping.me.uk wrote on Tue, 15 Jan 2013 22:58 +0000:\n> For reference, putting the version check in setup.py looks like this:\n> \n> -- >8 --\n> \n> diff --git a/git_remote_helpers/setup.py b/git_remote_helpers/setup.py\n> index 6de41de..2c21eb5 100644\n> --- a/git_remote_helpers/setup.py\n> +++ b/git_remote_helpers/setup.py\n> @@ -3,6 +3,7 @@\n>  \"\"\"Distutils build/install script for the git_remote_helpers package.\"\"\"\n>  \n>  from distutils.core import setup\n> +import sys\n>  \n>  # If building under Python3 we need to run 2to3 on the code, do this by\n>  # trying to import distutils' 2to3 builder, which is only available in\n> @@ -13,6 +14,24 @@ except ImportError:\n>      # 2.x\n>      from distutils.command.build_py import build_py\n>  \n> +\n> +current_version = '%d.%d' % sys.version_info[:2]\n> +try:\n> +    f = open('GIT-PYTHON_VERSION', 'r')\n> +    latest_version = f.read().strip()\n> +    f.close()\n> +\n> +    if latest_version != current_version:\n> +        if not '--force' in sys.argv:\n> +            sys.argv.insert(0, '--force')\n> +except IOError:\n> +    pass\n> +\n> +f = open('GIT-PYTHON_VERSION', 'w')\n> +f.write(current_version)\n> +f.close()\n> +\n> +\n>  setup(\n>      name = 'git_remote_helpers',\n>      version = '0.1.0',\n> \n\nThat's about the same overhead as doing it in the Makefile,\nand a bit more obscure.  I don't mind your initial version\nso much anymore.  Thanks for thinking about it.\n\n\t\t-- Pete\n"},{"id":"207124","messageId":"20130117002955.GB15517@padd.com","threadId":"32609","inReplyTo":"20130116094418.GA9089@river","subject":"Re: [RFC/PATCH 2/8 v3] git_remote_helpers: fix input when running under Python 3","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-01-17T00:29:55Z","receivedAt":"2013-01-17T00:29:55Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"john@keeping.me.uk wrote on Wed, 16 Jan 2013 09:45 +0000:\n> On Tue, Jan 15, 2013 at 07:03:16PM -0500, Pete Wyckoff wrote:\n> > I'd suggest for this Python conundrum using byte-string literals, e.g.:\n> > \n> >         lines = check_output(args).strip().split(b'\\n')\n> > \tvalue, name = line.split(b' ')\n> > \tname = name.strip(b'commit\\t')\n> > \n> > Essentially identical to what you have, but avoids naming \"utf-8\" as\n> > the encoding.  It instead relies on Python's interpretation of\n> > ASCII characters in string context, which is exactly what C does.\n> \n> The problem is that AFAICT the byte-string prefix is only available in\n> Python 2.7 and later (compare [1] and [2]).  I think we need this more\n> convoluted code if we want to keep supporting Python 2.6 (although\n> perhaps 'ascii' would be a better choice than 'utf-8').\n> \n> [1] http://docs.python.org/2.6/reference/lexical_analysis.html#literals\n> [2] http://docs.python.org/2.7/reference/lexical_analysis.html#literals\n\nDrat.  The b'' syntax seems to work on 2.6.8, in spite of\nthe docs, but certainly isn't in 2.5.\n\nI think you had hit on the best compromise with encoding,\nbut maybe ascii is a little less presumptuous than utf-8,\nand more indicative of the encoding assumption.\n\n\t\t-- Pete\n"},{"id":"207154","messageId":"cover.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358018078.git.john@keeping.me.uk","subject":"[PATCH v2 0/8] Initial Python 3 support","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:53:53Z","receivedAt":"2013-01-17T18:53:53Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"This series does enough so that everything except git-p4 runs under\nPython 3.\n\nAs discussed with Pete, it may not make sense to change git-p4 to\nsupport Python 3 until Perforce's Python output mode is changed.  So\ndoes it make sense to merge this now and say \"use Python 2 if you want\ngit-p4\"?\n\nChanges since v1:\n\n* rebased on master after fc/remote-testgit-feature-done was merged,\n  leading to an extra change in patch 8 (git-remote-testpy: call print\n  as a function)\n* changed patch 2 (git_remote_helpers: fix input when running under\n  Python 3) to treat ref names as byte strings\n\nJohn Keeping (8):\n  git_remote_helpers: Allow building with Python 3\n  git_remote_helpers: fix input when running under Python 3\n  git_remote_helpers: Force rebuild if python version changes\n  git_remote_helpers: Use 2to3 if building with Python 3\n  svn-fe: allow svnrdump_sim.py to run with Python 3\n  git-remote-testpy: hash bytes explicitly\n  git-remote-testpy: don't do unbuffered text I/O\n  git-remote-testpy: call print as a function\n\n contrib/svn-fe/svnrdump_sim.py     |  4 ++--\n git-remote-testpy.py               | 42 +++++++++++++++++++-------------------\n git_remote_helpers/.gitignore      |  1 +\n git_remote_helpers/Makefile        | 10 +++++++--\n git_remote_helpers/git/importer.py |  9 +++++---\n git_remote_helpers/setup.py        | 10 +++++++++\n 6 files changed, 48 insertions(+), 28 deletions(-)\n\n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207155","messageId":"e450e2711f963ed46fabbeccafc5fbea02fdc834.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358448207.git.john@keeping.me.uk","subject":"[PATCH v2 1/8] git_remote_helpers: allow building with Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:53:54Z","receivedAt":"2013-01-17T18:53:54Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Change inline Python to call \"print\" as a function not a statement.\n\nThis is harmless because Python 2 will see the parentheses as redundant\ngrouping but they are necessary to run this code with Python 3.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git_remote_helpers/Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\nindex 74b05dc..f65f064 100644\n--- a/git_remote_helpers/Makefile\n+++ b/git_remote_helpers/Makefile\n@@ -23,7 +23,7 @@ endif\n \n PYLIBDIR=$(shell $(PYTHON_PATH) -c \\\n \t \"import sys; \\\n-\t print 'lib/python%i.%i/site-packages' % sys.version_info[:2]\")\n+\t print('lib/python%i.%i/site-packages' % sys.version_info[:2])\")\n \n all: $(pysetupfile)\n \t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207156","messageId":"68095fcf00aac8fbae37f52d06a6397b074248a3.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358448207.git.john@keeping.me.uk","subject":"[PATCH v2 2/8] git_remote_helpers: fix input when running under Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:53:55Z","receivedAt":"2013-01-17T18:53:55Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Although 2to3 will fix most issues in Python 2 code to make it run under\nPython 3, it does not handle the new strict separation between byte\nstrings and unicode strings.  There is one instance in\ngit_remote_helpers where we are caught by this, which is when reading\nrefs from \"git for-each-ref\".\n\nFix this by operating on the returned string as a byte string rather\nthan a unicode string.  As this method is currently only used internally\nby the class this does not affect code anywhere else.\n\nNote that we cannot use byte strings in the source as the 'b' prefix is\nnot supported before Python 2.7 so in order to maintain compatibility\nwith the maximum range of Python versions we use an explicit call to\nencode().\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git_remote_helpers/git/importer.py | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/git_remote_helpers/git/importer.py b/git_remote_helpers/git/importer.py\nindex e28cc8f..d3f90e1 100644\n--- a/git_remote_helpers/git/importer.py\n+++ b/git_remote_helpers/git/importer.py\n@@ -18,13 +18,16 @@ class GitImporter(object):\n \n     def get_refs(self, gitdir):\n         \"\"\"Returns a dictionary with refs.\n+\n+        Note that the keys in the returned dictionary are byte strings as\n+        read from git.\n         \"\"\"\n         args = [\"git\", \"--git-dir=\" + gitdir, \"for-each-ref\", \"refs/heads\"]\n-        lines = check_output(args).strip().split('\\n')\n+        lines = check_output(args).strip().split('\\n'.encode('ascii'))\n         refs = {}\n         for line in lines:\n-            value, name = line.split(' ')\n-            name = name.strip('commit\\t')\n+            value, name = line.split(' '.encode('ascii'))\n+            name = name.strip('commit\\t'.encode('ascii'))\n             refs[name] = value\n         return refs\n \n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207157","messageId":"3dd93f736c6e78b9ac9e902cc2321209d0a78fd5.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358448207.git.john@keeping.me.uk","subject":"[PATCH v2 3/8] git_remote_helpers: force rebuild if python version changes","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:53:56Z","receivedAt":"2013-01-17T18:53:56Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"When different version of python are used to build via distutils, the\nbehaviour can change.  Detect changes in version and pass --force in\nthis case.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git_remote_helpers/.gitignore | 1 +\n git_remote_helpers/Makefile   | 8 +++++++-\n 2 files changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/git_remote_helpers/.gitignore b/git_remote_helpers/.gitignore\nindex 2247d5f..06c664f 100644\n--- a/git_remote_helpers/.gitignore\n+++ b/git_remote_helpers/.gitignore\n@@ -1,2 +1,3 @@\n+/GIT-PYTHON_VERSION\n /build\n /dist\ndiff --git a/git_remote_helpers/Makefile b/git_remote_helpers/Makefile\nindex f65f064..91f458f 100644\n--- a/git_remote_helpers/Makefile\n+++ b/git_remote_helpers/Makefile\n@@ -25,8 +25,14 @@ PYLIBDIR=$(shell $(PYTHON_PATH) -c \\\n \t \"import sys; \\\n \t print('lib/python%i.%i/site-packages' % sys.version_info[:2])\")\n \n+py_version=$(shell $(PYTHON_PATH) -c \\\n+\t'import sys; print(\"%i.%i\" % sys.version_info[:2])')\n+\n all: $(pysetupfile)\n-\t$(QUIET)$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build\n+\t$(QUIET)test \"$$(cat GIT-PYTHON_VERSION 2>/dev/null)\" = \"$(py_version)\" || \\\n+\tflags=--force; \\\n+\t$(PYTHON_PATH) $(pysetupfile) $(QUIETSETUP) build $$flags\n+\t$(QUIET)echo \"$(py_version)\" >GIT-PYTHON_VERSION\n \n install: $(pysetupfile)\n \t$(PYTHON_PATH) $(pysetupfile) install --prefix $(DESTDIR_SQ)$(prefix)\n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207158","messageId":"bcef80fb913ca829bd2d08284e364ebd55b7297e.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358448207.git.john@keeping.me.uk","subject":"[PATCH v2 4/8] git_remote_helpers: use 2to3 if building with Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:53:57Z","receivedAt":"2013-01-17T18:53:57Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Using the approach detailed on the Python wiki[1], run 2to3 on the code\nas part of the build if building with Python 3.\n\nThe code itself requires no changes to convert cleanly.\n\n[1] http://wiki.python.org/moin/PortingPythonToPy3k\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git_remote_helpers/setup.py | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/git_remote_helpers/setup.py b/git_remote_helpers/setup.py\nindex 4d434b6..6de41de 100644\n--- a/git_remote_helpers/setup.py\n+++ b/git_remote_helpers/setup.py\n@@ -4,6 +4,15 @@\n \n from distutils.core import setup\n \n+# If building under Python3 we need to run 2to3 on the code, do this by\n+# trying to import distutils' 2to3 builder, which is only available in\n+# Python3.\n+try:\n+    from distutils.command.build_py import build_py_2to3 as build_py\n+except ImportError:\n+    # 2.x\n+    from distutils.command.build_py import build_py\n+\n setup(\n     name = 'git_remote_helpers',\n     version = '0.1.0',\n@@ -14,4 +23,5 @@ setup(\n     url = 'http://www.git-scm.com/',\n     package_dir = {'git_remote_helpers': ''},\n     packages = ['git_remote_helpers', 'git_remote_helpers.git'],\n+    cmdclass = {'build_py': build_py},\n )\n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207159","messageId":"649f0d65308358637d923483df6b780ebcce5d82.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358448207.git.john@keeping.me.uk","subject":"[PATCH v2 5/8] svn-fe: allow svnrdump_sim.py to run with Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:53:58Z","receivedAt":"2013-01-17T18:53:58Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"The changes to allow this script to run with Python 3 are minimal and do\nnot affect its functionality on the versions of Python 2 that are\nalready supported (2.4 onwards).\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n contrib/svn-fe/svnrdump_sim.py | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/svn-fe/svnrdump_sim.py b/contrib/svn-fe/svnrdump_sim.py\nindex 17cf6f9..4e78a1c 100755\n--- a/contrib/svn-fe/svnrdump_sim.py\n+++ b/contrib/svn-fe/svnrdump_sim.py\n@@ -14,7 +14,7 @@ if sys.hexversion < 0x02040000:\n \n def getrevlimit():\n         var = 'SVNRMAX'\n-        if os.environ.has_key(var):\n+        if var in os.environ:\n                 return os.environ[var]\n         return None\n \n@@ -44,7 +44,7 @@ def writedump(url, lower, upper):\n \n if __name__ == \"__main__\":\n         if not (len(sys.argv) in (3, 4, 5)):\n-                print \"usage: %s dump URL -rLOWER:UPPER\"\n+                print(\"usage: %s dump URL -rLOWER:UPPER\")\n                 sys.exit(1)\n         if not sys.argv[1] == 'dump': raise NotImplementedError('only \"dump\" is suppported.')\n         url = sys.argv[2]\n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207160","messageId":"66c42ff65eddde494f40d0a582e89a081b4ab8e8.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358448207.git.john@keeping.me.uk","subject":"[PATCH v2 6/8] git-remote-testpy: hash bytes explicitly","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:53:59Z","receivedAt":"2013-01-17T18:53:59Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Under Python 3 'hasher.update(...)' must take a byte string and not a\nunicode string.  Explicitly encode the argument to this method as UTF-8\nso that this code works under Python 3.\n\nThis moves the required Python version forward to 2.0.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git-remote-testpy.py | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/git-remote-testpy.py b/git-remote-testpy.py\nindex d94a66a..f8dc196 100644\n--- a/git-remote-testpy.py\n+++ b/git-remote-testpy.py\n@@ -31,9 +31,9 @@ from git_remote_helpers.git.exporter import GitExporter\n from git_remote_helpers.git.importer import GitImporter\n from git_remote_helpers.git.non_local import NonLocalGit\n \n-if sys.hexversion < 0x01050200:\n-    # os.makedirs() is the limiter\n-    sys.stderr.write(\"git-remote-testgit: requires Python 1.5.2 or later.\\n\")\n+if sys.hexversion < 0x02000000:\n+    # string.encode() is the limiter\n+    sys.stderr.write(\"git-remote-testgit: requires Python 2.0 or later.\\n\")\n     sys.exit(1)\n \n def get_repo(alias, url):\n@@ -45,7 +45,7 @@ def get_repo(alias, url):\n     repo.get_head()\n \n     hasher = _digest()\n-    hasher.update(repo.path)\n+    hasher.update(repo.path.encode('utf-8'))\n     repo.hash = hasher.hexdigest()\n \n     repo.get_base_path = lambda base: os.path.join(\n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207161","messageId":"6bc90f3afc86d53eb6e4b4d6b87f6afd20023769.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358448207.git.john@keeping.me.uk","subject":"[PATCH v2 7/8] git-remote-testpy: don't do unbuffered text I/O","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:54:00Z","receivedAt":"2013-01-17T18:54:00Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"Python 3 forbids unbuffered I/O in text mode.  Change the reading of\nstdin in git-remote-testpy so that we read the lines as bytes and then\ndecode them a line at a time.\n\nThis allows us to keep the I/O unbuffered in order to avoid\nreintroducing the bug fixed by commit 7fb8e16 (git-remote-testgit: fix\nrace when spawning fast-import).\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git-remote-testpy.py | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/git-remote-testpy.py b/git-remote-testpy.py\nindex f8dc196..bc5e3cf 100644\n--- a/git-remote-testpy.py\n+++ b/git-remote-testpy.py\n@@ -154,7 +154,7 @@ def do_import(repo, args):\n     refs = [ref]\n \n     while True:\n-        line = sys.stdin.readline()\n+        line = sys.stdin.readline().decode()\n         if line == '\\n':\n             break\n         if not line.startswith('import '):\n@@ -225,7 +225,7 @@ def read_one_line(repo):\n \n     line = sys.stdin.readline()\n \n-    cmdline = line\n+    cmdline = line.decode()\n \n     if not cmdline:\n         warn(\"Unexpected EOF\")\n@@ -277,7 +277,7 @@ def main(args):\n \n     more = True\n \n-    sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n+    sys.stdin = os.fdopen(sys.stdin.fileno(), 'rb', 0)\n     while (more):\n         more = read_one_line(repo)\n \n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207162","messageId":"88502ee7b090454352b46d496741029817b27482.1358448207.git.john@keeping.me.uk","threadId":"32609","inReplyTo":"cover.1358448207.git.john@keeping.me.uk","subject":"[PATCH v2 8/8] git-remote-testpy: call print as a function","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T18:54:01Z","receivedAt":"2013-01-17T18:54:01Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"This is harmless in Python 2, which sees the parentheses as redundant\ngrouping, but is required for Python 3.  Since this is the only change\nrequired to make this script just run under Python 3 without needing\n2to3 it seems worthwhile.\n\nThe case of an empty print must be handled specially because in that\ncase Python 2 will interpret '()' as an empty tuple and print it as\n'()'; inserting an empty string fixes this.\n\nSigned-off-by: John Keeping <john@keeping.me.uk>\n---\n git-remote-testpy.py | 28 ++++++++++++++--------------\n 1 file changed, 14 insertions(+), 14 deletions(-)\n\ndiff --git a/git-remote-testpy.py b/git-remote-testpy.py\nindex bc5e3cf..ccdb2dc 100644\n--- a/git-remote-testpy.py\n+++ b/git-remote-testpy.py\n@@ -87,9 +87,9 @@ def do_capabilities(repo, args):\n     \"\"\"Prints the supported capabilities.\n     \"\"\"\n \n-    print \"import\"\n-    print \"export\"\n-    print \"refspec refs/heads/*:%s*\" % repo.prefix\n+    print(\"import\")\n+    print(\"export\")\n+    print(\"refspec refs/heads/*:%s*\" % repo.prefix)\n \n     dirname = repo.get_base_path(repo.gitdir)\n \n@@ -98,11 +98,11 @@ def do_capabilities(repo, args):\n \n     path = os.path.join(dirname, 'git.marks')\n \n-    print \"*export-marks %s\" % path\n+    print(\"*export-marks %s\" % path)\n     if os.path.exists(path):\n-        print \"*import-marks %s\" % path\n+        print(\"*import-marks %s\" % path)\n \n-    print # end capabilities\n+    print('') # end capabilities\n \n \n def do_list(repo, args):\n@@ -115,16 +115,16 @@ def do_list(repo, args):\n \n     for ref in repo.revs:\n         debug(\"? refs/heads/%s\", ref)\n-        print \"? refs/heads/%s\" % ref\n+        print(\"? refs/heads/%s\" % ref)\n \n     if repo.head:\n         debug(\"@refs/heads/%s HEAD\" % repo.head)\n-        print \"@refs/heads/%s HEAD\" % repo.head\n+        print(\"@refs/heads/%s HEAD\" % repo.head)\n     else:\n         debug(\"@refs/heads/master HEAD\")\n-        print \"@refs/heads/master HEAD\"\n+        print(\"@refs/heads/master HEAD\")\n \n-    print # end list\n+    print('') # end list\n \n \n def update_local_repo(repo):\n@@ -164,7 +164,7 @@ def do_import(repo, args):\n         ref = line[7:].strip()\n         refs.append(ref)\n \n-    print \"feature done\"\n+    print(\"feature done\")\n \n     if os.environ.get(\"GIT_REMOTE_TESTGIT_FAILURE\"):\n         die('Told to fail')\n@@ -172,7 +172,7 @@ def do_import(repo, args):\n     repo = update_local_repo(repo)\n     repo.exporter.export_repo(repo.gitdir, refs)\n \n-    print \"done\"\n+    print(\"done\")\n \n \n def do_export(repo, args):\n@@ -192,8 +192,8 @@ def do_export(repo, args):\n         repo.non_local.push(repo.gitdir)\n \n     for ref in changed:\n-        print \"ok %s\" % ref\n-    print\n+        print(\"ok %s\" % ref)\n+    print('')\n \n \n COMMANDS = {\n-- \n1.8.1.1.260.g99b33f4.dirty\n"},{"id":"207167","messageId":"7vtxqftulq.fsf@alter.siamese.dyndns.org","threadId":"32609","inReplyTo":"66c42ff65eddde494f40d0a582e89a081b4ab8e8.1358448207.git.john@keeping.me.uk","subject":"Re: [PATCH v2 6/8] git-remote-testpy: hash bytes explicitly","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-17T20:36:33Z","receivedAt":"2013-01-17T20:36:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Keeping <john@keeping.me.uk> writes:\n\n> Under Python 3 'hasher.update(...)' must take a byte string and not a\n> unicode string.  Explicitly encode the argument to this method as UTF-8\n> so that this code works under Python 3.\n>\n> This moves the required Python version forward to 2.0.\n>\n> Signed-off-by: John Keeping <john@keeping.me.uk>\n> ---\n\nHmph.  So what happens when the path is _not_ encoded in UTF-8?\n\nIs the repo.hash (and local.hash that gets a copy of it) something\nthat needs to stay the same across multiple invocations of this\nremote helper, and between the currently shipped Git and the version\nof Git after applying this patch?  If that is not the case, and if\nthis is used only to get a randomly-looking 40-byte hexadecimal\nstring, then a lossy attempt to .encode('utf-8') and falling back to\nreplace or ignore bytes in the original that couldn't be interpreted\nas part of a UTF-8 string would be OK, but doesn't .encode('utf-8')\nthrow an exception if not told to 'ignore' or something?\n\n>  git-remote-testpy.py | 8 ++++----\n>  1 file changed, 4 insertions(+), 4 deletions(-)\n>\n> diff --git a/git-remote-testpy.py b/git-remote-testpy.py\n> index d94a66a..f8dc196 100644\n> --- a/git-remote-testpy.py\n> +++ b/git-remote-testpy.py\n> @@ -31,9 +31,9 @@ from git_remote_helpers.git.exporter import GitExporter\n>  from git_remote_helpers.git.importer import GitImporter\n>  from git_remote_helpers.git.non_local import NonLocalGit\n>  \n> -if sys.hexversion < 0x01050200:\n> -    # os.makedirs() is the limiter\n> -    sys.stderr.write(\"git-remote-testgit: requires Python 1.5.2 or later.\\n\")\n> +if sys.hexversion < 0x02000000:\n> +    # string.encode() is the limiter\n> +    sys.stderr.write(\"git-remote-testgit: requires Python 2.0 or later.\\n\")\n>      sys.exit(1)\n>  \n>  def get_repo(alias, url):\n> @@ -45,7 +45,7 @@ def get_repo(alias, url):\n>      repo.get_head()\n>  \n>      hasher = _digest()\n> -    hasher.update(repo.path)\n> +    hasher.update(repo.path.encode('utf-8'))\n>      repo.hash = hasher.hexdigest()\n>  \n>      repo.get_base_path = lambda base: os.path.join(\n"},{"id":"207168","messageId":"7vmww7tuat.fsf@alter.siamese.dyndns.org","threadId":"32609","inReplyTo":"7vtxqftulq.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v2 6/8] git-remote-testpy: hash bytes explicitly","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-17T20:43:06Z","receivedAt":"2013-01-17T20:43:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> John Keeping <john@keeping.me.uk> writes:\n>\n>> Under Python 3 'hasher.update(...)' must take a byte string and not a\n>> unicode string.  Explicitly encode the argument to this method as UTF-8\n>> so that this code works under Python 3.\n>>\n>> This moves the required Python version forward to 2.0.\n>>\n>> Signed-off-by: John Keeping <john@keeping.me.uk>\n>> ---\n>\n> Hmph.  So what happens when the path is _not_ encoded in UTF-8?\n\nOh, my brain was not working. Forget this part, and sorry for the\nnoise.  We are not decoding a bytestring to an array of unicode\ncharacters, but going the other way around here.\n"},{"id":"207170","messageId":"20130117210048.GI4574@serenity.lan","threadId":"32609","inReplyTo":"7vtxqftulq.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v2 6/8] git-remote-testpy: hash bytes explicitly","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T21:00:48Z","receivedAt":"2013-01-17T21:00:48Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Thu, Jan 17, 2013 at 12:36:33PM -0800, Junio C Hamano wrote:\n> John Keeping <john@keeping.me.uk> writes:\n> \n>> Under Python 3 'hasher.update(...)' must take a byte string and not a\n>> unicode string.  Explicitly encode the argument to this method as UTF-8\n>> so that this code works under Python 3.\n>>\n>> This moves the required Python version forward to 2.0.\n>>\n>> Signed-off-by: John Keeping <john@keeping.me.uk>\n>> ---\n> \n> Hmph.  So what happens when the path is _not_ encoded in UTF-8?\n\nDo you mean encodable?  As you say below it will currently throw an\nexception.\n\n> Is the repo.hash (and local.hash that gets a copy of it) something\n> that needs to stay the same across multiple invocations of this\n> remote helper, and between the currently shipped Git and the version\n> of Git after applying this patch?\n\nIt's used to specify the path of the repository for importing or\nexporting, so it should stay consistent across invocations.  However,\nthis is only an example remote helper so I don't think we should worry\nif it changes from one Git release to the next.\n\n>                                    If that is not the case, and if\n> this is used only to get a randomly-looking 40-byte hexadecimal\n> string, then a lossy attempt to .encode('utf-8') and falling back to\n> replace or ignore bytes in the original that couldn't be interpreted\n> as part of a UTF-8 string would be OK, but doesn't .encode('utf-8')\n> throw an exception if not told to 'ignore' or something?\n\nYou're right - I think we need to add \", errors='replace'\" to the call\nto encode.\n\n> >  git-remote-testpy.py | 8 ++++----\n> >  1 file changed, 4 insertions(+), 4 deletions(-)\n> >\n> > diff --git a/git-remote-testpy.py b/git-remote-testpy.py\n> > index d94a66a..f8dc196 100644\n> > --- a/git-remote-testpy.py\n> > +++ b/git-remote-testpy.py\n> > @@ -31,9 +31,9 @@ from git_remote_helpers.git.exporter import GitExporter\n> >  from git_remote_helpers.git.importer import GitImporter\n> >  from git_remote_helpers.git.non_local import NonLocalGit\n> >  \n> > -if sys.hexversion < 0x01050200:\n> > -    # os.makedirs() is the limiter\n> > -    sys.stderr.write(\"git-remote-testgit: requires Python 1.5.2 or later.\\n\")\n> > +if sys.hexversion < 0x02000000:\n> > +    # string.encode() is the limiter\n> > +    sys.stderr.write(\"git-remote-testgit: requires Python 2.0 or later.\\n\")\n> >      sys.exit(1)\n> >  \n> >  def get_repo(alias, url):\n> > @@ -45,7 +45,7 @@ def get_repo(alias, url):\n> >      repo.get_head()\n> >  \n> >      hasher = _digest()\n> > -    hasher.update(repo.path)\n> > +    hasher.update(repo.path.encode('utf-8'))\n> >      repo.hash = hasher.hexdigest()\n> >  \n> >      repo.get_base_path = lambda base: os.path.join(\n"},{"id":"207171","messageId":"20130117210506.GJ4574@serenity.lan","threadId":"32609","inReplyTo":"20130117210048.GI4574@serenity.lan","subject":"Re: [PATCH v2 6/8] git-remote-testpy: hash bytes explicitly","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T21:05:06Z","receivedAt":"2013-01-17T21:05:06Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Thu, Jan 17, 2013 at 09:00:48PM +0000, John Keeping wrote:\n> On Thu, Jan 17, 2013 at 12:36:33PM -0800, Junio C Hamano wrote:\n>> John Keeping <john@keeping.me.uk> writes:\n>> \n>>> Under Python 3 'hasher.update(...)' must take a byte string and not a\n>>> unicode string.  Explicitly encode the argument to this method as UTF-8\n>>> so that this code works under Python 3.\n>>>\n>>> This moves the required Python version forward to 2.0.\n>>>\n>>> Signed-off-by: John Keeping <john@keeping.me.uk>\n>>> ---\n>> \n>> Hmph.  So what happens when the path is _not_ encoded in UTF-8?\n> \n> Do you mean encodable?  As you say below it will currently throw an\n> exception.\n\nNow my brain's not working - we shouldn't get an error converting from a\nUnicode string to UTF-8, so I think this patch is OK as it is.\n\n> > Is the repo.hash (and local.hash that gets a copy of it) something\n> > that needs to stay the same across multiple invocations of this\n> > remote helper, and between the currently shipped Git and the version\n> > of Git after applying this patch?\n> \n> It's used to specify the path of the repository for importing or\n> exporting, so it should stay consistent across invocations.  However,\n> this is only an example remote helper so I don't think we should worry\n> if it changes from one Git release to the next.\n"},{"id":"207177","messageId":"7v622vtplm.fsf@alter.siamese.dyndns.org","threadId":"32609","inReplyTo":"20130117210048.GI4574@serenity.lan","subject":"Re: [PATCH v2 6/8] git-remote-testpy: hash bytes explicitly","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-17T22:24:37Z","receivedAt":"2013-01-17T22:24:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Keeping <john@keeping.me.uk> writes:\n\n> You're right - I think we need to add \", errors='replace'\" to the call\n> to encode.\n\nOf if it is used just as a opaque token, you can .encode('hex') or\nsomething to punt on the whole issue, no?\n\n>\n>> >  git-remote-testpy.py | 8 ++++----\n>> >  1 file changed, 4 insertions(+), 4 deletions(-)\n>> >\n>> > diff --git a/git-remote-testpy.py b/git-remote-testpy.py\n>> > index d94a66a..f8dc196 100644\n>> > --- a/git-remote-testpy.py\n>> > +++ b/git-remote-testpy.py\n>> > @@ -31,9 +31,9 @@ from git_remote_helpers.git.exporter import GitExporter\n>> >  from git_remote_helpers.git.importer import GitImporter\n>> >  from git_remote_helpers.git.non_local import NonLocalGit\n>> >  \n>> > -if sys.hexversion < 0x01050200:\n>> > -    # os.makedirs() is the limiter\n>> > -    sys.stderr.write(\"git-remote-testgit: requires Python 1.5.2 or later.\\n\")\n>> > +if sys.hexversion < 0x02000000:\n>> > +    # string.encode() is the limiter\n>> > +    sys.stderr.write(\"git-remote-testgit: requires Python 2.0 or later.\\n\")\n>> >      sys.exit(1)\n>> >  \n>> >  def get_repo(alias, url):\n>> > @@ -45,7 +45,7 @@ def get_repo(alias, url):\n>> >      repo.get_head()\n>> >  \n>> >      hasher = _digest()\n>> > -    hasher.update(repo.path)\n>> > +    hasher.update(repo.path.encode('utf-8'))\n>> >      repo.hash = hasher.hexdigest()\n>> >  \n>> >      repo.get_base_path = lambda base: os.path.join(\n"},{"id":"207178","messageId":"20130117223050.GL4574@serenity.lan","threadId":"32609","inReplyTo":"7v622vtplm.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v2 6/8] git-remote-testpy: hash bytes explicitly","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-17T22:30:50Z","receivedAt":"2013-01-17T22:30:50Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Thu, Jan 17, 2013 at 02:24:37PM -0800, Junio C Hamano wrote:\n> John Keeping <john@keeping.me.uk> writes:\n> \n>> You're right - I think we need to add \", errors='replace'\" to the call\n>> to encode.\n> \n> Of if it is used just as a opaque token, you can .encode('hex') or\n> something to punt on the whole issue, no?\n\nEven better.  Are you happy to squash that in (assuming nothing else\ncomes up) or shall I resend?\n\n>>>>  git-remote-testpy.py | 8 ++++----\n>>>>  1 file changed, 4 insertions(+), 4 deletions(-)\n>>>>\n>>>> diff --git a/git-remote-testpy.py b/git-remote-testpy.py\n>>>> index d94a66a..f8dc196 100644\n>>>> --- a/git-remote-testpy.py\n>>>> +++ b/git-remote-testpy.py\n>>>> @@ -31,9 +31,9 @@ from git_remote_helpers.git.exporter import GitExporter\n>>>>  from git_remote_helpers.git.importer import GitImporter\n>>>>  from git_remote_helpers.git.non_local import NonLocalGit\n>>>>  \n>>>> -if sys.hexversion < 0x01050200:\n>>>> -    # os.makedirs() is the limiter\n>>>> -    sys.stderr.write(\"git-remote-testgit: requires Python 1.5.2 or later.\\n\")\n>>>> +if sys.hexversion < 0x02000000:\n>>>> +    # string.encode() is the limiter\n>>>> +    sys.stderr.write(\"git-remote-testgit: requires Python 2.0 or later.\\n\")\n>>>>      sys.exit(1)\n>>>>  \n>>>>  def get_repo(alias, url):\n>>>> @@ -45,7 +45,7 @@ def get_repo(alias, url):\n>>>>      repo.get_head()\n>>>>  \n>>>>      hasher = _digest()\n>>>> -    hasher.update(repo.path)\n>>>> +    hasher.update(repo.path.encode('utf-8'))\n>>>>      repo.hash = hasher.hexdigest()\n>>>>  \n>>>>      repo.get_base_path = lambda base: os.path.join(\n"},{"id":"207180","messageId":"7v1udjto3a.fsf@alter.siamese.dyndns.org","threadId":"32609","inReplyTo":"20130117223050.GL4574@serenity.lan","subject":"Re: [PATCH v2 6/8] git-remote-testpy: hash bytes explicitly","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-17T22:57:13Z","receivedAt":"2013-01-17T22:57:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Keeping <john@keeping.me.uk> writes:\n\n> On Thu, Jan 17, 2013 at 02:24:37PM -0800, Junio C Hamano wrote:\n>> John Keeping <john@keeping.me.uk> writes:\n>> \n>>> You're right - I think we need to add \", errors='replace'\" to the call\n>>> to encode.\n>> \n>> Of if it is used just as a opaque token, you can .encode('hex') or\n>> something to punt on the whole issue, no?\n>\n> Even better.  Are you happy to squash that in (assuming nothing else\n> comes up) or shall I resend?\n\nIf you go the .encode('hex') route, the log message needs to explain\nwhy the hashed values are now different from the old implementation\nand justify why it is safe to do so.  I do not think I want to do\nthat myself ;-).\n\nThanks.\n\n\n>\n>>>>>  git-remote-testpy.py | 8 ++++----\n>>>>>  1 file changed, 4 insertions(+), 4 deletions(-)\n>>>>>\n>>>>> diff --git a/git-remote-testpy.py b/git-remote-testpy.py\n>>>>> index d94a66a..f8dc196 100644\n>>>>> --- a/git-remote-testpy.py\n>>>>> +++ b/git-remote-testpy.py\n>>>>> @@ -31,9 +31,9 @@ from git_remote_helpers.git.exporter import GitExporter\n>>>>>  from git_remote_helpers.git.importer import GitImporter\n>>>>>  from git_remote_helpers.git.non_local import NonLocalGit\n>>>>>  \n>>>>> -if sys.hexversion < 0x01050200:\n>>>>> -    # os.makedirs() is the limiter\n>>>>> -    sys.stderr.write(\"git-remote-testgit: requires Python 1.5.2 or later.\\n\")\n>>>>> +if sys.hexversion < 0x02000000:\n>>>>> +    # string.encode() is the limiter\n>>>>> +    sys.stderr.write(\"git-remote-testgit: requires Python 2.0 or later.\\n\")\n>>>>>      sys.exit(1)\n>>>>>  \n>>>>>  def get_repo(alias, url):\n>>>>> @@ -45,7 +45,7 @@ def get_repo(alias, url):\n>>>>>      repo.get_head()\n>>>>>  \n>>>>>      hasher = _digest()\n>>>>> -    hasher.update(repo.path)\n>>>>> +    hasher.update(repo.path.encode('utf-8'))\n>>>>>      repo.hash = hasher.hexdigest()\n>>>>>  \n>>>>>      repo.get_base_path = lambda base: os.path.join(\n"},{"id":"207190","messageId":"CAGdFq_hbstcUUUab42485oacdov4TS-f48e=spBSgi_jvBPCjw@mail.gmail.com","threadId":"32609","inReplyTo":"88502ee7b090454352b46d496741029817b27482.1358448207.git.john@keeping.me.uk","subject":"Re: [PATCH v2 8/8] git-remote-testpy: call print as a function","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2013-01-18T03:48:49Z","receivedAt":"2013-01-18T03:48:49Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Looks harmless enough.\n\nAcked-by: Sverre Rabbelier <srabbelier@gmail.com>\n\nOn Thu, Jan 17, 2013 at 10:54 AM, John Keeping <john@keeping.me.uk> wrote:\n> This is harmless in Python 2, which sees the parentheses as redundant\n> grouping, but is required for Python 3.  Since this is the only change\n> required to make this script just run under Python 3 without needing\n> 2to3 it seems worthwhile.\n>\n> The case of an empty print must be handled specially because in that\n> case Python 2 will interpret '()' as an empty tuple and print it as\n> '()'; inserting an empty string fixes this.\n>\n> Signed-off-by: John Keeping <john@keeping.me.uk>\n> ---\n>  git-remote-testpy.py | 28 ++++++++++++++--------------\n>  1 file changed, 14 insertions(+), 14 deletions(-)\n>\n> diff --git a/git-remote-testpy.py b/git-remote-testpy.py\n> index bc5e3cf..ccdb2dc 100644\n> --- a/git-remote-testpy.py\n> +++ b/git-remote-testpy.py\n> @@ -87,9 +87,9 @@ def do_capabilities(repo, args):\n>      \"\"\"Prints the supported capabilities.\n>      \"\"\"\n>\n> -    print \"import\"\n> -    print \"export\"\n> -    print \"refspec refs/heads/*:%s*\" % repo.prefix\n> +    print(\"import\")\n> +    print(\"export\")\n> +    print(\"refspec refs/heads/*:%s*\" % repo.prefix)\n>\n>      dirname = repo.get_base_path(repo.gitdir)\n>\n> @@ -98,11 +98,11 @@ def do_capabilities(repo, args):\n>\n>      path = os.path.join(dirname, 'git.marks')\n>\n> -    print \"*export-marks %s\" % path\n> +    print(\"*export-marks %s\" % path)\n>      if os.path.exists(path):\n> -        print \"*import-marks %s\" % path\n> +        print(\"*import-marks %s\" % path)\n>\n> -    print # end capabilities\n> +    print('') # end capabilities\n>\n>\n>  def do_list(repo, args):\n> @@ -115,16 +115,16 @@ def do_list(repo, args):\n>\n>      for ref in repo.revs:\n>          debug(\"? refs/heads/%s\", ref)\n> -        print \"? refs/heads/%s\" % ref\n> +        print(\"? refs/heads/%s\" % ref)\n>\n>      if repo.head:\n>          debug(\"@refs/heads/%s HEAD\" % repo.head)\n> -        print \"@refs/heads/%s HEAD\" % repo.head\n> +        print(\"@refs/heads/%s HEAD\" % repo.head)\n>      else:\n>          debug(\"@refs/heads/master HEAD\")\n> -        print \"@refs/heads/master HEAD\"\n> +        print(\"@refs/heads/master HEAD\")\n>\n> -    print # end list\n> +    print('') # end list\n>\n>\n>  def update_local_repo(repo):\n> @@ -164,7 +164,7 @@ def do_import(repo, args):\n>          ref = line[7:].strip()\n>          refs.append(ref)\n>\n> -    print \"feature done\"\n> +    print(\"feature done\")\n>\n>      if os.environ.get(\"GIT_REMOTE_TESTGIT_FAILURE\"):\n>          die('Told to fail')\n> @@ -172,7 +172,7 @@ def do_import(repo, args):\n>      repo = update_local_repo(repo)\n>      repo.exporter.export_repo(repo.gitdir, refs)\n>\n> -    print \"done\"\n> +    print(\"done\")\n>\n>\n>  def do_export(repo, args):\n> @@ -192,8 +192,8 @@ def do_export(repo, args):\n>          repo.non_local.push(repo.gitdir)\n>\n>      for ref in changed:\n> -        print \"ok %s\" % ref\n> -    print\n> +        print(\"ok %s\" % ref)\n> +    print('')\n>\n>\n>  COMMANDS = {\n> --\n> 1.8.1.1.260.g99b33f4.dirty\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n\n\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"207191","messageId":"CAGdFq_jcp-XKoTkz0iVKiVb0csrfcmCWEEiTBC-c0Bg2HP+e3g@mail.gmail.com","threadId":"32609","inReplyTo":"6bc90f3afc86d53eb6e4b4d6b87f6afd20023769.1358448207.git.john@keeping.me.uk","subject":"Re: [PATCH v2 7/8] git-remote-testpy: don't do unbuffered text I/O","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2013-01-18T03:50:19Z","receivedAt":"2013-01-18T03:50:19Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"On Thu, Jan 17, 2013 at 10:54 AM, John Keeping <john@keeping.me.uk> wrote:\n> -    sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0)\n> +    sys.stdin = os.fdopen(sys.stdin.fileno(), 'rb', 0)\n\nIt is not immediately obvious why you would open stdin in rb mode,\nplease add a comment.\n\n--\nCheers,\n\nSverre Rabbelier\n"},{"id":"207194","messageId":"CAGdFq_gew1-YmeUh=brWREHSYQvaV7vRBmEo0KFzi-ViqzOnaw@mail.gmail.com","threadId":"32609","inReplyTo":"bcef80fb913ca829bd2d08284e364ebd55b7297e.1358448207.git.john@keeping.me.uk","subject":"Re: [PATCH v2 4/8] git_remote_helpers: use 2to3 if building with Python 3","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2013-01-18T05:15:08Z","receivedAt":"2013-01-18T05:15:08Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"On Thu, Jan 17, 2013 at 10:53 AM, John Keeping <john@keeping.me.uk> wrote:\n> [1] http://wiki.python.org/moin/PortingPythonToPy3k\n\nThis link seems dead.\n\n--\nCheers,\n\nSverre Rabbelier\n"},{"id":"207199","messageId":"20130118103241.GM4574@serenity.lan","threadId":"32609","inReplyTo":"CAGdFq_gew1-YmeUh=brWREHSYQvaV7vRBmEo0KFzi-ViqzOnaw@mail.gmail.com","subject":"Re: [PATCH v2 4/8] git_remote_helpers: use 2to3 if building with Python 3","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-18T10:32:41Z","receivedAt":"2013-01-18T10:32:41Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Thu, Jan 17, 2013 at 09:15:08PM -0800, Sverre Rabbelier wrote:\n> On Thu, Jan 17, 2013 at 10:53 AM, John Keeping <john@keeping.me.uk> wrote:\n> > [1] http://wiki.python.org/moin/PortingPythonToPy3k\n> \n> This link seems dead.\n\nLooks like the Python wiki is down [1].\n\nI'll replace it with [2] since the content is similar and it should be\neasier to find a mirror of the Python documentation than of the wiki.\n\n[1] http://pyfound.blogspot.co.uk/2013/01/wikipythonorg-compromised.html\n[2] http://docs.python.org/3.3/howto/pyporting.html#during-installation\n\n\nJohn\n"},{"id":"207249","messageId":"CAGdFq_gwNUxug+AKMWebf4=-7D0PSdSm87mwzb3L3XGsg+MkTA@mail.gmail.com","threadId":"32609","inReplyTo":"20130118103241.GM4574@serenity.lan","subject":"Re: [PATCH v2 4/8] git_remote_helpers: use 2to3 if building with Python 3","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2013-01-19T07:52:16Z","receivedAt":"2013-01-19T07:52:16Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Assuming you tried this out on both 2.x and 3.x:\n\nAcked-by: Sverre Rabbelier <srabbelier@gmail.com>\n\nOn Fri, Jan 18, 2013 at 2:32 AM, John Keeping <john@keeping.me.uk> wrote:\n> On Thu, Jan 17, 2013 at 09:15:08PM -0800, Sverre Rabbelier wrote:\n>> On Thu, Jan 17, 2013 at 10:53 AM, John Keeping <john@keeping.me.uk> wrote:\n>> > [1] http://wiki.python.org/moin/PortingPythonToPy3k\n>>\n>> This link seems dead.\n>\n> Looks like the Python wiki is down [1].\n>\n> I'll replace it with [2] since the content is similar and it should be\n> easier to find a mirror of the Python documentation than of the wiki.\n>\n> [1] http://pyfound.blogspot.co.uk/2013/01/wikipythonorg-compromised.html\n> [2] http://docs.python.org/3.3/howto/pyporting.html#during-installation\n>\n>\n> John\n\n\n\n-- \nCheers,\n\nSverre Rabbelier\n"}]}