git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 05/14] git-p4: encode/decode communication with git for python3

From
Yang Zhao <yang.zhao@skyboxlabs.com>
Date
Dec 13, 2019, 23:52 UTC
Message-ID
<20191213235247.23660-7-yang.zhao@skyboxlabs.com>
In-Reply-To
<20191213235247.23660-1-yang.zhao@skyboxlabs.com>

Under python3, calls to write() on the stream to `git fast-import` must be encoded. This patch wraps the IO object such that this encoding is done transparently.

Conversely, any text data read from subprocesses must also be decoded before running through the rest of the pipeline.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
---
 git-p4.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/git-p4.py b/git-p4.py
index ca891e3d5d..d62fb05989 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -183,10 +183,12 @@ def read_pipe_full(c):
     (out, err) = p.communicate()
     return (p.returncode, out, decode_text_stream(err))
 
-def read_pipe(c, ignore_error=False):
+def read_pipe(c, ignore_error=False, raw=False):
     """ Read output from  command. Returns the output text on
         success. On failure, terminates execution, unless
         ignore_error is True, when it returns an empty string.
+
+        If raw is True, do not attempt to decode output text.
     """
     (retcode, out, err) = read_pipe_full(c)
     if retcode != 0:
@@ -194,6 +196,8 @@ def read_pipe(c, ignore_error=False):
             out = ""
         else:
             die('Command failed: %s\nError: %s' % (str(c), err))
+    if not raw:
+        out = decode_text_stream(out)
     return out
 
 def read_pipe_text(c):
@@ -220,7 +224,6 @@ def read_pipe_lines(c):
     val = [decode_text_stream(line) for line in pipe.readlines()]
     if pipe.close() or p.wait():
         die('Command failed: %s' % str(c))
-
     return val
 
 def p4_read_pipe_lines(c):
@@ -616,7 +619,8 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,
             stdin_file.write(stdin)
         else:
             for i in stdin:
-                stdin_file.write(i + '\n')
+                stdin_file.write(encode_text_stream(i))
+                stdin_file.write(b'\n')
         stdin_file.flush()
         stdin_file.seek(0)
 
@@ -1245,7 +1249,7 @@ def generatePointer(self, contentFile):
             ['git', 'lfs', 'pointer', '--file=' + contentFile],
             stdout=subprocess.PIPE
         )
-        pointerFile = pointerProcess.stdout.read()
+        pointerFile = decode_text_stream(pointerProcess.stdout.read())
         if pointerProcess.wait():
             os.remove(contentFile)
             die('git-lfs pointer command failed. Did you install the extension?')
@@ -3538,6 +3542,15 @@ def openStreams(self):
         self.gitStream = self.importProcess.stdin
         self.gitError = self.importProcess.stderr
 
+        if bytes is not str:
+            # Wrap gitStream.write() so that it can be called using `str` arguments
+            def make_encoded_write(write):
+                def encoded_write(s):
+                    return write(s.encode() if isinstance(s, str) else s)
+                return encoded_write
+
+            self.gitStream.write = make_encoded_write(self.gitStream.write)
+
     def closeStreams(self):
         self.gitStream.close()
         if self.importProcess.wait() != 0:
-- 
2.21.0.windows.1
Previous: Yang ZhaoNext: Yang Zhao
Message 10 of 22 in “git-p4: python3 compatibility”
  1. 00/14 git-p4: python3 compatibilityYang Zhao, Dec 13, 2019
  2. Yang ZhaoJan 17, 2020
  3. Luke DiamandJan 24, 2020
  4. Luke DiamandJan 30, 2020
  5. Luke DiamandFeb 3, 2020
  6. Yang ZhaoFeb 3, 2020
  7. Yang ZhaoFeb 4, 2020
  8. 01/14 git-p4: make python2.7 the oldest supported versionYang Zhao, Dec 13, 2019
  9. 02/14 git-p4: change the expansion test from basestring to listYang Zhao, Dec 13, 2019
  10. 05/14 git-p4: encode/decode communication with git for python3Yang Zhao, Dec 13, 2019
  11. 03/14 git-p4: remove string type aliasingYang Zhao, Dec 13, 2019
  12. 04/14 git-p4: encode/decode communication with p4 for python3Yang Zhao, Dec 13, 2019
  13. Junio C HamanoDec 17, 2019
  14. 08/14 git-p4: use marshal format version 2 when sending to p4Yang Zhao, Dec 13, 2019
  15. 07/14 git-p4: open .gitp4-usercache.txt in text modeYang Zhao, Dec 13, 2019
  16. 06/14 git-p4: convert path to unicode before processing themYang Zhao, Dec 13, 2019
  17. 09/14 git-p4: fix freezing while waiting for fast-import progressYang Zhao, Dec 13, 2019
  18. 10/14 git-p4: use functools.reduce instead of reduceYang Zhao, Dec 13, 2019
  19. 12/14 git-p4: simplify regex pattern generation for parsing diff-treeYang Zhao, Dec 13, 2019
  20. 11/14 git-p4: use dict.items() iteration for python3 compatibilityYang Zhao, Dec 13, 2019
  21. 14/14 ci: also run linux-gcc pipeline with python3.5 environmentYang Zhao, Dec 13, 2019
  22. 13/14 git-p4: use python3's input() everywhereYang Zhao, Dec 13, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.