git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 05/13] git-p4: properly encode/decode communication with git for python 3

From
Yang Zhao <yang.zhao@skyboxlabs.com>
Date
Dec 7, 2019, 00:33 UTC
Message-ID
<20191207003333.3228-6-yang.zhao@skyboxlabs.com>
In-Reply-To
<20191207003333.3228-1-yang.zhao@skyboxlabs.com>

Under python3, calls to write() on the stream to `git fast-import` must be encoded. This patch wraps the IO object such that this encoding is done transparently.

Conversely, any text data read from subprocesses must also be decoded before running through the rest of the pipeline.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
---
 git-p4.py | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/git-p4.py b/git-p4.py
index 6720c7b24a..fefa716b17 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -201,10 +201,12 @@ def read_pipe_full(c):
     (out, err) = p.communicate()
     return (p.returncode, out, decode_text_stream(err))
 
-def read_pipe(c, ignore_error=False):
+def read_pipe(c, ignore_error=False, raw=False):
     """ Read output from  command. Returns the output text on
         success. On failure, terminates execution, unless
         ignore_error is True, when it returns an empty string.
+
+        If raw is True, do not attempt to decode output text.
     """
     (retcode, out, err) = read_pipe_full(c)
     if retcode != 0:
@@ -212,6 +214,8 @@ def read_pipe(c, ignore_error=False):
             out = ""
         else:
             die('Command failed: %s\nError: %s' % (str(c), err))
+    if not raw:
+        out = decode_text_stream(out)
     return out
 
 def read_pipe_text(c):
@@ -238,7 +242,6 @@ def read_pipe_lines(c):
     val = [decode_text_stream(line) for line in pipe.readlines()]
     if pipe.close() or p.wait():
         die('Command failed: %s' % str(c))
-
     return val
 
 def p4_read_pipe_lines(c):
@@ -634,7 +637,8 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,
             stdin_file.write(stdin)
         else:
             for i in stdin:
-                stdin_file.write(i + '\n')
+                stdin_file.write(encode_text_stream(i))
+                stdin_file.write(b'\n')
         stdin_file.flush()
         stdin_file.seek(0)
 
@@ -3556,6 +3560,15 @@ def openStreams(self):
         self.gitStream = self.importProcess.stdin
         self.gitError = self.importProcess.stderr
 
+        if bytes is not str:
+            # Wrap gitStream.write() so that it can be called using `str` arguments
+            def make_encoded_write(write):
+                def encoded_write(s):
+                    return write(s.encode() if isinstance(s, str) else s)
+                return encoded_write
+
+            self.gitStream.write = make_encoded_write(self.gitStream.write)
+
     def closeStreams(self):
         self.gitStream.close()
         if self.importProcess.wait() != 0:
-- 
2.21.0.windows.1
Previous: Yang ZhaoNext: Yang Zhao
Message 12 of 33 in “git-p4: python3 compatibility”
  1. 00/13 git-p4: python3 compatibilityYang Zhao, Dec 7, 2019
  2. 01/13 ci: also run linux-gcc pipeline with python-3.7 environmentYang Zhao, Dec 7, 2019
  3. SZEDER GáborDec 10, 2019
  4. Yang ZhaoDec 10, 2019
  5. SZEDER GáborDec 12, 2019
  6. Yang ZhaoDec 12, 2019
  7. SZEDER GáborDec 12, 2019
  8. Yang ZhaoDec 12, 2019
  9. 02/13 git-p4: make python-2.7 the oldest supported versionYang Zhao, Dec 7, 2019
  10. 03/13 git-p4: simplify python version detectionYang Zhao, Dec 7, 2019
  11. 04/13 git-p4: decode response from p4 to str for python3Yang Zhao, Dec 7, 2019
  12. 05/13 git-p4: properly encode/decode communication with git for python 3Yang Zhao, Dec 7, 2019
  13. 06/13 git-p4: open .gitp4-usercache.txt in text modeYang Zhao, Dec 7, 2019
  14. 06/13 git-p4: convert path to unicode before processing themYang Zhao, Dec 7, 2019
  15. 07/13 git-p4: convert path to unicode before processing themYang Zhao, Dec 7, 2019
  16. 07/13 git-p4: open .gitp4-usercache.txt in text modeYang Zhao, Dec 7, 2019
  17. 09/13 git-p4: fix freezing while waiting for fast-import progressYang Zhao, Dec 7, 2019
  18. 08/13 git-p4: use marshal format version 2 when sending to p4Yang Zhao, Dec 7, 2019
  19. 10/13 git-p4: use functools.reduce instead of reduceYang Zhao, Dec 7, 2019
  20. 11/13 git-p4: use dict.items() iteration for python3 compatibilityYang Zhao, Dec 7, 2019
  21. 13/13 git-p4: use python3's input() everywhereYang Zhao, Dec 7, 2019
  22. 12/13 git-p4: simplify regex pattern generation for parsing diff-treeYang Zhao, Dec 7, 2019
  23. Denton LiuDec 7, 2019
  24. Yang ZhaoDec 7, 2019
  25. Ben KeeneDec 7, 2019
  26. Yang ZhaoDec 7, 2019
  27. Ben KeeneDec 9, 2019
  28. Ben KeeneDec 9, 2019
  29. Johannes SchindelinDec 9, 2019
  30. Ben KeeneDec 10, 2019
  31. Yang ZhaoDec 9, 2019
  32. Ben KeeneDec 13, 2019
  33. Yang ZhaoDec 7, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.