threads / patch / 51498

patchTravis CI: Lint for Python syntax errors and undefined names

Subject: [PATCH 0/1] Travis CI: Lint for Python syntax errors and undefined names

## tl;dr

4 messages between Jul 19, 2019 and Jul 20, 2019. Diffs are folded; open one to read it.

replies: 3people: 4as markdown or json

Christian Clauss via GitGitGadget· Jul 19, 2019, 14:18 UTC · lore

Several things were changed between Python 2 and Python 3. Here we are making changes to make the code run on both Py2 and Py3. We are doing this because the end of life of Python 2 is in 167 days. We are using print() function because legacy print statements are syntax errors on Py3.reduce() was moved in Python 3 and raw_input() was removed so we make changes to avoid NameErrors at runtime. Signed-off-by: Christian Clauss cclauss@me.com [cclauss@me.com]

cclauss (1):
  Travis CI: Lint for Python syntax errors and undefined names
 .travis.yml                        |  4 +++
 contrib/fast-import/import-zips.py |  3 +-
 contrib/hg-to-git/hg-to-git.py     | 47 +++++++++++++++---------------
 git-p4.py                          |  3 ++
 4 files changed, 33 insertions(+), 24 deletions(-)
base-commit: 9d418600f4d10dcbbfb0b5fdbc71d509e03ba719
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-290%2Fcclauss%2Fpatch-1-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-290/cclauss/patch-1-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/290
-- 
gitgitgadget
cclauss via GitGitGadget· Jul 19, 2019, 14:18 UTC · re: Christian Clauss via GitGitGadget · lore

[PATCH 1/1] Travis CI: Lint for Python syntax errors and undefined names

From: cclauss <cclauss@me.com>

Several things were changed between Python 2 and Python 3. There are a few Python 3 incompatibilities to work on. Here we are making changes to make the code run on both Py2 and Py3. We are doing this because the end of life of Python 2 is in 167 days. We are using print() function because legacy print statements are syntax errors on Py3. reduce() was moved in Python 3 and raw_input() was removed so we make changes to avoid NameErrors being raised at runtime. We are also putting flake8 lint tests in place on Travis CI to avoid any backsliding on future pull requests.

Signed-off-by: cclauss <cclauss@me.com>
---
 .travis.yml                        |  4 +++
 contrib/fast-import/import-zips.py |  3 +-
 contrib/hg-to-git/hg-to-git.py     | 47 +++++++++++++++---------------
 git-p4.py                          |  3 ++
 4 files changed, 33 insertions(+), 24 deletions(-)
Show changes to 4 files +33 −24

.travis.yml, contrib/fast-import/import-zips.py, contrib/hg-to-git/hg-to-git.py, git-p4.py

diff --git a/.travis.yml b/.travis.yml
index ffb1bc46f2..62557dc8fd 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -39,6 +39,10 @@ matrix:
       compiler:
       script: ci/test-documentation.sh
       after_failure:
+    - env: jobname=LintPython
+      language: python
+      before_install: pip install flake8
+      script: flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics
 
 before_install: ci/install-dependencies.sh
 script: ci/run-build-and-tests.sh
diff --git a/contrib/fast-import/import-zips.py b/contrib/fast-import/import-zips.py
index d12c296223..138194475c 100755
--- a/contrib/fast-import/import-zips.py
+++ b/contrib/fast-import/import-zips.py
@@ -8,6 +8,7 @@
 ##  python import-zips.py *.zip
 ##  git log --stat import-zips
 
+from __future__ import print_function
 from os import popen, path
 from sys import argv, exit, hexversion, stderr
 from time import mktime
@@ -19,7 +20,7 @@
     exit(1)
 
 if len(argv) < 2:
-    print 'usage:', argv[0], '<zipfile>...'
+    print('usage:', argv[0], '<zipfile>...')
     exit(1)
 
 branch_ref = 'refs/heads/import-zips'
diff --git a/contrib/hg-to-git/hg-to-git.py b/contrib/hg-to-git/hg-to-git.py
index de3f81667e..1d5e5156a4 100755
--- a/contrib/hg-to-git/hg-to-git.py
+++ b/contrib/hg-to-git/hg-to-git.py
@@ -18,6 +18,7 @@
     along with this program; if not, see <http://www.gnu.org/licenses/>.
 """
 
+from __future__ import print_function
 import os, os.path, sys
 import tempfile, pickle, getopt
 import re
@@ -42,7 +43,7 @@
 
 def usage():
 
-        print """\
+        print("""\
 %s: [OPTIONS] <hgprj>
 
 options:
@@ -54,7 +55,7 @@ def usage():
 
 required:
     hgprj:  name of the HG project to import (directory)
-""" % sys.argv[0]
+""" % sys.argv[0])
 
 #------------------------------------------------------------------------------
 
@@ -104,22 +105,22 @@ def getgitenv(user, date):
 if state:
     if os.path.exists(state):
         if verbose:
-            print 'State does exist, reading'
+            print('State does exist, reading')
         f = open(state, 'r')
         hgvers = pickle.load(f)
     else:
-        print 'State does not exist, first run'
+        print('State does not exist, first run')
 
 sock = os.popen('hg tip --template "{rev}"')
 tip = sock.read()
 if sock.close():
     sys.exit(1)
 if verbose:
-    print 'tip is', tip
+    print('tip is', tip)
 
 # Calculate the branches
 if verbose:
-    print 'analysing the branches...'
+    print('analysing the branches...')
 hgchildren["0"] = ()
 hgparents["0"] = (None, None)
 hgbranch["0"] = "master"
@@ -155,7 +156,7 @@ def getgitenv(user, date):
             hgbranch[str(cset)] = "branch-" + str(cset)
 
 if not hgvers.has_key("0"):
-    print 'creating repository'
+    print('creating repository')
     os.system('git init')
 
 # loop through every hg changeset
@@ -180,27 +181,27 @@ def getgitenv(user, date):
     os.write(fdcomment, csetcomment)
     os.close(fdcomment)
 
-    print '-----------------------------------------'
-    print 'cset:', cset
-    print 'branch:', hgbranch[str(cset)]
-    print 'user:', user
-    print 'date:', date
-    print 'comment:', csetcomment
+    print('-----------------------------------------')
+    print('cset:', cset)
+    print('branch:', hgbranch[str(cset)])
+    print('user:', user)
+    print('date:', date)
+    print('comment:', csetcomment)
     if parent:
-	print 'parent:', parent
+	    print('parent:', parent)
     if mparent:
-        print 'mparent:', mparent
+        print('mparent:', mparent)
     if tag:
-        print 'tag:', tag
-    print '-----------------------------------------'
+        print('tag:', tag)
+    print('-----------------------------------------')
 
     # checkout the parent if necessary
     if cset != 0:
         if hgbranch[str(cset)] == "branch-" + str(cset):
-            print 'creating new branch', hgbranch[str(cset)]
+            print('creating new branch', hgbranch[str(cset)])
             os.system('git checkout -b %s %s' % (hgbranch[str(cset)], hgvers[parent]))
         else:
-            print 'checking out branch', hgbranch[str(cset)]
+            print('checking out branch', hgbranch[str(cset)])
             os.system('git checkout %s' % hgbranch[str(cset)])
 
     # merge
@@ -209,7 +210,7 @@ def getgitenv(user, date):
             otherbranch = hgbranch[mparent]
         else:
             otherbranch = hgbranch[parent]
-        print 'merging', otherbranch, 'into', hgbranch[str(cset)]
+        print('merging', otherbranch, 'into', hgbranch[str(cset)])
         os.system(getgitenv(user, date) + 'git merge --no-commit -s ours "" %s %s' % (hgbranch[str(cset)], otherbranch))
 
     # remove everything except .git and .hg directories
@@ -233,12 +234,12 @@ def getgitenv(user, date):
 
     # delete branch if not used anymore...
     if mparent and len(hgchildren[str(cset)]):
-        print "Deleting unused branch:", otherbranch
+        print("Deleting unused branch:", otherbranch)
         os.system('git branch -d %s' % otherbranch)
 
     # retrieve and record the version
     vvv = os.popen('git show --quiet --pretty=format:%H').read()
-    print 'record', cset, '->', vvv
+    print('record', cset, '->', vvv)
     hgvers[str(cset)] = vvv
 
 if hgnewcsets >= opt_nrepack and opt_nrepack != -1:
@@ -247,7 +248,7 @@ def getgitenv(user, date):
 # write the state for incrementals
 if state:
     if verbose:
-        print 'Writing state'
+        print('Writing state')
     f = open(state, 'w')
     pickle.dump(hgvers, f)
 
diff --git a/git-p4.py b/git-p4.py
index 3991e7d1a7..9faee25db2 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -36,6 +36,8 @@
     unicode = str
     bytes = bytes
     basestring = (str,bytes)
+    raw_input = input
+    from functools import reduce
 else:
     # 'unicode' exists, must be Python 2
     str = str
@@ -3968,6 +3970,7 @@ def renameBranch(self, branch_name):
                 break
 
         if not found:
+            sync = P4Sync()
             sys.exit("gave up trying to rename existing branch {0}".format(sync.branch))
 
     def findLastP4Revision(self, starting_point):
-- 
gitgitgadget
Junio C Hamano· Jul 19, 2019, 19:44 UTC · re: cclauss via GitGitGadget · lore

Re: [PATCH 1/1] Travis CI: Lint for Python syntax errors and undefined names

"cclauss via GitGitGadget" <gitgitgadget@gmail.com> writes:
> From: cclauss <cclauss@me.com>

I'll tweak this line (and your sign-off) to read as "Christian Clauss <cclauss@me.com>" as you had in your cover letter.

> Several things were changed between Python 2 and Python 3.
> There are a few Python 3 incompatibilities to work on.
> Here we are making changes to make the code run on both Py2 and Py3.
> We are doing this because the end of life of Python 2 is in 167 days.

All sounds sensible, and the above is quite a good problem description.

Show 6 quoted lines
> We are using print() function because legacy print statements are syntax
> errors on Py3.
> reduce() was moved in Python 3 and raw_input() was removed so we make
> changes to avoid NameErrors being raised at runtime.
> We are also putting flake8 lint tests in place on Travis CI to avoid
> any backsliding on future pull requests.

Nothing is wrong here, but the convention in our codebase is to describe the changes as if we are giving orders to the codebase "to be like so". And as you have enumeration, I would write something like this if I were describing this change:

     - Use the `print()` function, because Py3 no longer has the `print`
       statement.
     - Import `reduce()` from functools, because Py3 requires this, and
       importing also works with Py2 (even though it wasn't necessary).
     - Use `input()` instead of `raw_input()`, as the former can be used
       with both but the latter was removed in Py3.
    Also add a CI job to Travis CI to run flake8 lint to avoid an
    backsliding on future pull requests.

https://python-future.org/compatible_idioms.html#raw-input seems to suggest, just like you import reduce from functools, you need to import input from builtins. Is it not the case?

> +      script: flake8 . --count --select=E9,F63,F72,F82 --show-source --statistics

How was the set of "select"ed violations to catch chosen? How are we going to maintain this list going forward?

The rest of the patch looked sensible.
Thanks.
SZEDER Gábor· Jul 20, 2019, 12:51 UTC · re: cclauss via GitGitGadget · lore

Re: [PATCH 1/1] Travis CI: Lint for Python syntax errors and undefined names

On Fri, Jul 19, 2019 at 07:18:55AM -0700, cclauss via GitGitGadget wrote:
Show 10 quoted lines
> Several things were changed between Python 2 and Python 3.
> There are a few Python 3 incompatibilities to work on.
> Here we are making changes to make the code run on both Py2 and Py3.
> We are doing this because the end of life of Python 2 is in 167 days.
> We are using print() function because legacy print statements are syntax
> errors on Py3.
> reduce() was moved in Python 3 and raw_input() was removed so we make
> changes to avoid NameErrors being raised at runtime.
> We are also putting flake8 lint tests in place on Travis CI to avoid
> any backsliding on future pull requests.

It seems to me that this patch does too many things at once, and perhaps it would be better to split it into a couple of smaller patches that do only one thing, e.g.:

  - use print function instead of statement in 'hg-to-git' (which
    constitutes the bulk of this patch),
  - do the same in 'contrib/fast-import/import-zips.py'
  - import 'reduce' and fix 'raw_input' in 'git-p4'
  - and once all that is done and the linter runs clean, add the
    linter job to Travis CI.

This would ease the job of the reader, now and in the future, and it would better stand out that ...

> diff --git a/git-p4.py b/git-p4.py
> index 3991e7d1a7..9faee25db2 100755
> --- a/git-p4.py
> +++ b/git-p4.py
Show 5 quoted lines
> @@ -3968,6 +3970,7 @@ def renameBranch(self, branch_name):
>                  break
>  
>          if not found:
> +            sync = P4Sync()
... this change is not mentioned in the commit message.

Is this something the linter complains about? It doesn't look like a Python 2 vs. 3 compatibility fix to me, so it might deserve a dedicated patch.

Cc-ing Luke for this bit.
Show 5 quoted lines
>              sys.exit("gave up trying to rename existing branch {0}".format(sync.branch))
>  
>      def findLastP4Revision(self, starting_point):
> -- 
> gitgitgadget

← back to recent threads