git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCH 2/2] git-p4: support loading changelist descriptions from files

From
MAMazo, Andrey <amazo@checkvideo.com>
Date
Mar 22, 2019, 19:54 UTC
Message-ID
<bb3e14a3897c98762b0e656d583eaa408a6aba60.1553283214.git.amazo@checkvideo.com>
In-Reply-To
<cover.1553283214.git.amazo@checkvideo.com>
Our Perforce server experienced some kind of database corruption a few years ago.
While the file data and revision history are mostly intact,
some metadata for several changesets got lost.
For example, inspecting certain changelists produces errors.
"""
$ p4 describe -s 12345
Date 2019/02/26 16:46:17:
Operation: user-describe
Operation 'user-describe' failed.
Change 12345 description missing!
"""
While some metadata (like changeset descriptions) is obviously lost,
most of it can be reconstructed via other commands:
 * `p4 changes -l -t //...@12345,12345` --
   to obtain date+time, author, beginning of changeset description;
 * `p4 files -a //...@12345,12345` --
   to obtain file revisions, file types, file actions;
 * `p4 diff2 -u //...@12344 //...@12345` --
   to get a unified diff of text files in a changeset;
 * `p4 print -o binary.blob@12345 //depot/binary.blob@12345` --
   to get a revision of a binary file.

It might be possible to teach git-p4 to fallback to other methods if `p4 describe` fails, but it's probably too special-cased (really depends on kind and scale of DB corruption), so some manual intervention is perhaps acceptable.

So, with some manual work, it's possible to reconstruct `p4 -G describe ...` output manually. In our case, once git-p4 passes `p4 describe` stage, it can proceed further just fine. Thus, it's tempting to feed resurrected metadata to git-p4 when a normal `p4 describe` would fail.

This functionality may be useful to cache changelist information, or to make some changes to changelist info before feeding it to git-p4.

A new config parameter is introduced to tell git-p4 to load certain changelist descriptions from files instead of from a server. For simplicity, it's one pickled file per changelist. ``` git config --add git-p4.damagedChangelists 12345.pickled git config --add git-p4.damagedChangelists 12346.pickled ```

The following trivial script may be used to produce pickled `p4 -G describe`-compatible output.
"""
 #!/usr/bin/python2
 import pickle
 import time
 # recovered commits of interest
 changes = [
     {
         'change':     '12345',
         'status':     'submitted',
         'code':       'stat',
         'user':       'username1',
         'time':       str(int(time.mktime(time.strptime('2019/02/28 16:00:30', '%Y/%m/%d %H:%M:%S')))),
         'client':     'username1_hostname1',
         'desc':       'A bug is fixed.\nDetails are below:<lost>\n',
         'depotFile0': '//depot/branch1/foo.sh',
         'action0':    'edit',
         'rev0':       '28',
         'type0':      'xtext',
         'depotFile1': '//depot/branch1/bar.py',
         'action1':    'edit',
         'rev1':       '43',
         'type1':      'text',
         'depotFile2': '//depot/branch1/baz.doc',
         'action2':    'edit',
         'rev2':       '8',
         'type2':      'binary',
         'depotFile3': '//depot/branch1/qqq.c',
         'action3':    'edit',
         'rev3':       '6',
         'type3':      'ktext',
     },
 ]
 for change in changes:
     pickle.dump(change, open('{0}.pickled'.format(change['change']), 'wb'))
"""
Signed-off-by: Andrey Mazo <amazo@checkvideo.com>
---
Notes:
    Documentation changes and tests are obviously missing,
    but I hoped to get some feedback on the idea overall
    before working on those.
 git-p4.py | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
diff --git a/git-p4.py b/git-p4.py
index 40bc84573b..3133419280 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -24,10 +24,11 @@
 import stat
 import zipfile
 import zlib
 import ctypes
 import errno
+import pickle
 
 # support basestring in python3
 try:
     unicode = unicode
 except NameError:
@@ -2615,10 +2616,12 @@ def __init__(self):
         self.knownAlienLabelBranches = {}
 
         self.tz = "%+03d%02d" % (- time.timezone / 3600, ((- time.timezone % 3600) / 60))
         self.labels = {}
 
+        self.damagedChangelists = {}
+
     # Force a checkpoint in fast-import and wait for it to finish
     def checkpoint(self):
         self.gitStream.write("checkpoint\n\n")
         self.gitStream.write("progress checkpoint\n\n")
         out = self.gitOutput.readline()
@@ -3312,10 +3315,25 @@ def getAlienLabelBranchMapping(self):
         for mapping in alienLabelBranches:
             if mapping:
                 (alien, ours) = mapping.split(":")
                 self.knownAlienLabelBranches[alien] = ours
 
+    def loadDamagedChangelists(self):
+        damagedChangelists = gitConfigList("git-p4.damagedChangelists")
+        for clPickled in damagedChangelists:
+            if not clPickled:
+                continue
+
+            try:
+                clDesc = pickle.load(open(clPickled, 'rb'))
+                if not ("status" in clDesc and "user" in clDesc and "time" in clDesc and "change" in clDesc):
+                    die("Changelist description read from {0} doesn't have required fields".format(clPickled))
+            except (IOError, TypeError) as e:
+                die("Can't read changelist description dict from {0}: {1}".format(clPickled, str(e)))
+
+            self.damagedChangelists[int(clDesc["change"])] = clDesc
+
     def updateOptionDict(self, d):
         option_keys = {}
         if self.keepRepoPath:
             option_keys['keepRepoPath'] = 1
 
@@ -3413,11 +3431,14 @@ def searchParent(self, parent, branch, target):
             return None
 
     def importChanges(self, changes, origin_revision=0):
         cnt = 1
         for change in changes:
-            description = p4_describe(change)
+            if change in self.damagedChangelists:
+                description = self.damagedChangelists[change]
+            else:
+                description = p4_describe(change)
             self.updateOptionDict(description)
 
             if not self.silent:
                 sys.stdout.write("\rImporting revision %s (%s%%)" % (change, cnt * 100 / len(changes)))
                 sys.stdout.flush()
@@ -3704,10 +3725,12 @@ def run(self, args):
                     bad_changesfile = True
                     break
         if bad_changesfile:
             die("Option --changesfile is incompatible with revision specifiers")
 
+        self.loadDamagedChangelists()
+
         newPaths = []
         for p in self.depotPaths:
             if p.find("@") != -1:
                 atIdx = p.index("@")
                 self.changeRange = p[atIdx:]
-- 
2.19.2
Previous: Mazo, AndreyNext: Luke Diamand
Message 22 of 40 in “git-p4: a few assorted fixes for branches, excludes”
  1. 0/5 git-p4: a few assorted fixes for branches, excludesMazo, Andrey, Mar 4, 2019
  2. 1/5 git-p4: detect/prevent infinite loop in gitCommitByP4Change()Mazo, Andrey, Mar 4, 2019
  3. 2/5 git-p4: match branches case insensitively if configuredMazo, Andrey, Mar 4, 2019
  4. 3/5 git-p4: don't groom exclude path list on every commitMazo, Andrey, Mar 4, 2019
  5. 4/5 git-p4: add failing test for "don't exclude other files with same prefix"Mazo, Andrey, Mar 4, 2019
  6. 5/5 git-p4: don't exclude other files with same prefixMazo, Andrey, Mar 4, 2019
  7. 0/7 git-p4: a few assorted fixes for branches, excludesMazo, Andrey, Mar 21, 2019
  8. 2/7 git-p4: match branches case insensitively if configuredMazo, Andrey, Mar 21, 2019
  9. Luke DiamandMar 23, 2019
  10. Mazo, AndreyMar 25, 2019
  11. 1/7 git-p4: detect/prevent infinite loop in gitCommitByP4Change()Mazo, Andrey, Mar 21, 2019
  12. 3/7 git-p4: don't groom exclude path list on every commitMazo, Andrey, Mar 21, 2019
  13. 4/7 git-p4: add failing test for "don't exclude other files with same prefix"Mazo, Andrey, Mar 21, 2019
  14. 5/7 git-p4: don't exclude other files with same prefixMazo, Andrey, Mar 21, 2019
  15. 6/7 git-p4: add failing test for "git-p4: respect excluded paths when detecting branches"Mazo, Andrey, Mar 21, 2019
  16. 7/7 git-p4: respect excluded paths when detecting branchesMazo, Andrey, Mar 21, 2019
  17. 0/2 git-p4: "alien" branches and load changelist info from fileMazo, Andrey, Mar 22, 2019
  18. 1/2 git-p4: introduce alien branch mappingsMazo, Andrey, Mar 22, 2019
  19. Luke DiamandMar 23, 2019
  20. Mazo, AndreyMar 26, 2019
  21. 1/1 git-p4: inexact label detectionMazo, Andrey, Mar 27, 2019
  22. 2/2 git-p4: support loading changelist descriptions from filesMazo, Andrey, Mar 22, 2019
  23. Luke DiamandMar 23, 2019
  24. Mazo, AndreyMar 25, 2019
  25. 0/8 git-p4: a few assorted fixes for branches, excludesMazo, Andrey, Apr 1, 2019
  26. 1/8 git-p4: detect/prevent infinite loop in gitCommitByP4Change()Mazo, Andrey, Apr 1, 2019
  27. 2/8 git-p4: add failing test for "git-p4: match branches case insensitively if configured"Mazo, Andrey, Apr 1, 2019
  28. SZEDER GáborApr 2, 2019
  29. Mazo, AndreyApr 2, 2019
  30. Junio C HamanoApr 3, 2019
  31. 4/8 git-p4: don't groom exclude path list on every commitMazo, Andrey, Apr 1, 2019
  32. 3/8 git-p4: match branches case insensitively if configuredMazo, Andrey, Apr 1, 2019
  33. 5/8 git-p4: add failing test for "don't exclude other files with same prefix"Mazo, Andrey, Apr 1, 2019
  34. 6/8 git-p4: don't exclude other files with same prefixMazo, Andrey, Apr 1, 2019
  35. 7/8 git-p4: add failing test for "git-p4: respect excluded paths when detecting branches"Mazo, Andrey, Apr 1, 2019
  36. 8/8 git-p4: respect excluded paths when detecting branchesMazo, Andrey, Apr 1, 2019
  37. Mazo, AndreyApr 1, 2019
  38. 0/2 git-p4: inexact labels and load changelist description from fileMazo, Andrey, Apr 2, 2019
  39. 1/2 git-p4: inexact label detectionMazo, Andrey, Apr 2, 2019
  40. 2/2 git-p4: support loading changelist descriptions from filesMazo, Andrey, Apr 2, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.