{"thread":{"id":"40347","subject":"[PATCH v5 0/7] git-p4: add support for large file systems","startedAt":"2015-09-14T13:26:27Z","lastAt":"2015-09-20T21:26:24Z","messageCount":14,"participants":["larsxschneider@gmail.com","Luke Diamand","Lars Schneider","Eric Sunshine","Junio C Hamano"],"isPatch":true,"patchVersion":5,"patchTotal":7},"messages":[{"id":"269916","messageId":"1442237194-49624-1-git-send-email-larsxschneider@gmail.com","threadId":"40347","inReplyTo":null,"subject":"[PATCH v5 0/7] git-p4: add support for large file systems","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-14T13:26:27Z","receivedAt":"2015-09-14T13:26:27Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nDiff to v4:\n* Add streaming progress in verbose mode.\n* Add check for free disk space.\n* Remove the limitation that no .gitattribues must be present.\n* Refactor large file system classes. The base implementation does not assume a \".gitattributes mechanism\" anymore.\n* Throw error if the large file system is used with 'git p4 submit'. Add warning in the docs, too.\n\nThanks to Junio and Luke feedback!\n\nLars Schneider (7):\n git-p4: add optional type specifier to gitConfig reader\n git-p4: add gitConfigInt reader\n git-p4: return an empty list if a list config has no values\n git-p4: add file streaming progress in verbose mode\n git-p4: check free space during streaming\n git-p4: add support for large file systems\n git-p4: add Git LFS backend for large file system\n\n Documentation/git-p4.txt | 32 +++++\n git-p4.py | 279 ++++++++++++++++++++++++++++++++++++++++---\n t/t9823-git-p4-mock-lfs.sh | 96 +++++++++++++++\n t/t9824-git-p4-git-lfs.sh | 288 +++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 678 insertions(+), 17 deletions(-)\n create mode 100755 t/t9823-git-p4-mock-lfs.sh\n create mode 100755 t/t9824-git-p4-git-lfs.sh\n\n--\n2.5.1\n"},{"id":"269924","messageId":"1442237194-49624-2-git-send-email-larsxschneider@gmail.com","threadId":"40347","inReplyTo":"1442237194-49624-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v5 1/7] git-p4: add optional type specifier to gitConfig reader","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-14T13:26:28Z","receivedAt":"2015-09-14T13:26:28Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nThe functions “gitConfig” and “gitConfigBool” are almost identical. Make “gitConfig” more generic by adding an optional type specifier. Use the type specifier “—bool” with “gitConfig” to implement “gitConfigBool. This prepares the implementation of other type specifiers such as “—int”.\n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 073f87b..c139cab 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -604,9 +604,12 @@ def gitBranchExists(branch):\n \n _gitConfig = {}\n \n-def gitConfig(key):\n+def gitConfig(key, typeSpecifier=None):\n if not _gitConfig.has_key(key):\n- cmd = [ \"git\", \"config\", key ]\n+ cmd = [ \"git\", \"config\" ]\n+ if typeSpecifier:\n+ cmd += [ typeSpecifier ]\n+ cmd += [ key ]\n s = read_pipe(cmd, ignore_error=True)\n _gitConfig[key] = s.strip()\n return _gitConfig[key]\n@@ -617,10 +620,7 @@ def gitConfigBool(key):\n in the config.\"\"\"\n \n if not _gitConfig.has_key(key):\n- cmd = [ \"git\", \"config\", \"--bool\", key ]\n- s = read_pipe(cmd, ignore_error=True)\n- v = s.strip()\n- _gitConfig[key] = v == \"true\"\n+ _gitConfig[key] = gitConfig(key, '--bool') == \"true\"\n return _gitConfig[key]\n \n def gitConfigList(key):\n-- \n2.5.1\n"},{"id":"269917","messageId":"1442237194-49624-3-git-send-email-larsxschneider@gmail.com","threadId":"40347","inReplyTo":"1442237194-49624-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v5 2/7] git-p4: add gitConfigInt reader","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-14T13:26:29Z","receivedAt":"2015-09-14T13:26:29Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nAdd a git config reader for integer variables. Please note that the git config implementation automatically supports k, m, and g suffixes.\n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex c139cab..40ad4ae 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -623,6 +623,17 @@ def gitConfigBool(key):\n _gitConfig[key] = gitConfig(key, '--bool') == \"true\"\n return _gitConfig[key]\n \n+def gitConfigInt(key):\n+ if not _gitConfig.has_key(key):\n+ cmd = [ \"git\", \"config\", \"--int\", key ]\n+ s = read_pipe(cmd, ignore_error=True)\n+ v = s.strip()\n+ try:\n+ _gitConfig[key] = int(gitConfig(key, '--int'))\n+ except ValueError:\n+ _gitConfig[key] = None\n+ return _gitConfig[key]\n+\n def gitConfigList(key):\n if not _gitConfig.has_key(key):\n s = read_pipe([\"git\", \"config\", \"--get-all\", key], ignore_error=True)\n-- \n2.5.1\n"},{"id":"269923","messageId":"1442237194-49624-4-git-send-email-larsxschneider@gmail.com","threadId":"40347","inReplyTo":"1442237194-49624-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v5 3/7] git-p4: return an empty list if a list config has no values","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-14T13:26:30Z","receivedAt":"2015-09-14T13:26:30Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 40ad4ae..90d3b90 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -638,6 +638,8 @@ def gitConfigList(key):\n if not _gitConfig.has_key(key):\n s = read_pipe([\"git\", \"config\", \"--get-all\", key], ignore_error=True)\n _gitConfig[key] = s.strip().split(os.linesep)\n+ if _gitConfig[key] == ['']:\n+ _gitConfig[key] = []\n return _gitConfig[key]\n \n def p4BranchesInGit(branchesAreInRemotes=True):\n-- \n2.5.1\n"},{"id":"269919","messageId":"1442237194-49624-5-git-send-email-larsxschneider@gmail.com","threadId":"40347","inReplyTo":"1442237194-49624-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v5 4/7] git-p4: add file streaming progress in verbose mode","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-14T13:26:31Z","receivedAt":"2015-09-14T13:26:31Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nIf a file is streamed from P4 to Git then the verbose mode prints\ncontinuously the progress as percentage like this:\n//depot/file.bin 20% (10 MB)\n\nUpon completion the progress is overwritten with depot source, local\nfile and size like this:\n//depot/file.bin --> local/file.bin (10 MB)\n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 20 ++++++++++++++++++--\n 1 file changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 90d3b90..e7a7ea4 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2164,7 +2164,9 @@ class P4Sync(Command, P4UserMap):\n def streamOneP4File(self, file, contents):\n relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes)\n if verbose:\n- sys.stderr.write(\"%s\\n\" % relPath)\n+ size = int(self.stream_file['fileSize'])\n+ sys.stdout.write('\\r%s --> %s (%i MB)\\n' % (file['depotFile'], relPath, size/1024/1024))\n+ sys.stdout.flush()\n \n (type_base, type_mods) = split_p4_type(file[\"type\"])\n \n@@ -2241,7 +2243,8 @@ class P4Sync(Command, P4UserMap):\n def streamOneP4Deletion(self, file):\n relPath = self.stripRepoPath(file['path'], self.branchPrefixes)\n if verbose:\n- sys.stderr.write(\"delete %s\\n\" % relPath)\n+ sys.stdout.write(\"delete %s\\n\" % relPath)\n+ sys.stdout.flush()\n self.gitStream.write(\"D %s\\n\" % relPath)\n \n # handle another chunk of streaming data\n@@ -2281,10 +2284,23 @@ class P4Sync(Command, P4UserMap):\n # 'data' field we need to append to our array\n for k in marshalled.keys():\n if k == 'data':\n+ if 'streamContentSize' not in self.stream_file:\n+ self.stream_file['streamContentSize'] = 0\n+ self.stream_file['streamContentSize'] += len(marshalled['data'])\n self.stream_contents.append(marshalled['data'])\n else:\n self.stream_file[k] = marshalled[k]\n \n+ if (verbose and\n+ 'streamContentSize' in self.stream_file and\n+ 'fileSize' in self.stream_file and\n+ 'depotFile' in self.stream_file):\n+ size = int(self.stream_file[\"fileSize\"])\n+ if size > 0:\n+ progress = 100*self.stream_file['streamContentSize']/size\n+ sys.stdout.write('\\r%s %d%% (%i MB)' % (self.stream_file['depotFile'], progress, int(size/1024/1024)))\n+ sys.stdout.flush()\n+\n self.stream_have_file_info = True\n \n # Stream directly from \"p4 files\" into \"git fast-import\"\n-- \n2.5.1\n"},{"id":"269918","messageId":"1442237194-49624-6-git-send-email-larsxschneider@gmail.com","threadId":"40347","inReplyTo":"1442237194-49624-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v5 5/7] git-p4: check free space during streaming","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-14T13:26:32Z","receivedAt":"2015-09-14T13:26:32Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\ngit-p4 will just halt if there is not enough disk space while\nstreaming content from P4 to Git. Add a check to ensure at least\n4 times (arbitrarily chosen) the size of a streamed file is available.\n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 27 ++++++++++++++++++++++++++-\n 1 file changed, 26 insertions(+), 1 deletion(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex e7a7ea4..b465356 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -104,6 +104,16 @@ def chdir(path, is_client_path=False):\n path = os.getcwd()\n os.environ['PWD'] = path\n \n+def calcDiskFree(dirname):\n+ \"\"\"Return free space in bytes on the disk of the given dirname.\"\"\"\n+ if platform.system() == 'Windows':\n+ free_bytes = ctypes.c_ulonglong(0)\n+ ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(dirname), None, None, ctypes.pointer(free_bytes))\n+ return free_bytes.value\n+ else:\n+ st = os.statvfs(dirname)\n+ return st.f_bavail * st.f_frsize\n+\n def die(msg):\n if verbose:\n raise Exception(msg)\n@@ -2038,6 +2048,14 @@ class P4Sync(Command, P4UserMap):\n self.clientSpecDirs = None\n self.tempBranches = []\n self.tempBranchLocation = \"git-p4-tmp\"\n+ self.largeFileSystem = None\n+ self.cloneDestination = None\n+\n+ if gitConfig('git-p4.largeFileSystem'):\n+ largeFileSystemConstructor = globals()[gitConfig('git-p4.largeFileSystem')]\n+ self.largeFileSystem = largeFileSystemConstructor(\n+ lambda git_mode, relPath, contents: self.writeToGitStream(git_mode, relPath, contents)\n+ )\n \n if gitConfig(\"git-p4.syncFromOrigin\") == \"false\":\n self.syncWithOrigin = False\n@@ -2256,6 +2274,14 @@ class P4Sync(Command, P4UserMap):\n if marshalled[\"code\"] == \"error\":\n if \"data\" in marshalled:\n err = marshalled[\"data\"].rstrip()\n+\n+ if not err and self.cloneDestination and 'fileSize' in self.stream_file:\n+ required_bytes = int((4 * int(self.stream_file[\"fileSize\"])) - calcDiskFree(self.cloneDestination))\n+ if required_bytes > 0:\n+ err = 'Not enough space left on %s! Free at least %i MB.' % (\n+ os.path.abspath(self.cloneDestination), required_bytes/1024/1024\n+ )\n+\n if err:\n f = None\n if self.stream_have_file_info:\n@@ -3218,7 +3244,6 @@ class P4Clone(P4Sync):\n optparse.make_option(\"--bare\", dest=\"cloneBare\",\n action=\"store_true\", default=False),\n ]\n- self.cloneDestination = None\n self.needsGit = False\n self.cloneBare = False\n \n-- \n2.5.1\n"},{"id":"269922","messageId":"1442237194-49624-7-git-send-email-larsxschneider@gmail.com","threadId":"40347","inReplyTo":"1442237194-49624-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v5 6/7] git-p4: add support for large file systems","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-14T13:26:33Z","receivedAt":"2015-09-14T13:26:33Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nPerforce repositories can contain large (binary) files. Migrating these\nrepositories to Git generates very large local clones. External storage\nsystems such as Git LFS [1], Git Fat [2], Git Media [3], git-annex [4]\ntry to address this problem.\n\nAdd a generic mechanism to detect large files based on extension,\nuncompressed size, and/or compressed size.\n\n[1] https://git-lfs.github.com/\n[2] https://github.com/jedbrown/git-fat\n[3] https://github.com/alebedev/git-media\n[4] https://git-annex.branchable.com/\n\nSigned-off-by: Lars Schneider \n---\n Documentation/git-p4.txt | 32 +++++++++++\n git-p4.py | 139 +++++++++++++++++++++++++++++++++++++++++----\n t/t9823-git-p4-mock-lfs.sh | 96 +++++++++++++++++++++++++++++++\n 3 files changed, 257 insertions(+), 10 deletions(-)\n create mode 100755 t/t9823-git-p4-mock-lfs.sh\n\ndiff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\nindex 82aa5d6..3f21e95 100644\n--- a/Documentation/git-p4.txt\n+++ b/Documentation/git-p4.txt\n@@ -510,6 +510,38 @@ git-p4.useClientSpec::\n \toption '--use-client-spec'. See the \"CLIENT SPEC\" section above.\n \tThis variable is a boolean, not the name of a p4 client.\n \n+git-p4.largeFileSystem::\n+\tSpecify the system that is used for large (binary) files. Please note\n+\tthat large file systems do not support the 'git p4 submit' command.\n+\tOnly Git LFS [1] is implemented right now. Download\n+\tand install the Git LFS command line extension to use this option\n+\tand configure it like this:\n++\n+-------------\n+git config git-p4.largeFileSystem GitLFS\n+-------------\n++\n+\t[1] https://git-lfs.github.com/\n+\n+git-p4.largeFileExtensions::\n+\tAll files matching a file extension in the list will be processed\n+\tby the large file system. Do not prefix the extensions with '.'.\n+\n+git-p4.largeFileThreshold::\n+\tAll files with an uncompressed size exceeding the threshold will be\n+\tprocessed by the large file system. By default the threshold is\n+\tdefined in bytes. Add the suffix k, m, or g to change the unit.\n+\n+git-p4.largeFileCompressedThreshold::\n+\tAll files with a compressed size exceeding the threshold will be\n+\tprocessed by the large file system. This option might slow down\n+\tyour clone/sync process. By default the threshold is defined in\n+\tbytes. Add the suffix k, m, or g to change the unit.\n+\n+git-p4.pushLargeFiles::\n+\tBoolean variable which defines if large files are automatically\n+\tpushed to a server.\n+\n Submit variables\n ~~~~~~~~~~~~~~~~\n git-p4.detectRenames::\ndiff --git a/git-p4.py b/git-p4.py\nindex b465356..bfe71b5 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -22,6 +22,8 @@ import platform\n import re\n import shutil\n import stat\n+import zipfile\n+import zlib\n \n try:\n from subprocess import CalledProcessError\n@@ -932,6 +934,111 @@ def wildcard_present(path):\n m = re.search(\"[*#@%]\", path)\n return m is not None\n \n+class LargeFileSystem(object):\n+ \"\"\"Base class for large file system support.\"\"\"\n+\n+ def __init__(self, writeToGitStream):\n+ self.largeFiles = set()\n+ self.writeToGitStream = writeToGitStream\n+\n+ def generatePointer(self, cloneDestination, contentFile):\n+ \"\"\"Return the content of a pointer file that is stored in Git instead of\n+ the actual content.\"\"\"\n+ assert False, \"Method 'generatePointer' required in \" + self.__class__.__name__\n+\n+ def pushFile(self, localLargeFile):\n+ \"\"\"Push the actual content which is not stored in the Git repository to\n+ a server.\"\"\"\n+ assert False, \"Method 'pushFile' required in \" + self.__class__.__name__\n+\n+ def hasLargeFileExtension(self, relPath):\n+ return reduce(\n+ lambda a, b: a or b,\n+ [relPath.endswith('.' + e) for e in gitConfigList('git-p4.largeFileExtensions')],\n+ False\n+ )\n+\n+ def generateTempFile(self, contents):\n+ contentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n+ for d in contents:\n+ contentFile.write(d)\n+ contentFile.flush()\n+ contentFile.close()\n+ return contentFile.name\n+\n+ def exceedsLargeFileThreshold(self, relPath, contents):\n+ if gitConfigInt('git-p4.largeFileThreshold'):\n+ contentsSize = sum(len(d) for d in contents)\n+ if contentsSize > gitConfigInt('git-p4.largeFileThreshold'):\n+ return True\n+ if gitConfigInt('git-p4.largeFileCompressedThreshold'):\n+ contentsSize = sum(len(d) for d in contents)\n+ if contentsSize <= gitConfigInt('git-p4.largeFileCompressedThreshold'):\n+ return False\n+ contentTempFile = self.generateTempFile(contents)\n+ compressedContentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n+ zf = zipfile.ZipFile(compressedContentFile.name, mode='w')\n+ zf.write(contentTempFile, compress_type=zipfile.ZIP_DEFLATED)\n+ zf.close()\n+ compressedContentsSize = zf.infolist()[0].compress_size\n+ os.remove(contentTempFile)\n+ os.remove(compressedContentFile.name)\n+ if compressedContentsSize > gitConfigInt('git-p4.largeFileCompressedThreshold'):\n+ return True\n+ return False\n+\n+ def addLargeFile(self, relPath):\n+ self.largeFiles.add(relPath)\n+\n+ def removeLargeFile(self, relPath):\n+ self.largeFiles.remove(relPath)\n+\n+ def isLargeFile(self, relPath):\n+ return relPath in self.largeFiles\n+\n+ def processContent(self, cloneDestination, git_mode, relPath, contents):\n+ \"\"\"Processes the content of git fast import. This method decides if a\n+ file is stored in the large file system and handles all necessary\n+ steps.\"\"\"\n+ if self.exceedsLargeFileThreshold(relPath, contents) or self.hasLargeFileExtension(relPath):\n+ contentTempFile = self.generateTempFile(contents)\n+ (git_mode, contents, localLargeFile) = self.generatePointer(cloneDestination, contentTempFile)\n+\n+ # Move temp file to final location in large file system\n+ largeFileDir = os.path.dirname(localLargeFile)\n+ if not os.path.isdir(largeFileDir):\n+ os.makedirs(largeFileDir)\n+ shutil.move(contentTempFile, localLargeFile)\n+ self.addLargeFile(relPath)\n+ if gitConfigBool('git-p4.pushLargeFiles'):\n+ self.pushFile(localLargeFile)\n+ if verbose:\n+ sys.stderr.write(\"%s moved to large file system (%s)\\n\" % (relPath, localLargeFile))\n+ return (git_mode, contents)\n+\n+class MockLFS(LargeFileSystem):\n+ \"\"\"Mock large file system for testing.\"\"\"\n+\n+ def generatePointer(self, cloneDestination, contentFile):\n+ \"\"\"The pointer content is the original content prefixed with \"pointer-\".\n+ The local filename of the large file storage is derived from the file content.\n+ \"\"\"\n+ with open(contentFile, 'r') as f:\n+ content = next(f)\n+ gitMode = '100644'\n+ pointerContents = 'pointer-' + content\n+ localLargeFile = os.path.join(cloneDestination, '.git', 'mock-storage', 'local', content[:-1])\n+ return (gitMode, pointerContents, localLargeFile)\n+\n+ def pushFile(self, localLargeFile):\n+ \"\"\"The remote filename of the large file storage is the same as the local\n+ one but in a different directory.\n+ \"\"\"\n+ remotePath = os.path.join(os.path.dirname(localLargeFile), '..', 'remote')\n+ if not os.path.exists(remotePath):\n+ os.makedirs(remotePath)\n+ shutil.copyfile(localLargeFile, os.path.join(remotePath, os.path.basename(localLargeFile)))\n+\n class Command:\n def __init__(self):\n self.usage = \"usage: %prog [options]\"\n@@ -1105,6 +1212,9 @@ class P4Submit(Command, P4UserMap):\n self.p4HasMoveCommand = p4_has_move_command()\n self.branch = None\n \n+ if gitConfig('git-p4.largeFileSystem'):\n+ die(\"Large file system not supported for git-p4 submit command. Please remove it from config.\")\n+\n def check(self):\n if len(p4CmdList(\"opened ...\")) > 0:\n die(\"You have files opened with perforce! Close them before starting the sync.\")\n@@ -2057,6 +2167,12 @@ class P4Sync(Command, P4UserMap):\n lambda git_mode, relPath, contents: self.writeToGitStream(git_mode, relPath, contents)\n )\n \n+ if gitConfig('git-p4.largeFileSystem'):\n+ largeFileSystemConstructor = globals()[gitConfig('git-p4.largeFileSystem')]\n+ self.largeFileSystem = largeFileSystemConstructor(\n+ lambda git_mode, relPath, contents: self.writeToGitStream(git_mode, relPath, contents)\n+ )\n+\n if gitConfig(\"git-p4.syncFromOrigin\") == \"false\":\n self.syncWithOrigin = False\n \n@@ -2176,6 +2292,13 @@ class P4Sync(Command, P4UserMap):\n \n return branches\n \n+ def writeToGitStream(self, gitMode, relPath, contents):\n+ self.gitStream.write('M %s inline %s\\n' % (gitMode, relPath))\n+ self.gitStream.write('data %d\\n' % sum(len(d) for d in contents))\n+ for d in contents:\n+ self.gitStream.write(d)\n+ self.gitStream.write('\\n')\n+\n # output one file from the P4 stream\n # - helper for streamP4Files\n \n@@ -2246,17 +2369,10 @@ class P4Sync(Command, P4UserMap):\n text = regexp.sub(r'$\\1$', text)\n contents = [ text ]\n \n- self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n+ if self.largeFileSystem:\n+ (git_mode, contents) = self.largeFileSystem.processContent(self.cloneDestination, git_mode, relPath, contents)\n \n- # total length...\n- length = 0\n- for d in contents:\n- length = length + len(d)\n-\n- self.gitStream.write(\"data %d\\n\" % length)\n- for d in contents:\n- self.gitStream.write(d)\n- self.gitStream.write(\"\\n\")\n+ self.writeToGitStream(git_mode, relPath, contents)\n \n def streamOneP4Deletion(self, file):\n relPath = self.stripRepoPath(file['path'], self.branchPrefixes)\n@@ -2265,6 +2381,9 @@ class P4Sync(Command, P4UserMap):\n sys.stdout.flush()\n self.gitStream.write(\"D %s\\n\" % relPath)\n \n+ if self.largeFileSystem and self.largeFileSystem.isLargeFile(relPath):\n+ self.largeFileSystem.removeLargeFile(relPath)\n+\n # handle another chunk of streaming data\n def streamP4FilesCb(self, marshalled):\n \ndiff --git a/t/t9823-git-p4-mock-lfs.sh b/t/t9823-git-p4-mock-lfs.sh\nnew file mode 100755\nindex 0000000..8582857\n--- /dev/null\n+++ b/t/t9823-git-p4-mock-lfs.sh\n@@ -0,0 +1,96 @@\n+#!/bin/sh\n+\n+test_description='Clone repositories and store files in Mock LFS'\n+\n+. ./lib-git-p4.sh\n+\n+test_file_in_mock () {\n+\tFILE=\"$1\"\n+\tCONTENT=\"$2\"\n+\tLOCAL_STORAGE=\".git/mock-storage/local/$CONTENT\"\n+\tSERVER_STORAGE=\".git/mock-storage/remote/$CONTENT\"\n+\techo \"pointer-$CONTENT\" >expect_pointer\n+\techo \"$CONTENT\" >expect_content\n+\ttest_path_is_file \"$FILE\" &&\n+\ttest_path_is_file \"$LOCAL_STORAGE\" &&\n+\ttest_path_is_file \"$SERVER_STORAGE\" &&\n+\ttest_cmp expect_pointer \"$FILE\" &&\n+\ttest_cmp expect_content \"$LOCAL_STORAGE\" &&\n+\ttest_cmp expect_content \"$SERVER_STORAGE\"\n+}\n+\n+test_file_count_in_dir () {\n+\tDIR=\"$1\"\n+\tEXPECTED_COUNT=\"$2\"\n+\tfind \"$DIR\" -type f >actual\n+\ttest_line_count = $EXPECTED_COUNT actual\n+}\n+\n+test_expect_success 'start p4d' '\n+\tstart_p4d\n+'\n+\n+test_expect_success 'Create repo with binary files' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\n+\t\techo \"content 1 txt 23 bytes\" >file1.txt &&\n+\t\tp4 add file1.txt &&\n+\t\techo \"content 2-3 bin 25 bytes\" >file2.dat &&\n+\t\tp4 add file2.dat &&\n+\t\tp4 submit -d \"Add text and binary file\" &&\n+\n+\t\tmkdir \"path with spaces\" &&\n+\t\techo \"content 2-3 bin 25 bytes\" >\"path with spaces/file3.bin\" &&\n+\t\tp4 add \"path with spaces/file3.bin\" &&\n+\t\tp4 submit -d \"Add another binary file with same content and spaces in path\" &&\n+\n+\t\techo \"content 4 bin 26 bytes XX\" >file4.bin &&\n+\t\tp4 add file4.bin &&\n+\t\tp4 submit -d \"Add another binary file with different content\"\n+\t)\n+'\n+\n+test_expect_success 'Store files in Mock based on size (>24 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem MockLFS &&\n+\t\tgit config git-p4.largeFileThreshold 24 &&\n+\t\tgit config git-p4.pushLargeFiles True &&\n+\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_mock file2.dat \"content 2-3 bin 25 bytes\" &&\n+\t\ttest_file_in_mock \"path with spaces/file3.bin\" \"content 2-3 bin 25 bytes\" &&\n+\t\ttest_file_in_mock file4.bin \"content 4 bin 26 bytes XX\" &&\n+\n+\t\ttest_file_count_in_dir \".git/mock-storage/local\" 2 &&\n+\t\ttest_file_count_in_dir \".git/mock-storage/remote\" 2\n+\t)\n+'\n+\n+test_expect_success 'Run git p4 submit in repo configured with large file system' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem MockLFS &&\n+\t\tgit config git-p4.largeFileThreshold 24 &&\n+\t\tgit config git-p4.pushLargeFiles True &&\n+\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_must_fail git p4 submit\n+\t)\n+'\n+\n+test_expect_success 'kill p4d' '\n+\tkill_p4d\n+'\n+\n+test_done\n-- \n2.5.1\n"},{"id":"269920","messageId":"1442237194-49624-8-git-send-email-larsxschneider@gmail.com","threadId":"40347","inReplyTo":"1442237194-49624-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v5 7/7] git-p4: add Git LFS backend for large file system","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-14T13:26:34Z","receivedAt":"2015-09-14T13:26:34Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nAdd example implementation including test cases for the large file\nsystem using Git LFS.\n\nPushing files to the Git LFS server is not tested.\n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 72 ++++++++++++\n t/t9824-git-p4-git-lfs.sh | 288 ++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 360 insertions(+)\n create mode 100755 t/t9824-git-p4-git-lfs.sh\n\ndiff --git a/git-p4.py b/git-p4.py\nindex bfe71b5..1f2ee97 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -1039,6 +1039,78 @@ class MockLFS(LargeFileSystem):\n os.makedirs(remotePath)\n shutil.copyfile(localLargeFile, os.path.join(remotePath, os.path.basename(localLargeFile)))\n \n+class GitLFS(LargeFileSystem):\n+ \"\"\"Git LFS as backend for the git-p4 large file system.\n+ See https://git-lfs.github.com/ for details.\"\"\"\n+\n+ def __init__(self, *args):\n+ LargeFileSystem.__init__(self, *args)\n+ self.baseGitAttributes = []\n+\n+ def generatePointer(self, cloneDestination, contentFile):\n+ \"\"\"Generate a Git LFS pointer for the content. Return LFS Pointer file\n+ mode and content which is stored in the Git repository instead of\n+ the actual content. Return also the new location of the actual\n+ content.\n+ \"\"\"\n+ pointerProcess = subprocess.Popen(\n+ ['git', 'lfs', 'pointer', '--file=' + contentFile],\n+ stdout=subprocess.PIPE\n+ )\n+ pointerFile = pointerProcess.stdout.read()\n+ if pointerProcess.wait():\n+ os.remove(contentFile)\n+ die('git-lfs pointer command failed. Did you install the extension?')\n+ pointerContents = [i+'\\n' for i in pointerFile.split('\\n')[2:][:-1]]\n+ oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]\n+ localLargeFile = os.path.join(\n+ cloneDestination,\n+ '.git', 'lfs', 'objects', oid[:2], oid[2:4],\n+ oid,\n+ )\n+ # LFS Spec states that pointer files should not have the executable bit set.\n+ gitMode = '100644'\n+ return (gitMode, pointerContents, localLargeFile)\n+\n+ def pushFile(self, localLargeFile):\n+ uploadProcess = subprocess.Popen(\n+ ['git', 'lfs', 'push', '--object-id', 'origin', os.path.basename(localLargeFile)]\n+ )\n+ if uploadProcess.wait():\n+ die('git-lfs push command failed. Did you define a remote?')\n+\n+ def generateGitAttributes(self):\n+ return (\n+ self.baseGitAttributes +\n+ [\n+ '\\n',\n+ '#\\n',\n+ '# Git LFS (see https://git-lfs.github.com/)\\n',\n+ '#\\n',\n+ ] +\n+ ['*.' + f.replace(' ', '[:space:]') + ' filter=lfs -text\\n'\n+ for f in sorted(gitConfigList('git-p4.largeFileExtensions'))\n+ ] +\n+ ['/' + f.replace(' ', '[:space:]') + ' filter=lfs -text\\n'\n+ for f in sorted(self.largeFiles) if not self.hasLargeFileExtension(f)\n+ ]\n+ )\n+\n+ def addLargeFile(self, relPath):\n+ LargeFileSystem.addLargeFile(self, relPath)\n+ self.writeToGitStream('100644', '.gitattributes', self.generateGitAttributes())\n+\n+ def removeLargeFile(self, relPath):\n+ LargeFileSystem.removeLargeFile(self, relPath)\n+ self.writeToGitStream('100644', '.gitattributes', self.generateGitAttributes())\n+\n+ def processContent(self, cloneDestination, git_mode, relPath, contents):\n+ if relPath == '.gitattributes':\n+ self.baseGitAttributes = contents\n+ return (git_mode, self.generateGitAttributes())\n+ else:\n+ return LargeFileSystem.processContent(self, cloneDestination, git_mode, relPath, contents)\n+\n class Command:\n def __init__(self):\n self.usage = \"usage: %prog [options]\"\ndiff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh\nnew file mode 100755\nindex 0000000..bf34efa\n--- /dev/null\n+++ b/t/t9824-git-p4-git-lfs.sh\n@@ -0,0 +1,288 @@\n+#!/bin/sh\n+\n+test_description='Clone repositories and store files in Git LFS'\n+\n+. ./lib-git-p4.sh\n+\n+git lfs help >/dev/null 2>&1 || {\n+\tskip_all='skipping git p4 Git LFS tests; Git LFS not found'\n+\ttest_done\n+}\n+\n+test_file_in_lfs () {\n+\tFILE=\"$1\"\n+\tSIZE=\"$2\"\n+\tEXPECTED_CONTENT=\"$3\"\n+\tcat \"$FILE\" | grep \"size $SIZE\"\n+\tHASH=$(cat \"$FILE\" | grep \"oid sha256:\" | sed -e 's/oid sha256://g')\n+\tLFS_FILE=\".git/lfs/objects/${HASH:0:2}/${HASH:2:2}/$HASH\"\n+\techo $EXPECTED_CONTENT >expect\n+\ttest_path_is_file \"$FILE\" &&\n+\ttest_path_is_file \"$LFS_FILE\" &&\n+\ttest_cmp expect \"$LFS_FILE\"\n+}\n+\n+test_file_count_in_dir () {\n+\tDIR=\"$1\"\n+\tEXPECTED_COUNT=\"$2\"\n+\tfind \"$DIR\" -type f >actual\n+\ttest_line_count = $EXPECTED_COUNT actual\n+}\n+\n+test_expect_success 'start p4d' '\n+\tstart_p4d\n+'\n+\n+test_expect_success 'Create repo with binary files' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\n+\t\techo \"content 1 txt 23 bytes\" >file1.txt &&\n+\t\tp4 add file1.txt &&\n+\t\techo \"content 2-3 bin 25 bytes\" >file2.dat &&\n+\t\tp4 add file2.dat &&\n+\t\tp4 submit -d \"Add text and binary file\" &&\n+\n+\t\tmkdir \"path with spaces\" &&\n+\t\techo \"content 2-3 bin 25 bytes\" >\"path with spaces/file3.bin\" &&\n+\t\tp4 add \"path with spaces/file3.bin\" &&\n+\t\tp4 submit -d \"Add another binary file with same content and spaces in path\" &&\n+\n+\t\techo \"content 4 bin 26 bytes XX\" >file4.bin &&\n+\t\tp4 add file4.bin &&\n+\t\tp4 submit -d \"Add another binary file with different content\"\n+\t)\n+'\n+\n+test_expect_success 'Store files in LFS based on size (>24 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileThreshold 24 &&\n+\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 25 \"content 2-3 bin 25 bytes\" &&\n+\t\ttest_file_in_lfs \"path with spaces/file3.bin\" 25 \"content 2-3 bin 25 bytes\" &&\n+\t\ttest_file_in_lfs file4.bin 26 \"content 4 bin 26 bytes XX\" &&\n+\n+\t\ttest_file_count_in_dir \".git/lfs/objects\" 2 &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file2.dat filter=lfs -text\n+\t\t/file4.bin filter=lfs -text\n+\t\t/path[:space:]with[:space:]spaces/file3.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Store files in LFS based on size (>25 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileThreshold 25 &&\n+\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file4.bin 26 \"content 4 bin 26 bytes XX\" &&\n+\t\ttest_file_count_in_dir \".git/lfs/objects\" 1 &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file4.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Store files in LFS based on extension (dat)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileExtensions dat &&\n+\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 25 \"content 2-3 bin 25 bytes\" &&\n+\t\ttest_file_count_in_dir \".git/lfs/objects\" 1 &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t*.dat filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Store files in LFS based on size (>25 bytes) and extension (dat)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileExtensions dat &&\n+\t\tgit config git-p4.largeFileThreshold 25 &&\n+\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 25 \"content 2-3 bin 25 bytes\" &&\n+\t\ttest_file_in_lfs file4.bin 26 \"content 4 bin 26 bytes XX\" &&\n+\t\ttest_file_count_in_dir \".git/lfs/objects\" 2 &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t*.dat filter=lfs -text\n+\t\t/file4.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Remove file from repo and store files in LFS based on size (>24 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\t\tp4 delete file4.bin &&\n+\t\tp4 submit -d \"Remove file\"\n+\t) &&\n+\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileThreshold 24 &&\n+\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 25 \"content 2-3 bin 25 bytes\" &&\n+\t\ttest_file_in_lfs \"path with spaces/file3.bin\" 25 \"content 2-3 bin 25 bytes\" &&\n+\t\t! test_path_is_file file4.bin &&\n+\t\ttest_file_count_in_dir \".git/lfs/objects\" 2 &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file2.dat filter=lfs -text\n+\t\t/path[:space:]with[:space:]spaces/file3.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Add .gitattributes and store files in LFS based on size (>24 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\t\techo \"*.txt text\" >.gitattributes &&\n+\t\tp4 add .gitattributes &&\n+\t\tp4 submit -d \"Add .gitattributes\"\n+\t) &&\n+\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileThreshold 24 &&\n+\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 25 \"content 2-3 bin 25 bytes\" &&\n+\t\ttest_file_in_lfs \"path with spaces/file3.bin\" 25 \"content 2-3 bin 25 bytes\" &&\n+\t\t! test_path_is_file file4.bin &&\n+\t\ttest_file_count_in_dir \".git/lfs/objects\" 2 &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\t*.txt text\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file2.dat filter=lfs -text\n+\t\t/path[:space:]with[:space:]spaces/file3.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Add big files to repo and store files in LFS based on compressed size (>28 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\t\techo \"content 5 bin 40 bytes XXXXXXXXXXXXXXXX\" >file5.bin &&\n+\t\tp4 add file5.bin &&\n+\t\tp4 submit -d \"Add file with small footprint after compression\" &&\n+\n+\t\techo \"content 6 bin 39 bytes XXXXXYYYYYZZZZZ\" >file6.bin &&\n+\t\tp4 add file6.bin &&\n+\t\tp4 submit -d \"Add file with large footprint after compression\"\n+\t) &&\n+\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.useClientSpec true &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileCompressedThreshold 28 &&\n+\t\t# We only import HEAD here (\"@all\" is missing!)\n+\t\tgit p4 clone --destination=\"$git\" //depot &&\n+\n+\t\ttest_file_in_lfs file6.bin 13 \"content 6 bin 39 bytes XXXXXYYYYYZZZZZ\"\n+\t\ttest_file_count_in_dir \".git/lfs/objects\" 1 &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\t*.txt text\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file6.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'kill p4d' '\n+\tkill_p4d\n+'\n+\n+test_done\n-- \n2.5.1\n"},{"id":"269988","messageId":"55F7CA0B.2010609@diamand.org","threadId":"40347","inReplyTo":"1442237194-49624-2-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH v5 1/7] git-p4: add optional type specifier to gitConfig reader","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2015-09-15T07:34:35Z","receivedAt":"2015-09-15T07:34:35Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On 14/09/15 14:26, larsxschneider@gmail.com wrote:\n> From: Lars Schneider \n>\n> The functions “gitConfig” and “gitConfigBool” are almost identical. Make “gitConfig” more generic by adding an optional type specifier. Use the type specifier “—bool” with “gitConfig” to implement “gitConfigBool. This prepares the implementation of other type specifiers such as “—int”.\n\nLooks good to me, Ack.\n\n\n>\n> Signed-off-by: Lars Schneider \n> ---\n> git-p4.py | 12 ++++++------\n> 1 file changed, 6 insertions(+), 6 deletions(-)\n>\n> diff --git a/git-p4.py b/git-p4.py\n> index 073f87b..c139cab 100755\n> --- a/git-p4.py\n> +++ b/git-p4.py\n> @@ -604,9 +604,12 @@ def gitBranchExists(branch):\n>\n> _gitConfig = {}\n>\n> -def gitConfig(key):\n> +def gitConfig(key, typeSpecifier=None):\n> if not _gitConfig.has_key(key):\n> - cmd = [ \"git\", \"config\", key ]\n> + cmd = [ \"git\", \"config\" ]\n> + if typeSpecifier:\n> + cmd += [ typeSpecifier ]\n> + cmd += [ key ]\n> s = read_pipe(cmd, ignore_error=True)\n> _gitConfig[key] = s.strip()\n> return _gitConfig[key]\n> @@ -617,10 +620,7 @@ def gitConfigBool(key):\n> in the config.\"\"\"\n>\n> if not _gitConfig.has_key(key):\n> - cmd = [ \"git\", \"config\", \"--bool\", key ]\n> - s = read_pipe(cmd, ignore_error=True)\n> - v = s.strip()\n> - _gitConfig[key] = v == \"true\"\n> + _gitConfig[key] = gitConfig(key, '--bool') == \"true\"\n> return _gitConfig[key]\n>\n> def gitConfigList(key):\n>\n"},{"id":"270104","messageId":"55F92A1E.1090002@diamand.org","threadId":"40347","inReplyTo":"1442237194-49624-7-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH v5 6/7] git-p4: add support for large file systems","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2015-09-16T08:36:46Z","receivedAt":"2015-09-16T08:36:46Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On 14/09/15 14:26, larsxschneider@gmail.com wrote:\n> From: Lars Schneider \n>\n> Perforce repositories can contain large (binary) files. Migrating these\n> repositories to Git generates very large local clones. External storage\n> systems such as Git LFS [1], Git Fat [2], Git Media [3], git-annex [4]\n> try to address this problem.\n>\n> Add a generic mechanism to detect large files based on extension,\n> uncompressed size, and/or compressed size.\n\nLooks excellent! I've got a small few comments (below) and I need to \ncome back and have another look through if that's OK.\n\nThanks!\nLuke\n\n>\n> [1] https://git-lfs.github.com/\n> [2] https://github.com/jedbrown/git-fat\n> [3] https://github.com/alebedev/git-media\n> [4] https://git-annex.branchable.com/\n>\n> Signed-off-by: Lars Schneider \n> ---\n> Documentation/git-p4.txt | 32 +++++++++++\n> git-p4.py | 139 +++++++++++++++++++++++++++++++++++++++++----\n> t/t9823-git-p4-mock-lfs.sh | 96 +++++++++++++++++++++++++++++++\n> 3 files changed, 257 insertions(+), 10 deletions(-)\n> create mode 100755 t/t9823-git-p4-mock-lfs.sh\n>\n> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\n> index 82aa5d6..3f21e95 100644\n> --- a/Documentation/git-p4.txt\n> +++ b/Documentation/git-p4.txt\n> @@ -510,6 +510,38 @@ git-p4.useClientSpec::\n> \toption '--use-client-spec'. See the \"CLIENT SPEC\" section above.\n> \tThis variable is a boolean, not the name of a p4 client.\n>\n> +git-p4.largeFileSystem::\n> +\tSpecify the system that is used for large (binary) files. Please note\n> +\tthat large file systems do not support the 'git p4 submit' command.\n\nWhy is that? Is it just that you haven't implemented support, or is it \nfundamentally impossible?\n\n> +\tOnly Git LFS [1] is implemented right now. Download\n> +\tand install the Git LFS command line extension to use this option\n> +\tand configure it like this:\n> ++\n> +-------------\n> +git config git-p4.largeFileSystem GitLFS\n> +-------------\n> ++\n> +\t[1] https://git-lfs.github.com/\n> +\n> +git-p4.largeFileExtensions::\n> +\tAll files matching a file extension in the list will be processed\n> +\tby the large file system. Do not prefix the extensions with '.'.\n> +\n> +git-p4.largeFileThreshold::\n> +\tAll files with an uncompressed size exceeding the threshold will be\n> +\tprocessed by the large file system. By default the threshold is\n> +\tdefined in bytes. Add the suffix k, m, or g to change the unit.\n> +\n> +git-p4.largeFileCompressedThreshold::\n> +\tAll files with a compressed size exceeding the threshold will be\n> +\tprocessed by the large file system. This option might slow down\n> +\tyour clone/sync process. By default the threshold is defined in\n> +\tbytes. Add the suffix k, m, or g to change the unit.\n> +\n> +git-p4.pushLargeFiles::\n> +\tBoolean variable which defines if large files are automatically\n> +\tpushed to a server.\n> +\n> Submit variables\n> ~~~~~~~~~~~~~~~~\n> git-p4.detectRenames::\n> diff --git a/git-p4.py b/git-p4.py\n> index b465356..bfe71b5 100755\n> --- a/git-p4.py\n> +++ b/git-p4.py\n> @@ -22,6 +22,8 @@ import platform\n> import re\n> import shutil\n> import stat\n> +import zipfile\n> +import zlib\n>\n> try:\n> from subprocess import CalledProcessError\n> @@ -932,6 +934,111 @@ def wildcard_present(path):\n> m = re.search(\"[*#@%]\", path)\n> return m is not None\n>\n> +class LargeFileSystem(object):\n> + \"\"\"Base class for large file system support.\"\"\"\n> +\n> + def __init__(self, writeToGitStream):\n> + self.largeFiles = set()\n> + self.writeToGitStream = writeToGitStream\n> +\n> + def generatePointer(self, cloneDestination, contentFile):\n> + \"\"\"Return the content of a pointer file that is stored in Git instead of\n> + the actual content.\"\"\"\n> + assert False, \"Method 'generatePointer' required in \" + self.__class__.__name__\n> +\n> + def pushFile(self, localLargeFile):\n> + \"\"\"Push the actual content which is not stored in the Git repository to\n> + a server.\"\"\"\n> + assert False, \"Method 'pushFile' required in \" + self.__class__.__name__\n> +\n> + def hasLargeFileExtension(self, relPath):\n> + return reduce(\n> + lambda a, b: a or b,\n> + [relPath.endswith('.' + e) for e in gitConfigList('git-p4.largeFileExtensions')],\n> + False\n> + )\n> +\n> + def generateTempFile(self, contents):\n> + contentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n> + for d in contents:\n> + contentFile.write(d)\n> + contentFile.flush()\n> + contentFile.close()\n\nclose() should flush() automatically I would have thought.\n\n> + return contentFile.name\n> +\n> + def exceedsLargeFileThreshold(self, relPath, contents):\n> + if gitConfigInt('git-p4.largeFileThreshold'):\n> + contentsSize = sum(len(d) for d in contents)\n> + if contentsSize > gitConfigInt('git-p4.largeFileThreshold'):\n> + return True\n> + if gitConfigInt('git-p4.largeFileCompressedThreshold'):\n> + contentsSize = sum(len(d) for d in contents)\n> + if contentsSize <= gitConfigInt('git-p4.largeFileCompressedThreshold'):\n> + return False\n> + contentTempFile = self.generateTempFile(contents)\n> + compressedContentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n> + zf = zipfile.ZipFile(compressedContentFile.name, mode='w')\n> + zf.write(contentTempFile, compress_type=zipfile.ZIP_DEFLATED)\n> + zf.close()\n> + compressedContentsSize = zf.infolist()[0].compress_size\n> + os.remove(contentTempFile)\n> + os.remove(compressedContentFile.name)\n> + if compressedContentsSize > gitConfigInt('git-p4.largeFileCompressedThreshold'):\n> + return True\n> + return False\n> +\n> + def addLargeFile(self, relPath):\n> + self.largeFiles.add(relPath)\n> +\n> + def removeLargeFile(self, relPath):\n> + self.largeFiles.remove(relPath)\n> +\n> + def isLargeFile(self, relPath):\n> + return relPath in self.largeFiles\n> +\n> + def processContent(self, cloneDestination, git_mode, relPath, contents):\n> + \"\"\"Processes the content of git fast import. This method decides if a\n> + file is stored in the large file system and handles all necessary\n> + steps.\"\"\"\n> + if self.exceedsLargeFileThreshold(relPath, contents) or self.hasLargeFileExtension(relPath):\n> + contentTempFile = self.generateTempFile(contents)\n> + (git_mode, contents, localLargeFile) = self.generatePointer(cloneDestination, contentTempFile)\n> +\n> + # Move temp file to final location in large file system\n> + largeFileDir = os.path.dirname(localLargeFile)\n> + if not os.path.isdir(largeFileDir):\n> + os.makedirs(largeFileDir)\n> + shutil.move(contentTempFile, localLargeFile)\n> + self.addLargeFile(relPath)\n> + if gitConfigBool('git-p4.pushLargeFiles'):\n> + self.pushFile(localLargeFile)\n> + if verbose:\n> + sys.stderr.write(\"%s moved to large file system (%s)\\n\" % (relPath, localLargeFile))\n> + return (git_mode, contents)\n> +\n> +class MockLFS(LargeFileSystem):\n> + \"\"\"Mock large file system for testing.\"\"\"\n> +\n> + def generatePointer(self, cloneDestination, contentFile):\n> + \"\"\"The pointer content is the original content prefixed with \"pointer-\".\n> + The local filename of the large file storage is derived from the file content.\n> + \"\"\"\n> + with open(contentFile, 'r') as f:\n> + content = next(f)\n> + gitMode = '100644'\n> + pointerContents = 'pointer-' + content\n> + localLargeFile = os.path.join(cloneDestination, '.git', 'mock-storage', 'local', content[:-1])\n> + return (gitMode, pointerContents, localLargeFile)\n> +\n> + def pushFile(self, localLargeFile):\n> + \"\"\"The remote filename of the large file storage is the same as the local\n> + one but in a different directory.\n> + \"\"\"\n> + remotePath = os.path.join(os.path.dirname(localLargeFile), '..', 'remote')\n> + if not os.path.exists(remotePath):\n> + os.makedirs(remotePath)\n> + shutil.copyfile(localLargeFile, os.path.join(remotePath, os.path.basename(localLargeFile)))\n> +\n> class Command:\n> def __init__(self):\n> self.usage = \"usage: %prog [options]\"\n> @@ -1105,6 +1212,9 @@ class P4Submit(Command, P4UserMap):\n> self.p4HasMoveCommand = p4_has_move_command()\n> self.branch = None\n>\n> + if gitConfig('git-p4.largeFileSystem'):\n> + die(\"Large file system not supported for git-p4 submit command. Please remove it from config.\")\n> +\n> def check(self):\n> if len(p4CmdList(\"opened ...\")) > 0:\n> die(\"You have files opened with perforce! Close them before starting the sync.\")\n> @@ -2057,6 +2167,12 @@ class P4Sync(Command, P4UserMap):\n> lambda git_mode, relPath, contents: self.writeToGitStream(git_mode, relPath, contents)\n> )\n>\n> + if gitConfig('git-p4.largeFileSystem'):\n> + largeFileSystemConstructor = globals()[gitConfig('git-p4.largeFileSystem')]\n> + self.largeFileSystem = largeFileSystemConstructor(\n> + lambda git_mode, relPath, contents: self.writeToGitStream(git_mode, relPath, contents)\n> + )\n> +\n> if gitConfig(\"git-p4.syncFromOrigin\") == \"false\":\n> self.syncWithOrigin = False\n>\n> @@ -2176,6 +2292,13 @@ class P4Sync(Command, P4UserMap):\n>\n> return branches\n>\n> + def writeToGitStream(self, gitMode, relPath, contents):\n> + self.gitStream.write('M %s inline %s\\n' % (gitMode, relPath))\n> + self.gitStream.write('data %d\\n' % sum(len(d) for d in contents))\n> + for d in contents:\n> + self.gitStream.write(d)\n> + self.gitStream.write('\\n')\n> +\n> # output one file from the P4 stream\n> # - helper for streamP4Files\n>\n> @@ -2246,17 +2369,10 @@ class P4Sync(Command, P4UserMap):\n> text = regexp.sub(r'$\\1$', text)\n> contents = [ text ]\n>\n> - self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n> + if self.largeFileSystem:\n> + (git_mode, contents) = self.largeFileSystem.processContent(self.cloneDestination, git_mode, relPath, contents)\n>\n> - # total length...\n> - length = 0\n> - for d in contents:\n> - length = length + len(d)\n> -\n> - self.gitStream.write(\"data %d\\n\" % length)\n> - for d in contents:\n> - self.gitStream.write(d)\n> - self.gitStream.write(\"\\n\")\n> + self.writeToGitStream(git_mode, relPath, contents)\n>\n> def streamOneP4Deletion(self, file):\n> relPath = self.stripRepoPath(file['path'], self.branchPrefixes)\n> @@ -2265,6 +2381,9 @@ class P4Sync(Command, P4UserMap):\n> sys.stdout.flush()\n> self.gitStream.write(\"D %s\\n\" % relPath)\n>\n> + if self.largeFileSystem and self.largeFileSystem.isLargeFile(relPath):\n> + self.largeFileSystem.removeLargeFile(relPath)\n> +\n> # handle another chunk of streaming data\n> def streamP4FilesCb(self, marshalled):\n>\n> diff --git a/t/t9823-git-p4-mock-lfs.sh b/t/t9823-git-p4-mock-lfs.sh\n> new file mode 100755\n> index 0000000..8582857\n> --- /dev/null\n> +++ b/t/t9823-git-p4-mock-lfs.sh\n> @@ -0,0 +1,96 @@\n> +#!/bin/sh\n> +\n> +test_description='Clone repositories and store files in Mock LFS'\n> +\n> +. ./lib-git-p4.sh\n> +\n> +test_file_in_mock () {\n> +\tFILE=\"$1\"\nMissing &&\nPlus the next few lines\n\n> +\tCONTENT=\"$2\"\n> +\tLOCAL_STORAGE=\".git/mock-storage/local/$CONTENT\"\n> +\tSERVER_STORAGE=\".git/mock-storage/remote/$CONTENT\"\n> +\techo \"pointer-$CONTENT\" >expect_pointer\n> +\techo \"$CONTENT\" >expect_content\n> +\ttest_path_is_file \"$FILE\" &&\n> +\ttest_path_is_file \"$LOCAL_STORAGE\" &&\n> +\ttest_path_is_file \"$SERVER_STORAGE\" &&\n> +\ttest_cmp expect_pointer \"$FILE\" &&\n> +\ttest_cmp expect_content \"$LOCAL_STORAGE\" &&\n> +\ttest_cmp expect_content \"$SERVER_STORAGE\"\n> +}\n> +\n> +test_file_count_in_dir () {\n> +\tDIR=\"$1\"\n> +\tEXPECTED_COUNT=\"$2\"\n> +\tfind \"$DIR\" -type f >actual\n> +\ttest_line_count = $EXPECTED_COUNT actual\n> +}\n> +\n> +test_expect_success 'start p4d' '\n> +\tstart_p4d\n> +'\n> +\n> +test_expect_success 'Create repo with binary files' '\n> +\tclient_view \"//depot/... //client/...\" &&\n> +\t(\n> +\t\tcd \"$cli\" &&\n> +\n> +\t\techo \"content 1 txt 23 bytes\" >file1.txt &&\n> +\t\tp4 add file1.txt &&\n> +\t\techo \"content 2-3 bin 25 bytes\" >file2.dat &&\n> +\t\tp4 add file2.dat &&\n> +\t\tp4 submit -d \"Add text and binary file\" &&\n> +\n> +\t\tmkdir \"path with spaces\" &&\n> +\t\techo \"content 2-3 bin 25 bytes\" >\"path with spaces/file3.bin\" &&\n\nNice putting spaces in the path!\n\n> +\t\tp4 add \"path with spaces/file3.bin\" &&\n> +\t\tp4 submit -d \"Add another binary file with same content and spaces in path\" &&\n> +\n> +\t\techo \"content 4 bin 26 bytes XX\" >file4.bin &&\n> +\t\tp4 add file4.bin &&\n> +\t\tp4 submit -d \"Add another binary file with different content\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'Store files in Mock based on size (>24 bytes)' '\n> +\tclient_view \"//depot/... //client/...\" &&\n> +\ttest_when_finished cleanup_git &&\n> +\t(\n> +\t\tcd \"$git\" &&\n> +\t\tgit init . &&\n> +\t\tgit config git-p4.useClientSpec true &&\n> +\t\tgit config git-p4.largeFileSystem MockLFS &&\n> +\t\tgit config git-p4.largeFileThreshold 24 &&\n> +\t\tgit config git-p4.pushLargeFiles True &&\n> +\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n> +\n> +\t\ttest_file_in_mock file2.dat \"content 2-3 bin 25 bytes\" &&\n> +\t\ttest_file_in_mock \"path with spaces/file3.bin\" \"content 2-3 bin 25 bytes\" &&\n> +\t\ttest_file_in_mock file4.bin \"content 4 bin 26 bytes XX\" &&\n> +\n> +\t\ttest_file_count_in_dir \".git/mock-storage/local\" 2 &&\n> +\t\ttest_file_count_in_dir \".git/mock-storage/remote\" 2\n> +\t)\n> +'\n> +\n> +test_expect_success 'Run git p4 submit in repo configured with large file system' '\n> +\tclient_view \"//depot/... //client/...\" &&\n> +\ttest_when_finished cleanup_git &&\n> +\t(\n> +\t\tcd \"$git\" &&\n> +\t\tgit init . &&\n> +\t\tgit config git-p4.useClientSpec true &&\n> +\t\tgit config git-p4.largeFileSystem MockLFS &&\n> +\t\tgit config git-p4.largeFileThreshold 24 &&\n> +\t\tgit config git-p4.pushLargeFiles True &&\n> +\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n> +\n> +\t\ttest_must_fail git p4 submit\n\nNo test for deletion, does that matter?\n\n\n\n> +\t)\n> +'\n> +\n> +test_expect_success 'kill p4d' '\n> +\tkill_p4d\n> +'\n> +\n> +test_done\n>\n"},{"id":"270120","messageId":"5329966D-1A0C-42A1-9099-AC449D50AA52@gmail.com","threadId":"40347","inReplyTo":"55F92A1E.1090002@diamand.org","subject":"Re: [PATCH v5 6/7] git-p4: add support for large file systems","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-16T12:05:40Z","receivedAt":"2015-09-16T12:05:40Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 16 Sep 2015, at 10:36, Luke Diamand wrote:\n\n> On 14/09/15 14:26, larsxschneider@gmail.com wrote:\n>> From: Lars Schneider \n>> \n>> Perforce repositories can contain large (binary) files. Migrating these\n>> repositories to Git generates very large local clones. External storage\n>> systems such as Git LFS [1], Git Fat [2], Git Media [3], git-annex [4]\n>> try to address this problem.\n>> \n>> Add a generic mechanism to detect large files based on extension,\n>> uncompressed size, and/or compressed size.\n> \n> Looks excellent! I've got a small few comments (below) and I need to come back and have another look through if that's OK.\nSure! Thank you :-)\n\n> \n> Thanks!\n> Luke\n> \n>> \n>> [1] https://git-lfs.github.com/\n>> [2] https://github.com/jedbrown/git-fat\n>> [3] https://github.com/alebedev/git-media\n>> [4] https://git-annex.branchable.com/\n>> \n>> Signed-off-by: Lars Schneider \n>> ---\n>> Documentation/git-p4.txt | 32 +++++++++++\n>> git-p4.py | 139 +++++++++++++++++++++++++++++++++++++++++----\n>> t/t9823-git-p4-mock-lfs.sh | 96 +++++++++++++++++++++++++++++++\n>> 3 files changed, 257 insertions(+), 10 deletions(-)\n>> create mode 100755 t/t9823-git-p4-mock-lfs.sh\n>> \n>> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\n>> index 82aa5d6..3f21e95 100644\n>> --- a/Documentation/git-p4.txt\n>> +++ b/Documentation/git-p4.txt\n>> @@ -510,6 +510,38 @@ git-p4.useClientSpec::\n>> \toption '--use-client-spec'. See the \"CLIENT SPEC\" section above.\n>> \tThis variable is a boolean, not the name of a p4 client.\n>> \n>> +git-p4.largeFileSystem::\n>> +\tSpecify the system that is used for large (binary) files. Please note\n>> +\tthat large file systems do not support the 'git p4 submit' command.\n> \n> Why is that? Is it just that you haven't implemented support, or is it fundamentally impossible?\n\nIf we detect LFS files only by file extension then we could make it work. But then we must not use any git-p4 settings. We would need to rely only on the “.gitattributes” file that is stored in the P4 repository. My implementation also looks at the file size and decides on a individual file basis if a file is stored in LFS. That means all clients need the same file size threshold. \n\nJunio explained the problem in the v4 thread:\n> How is collaboration between those who talk to the same p4 depot\n> backed by GitHub LFS expected to work? You use config to set size\n> limits and list of file extensions in your repository, and grab new\n> changes from p4 and turn them into Git commits (with pointers to LFS\n> and the .gitattributes file that records your choice of the config).\n> I as a new member to the same project come, clone the resulting Git\n> repository from you and then what do I do before talking to the same\n> p4 depot to further update the Git history? Are the values recorded\n> in .gitattributes (which essentially were derived from your\n> configuration) somehow reflected back automatically to my config so\n> that our world view would become consistent? Perhaps you added\n> 'iso' to largeFileExtensions before I cloned from you, and that\n> would be present in the copy of .gitattributes I obtained from you.\n> I may be trying to add a new \".iso\" file, and I presume that an\n> existing .gitattributes entry you created based on the file\n> extension automatically covers it from the LFS side, but does my\n> \"git p4 submit\" also know what to do, or would it be broken until I\n> add a set of configrations that matches yours?\n \n\n\n> \n>> +\tOnly Git LFS [1] is implemented right now. Download\n>> +\tand install the Git LFS command line extension to use this option\n>> +\tand configure it like this:\n>> ++\n>> +-------------\n>> +git config git-p4.largeFileSystem GitLFS\n>> +-------------\n>> ++\n>> +\t[1] https://git-lfs.github.com/\n>> +\n>> +git-p4.largeFileExtensions::\n>> +\tAll files matching a file extension in the list will be processed\n>> +\tby the large file system. Do not prefix the extensions with '.'.\n>> +\n>> +git-p4.largeFileThreshold::\n>> +\tAll files with an uncompressed size exceeding the threshold will be\n>> +\tprocessed by the large file system. By default the threshold is\n>> +\tdefined in bytes. Add the suffix k, m, or g to change the unit.\n>> +\n>> +git-p4.largeFileCompressedThreshold::\n>> +\tAll files with a compressed size exceeding the threshold will be\n>> +\tprocessed by the large file system. This option might slow down\n>> +\tyour clone/sync process. By default the threshold is defined in\n>> +\tbytes. Add the suffix k, m, or g to change the unit.\n>> +\n>> +git-p4.pushLargeFiles::\n>> +\tBoolean variable which defines if large files are automatically\n>> +\tpushed to a server.\n>> +\n>> Submit variables\n>> ~~~~~~~~~~~~~~~~\n>> git-p4.detectRenames::\n>> diff --git a/git-p4.py b/git-p4.py\n>> index b465356..bfe71b5 100755\n>> --- a/git-p4.py\n>> +++ b/git-p4.py\n>> @@ -22,6 +22,8 @@ import platform\n>> import re\n>> import shutil\n>> import stat\n>> +import zipfile\n>> +import zlib\n>> \n>> try:\n>> from subprocess import CalledProcessError\n>> @@ -932,6 +934,111 @@ def wildcard_present(path):\n>> m = re.search(\"[*#@%]\", path)\n>> return m is not None\n>> \n>> +class LargeFileSystem(object):\n>> + \"\"\"Base class for large file system support.\"\"\"\n>> +\n>> + def __init__(self, writeToGitStream):\n>> + self.largeFiles = set()\n>> + self.writeToGitStream = writeToGitStream\n>> +\n>> + def generatePointer(self, cloneDestination, contentFile):\n>> + \"\"\"Return the content of a pointer file that is stored in Git instead of\n>> + the actual content.\"\"\"\n>> + assert False, \"Method 'generatePointer' required in \" + self.__class__.__name__\n>> +\n>> + def pushFile(self, localLargeFile):\n>> + \"\"\"Push the actual content which is not stored in the Git repository to\n>> + a server.\"\"\"\n>> + assert False, \"Method 'pushFile' required in \" + self.__class__.__name__\n>> +\n>> + def hasLargeFileExtension(self, relPath):\n>> + return reduce(\n>> + lambda a, b: a or b,\n>> + [relPath.endswith('.' + e) for e in gitConfigList('git-p4.largeFileExtensions')],\n>> + False\n>> + )\n>> +\n>> + def generateTempFile(self, contents):\n>> + contentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n>> + for d in contents:\n>> + contentFile.write(d)\n>> + contentFile.flush()\n>> + contentFile.close()\n> \n> close() should flush() automatically I would have thought.\nCorrect. SO thinks so, too https://stackoverflow.com/questions/2447143/does-close-imply-flush-in-python\n\n> \n>> + return contentFile.name\n>> +\n>> + def exceedsLargeFileThreshold(self, relPath, contents):\n>> + if gitConfigInt('git-p4.largeFileThreshold'):\n>> + contentsSize = sum(len(d) for d in contents)\n>> + if contentsSize > gitConfigInt('git-p4.largeFileThreshold'):\n>> + return True\n>> + if gitConfigInt('git-p4.largeFileCompressedThreshold'):\n>> + contentsSize = sum(len(d) for d in contents)\n>> + if contentsSize <= gitConfigInt('git-p4.largeFileCompressedThreshold'):\n>> + return False\n>> + contentTempFile = self.generateTempFile(contents)\n>> + compressedContentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n>> + zf = zipfile.ZipFile(compressedContentFile.name, mode='w')\n>> + zf.write(contentTempFile, compress_type=zipfile.ZIP_DEFLATED)\n>> + zf.close()\n>> + compressedContentsSize = zf.infolist()[0].compress_size\n>> + os.remove(contentTempFile)\n>> + os.remove(compressedContentFile.name)\n>> + if compressedContentsSize > gitConfigInt('git-p4.largeFileCompressedThreshold'):\n>> + return True\n>> + return False\n>> +\n>> + def addLargeFile(self, relPath):\n>> + self.largeFiles.add(relPath)\n>> +\n>> + def removeLargeFile(self, relPath):\n>> + self.largeFiles.remove(relPath)\n>> +\n>> + def isLargeFile(self, relPath):\n>> + return relPath in self.largeFiles\n>> +\n>> + def processContent(self, cloneDestination, git_mode, relPath, contents):\n>> + \"\"\"Processes the content of git fast import. This method decides if a\n>> + file is stored in the large file system and handles all necessary\n>> + steps.\"\"\"\n>> + if self.exceedsLargeFileThreshold(relPath, contents) or self.hasLargeFileExtension(relPath):\n>> + contentTempFile = self.generateTempFile(contents)\n>> + (git_mode, contents, localLargeFile) = self.generatePointer(cloneDestination, contentTempFile)\n>> +\n>> + # Move temp file to final location in large file system\n>> + largeFileDir = os.path.dirname(localLargeFile)\n>> + if not os.path.isdir(largeFileDir):\n>> + os.makedirs(largeFileDir)\n>> + shutil.move(contentTempFile, localLargeFile)\n>> + self.addLargeFile(relPath)\n>> + if gitConfigBool('git-p4.pushLargeFiles'):\n>> + self.pushFile(localLargeFile)\n>> + if verbose:\n>> + sys.stderr.write(\"%s moved to large file system (%s)\\n\" % (relPath, localLargeFile))\n>> + return (git_mode, contents)\n>> +\n>> +class MockLFS(LargeFileSystem):\n>> + \"\"\"Mock large file system for testing.\"\"\"\n>> +\n>> + def generatePointer(self, cloneDestination, contentFile):\n>> + \"\"\"The pointer content is the original content prefixed with \"pointer-\".\n>> + The local filename of the large file storage is derived from the file content.\n>> + \"\"\"\n>> + with open(contentFile, 'r') as f:\n>> + content = next(f)\n>> + gitMode = '100644'\n>> + pointerContents = 'pointer-' + content\n>> + localLargeFile = os.path.join(cloneDestination, '.git', 'mock-storage', 'local', content[:-1])\n>> + return (gitMode, pointerContents, localLargeFile)\n>> +\n>> + def pushFile(self, localLargeFile):\n>> + \"\"\"The remote filename of the large file storage is the same as the local\n>> + one but in a different directory.\n>> + \"\"\"\n>> + remotePath = os.path.join(os.path.dirname(localLargeFile), '..', 'remote')\n>> + if not os.path.exists(remotePath):\n>> + os.makedirs(remotePath)\n>> + shutil.copyfile(localLargeFile, os.path.join(remotePath, os.path.basename(localLargeFile)))\n>> +\n>> class Command:\n>> def __init__(self):\n>> self.usage = \"usage: %prog [options]\"\n>> @@ -1105,6 +1212,9 @@ class P4Submit(Command, P4UserMap):\n>> self.p4HasMoveCommand = p4_has_move_command()\n>> self.branch = None\n>> \n>> + if gitConfig('git-p4.largeFileSystem'):\n>> + die(\"Large file system not supported for git-p4 submit command. Please remove it from config.\")\n>> +\n>> def check(self):\n>> if len(p4CmdList(\"opened ...\")) > 0:\n>> die(\"You have files opened with perforce! Close them before starting the sync.\")\n>> @@ -2057,6 +2167,12 @@ class P4Sync(Command, P4UserMap):\n>> lambda git_mode, relPath, contents: self.writeToGitStream(git_mode, relPath, contents)\n>> )\n>> \n>> + if gitConfig('git-p4.largeFileSystem'):\n>> + largeFileSystemConstructor = globals()[gitConfig('git-p4.largeFileSystem')]\n>> + self.largeFileSystem = largeFileSystemConstructor(\n>> + lambda git_mode, relPath, contents: self.writeToGitStream(git_mode, relPath, contents)\n>> + )\n>> +\n>> if gitConfig(\"git-p4.syncFromOrigin\") == \"false\":\n>> self.syncWithOrigin = False\n>> \n>> @@ -2176,6 +2292,13 @@ class P4Sync(Command, P4UserMap):\n>> \n>> return branches\n>> \n>> + def writeToGitStream(self, gitMode, relPath, contents):\n>> + self.gitStream.write('M %s inline %s\\n' % (gitMode, relPath))\n>> + self.gitStream.write('data %d\\n' % sum(len(d) for d in contents))\n>> + for d in contents:\n>> + self.gitStream.write(d)\n>> + self.gitStream.write('\\n')\n>> +\n>> # output one file from the P4 stream\n>> # - helper for streamP4Files\n>> \n>> @@ -2246,17 +2369,10 @@ class P4Sync(Command, P4UserMap):\n>> text = regexp.sub(r'$\\1$', text)\n>> contents = [ text ]\n>> \n>> - self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n>> + if self.largeFileSystem:\n>> + (git_mode, contents) = self.largeFileSystem.processContent(self.cloneDestination, git_mode, relPath, contents)\n>> \n>> - # total length...\n>> - length = 0\n>> - for d in contents:\n>> - length = length + len(d)\n>> -\n>> - self.gitStream.write(\"data %d\\n\" % length)\n>> - for d in contents:\n>> - self.gitStream.write(d)\n>> - self.gitStream.write(\"\\n\")\n>> + self.writeToGitStream(git_mode, relPath, contents)\n>> \n>> def streamOneP4Deletion(self, file):\n>> relPath = self.stripRepoPath(file['path'], self.branchPrefixes)\n>> @@ -2265,6 +2381,9 @@ class P4Sync(Command, P4UserMap):\n>> sys.stdout.flush()\n>> self.gitStream.write(\"D %s\\n\" % relPath)\n>> \n>> + if self.largeFileSystem and self.largeFileSystem.isLargeFile(relPath):\n>> + self.largeFileSystem.removeLargeFile(relPath)\n>> +\n>> # handle another chunk of streaming data\n>> def streamP4FilesCb(self, marshalled):\n>> \n>> diff --git a/t/t9823-git-p4-mock-lfs.sh b/t/t9823-git-p4-mock-lfs.sh\n>> new file mode 100755\n>> index 0000000..8582857\n>> --- /dev/null\n>> +++ b/t/t9823-git-p4-mock-lfs.sh\n>> @@ -0,0 +1,96 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='Clone repositories and store files in Mock LFS'\n>> +\n>> +. ./lib-git-p4.sh\n>> +\n>> +test_file_in_mock () {\n>> +\tFILE=\"$1\"\n> Missing &&\n> Plus the next few lines\nAre they strictly necessary? I believe you can define variables all in “one line”. I found it like that in existing tests:\nhttps://github.com/git/git/blob/f4d9753a89bf04011c00e943d85211906e86a0f6/t/t9402-git-cvsserver-refs.sh#L18-L20\n\nNevertheless I will add them in the next role!\n\n> \n>> +\tCONTENT=\"$2\"\n>> +\tLOCAL_STORAGE=\".git/mock-storage/local/$CONTENT\"\n>> +\tSERVER_STORAGE=\".git/mock-storage/remote/$CONTENT\"\n>> +\techo \"pointer-$CONTENT\" >expect_pointer\n>> +\techo \"$CONTENT\" >expect_content\n>> +\ttest_path_is_file \"$FILE\" &&\n>> +\ttest_path_is_file \"$LOCAL_STORAGE\" &&\n>> +\ttest_path_is_file \"$SERVER_STORAGE\" &&\n>> +\ttest_cmp expect_pointer \"$FILE\" &&\n>> +\ttest_cmp expect_content \"$LOCAL_STORAGE\" &&\n>> +\ttest_cmp expect_content \"$SERVER_STORAGE\"\n>> +}\n>> +\n>> +test_file_count_in_dir () {\n>> +\tDIR=\"$1\"\n>> +\tEXPECTED_COUNT=\"$2\"\n>> +\tfind \"$DIR\" -type f >actual\n>> +\ttest_line_count = $EXPECTED_COUNT actual\n>> +}\n>> +\n>> +test_expect_success 'start p4d' '\n>> +\tstart_p4d\n>> +'\n>> +\n>> +test_expect_success 'Create repo with binary files' '\n>> +\tclient_view \"//depot/... //client/...\" &&\n>> +\t(\n>> +\t\tcd \"$cli\" &&\n>> +\n>> +\t\techo \"content 1 txt 23 bytes\" >file1.txt &&\n>> +\t\tp4 add file1.txt &&\n>> +\t\techo \"content 2-3 bin 25 bytes\" >file2.dat &&\n>> +\t\tp4 add file2.dat &&\n>> +\t\tp4 submit -d \"Add text and binary file\" &&\n>> +\n>> +\t\tmkdir \"path with spaces\" &&\n>> +\t\techo \"content 2-3 bin 25 bytes\" >\"path with spaces/file3.bin\" &&\n> \n> Nice putting spaces in the path!\nI stumbled across this case in a production migration :-)\n\n> \n>> +\t\tp4 add \"path with spaces/file3.bin\" &&\n>> +\t\tp4 submit -d \"Add another binary file with same content and spaces in path\" &&\n>> +\n>> +\t\techo \"content 4 bin 26 bytes XX\" >file4.bin &&\n>> +\t\tp4 add file4.bin &&\n>> +\t\tp4 submit -d \"Add another binary file with different content\"\n>> +\t)\n>> +'\n>> +\n>> +test_expect_success 'Store files in Mock based on size (>24 bytes)' '\n>> +\tclient_view \"//depot/... //client/...\" &&\n>> +\ttest_when_finished cleanup_git &&\n>> +\t(\n>> +\t\tcd \"$git\" &&\n>> +\t\tgit init . &&\n>> +\t\tgit config git-p4.useClientSpec true &&\n>> +\t\tgit config git-p4.largeFileSystem MockLFS &&\n>> +\t\tgit config git-p4.largeFileThreshold 24 &&\n>> +\t\tgit config git-p4.pushLargeFiles True &&\n>> +\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n>> +\n>> +\t\ttest_file_in_mock file2.dat \"content 2-3 bin 25 bytes\" &&\n>> +\t\ttest_file_in_mock \"path with spaces/file3.bin\" \"content 2-3 bin 25 bytes\" &&\n>> +\t\ttest_file_in_mock file4.bin \"content 4 bin 26 bytes XX\" &&\n>> +\n>> +\t\ttest_file_count_in_dir \".git/mock-storage/local\" 2 &&\n>> +\t\ttest_file_count_in_dir \".git/mock-storage/remote\" 2\n>> +\t)\n>> +'\n>> +\n>> +test_expect_success 'Run git p4 submit in repo configured with large file system' '\n>> +\tclient_view \"//depot/... //client/...\" &&\n>> +\ttest_when_finished cleanup_git &&\n>> +\t(\n>> +\t\tcd \"$git\" &&\n>> +\t\tgit init . &&\n>> +\t\tgit config git-p4.useClientSpec true &&\n>> +\t\tgit config git-p4.largeFileSystem MockLFS &&\n>> +\t\tgit config git-p4.largeFileThreshold 24 &&\n>> +\t\tgit config git-p4.pushLargeFiles True &&\n>> +\t\tgit p4 clone --destination=\"$git\" //depot@all &&\n>> +\n>> +\t\ttest_must_fail git p4 submit\n> \n> No test for deletion, does that matter?\nOK, I will add one in the mock, too. I have a test for that in the “t9828-git-p4-lfs.sh” named “Remove file from repo and store files in LFS based on size (>24 bytes)”.\n\nThanks,\nLars"},{"id":"270124","messageId":"20150916145956.GA515@flurp.local","threadId":"40347","inReplyTo":"5329966D-1A0C-42A1-9099-AC449D50AA52@gmail.com","subject":"Re: [PATCH v5 6/7] git-p4: add support for large file systems","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-09-16T14:59:56Z","receivedAt":"2015-09-16T14:59:56Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Sep 16, 2015 at 02:05:40PM +0200, Lars Schneider wrote:\n> On 16 Sep 2015, at 10:36, Luke Diamand wrote:\n> > On 14/09/15 14:26, larsxschneider@gmail.com wrote:\n> >> +test_file_in_mock () {\n> >> +\tFILE=\"$1\"\n> > Missing &&\n> > Plus the next few lines\n> Are they strictly necessary? I believe you can define variables all\n> in ?one line?. I found it like that in existing tests:\n\nThe reason for keeping the &&-chain intact even on these variable\nassignment lines is to future-proof it against some programmer coming\nalong and inserting new code above or in between the assignment\nlines, and not realizing that the &&-chain is not intact.\n"},{"id":"270128","messageId":"xmqqzj0ms85n.fsf@gitster.mtv.corp.google.com","threadId":"40347","inReplyTo":"5329966D-1A0C-42A1-9099-AC449D50AA52@gmail.com","subject":"Re: [PATCH v5 6/7] git-p4: add support for large file systems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-16T15:20:20Z","receivedAt":"2015-09-16T15:20:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n>>> +git-p4.largeFileSystem::\n>>> +\tSpecify the system that is used for large (binary) files. Please note\n>>> +\tthat large file systems do not support the 'git p4 submit' command.\n>> \n>> Why is that? Is it just that you haven't implemented support, or\n>> is it fundamentally impossible?\n>\n> If we detect LFS files only by file extension then we could make\n> it work. But then we must not use any git-p4 settings. We would\n> need to rely only on the “.gitattributes” file that is stored in\n> the P4 repository. My implementation also looks at the file size\n> and decides on a individual file basis if a file is stored in\n> LFS. That means all clients need the same file size threshold.\n>\n> Junio explained the problem in the v4 thread:\n>> ...\n\nHmm, I am not sure if Luke's question was answered with the above,\nand I do not think I explained anything, either. I did point out\nthat with _your_ code I didn't see how \"submit\" would not work, but\nthat is quite different from the problem being fundamentally not\nsolvable.\n\n>>> +test_file_in_mock () {\n>>> +\tFILE=\"$1\"\n>> Missing &&\n>> Plus the next few lines\n\n> Are they strictly necessary? I believe you can define variables all in “one line”\n\nAbsolutely. Making multiple assignments as a single statment like\n\n\tX=A Y=B Z=C &&\n ...\n\nis fine, but do not break &&-chain.\n\n>> \n>>> +\tCONTENT=\"$2\"\n>>> +\tLOCAL_STORAGE=\".git/mock-storage/local/$CONTENT\"\n>>> +\tSERVER_STORAGE=\".git/mock-storage/remote/$CONTENT\"\n\n... and to be sure, these 'echo' also must not break &&-chain.\n\n>>> +\techo \"pointer-$CONTENT\" >expect_pointer\n>>> +\techo \"$CONTENT\" >expect_content\n\n>>> +test_file_count_in_dir () {\n>>> +\tDIR=\"$1\"\n>>> +\tEXPECTED_COUNT=\"$2\"\n>>> +\tfind \"$DIR\" -type f >actual\n\n... so are these.\n\n>>> +\ttest_line_count = $EXPECTED_COUNT actual\n>>> +}\n\n\nThanks.\n"},{"id":"270366","messageId":"107A7A4A-1B2C-4280-A66E-A4B3BED13AE0@gmail.com","threadId":"40347","inReplyTo":"xmqqzj0ms85n.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 6/7] git-p4: add support for large file systems","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-20T21:26:24Z","receivedAt":"2015-09-20T21:26:24Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 16 Sep 2015, at 17:20, Junio C Hamano wrote:\n\n> Lars Schneider writes:\n> \n>>>> +git-p4.largeFileSystem::\n>>>> +\tSpecify the system that is used for large (binary) files. Please note\n>>>> +\tthat large file systems do not support the 'git p4 submit' command.\n>>> \n>>> Why is that? Is it just that you haven't implemented support, or\n>>> is it fundamentally impossible?\n>> \n>> If we detect LFS files only by file extension then we could make\n>> it work. But then we must not use any git-p4 settings. We would\n>> need to rely only on the “.gitattributes” file that is stored in\n>> the P4 repository. My implementation also looks at the file size\n>> and decides on a individual file basis if a file is stored in\n>> LFS. That means all clients need the same file size threshold.\n>> \n>> Junio explained the problem in the v4 thread:\n>>> ...\n> \n> Hmm, I am not sure if Luke's question was answered with the above,\n> and I do not think I explained anything, either. I did point out\n> that with _your_ code I didn't see how \"submit\" would not work, but\n> that is quite different from the problem being fundamentally not\n> solvable.\n\nOK, to answer Luke’s question after some thoughts: I think it is possible but I haven’t implemented it.\n\nI see one issue right away:\nSome large file systems depend on gitattributes filters (e.g. Git-LFS). Git-p4 would need to run the filters on clone/sync/submit. Right now this does not happen and the user sees the raw LFS pointer files in the Git repository instead of the content after the initial git-p4 clone (the Git-LFS test cases shows this). This is no problem for a git-p4 one time usage as you usually upload the created Git repo to some Git server. Nevertheless, I was not able to find a quick and easy way to fix this. Anyone an idea?\n\nIf we get these filters working then we could create a new large file system similar to “Git-LFS”. Instead of pointing to large files on a “Git-LFS” server we could point right to their P4 location. That would be pretty neat.\n\n- Lars"}]}