{"thread":{"id":"40291","subject":"[PATCH v3 0/5] git-p4: add support for large file systems","startedAt":"2015-09-07T12:21:13Z","lastAt":"2015-09-08T04:25:33Z","messageCount":7,"participants":["larsxschneider@gmail.com","Luke Diamand"],"isPatch":true,"patchVersion":3,"patchTotal":5},"messages":[{"id":"269556","messageId":"1441628478-86503-1-git-send-email-larsxschneider@gmail.com","threadId":"40291","inReplyTo":null,"subject":"[PATCH v3 0/5] git-p4: add support for large file systems","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-07T12:21:13Z","receivedAt":"2015-09-07T12:21:13Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nDiff to v2:\n* add proper commit messages\n* split commits into generic large file system and GitLFS implementation\n* improve docs, mention clean/smduge filter and add example for clean checkout\n* fix spelling\n* add option to push large files to server\n* use ValueError for gitConfigInt exception\n* assert required functions for large file system classes\n* rename LFS to GitLFS\n* add Python docs for GitLFS class\n* move generic code from GitLFS class to git-p4 main code\n* make variable that keeps large files for .gitattributes creation a set() to avoid duplicated entries\n* sort large files in .gitattirubutes to minimize diffs between commits\n\nThanks to Luke and Junio for feedback!\n\nOne thing I don't like about the current implementation is that I don't see a\nway to test the \"git-p4.pushLargeFiles\" config. I could start a git lfs server\nlocally but that seems a bit too much, no?\n\nThanks,\nLars\n\nLars Schneider (5):\n git-p4: add optional type specifier to gitConfig reader\n git-p4: add gitConfigInt reader\n git-p4: return an empty list if a list config has no values\n git-p4: add support for large file systems\n git-p4: add Git LFS backend for large file system\n\n Documentation/git-p4.txt | 28 +++++\n git-p4.py | 187 ++++++++++++++++++++++++++++++---\n t/t9823-git-p4-lfs.sh | 263 +++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 465 insertions(+), 13 deletions(-)\n create mode 100755 t/t9823-git-p4-lfs.sh\n\n--\n2.5.1\n"},{"id":"269559","messageId":"1441628478-86503-2-git-send-email-larsxschneider@gmail.com","threadId":"40291","inReplyTo":"1441628478-86503-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v3 1/5] git-p4: add optional type specifier to gitConfig reader","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-07T12:21:14Z","receivedAt":"2015-09-07T12:21:14Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nThe functions “gitConfig” and “gitConfigBool” are almost identical. Make “gitConfig” more generic by adding an optional type specifier. Use the type specifier “—bool” with “gitConfig” to implement “gitConfigBool. This prepares the implementation of other type specifiers such as “—int”.\n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 073f87b..c139cab 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -604,9 +604,12 @@ def gitBranchExists(branch):\n \n _gitConfig = {}\n \n-def gitConfig(key):\n+def gitConfig(key, typeSpecifier=None):\n if not _gitConfig.has_key(key):\n- cmd = [ \"git\", \"config\", key ]\n+ cmd = [ \"git\", \"config\" ]\n+ if typeSpecifier:\n+ cmd += [ typeSpecifier ]\n+ cmd += [ key ]\n s = read_pipe(cmd, ignore_error=True)\n _gitConfig[key] = s.strip()\n return _gitConfig[key]\n@@ -617,10 +620,7 @@ def gitConfigBool(key):\n in the config.\"\"\"\n \n if not _gitConfig.has_key(key):\n- cmd = [ \"git\", \"config\", \"--bool\", key ]\n- s = read_pipe(cmd, ignore_error=True)\n- v = s.strip()\n- _gitConfig[key] = v == \"true\"\n+ _gitConfig[key] = gitConfig(key, '--bool') == \"true\"\n return _gitConfig[key]\n \n def gitConfigList(key):\n-- \n2.5.1\n"},{"id":"269558","messageId":"1441628478-86503-3-git-send-email-larsxschneider@gmail.com","threadId":"40291","inReplyTo":"1441628478-86503-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v3 2/5] git-p4: add gitConfigInt reader","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-07T12:21:15Z","receivedAt":"2015-09-07T12:21:15Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nAdd a git config reader for integer variables. Please note that the git config implementation automatically supports k, m, and g suffixes.\n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex c139cab..40ad4ae 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -623,6 +623,17 @@ def gitConfigBool(key):\n _gitConfig[key] = gitConfig(key, '--bool') == \"true\"\n return _gitConfig[key]\n \n+def gitConfigInt(key):\n+ if not _gitConfig.has_key(key):\n+ cmd = [ \"git\", \"config\", \"--int\", key ]\n+ s = read_pipe(cmd, ignore_error=True)\n+ v = s.strip()\n+ try:\n+ _gitConfig[key] = int(gitConfig(key, '--int'))\n+ except ValueError:\n+ _gitConfig[key] = None\n+ return _gitConfig[key]\n+\n def gitConfigList(key):\n if not _gitConfig.has_key(key):\n s = read_pipe([\"git\", \"config\", \"--get-all\", key], ignore_error=True)\n-- \n2.5.1\n"},{"id":"269560","messageId":"1441628478-86503-4-git-send-email-larsxschneider@gmail.com","threadId":"40291","inReplyTo":"1441628478-86503-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v3 3/5] git-p4: return an empty list if a list config has no values","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-07T12:21:16Z","receivedAt":"2015-09-07T12:21:16Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 40ad4ae..90d3b90 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -638,6 +638,8 @@ def gitConfigList(key):\n if not _gitConfig.has_key(key):\n s = read_pipe([\"git\", \"config\", \"--get-all\", key], ignore_error=True)\n _gitConfig[key] = s.strip().split(os.linesep)\n+ if _gitConfig[key] == ['']:\n+ _gitConfig[key] = []\n return _gitConfig[key]\n \n def p4BranchesInGit(branchesAreInRemotes=True):\n-- \n2.5.1\n"},{"id":"269557","messageId":"1441628478-86503-5-git-send-email-larsxschneider@gmail.com","threadId":"40291","inReplyTo":"1441628478-86503-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v3 4/5] git-p4: add support for large file systems","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-07T12:21:17Z","receivedAt":"2015-09-07T12:21:17Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nPerforce repositories can contain large (binary) files. Migrating these\nrepositories to Git generates very large local clones. External storage\nsystems such as Git LFS [1], Git Fat [2], or Git Media [2] try to\naddress this problem.\n\nAdd a generic mechanism to detect large files based on extension,\nuncompressed size, and/or compressed size.\n\n[1] https://git-lfs.github.com/\n[2] https://github.com/jedbrown/git-fat\n[3] https://github.com/alebedev/git-media\n\nSigned-off-by: Lars Schneider \n---\n Documentation/git-p4.txt | 26 +++++++++++\n git-p4.py | 114 +++++++++++++++++++++++++++++++++++++++++++----\n 2 files changed, 131 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\nindex 82aa5d6..e0d0239 100644\n--- a/Documentation/git-p4.txt\n+++ b/Documentation/git-p4.txt\n@@ -510,6 +510,32 @@ git-p4.useClientSpec::\n \toption '--use-client-spec'. See the \"CLIENT SPEC\" section above.\n \tThis variable is a boolean, not the name of a p4 client.\n \n+git-p4.largeFileSystem::\n+\tSpecify the system that is used used for large (binary) files.\n+\tPlease note that most of these large file systems depend on the\n+\tGit clean/smudge filters. These filters are not applied through\n+\tgit-p4. You need to create a fresh clone of the repository after\n+\trunning git-p4.\n+\n+git-p4.largeFileExtensions::\n+\tAll files matching a file extension in the list will be processed\n+\tby the large file system. Do not prefix the extensions with '.'.\n+\n+git-p4.largeFileThreshold::\n+\tAll files with an uncompressed size exceeding the threshold will be\n+\tprocessed by the large file system. By default the threshold is\n+\tdefined in bytes. Add the suffix k, m, or g to change the unit.\n+\n+git-p4.largeFileCompressedThreshold::\n+\tAll files with a compressed size exceeding the threshold will be\n+\tprocessed by the large file system. This option might significantly\n+\tslow down your clone/sync process. By default the threshold is\n+\tdefined in bytes. Add the suffix k, m, or g to change the unit.\n+\n+git-p4.pushLargeFiles::\n+\tBoolean variable which defines if large files are automatically\n+\tpushed to a server.\n+\n Submit variables\n ~~~~~~~~~~~~~~~~\n git-p4.detectRenames::\ndiff --git a/git-p4.py b/git-p4.py\nindex 90d3b90..06651a8 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -22,6 +22,8 @@ import platform\n import re\n import shutil\n import stat\n+import zipfile\n+import zlib\n \n try:\n from subprocess import CalledProcessError\n@@ -922,6 +924,17 @@ def wildcard_present(path):\n m = re.search(\"[*#@%]\", path)\n return m is not None\n \n+def largeFileSystem():\n+ try:\n+ largeFileSystem = getattr(sys.modules[__name__], gitConfig('git-p4.largeFileSystem'))\n+ assert(hasattr(largeFileSystem, \"attributeDescription\"))\n+ assert(hasattr(largeFileSystem, \"attributeFilter\"))\n+ assert(hasattr(largeFileSystem, \"generatePointer\"))\n+ assert(hasattr(largeFileSystem, \"pushFile\"))\n+ return largeFileSystem\n+ except AttributeError as e:\n+ die('Large file system not supported: %s' % gitConfig('git-p4.largeFileSystem'))\n+\n class Command:\n def __init__(self):\n self.usage = \"usage: %prog [options]\"\n@@ -2038,6 +2051,7 @@ class P4Sync(Command, P4UserMap):\n self.clientSpecDirs = None\n self.tempBranches = []\n self.tempBranchLocation = \"git-p4-tmp\"\n+ self.largeFiles = set()\n \n if gitConfig(\"git-p4.syncFromOrigin\") == \"false\":\n self.syncWithOrigin = False\n@@ -2158,6 +2172,85 @@ class P4Sync(Command, P4UserMap):\n \n return branches\n \n+ def writeToGitStream(self, gitMode, relPath, contents):\n+ self.gitStream.write('M %s inline %s\\n' % (gitMode, relPath))\n+ self.gitStream.write('data %d\\n' % sum(len(d) for d in contents))\n+ for d in contents:\n+ self.gitStream.write(d)\n+ self.gitStream.write('\\n')\n+\n+ def writeGitAttributesToStream(self):\n+ self.writeToGitStream(\n+ '100644',\n+ '.gitattributes',\n+ [\n+ '\\n',\n+ '#\\n',\n+ '# %s\\n' % largeFileSystem().attributeDescription(),\n+ '#\\n',\n+ ] +\n+ ['*.' + f.replace(' ', '[:space:]') + ' filter=%s -text\\n'\n+ % largeFileSystem().attributeFilter()\n+ for f in sorted(gitConfigList(\"git-p4.largeFileExtensions\"))\n+ ] +\n+ ['/' + f.replace(' ', '[:space:]') + ' filter=%s -text\\n'\n+ % largeFileSystem().attributeFilter()\n+ for f in sorted(self.largeFiles) if not self.hasLargeFileExtension(f)\n+ ]\n+ )\n+\n+ def hasLargeFileExtension(self, relPath):\n+ return reduce(\n+ lambda a, b: a or b,\n+ [relPath.endswith('.' + e) for e in gitConfigList('git-p4.largeFileExtensions')],\n+ False\n+ )\n+\n+ def exceedsLargeFileThreshold(self, relPath, contents):\n+ if gitConfigInt('git-p4.largeFileThreshold'):\n+ contentsSize = sum(len(d) for d in contents)\n+ if contentsSize > gitConfigInt('git-p4.largeFileThreshold'):\n+ return True\n+ if gitConfigInt('git-p4.largeFileCompressedThreshold'):\n+ contentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n+ for d in contents:\n+ contentFile.write(d)\n+ contentFile.flush()\n+ compressedContentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n+ zf = zipfile.ZipFile(compressedContentFile.name, mode='w')\n+ zf.write(contentFile.name, compress_type=zipfile.ZIP_DEFLATED)\n+ zf.close()\n+ compressedContentsSize = zf.infolist()[0].compress_size\n+ os.remove(contentFile.name)\n+ os.remove(compressedContentFile.name)\n+ if compressedContentsSize > gitConfigInt('git-p4.largeFileCompressedThreshold'):\n+ return True\n+ return False\n+\n+ def moveToLargeFileSystem(self, relPath, contents):\n+ # Write P4 content to temp file\n+ contentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)\n+ for d in contents:\n+ contentFile.write(d)\n+ contentFile.flush()\n+ contentFile.close()\n+ (git_mode, contents, largeFileTarget) = \\\n+ largeFileSystem().generatePointer(self.cloneDestination, contentFile.name)\n+ # Move temp file to final location in large file system\n+ largeFileDir = os.path.dirname(largeFileTarget)\n+ if not os.path.isdir(largeFileDir):\n+ os.makedirs(largeFileDir)\n+ shutil.move(contentFile.name, largeFileTarget)\n+ if verbose:\n+ sys.stderr.write(\"%s moved to large file system (%s)\\n\" % (relPath, largeFileTarget))\n+\n+ if gitConfigBool('git-p4.pushLargeFiles'):\n+ largeFileSystem().pushFile(largeFileTarget)\n+\n+ self.largeFiles.add(relPath)\n+ self.writeGitAttributesToStream()\n+ return (git_mode, contents)\n+\n # output one file from the P4 stream\n # - helper for streamP4Files\n \n@@ -2226,17 +2319,16 @@ class P4Sync(Command, P4UserMap):\n text = regexp.sub(r'$\\1$', text)\n contents = [ text ]\n \n- self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n+ if relPath == '.gitattributes':\n+ die('.gitattributes already exists in P4.')\n \n- # total length...\n- length = 0\n- for d in contents:\n- length = length + len(d)\n+ if (gitConfig('git-p4.largeFileSystem') and\n+ ( self.exceedsLargeFileThreshold(relPath, contents) or\n+ self.hasLargeFileExtension(relPath)\n+ )):\n+ (git_mode, contents) = self.moveToLargeFileSystem(relPath, contents)\n \n- self.gitStream.write(\"data %d\\n\" % length)\n- for d in contents:\n- self.gitStream.write(d)\n- self.gitStream.write(\"\\n\")\n+ self.writeToGitStream(git_mode, relPath, contents)\n \n def streamOneP4Deletion(self, file):\n relPath = self.stripRepoPath(file['path'], self.branchPrefixes)\n@@ -2244,6 +2336,10 @@ class P4Sync(Command, P4UserMap):\n sys.stderr.write(\"delete %s\\n\" % relPath)\n self.gitStream.write(\"D %s\\n\" % relPath)\n \n+ if relPath in self.largeFiles:\n+ self.largeFiles.remove(relPath)\n+ self.writeGitAttributesToStream()\n+\n # handle another chunk of streaming data\n def streamP4FilesCb(self, marshalled):\n \n-- \n2.5.1\n"},{"id":"269561","messageId":"1441628478-86503-6-git-send-email-larsxschneider@gmail.com","threadId":"40291","inReplyTo":"1441628478-86503-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v3 5/5] git-p4: add Git LFS backend for large file system","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-07T12:21:18Z","receivedAt":"2015-09-07T12:21:18Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nAdd example implementation including test cases for the large file\nsystem using Git LFS.\n\nPushing files to the Git LFS server is not tested.\n\nSigned-off-by: Lars Schneider \n---\n Documentation/git-p4.txt | 4 +-\n git-p4.py | 52 ++++++++++\n t/t9823-git-p4-lfs.sh | 263 +++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 318 insertions(+), 1 deletion(-)\n create mode 100755 t/t9823-git-p4-lfs.sh\n\ndiff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\nindex e0d0239..3168a64 100644\n--- a/Documentation/git-p4.txt\n+++ b/Documentation/git-p4.txt\n@@ -515,7 +515,9 @@ git-p4.largeFileSystem::\n \tPlease note that most of these large file systems depend on the\n \tGit clean/smudge filters. These filters are not applied through\n \tgit-p4. You need to create a fresh clone of the repository after\n-\trunning git-p4.\n+\trunning git-p4. Only Git LFS [1] is supported right now. Download\n+\tand install the Git LFS command line extension to use this option.\n+\t[1] https://git-lfs.github.com/\n \n git-p4.largeFileExtensions::\n \tAll files matching a file extension in the list will be processed\ndiff --git a/git-p4.py b/git-p4.py\nindex 06651a8..d24d84f 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -935,6 +935,58 @@ def largeFileSystem():\n except AttributeError as e:\n die('Large file system not supported: %s' % gitConfig('git-p4.largeFileSystem'))\n \n+class GitLFS:\n+ \"\"\"Git LFS as backend for the git-p4 large file system.\n+ See https://git-lfs.github.com/ for details.\"\"\"\n+\n+ @staticmethod\n+ def attributeDescription():\n+ \"\"\"Return a description which is used to mark LFS entries in the\n+ .gitattributes file.\"\"\"\n+ return 'Git LFS (see https://git-lfs.github.com/)'\n+\n+ @staticmethod\n+ def attributeFilter():\n+ \"\"\"Return the name of the filter which is used for LFS entries in the\n+ .gitattributes file.\"\"\"\n+ return 'lfs'\n+\n+ @staticmethod\n+ def generatePointer(cloneDestination, contentFile):\n+ \"\"\"Generate a Git LFS pointer for the content. Return LFS Pointer file\n+ mode and content which is stored in the Git repository instead of\n+ the actual content. Return also the new location of the actual\n+ content.\n+ \"\"\"\n+ pointerProcess = subprocess.Popen(\n+ ['git', 'lfs', 'pointer', '--file=' + contentFile],\n+ stdout=subprocess.PIPE\n+ )\n+ pointerFile = pointerProcess.stdout.read()\n+ if pointerProcess.wait():\n+ os.remove(contentFile)\n+ die('git-lfs pointer command failed. Did you install the extension?')\n+ pointerContents = [i+'\\n' for i in pointerFile.split('\\n')[2:][:-1]]\n+ oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]\n+ oidPath = os.path.join(\n+ cloneDestination,\n+ '.git', 'lfs', 'objects', oid[:2], oid[2:4],\n+ oid,\n+ )\n+ # LFS Spec states that pointer files should not have the executable bit set.\n+ gitMode = '100644'\n+ return (gitMode, pointerContents, oidPath)\n+\n+ @staticmethod\n+ def pushFile(contentFile):\n+ \"\"\"Push the actual content which is not stored in the Git repository to\n+ a server.\"\"\"\n+ uploadProcess = subprocess.Popen(\n+ ['git', 'lfs', 'push', '--object-id', 'origin', os.path.basename(contentFile)]\n+ )\n+ if uploadProcess.wait():\n+ die('git-lfs push command failed. Did you define a remote?')\n+\n class Command:\n def __init__(self):\n self.usage = \"usage: %prog [options]\"\ndiff --git a/t/t9823-git-p4-lfs.sh b/t/t9823-git-p4-lfs.sh\nnew file mode 100755\nindex 0000000..4d58cac\n--- /dev/null\n+++ b/t/t9823-git-p4-lfs.sh\n@@ -0,0 +1,263 @@\n+#!/bin/sh\n+\n+test_description='Clone repositories and store files in Git LFS'\n+\n+. ./lib-git-p4.sh\n+\n+git lfs help >/dev/null 2>&1 || {\n+\tskip_all='skipping git p4 Git LFS tests; Git LFS not found'\n+\ttest_done\n+}\n+\n+test_file_in_lfs() {\n+\tFILE=$1\n+\tSIZE=$2\n+\tEXPECTED_CONTENT=$3\n+\ttest_path_is_file $FILE &&\n+\ttest_line_count = 3 $FILE &&\n+\tcat $FILE | grep \"size $SIZE\" &&\n+\tHASH=$(cat $FILE | grep \"oid sha256:\" | sed -e 's/oid sha256://g') &&\n+\tLFS_FILE=\".git/lfs/objects/${HASH:0:2}/${HASH:2:2}/$HASH\"\n+\ttest_path_is_file $LFS_FILE &&\n+\techo $EXPECTED_CONTENT >expect\n+\ttest_cmp expect $LFS_FILE\n+}\n+\n+test_expect_success 'start p4d' '\n+\tstart_p4d\n+'\n+\n+test_expect_success 'Create repo with binary files' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\n+\t\techo \"txt 7b\" >file1.txt &&\n+\t\tp4 add file1.txt &&\n+\t\techo \"bin 13 bytes\" >file2.dat &&\n+\t\tp4 add file2.dat &&\n+\t\tp4 submit -d \"Add text and binary file\" &&\n+\n+\t\tmkdir \"path with spaces\" &&\n+\t\techo \"bin 13 bytes\" >\"path with spaces/file3.bin\" &&\n+\t\tp4 add \"path with spaces/file3.bin\" &&\n+\t\tp4 submit -d \"Add another binary file with same content and spaces in path\" &&\n+\n+\t\techo \"bin 14 bytesx\" >file4.bin &&\n+\t\tp4 add file4.bin &&\n+\t\tp4 submit -d \"Add another binary file with different content\"\n+\t)\n+'\n+\n+test_expect_success 'Store files in LFS based on size (>12 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileThreshold 12 &&\n+\t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 13 \"bin 13 bytes\" &&\n+\t\ttest_file_in_lfs \"path with spaces/file3.bin\" 13 \"bin 13 bytes\" &&\n+\t\ttest_file_in_lfs file4.bin 14 \"bin 14 bytesx\" &&\n+\n+\t\tfind \".git/lfs/objects\" -type f >actual &&\n+\t\ttest_line_count = 2 actual &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file2.dat filter=lfs -text\n+\t\t/path[:space:]with[:space:]spaces/file3.bin filter=lfs -text\n+\t\t/file4.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Store files in LFS based on size (>13 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileThreshold 13 &&\n+\t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file4.bin 14 \"bin 14 bytesx\" &&\n+\n+\t\tfind \".git/lfs/objects\" -type f >actual &&\n+\t\ttest_line_count = 1 actual &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file4.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Store files in LFS based on extension (dat)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileExtensions dat &&\n+\t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 13 \"bin 13 bytes\" &&\n+\n+\t\tfind \".git/lfs/objects\" -type f >actual &&\n+\t\ttest_line_count = 1 actual &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t*.dat filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Store files in LFS based on size (>13 bytes) and extension (dat)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileExtensions dat &&\n+\t\tgit config git-p4.largeFileThreshold 13 &&\n+\t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 13 \"bin 13 bytes\" &&\n+\t\ttest_file_in_lfs file4.bin 14 \"bin 14 bytesx\" &&\n+\n+\t\tfind \".git/lfs/objects\" -type f >actual &&\n+\t\ttest_line_count = 2 actual &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t*.dat filter=lfs -text\n+\t\t/file4.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Remove file from repo and store files in LFS based on size (>12 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\t\tp4 delete file4.bin &&\n+\t\tp4 submit -d \"Remove file\"\n+\t) &&\n+\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileThreshold 12 &&\n+\t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot@all &&\n+\n+\t\ttest_file_in_lfs file2.dat 13 \"bin 13 bytes\" &&\n+\t\ttest_file_in_lfs \"path with spaces/file3.bin\" 13 \"bin 13 bytes\" &&\n+\t\t! test_path_is_file file4.bin &&\n+\n+\t\tfind \".git/lfs/objects\" -type f >actual &&\n+\t\t# 3 is correct here as some commits in the tree still reference the deleted file\n+\t\ttest_line_count = 3 actual\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file2.dat filter=lfs -text\n+\t\t/path[:space:]with[:space:]spaces/file3.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Add big files to repo and store files in LFS based on compressed size (>19 bytes)' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\t\techo \"bin 24 bytesxxxxxxxxxxx\" >file5.bin &&\n+\t\tp4 add file5.bin &&\n+\t\tp4 submit -d \"Add another binary file with different content\" &&\n+\n+\t\techo \"bin 34 bytesxxxxxxxxxxxzzzzzzzzzz\" >file6.bin &&\n+\t\tp4 add file6.bin &&\n+\t\tp4 submit -d \"Add another binary file with different content\"\n+\t) &&\n+\n+\tclient_view \"//depot/... //client/...\" &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.largeFileSystem GitLFS &&\n+\t\tgit config git-p4.largeFileCompressedThreshold 19 &&\n+\t\t# We only import HEAD here (\"@all\" is missing!)\n+\t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot &&\n+\n+\t\ttest_file_in_lfs file6.bin 13 \"bin 34 bytesxxxxxxxxxxxzzzzzzzzzz\" &&\n+\n+\t\tfind \".git/lfs/objects\" -type f >actual &&\n+\t\ttest_line_count = 1 actual\n+\n+\t\tcat >expect <<-\\EOF &&\n+\n+\t\t#\n+\t\t# Git LFS (see https://git-lfs.github.com/)\n+\t\t#\n+\t\t/file6.bin filter=lfs -text\n+\t\tEOF\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_cmp expect .gitattributes\n+\t)\n+'\n+\n+test_expect_success 'Clone repo with existing .gitattributes file' '\n+\tclient_view \"//depot/... //client/...\" &&\n+\t(\n+\t\tcd \"$cli\" &&\n+\n+\t\techo \"*.txt text\" >.gitattributes &&\n+\t\tp4 add .gitattributes &&\n+\t\tp4 submit -d \"Add .gitattributes\"\n+\t) &&\n+\n+\ttest_must_fail git p4 clone --use-client-spec --destination=\"$git\" //depot 2>errs &&\n+\tgrep \".gitattributes already exists in P4.\" errs\n+'\n+\n+test_expect_success 'kill p4d' '\n+\tkill_p4d\n+'\n+\n+test_done\n-- \n2.5.1\n"},{"id":"269588","messageId":"CAE5ih7_9CfGw51Aw8J8d2m=s3Z9sf8LV9p84SynrpCW4k0Vbxg@mail.gmail.com","threadId":"40291","inReplyTo":"1441628478-86503-1-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH v3 0/5] git-p4: add support for large file systems","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2015-09-08T04:25:33Z","receivedAt":"2015-09-08T04:25:33Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On 7 September 2015 at 13:21, wrote:\n> From: Lars Schneider \n>\n> One thing I don't like about the current implementation is that I don't see a\n> way to test the \"git-p4.pushLargeFiles\" config. I could start a git lfs server\n> locally but that seems a bit too much, no?\n\nPerhaps add a trivial large-file derived class that just does enough\nto show that it's all working. Then the final bit of pushing the large\nfiles to LFS would remain forever untested, but it would be small\nenough and simple enough not to be a real problem. If it ever broke,\nit would indicate that either the trivial test class was a bit too\ntrivial, or that the abstraction was not abstract enough and there's\ntoo much common-code in the derived classes.\n\nLuke\n"}]}