threads / patch / 42078

patch, 2 partsgit-p4: fix Git LFS pointer parsing

Subject: [PATCH v1 0/2] git-p4: fix Git LFS pointer parsing

## tl;dr

13 messages between Apr 19, 2016 and Apr 24, 2016. Diffs are folded; open one to read it.

replies: 12people: 4as markdown or json

larsxschneider@gmail.com· Apr 19, 2016, 20:08 UTC · lore
From: Lars Schneider <larsxschneider@gmail.com>

Travis-CI uses 'brew' to always install the latest available version of Git LFS on the OS X build machines (on Linux the version is sticky). A change in Git LFS 1.2.0 [1] breaks the git-p4 LFS integration [2].

This mini series updates Travis-CI to the latest Git-LFS and Perforce versions on Linux and Mac and makes git-p4 work with the latest Git LFS again.

Thanks, Lars

[1] https://github.com/github/git-lfs/pull/1105 [2] https://travis-ci.org/git/git/jobs/124047942

Lars Schneider (2):
  travis-ci: update Git-LFS and P4 to the latest version
  git-p4: fix Git LFS pointer parsing
 .travis.yml              | 4 ++--
 Documentation/git-p4.txt | 3 ++-
 git-p4.py                | 6 +++---
 3 files changed, 7 insertions(+), 6 deletions(-)

-- 2.5.1

larsxschneider@gmail.com· Apr 19, 2016, 20:08 UTC · re: larsxschneider@gmail.com · lore

[PATCH v1 1/2] travis-ci: update Git-LFS and P4 to the latest version

From: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Show changes to .travis.yml +2 −2
diff --git a/.travis.yml b/.travis.yml
index 78e433b..4acf617 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,8 +22,8 @@ addons:
 env:
   global:
     - DEVELOPER=1
-    - P4_VERSION="15.2"
-    - GIT_LFS_VERSION="1.1.0"
+    - P4_VERSION="16.1"
+    - GIT_LFS_VERSION="1.2.0"
     - DEFAULT_TEST_TARGET=prove
     - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
     - GIT_TEST_OPTS="--verbose --tee"
-- 
2.5.1
larsxschneider@gmail.com· Apr 19, 2016, 20:08 UTC · re: larsxschneider@gmail.com · lore

[PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

From: Lars Schneider <larsxschneider@gmail.com>

Git LFS 1.2.0 removed a line from the output of the 'git lfs pointer' command [1] which broke the parsing of this output. Adjust the parser to the new output and add minimum Git LFS version to the docs.

[1] https://github.com/github/git-lfs/pull/1105
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 Documentation/git-p4.txt | 3 ++-
 git-p4.py                | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)
Show changes to 2 files +5 −5

Documentation/git-p4.txt, git-p4.py

diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt
index 88ba42b..b862cb9 100644
--- a/Documentation/git-p4.txt
+++ b/Documentation/git-p4.txt
@@ -522,7 +522,8 @@ git-p4.largeFileSystem::
 	that large file systems do not support the 'git p4 submit' command.
 	Only Git LFS is implemented right now (see https://git-lfs.github.com/
 	for more information). Download and install the Git LFS command line
-	extension to use this option and configure it like this:
+	extension (minimum version 1.2.0) to use this option and configure it
+	like this:
 +
 -------------
 git config       git-p4.largeFileSystem GitLFS
diff --git a/git-p4.py b/git-p4.py
index 527d44b..d2be574 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1064,8 +1064,8 @@ class GitLFS(LargeFileSystem):
         if pointerProcess.wait():
             os.remove(contentFile)
             die('git-lfs pointer command failed. Did you install the extension?')
-        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
-        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
+        oidEntry = [i for i in pointerFile.split('\n') if i.startswith('oid')]
+        oid = oidEntry[0].split(' ')[1].split(':')[1]
         localLargeFile = os.path.join(
             os.getcwd(),
             '.git', 'lfs', 'objects', oid[:2], oid[2:4],
@@ -1073,7 +1073,7 @@ class GitLFS(LargeFileSystem):
         )
         # LFS Spec states that pointer files should not have the executable bit set.
         gitMode = '100644'
-        return (gitMode, pointerContents, localLargeFile)
+        return (gitMode, pointerFile, localLargeFile)

     def pushFile(self, localLargeFile):
         uploadProcess = subprocess.Popen(
--
2.5.1
Junio C Hamano· Apr 19, 2016, 20:30 UTC · re: larsxschneider@gmail.com · lore

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

larsxschneider@gmail.com writes:
Show 5 quoted lines
> From: Lars Schneider <larsxschneider@gmail.com>
>
> Git LFS 1.2.0 removed a line from the output of the 'git lfs pointer'
> command [1] which broke the parsing of this output. Adjust the parser
> to the new output and add minimum Git LFS version to the docs.

Hmph, adjust to operate with both, or drop the support for the old one?

Show 49 quoted lines
>
> [1] https://github.com/github/git-lfs/pull/1105
>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
>  Documentation/git-p4.txt | 3 ++-
>  git-p4.py                | 6 +++---
>  2 files changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt
> index 88ba42b..b862cb9 100644
> --- a/Documentation/git-p4.txt
> +++ b/Documentation/git-p4.txt
> @@ -522,7 +522,8 @@ git-p4.largeFileSystem::
>  	that large file systems do not support the 'git p4 submit' command.
>  	Only Git LFS is implemented right now (see https://git-lfs.github.com/
>  	for more information). Download and install the Git LFS command line
> -	extension to use this option and configure it like this:
> +	extension (minimum version 1.2.0) to use this option and configure it
> +	like this:
>  +
>  -------------
>  git config       git-p4.largeFileSystem GitLFS
> diff --git a/git-p4.py b/git-p4.py
> index 527d44b..d2be574 100755
> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -1064,8 +1064,8 @@ class GitLFS(LargeFileSystem):
>          if pointerProcess.wait():
>              os.remove(contentFile)
>              die('git-lfs pointer command failed. Did you install the extension?')
> -        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
> -        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
> +        oidEntry = [i for i in pointerFile.split('\n') if i.startswith('oid')]
> +        oid = oidEntry[0].split(' ')[1].split(':')[1]
>          localLargeFile = os.path.join(
>              os.getcwd(),
>              '.git', 'lfs', 'objects', oid[:2], oid[2:4],
> @@ -1073,7 +1073,7 @@ class GitLFS(LargeFileSystem):
>          )
>          # LFS Spec states that pointer files should not have the executable bit set.
>          gitMode = '100644'
> -        return (gitMode, pointerContents, localLargeFile)
> +        return (gitMode, pointerFile, localLargeFile)
>
>      def pushFile(self, localLargeFile):
>          uploadProcess = subprocess.Popen(
> --
> 2.5.1
Lars Schneider· Apr 19, 2016, 20:54 UTC · re: Junio C Hamano · lore

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

Show 12 quoted lines
> On 19 Apr 2016, at 22:30, Junio C Hamano <gitster@pobox.com> wrote:
> 
> larsxschneider@gmail.com writes:
> 
>> From: Lars Schneider <larsxschneider@gmail.com>
>> 
>> Git LFS 1.2.0 removed a line from the output of the 'git lfs pointer'
>> command [1] which broke the parsing of this output. Adjust the parser
>> to the new output and add minimum Git LFS version to the docs.
> 
> Hmph, adjust to operate with both, or drop the support for the old
> one?
I dropped the support for the older version to keep the code as simple as possible (plus it would be cumbersome to test with an outdated Git LFS version). Since it is probably a niche feature I thought that might be acceptable.

Thanks, Lars

Show 49 quoted lines
>> 
>> [1] https://github.com/github/git-lfs/pull/1105
>> 
>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
>> ---
>> Documentation/git-p4.txt | 3 ++-
>> git-p4.py                | 6 +++---
>> 2 files changed, 5 insertions(+), 4 deletions(-)
>> 
>> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt
>> index 88ba42b..b862cb9 100644
>> --- a/Documentation/git-p4.txt
>> +++ b/Documentation/git-p4.txt
>> @@ -522,7 +522,8 @@ git-p4.largeFileSystem::
>>    that large file systems do not support the 'git p4 submit' command.
>>    Only Git LFS is implemented right now (see https://git-lfs.github.com/
>>    for more information). Download and install the Git LFS command line
>> -    extension to use this option and configure it like this:
>> +    extension (minimum version 1.2.0) to use this option and configure it
>> +    like this:
>> +
>> -------------
>> git config       git-p4.largeFileSystem GitLFS
>> diff --git a/git-p4.py b/git-p4.py
>> index 527d44b..d2be574 100755
>> --- a/git-p4.py
>> +++ b/git-p4.py
>> @@ -1064,8 +1064,8 @@ class GitLFS(LargeFileSystem):
>>         if pointerProcess.wait():
>>             os.remove(contentFile)
>>             die('git-lfs pointer command failed. Did you install the extension?')
>> -        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
>> -        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
>> +        oidEntry = [i for i in pointerFile.split('\n') if i.startswith('oid')]
>> +        oid = oidEntry[0].split(' ')[1].split(':')[1]
>>         localLargeFile = os.path.join(
>>             os.getcwd(),
>>             '.git', 'lfs', 'objects', oid[:2], oid[2:4],
>> @@ -1073,7 +1073,7 @@ class GitLFS(LargeFileSystem):
>>         )
>>         # LFS Spec states that pointer files should not have the executable bit set.
>>         gitMode = '100644'
>> -        return (gitMode, pointerContents, localLargeFile)
>> +        return (gitMode, pointerFile, localLargeFile)
>> 
>>     def pushFile(self, localLargeFile):
>>         uploadProcess = subprocess.Popen(
>> --
>> 2.5.1
Junio C Hamano· Apr 19, 2016, 21:04 UTC · re: Lars Schneider · lore

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

Lars Schneider <larsxschneider@gmail.com> writes:
Show 12 quoted lines
>> On 19 Apr 2016, at 22:30, Junio C Hamano <gitster@pobox.com> wrote:
>> 
>> larsxschneider@gmail.com writes:
>> 
>>> From: Lars Schneider <larsxschneider@gmail.com>
>>> 
>>> Git LFS 1.2.0 removed a line from the output of the 'git lfs pointer'
>>> command [1] which broke the parsing of this output. Adjust the parser
>>> to the new output and add minimum Git LFS version to the docs.
>> 
>> Hmph, adjust to operate with both, or drop the support for the old
>> one?
> I dropped the support for the older version to keep the code as
> simple as possible (plus it would be cumbersome to test with an
> outdated Git LFS version). Since it is probably a niche feature I
> thought that might be acceptable.

It is bad enough that clients need to be adjusted at all in the first place, but I would have found it very troubling if the problematic change to LFS thing were made in such a way that it makes backward compatible adjustment on the client code impossible.

But it seems that you could read their output and strip a line that begins with a known substring to make it compatible with both?

"git P4" itself may be niche and using it with the LFS thing may even be more so, but in Git land, traditionally we take the backward compatibility seriously. If it is not too much work, I'd prefer to see this done the right way.

Thanks.
Sebastian Schuberth· Apr 20, 2016, 08:32 UTC · re: Junio C Hamano · lore

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

On Tue, Apr 19, 2016 at 11:04 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 9 quoted lines
>> I dropped the support for the older version to keep the code as
>> simple as possible (plus it would be cumbersome to test with an
>> outdated Git LFS version). Since it is probably a niche feature I
>> thought that might be acceptable.
>
> It is bad enough that clients need to be adjusted at all in the
> first place, but I would have found it very troubling if the
> problematic change to LFS thing were made in such a way that it
> makes backward compatible adjustment on the client code impossible.

If clients rely on output targeted at human consumption it's not surprising that these clients need to be adjusted from time to time. What's troubling is not the change to git-lfs, but the very un-generic way git-p4 is implemented.

-- 
Sebastian Schuberth
Junio C Hamano· Apr 20, 2016, 15:30 UTC · re: Sebastian Schuberth · lore

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

Sebastian Schuberth <sschuberth@gmail.com> writes:
> If clients rely on output targeted at human consumption it's not
> surprising that these clients need to be adjusted from time to time.
> What's troubling is not the change to git-lfs, but the very un-generic
> way git-p4 is implemented.

Sounds like the subcommand they are using is not meant for scripting? What is the kosher way to get at the information they can use that is a supported interface for scripters?

Sebastian Schuberth· Apr 20, 2016, 15:40 UTC · re: Junio C Hamano · lore

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

On Wed, Apr 20, 2016 at 5:30 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 8 quoted lines
>> If clients rely on output targeted at human consumption it's not
>> surprising that these clients need to be adjusted from time to time.
>> What's troubling is not the change to git-lfs, but the very un-generic
>> way git-p4 is implemented.
>
> Sounds like the subcommand they are using is not meant for
> scripting?  What is the kosher way to get at the information they
> can use that is a supported interface for scripters?

The "pointer" subcommand indeed is listed under "Low level commands" by "git lfs" (without any arguments), and as such it probably can be considered for scripting use. However, before my fix in [1] the subcommand was printing both output targeted at humans and output targeted at scripts to stdout. After my fix, only output targeted at script goes to stdout, and output targeted at humans goes to stderr.

[1] https://github.com/github/git-lfs/pull/1105
-- 
Sebastian Schuberth
SZEDER Gábor· Apr 24, 2016, 11:50 UTC · re: larsxschneider@gmail.com · lore

[PATCH] t9824: fix broken &&-chain in a subshell

Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
---

As far as I can tell after eyeballing the test script, this was the only case where the &&-chain was broken.

 t/t9824-git-p4-git-lfs.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to t/t9824-git-p4-git-lfs.sh +1 −1
diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
index 0b664a377c45..64f8d18216d4 100755
--- a/t/t9824-git-p4-git-lfs.sh
+++ b/t/t9824-git-p4-git-lfs.sh
@@ -265,7 +265,7 @@ test_expect_success 'Add big files to repo and store files in LFS based on compr
 		# We only import HEAD here ("@all" is missing!)
 		git p4 clone --destination="$git" //depot &&
 
-		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ"
+		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ" &&
 		test_file_count_in_dir ".git/lfs/objects" 1 &&
 
 		cat >expect <<-\EOF &&
-- 
2.8.1.99.g5d5236f
Lars Schneider· Apr 24, 2016, 16:29 UTC · re: SZEDER Gábor · lore

Re: [PATCH] t9824: fix broken &&-chain in a subshell

On 24 Apr 2016, at 13:50, SZEDER Gábor <szeder@ira.uka.de> wrote:
Show 19 quoted lines
> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
> ---
> 
> As far as I can tell after eyeballing the test script, this was the
> only case where the &&-chain was broken.
> 
> t/t9824-git-p4-git-lfs.sh | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
> index 0b664a377c45..64f8d18216d4 100755
> --- a/t/t9824-git-p4-git-lfs.sh
> +++ b/t/t9824-git-p4-git-lfs.sh
> @@ -265,7 +265,7 @@ test_expect_success 'Add big files to repo and store files in LFS based on compr
> 		# We only import HEAD here ("@all" is missing!)
> 		git p4 clone --destination="$git" //depot &&
> 
> -		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ"
> +		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ" &&

Thanks! I believe I remember some script/flag to detect broken &&-chains. Do I imagine that or can anyone point me to it?

Thanks, Lars

Show 6 quoted lines
> 		test_file_count_in_dir ".git/lfs/objects" 1 &&
> 
> 		cat >expect <<-\EOF &&
> -- 
> 2.8.1.99.g5d5236f
> 
SZEDER Gábor· Apr 24, 2016, 16:37 UTC · re: Lars Schneider · lore

Re: [PATCH] t9824: fix broken &&-chain in a subshell

Quoting Lars Schneider <larsxschneider@gmail.com>:
Show 24 quoted lines
> On 24 Apr 2016, at 13:50, SZEDER Gábor <szeder@ira.uka.de> wrote:
>
>> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
>> ---
>>
>> As far as I can tell after eyeballing the test script, this was the
>> only case where the &&-chain was broken.
>>
>> t/t9824-git-p4-git-lfs.sh | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
>> index 0b664a377c45..64f8d18216d4 100755
>> --- a/t/t9824-git-p4-git-lfs.sh
>> +++ b/t/t9824-git-p4-git-lfs.sh
>> @@ -265,7 +265,7 @@ test_expect_success 'Add big files to repo and  
>> store files in LFS based on compr
>> 		# We only import HEAD here ("@all" is missing!)
>> 		git p4 clone --destination="$git" //depot &&
>>
>> -		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ"
>> +		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ" &&
> Thanks! I believe I remember some script/flag to detect broken &&-chains.
> Do I imagine that or can anyone point me to it?

It was introduced in bb79af9d0970 (t/test-lib: introduce --chain-lint option, 2015-03-20) and is enabled by default. Unfortunately, it can't "see" inside subshells, that's why it couldn't detect this case.

Show 6 quoted lines
>> 		test_file_count_in_dir ".git/lfs/objects" 1 &&
>>
>> 		cat >expect <<-\EOF &&
>> --
>> 2.8.1.99.g5d5236f
>>
Lars Schneider· Apr 24, 2016, 16:50 UTC · re: SZEDER Gábor · lore

Re: [PATCH] t9824: fix broken &&-chain in a subshell

On 24 Apr 2016, at 18:37, SZEDER Gábor <szeder@ira.uka.de> wrote:
Show 31 quoted lines
> 
> Quoting Lars Schneider <larsxschneider@gmail.com>:
> 
>> On 24 Apr 2016, at 13:50, SZEDER Gábor <szeder@ira.uka.de> wrote:
>> 
>>> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
>>> ---
>>> 
>>> As far as I can tell after eyeballing the test script, this was the
>>> only case where the &&-chain was broken.
>>> 
>>> t/t9824-git-p4-git-lfs.sh | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>> 
>>> diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
>>> index 0b664a377c45..64f8d18216d4 100755
>>> --- a/t/t9824-git-p4-git-lfs.sh
>>> +++ b/t/t9824-git-p4-git-lfs.sh
>>> @@ -265,7 +265,7 @@ test_expect_success 'Add big files to repo and
>>> store files in LFS based on compr
>>> 		# We only import HEAD here ("@all" is missing!)
>>> 		git p4 clone --destination="$git" //depot &&
>>> 
>>> -		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ"
>>> +		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ" &&
>> Thanks! I believe I remember some script/flag to detect broken &&-chains.
>> Do I imagine that or can anyone point me to it?
> 
> It was introduced in bb79af9d0970 (t/test-lib: introduce --chain-lint
> option, 2015-03-20) and is enabled by default.  Unfortunately, it
> can't "see" inside subshells, that's why it couldn't detect this case.
That explains it. Thank you!
- Lars

← back to recent threads