threads / discuss / 39925

unexplained behavior/issue with git archive?

Subject: unexplained behavior/issue with git archive?

## tl;dr

3 messages between Jul 23, 2015 and Jul 23, 2015.

replies: 2people: 3as markdown or json

Jan Vales· Jul 23, 2015, 15:23 UTC · lore

hi, i seem to trigger behavior i do not understand with git archive.

I have this little 3 liner (vmdiff.sh): #!/bin/bash git diff --name-status "$2" "$3" > "$1.files" git diff --name-only "$2" "$3" |xargs -d'\n' git archive -o "$1" "$3" --

For testing purpose, lets assume this call: # ./vmdiff.sh latest.zip HEAD^1 HEAD

# cat latest.zip.files | wc -l 149021

# cat latest.zip.files | egrep "^D" | wc -l 159

# mkdir empty; cd empty; unzip latest.zip ; find * | wc -l 1090

My goal is to basically diff (parts of) filesystems against each other and create an archive with all changed files + a file list to know what files were deleted. (I currently do not care about the files permissions+ownership, and it doesnt really matter in the current problem. Also dont ask, why one would store a root-filesystem in git :)

What I do not understand: why does the zip file only contains 1090 files+dirs if the wc -l shows like 150k files and only like 159 were deleted? There should be like 149k files in that archive.

Also only the few files are all from "var" and none from etc or srv where definitely files changed in too! (and show up in latest.zip.files)

Is there a limit of files git archive can process?

lg Jan Vales -- I only read plaintext emails.

Someone @ irc://irc.fsinf.at:6667/tuwien webIRC: https://frost.fsinf.at/iris/

John Keeping· Jul 23, 2015, 15:59 UTC · re: Jan Vales · lore

Re: unexplained behavior/issue with git archive?

On Thu, Jul 23, 2015 at 05:23:49PM +0200, Jan Vales wrote:
Show 35 quoted lines
> i seem to trigger behavior i do not understand with git archive.
> 
> I have this little 3 liner (vmdiff.sh):
> #!/bin/bash
> git diff --name-status "$2" "$3" > "$1.files"
> git diff --name-only "$2" "$3" |xargs -d'\n' git archive -o "$1" "$3" --
> 
> 
> For testing purpose, lets assume this call:
> # ./vmdiff.sh latest.zip HEAD^1 HEAD
> 
> # cat latest.zip.files | wc -l
> 149021
> 
> # cat latest.zip.files | egrep "^D" | wc -l
> 159
> 
> # mkdir empty; cd empty; unzip latest.zip ; find * | wc -l
> 1090
> 
> My goal is to basically diff (parts of) filesystems against each other
> and create an archive with all changed files + a file list to know what
> files were deleted. (I currently do not care about the files
> permissions+ownership, and it doesnt really matter in the current
> problem. Also dont ask, why one would store a root-filesystem in git :)
> 
> What I do not understand: why does the zip file only contains 1090
> files+dirs if the wc -l shows like 150k files and only like 159 were
> deleted?
> There should be like 149k files in that archive.
> 
> Also only the few files are all from "var" and none from etc or srv
> where definitely files changed in too! (and show up in latest.zip.files)
> 
> Is there a limit of files git archive can process?

Not explicitly, but there is a limit on the size of command lines and xargs will invoke the command multiple times if enough arguments are given.

What happens if you do:
	git diff --name-only HEAD^ HEAD | xargs -d'\n' echo | wc -l
?

With a small number of items, there should only be one output line, but if xargs invokes the command multiple times there will be multiple lines. For example (using -L2 to force a maximum of two arguments per invocation):

	$ printf '%s\n' a b c | xargs -d'\n' echo | wc -l
	1
	$ printf '%s\n' a b c | xargs -d'\n' -L2 echo | wc -l
	2
Junio C Hamano· Jul 23, 2015, 17:21 UTC · re: John Keeping · lore

Re: unexplained behavior/issue with git archive?

John Keeping <john@keeping.me.uk> writes:
Show 9 quoted lines
> With a small number of items, there should only be one output line, but
> if xargs invokes the command multiple times there will be multiple
> lines.  For example (using -L2 to force a maximum of two arguments per
> invocation):
>
> 	$ printf '%s\n' a b c | xargs -d'\n' echo | wc -l
> 	1
> 	$ printf '%s\n' a b c | xargs -d'\n' -L2 echo | wc -l
> 	2

Yup, I think this thread is mistitled; it looks like an "unexpected behaviour with xargs". Or "common pitfalls with xargs", perhaps.

Now, what would be a reasonable workaround. To work around command line length limits (not necessarily for xargs, but the exact same issue would arise if you are trying to specify too many pathspecs on the command line), many of our commands take paths from their standard input. Would it be reasonable to teach "git archive" to also do so?

Or would it make sense to teach "git archive -o" a new mode to append to an existing archive, so that repeated invocations of "git archive" via such a use of "xargs" would create in the first invocation and then keep appending to the same archive in the subsequent invocations?

← back to recent threads