{"thread":{"id":"39925","subject":"unexplained behavior/issue with git archive?","startedAt":"2015-07-23T15:23:49Z","lastAt":"2015-07-23T17:21:12Z","messageCount":3,"participants":["Jan Vales","John Keeping","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"266662","messageId":"55B10705.6090303@jvales.net","threadId":"39925","inReplyTo":null,"subject":"unexplained behavior/issue with git archive?","fromName":"Jan Vales","fromEmail":"jan@jvales.net","sentAt":"2015-07-23T15:23:49Z","receivedAt":"2015-07-23T15:23:49Z","isPatch":false,"sender":{"key":"jan@jvales.net","avatar":null},"body":"hi,\ni seem to trigger behavior i do not understand with git archive.\n\nI have this little 3 liner (vmdiff.sh):\n#!/bin/bash\ngit diff --name-status \"$2\" \"$3\" > \"$1.files\"\ngit diff --name-only \"$2\" \"$3\" |xargs -d'\\n' git archive -o \"$1\" \"$3\" --\n\n\nFor testing purpose, lets assume this call:\n# ./vmdiff.sh latest.zip HEAD^1 HEAD\n\n# cat latest.zip.files | wc -l\n149021\n\n# cat latest.zip.files | egrep \"^D\" | wc -l\n159\n\n# mkdir empty; cd empty; unzip latest.zip ; find * | wc -l\n1090\n\nMy goal is to basically diff (parts of) filesystems against each other\nand create an archive with all changed files + a file list to know what\nfiles were deleted. (I currently do not care about the files\npermissions+ownership, and it doesnt really matter in the current\nproblem. Also dont ask, why one would store a root-filesystem in git :)\n\nWhat I do not understand: why does the zip file only contains 1090\nfiles+dirs if the wc -l shows like 150k files and only like 159 were\ndeleted?\nThere should be like 149k files in that archive.\n\nAlso only the few files are all from \"var\" and none from etc or srv\nwhere definitely files changed in too! (and show up in latest.zip.files)\n\nIs there a limit of files git archive can process?\n\nlg\nJan Vales\n--\nI only read plaintext emails.\n\nSomeone @ irc://irc.fsinf.at:6667/tuwien\nwebIRC: https://frost.fsinf.at/iris/\n\n"},{"id":"266663","messageId":"20150723155936.GC14935@serenity.lan","threadId":"39925","inReplyTo":"55B10705.6090303@jvales.net","subject":"Re: unexplained behavior/issue with git archive?","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2015-07-23T15:59:36Z","receivedAt":"2015-07-23T15:59:36Z","isPatch":false,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Thu, Jul 23, 2015 at 05:23:49PM +0200, Jan Vales wrote:\n> i seem to trigger behavior i do not understand with git archive.\n> \n> I have this little 3 liner (vmdiff.sh):\n> #!/bin/bash\n> git diff --name-status \"$2\" \"$3\" > \"$1.files\"\n> git diff --name-only \"$2\" \"$3\" |xargs -d'\\n' git archive -o \"$1\" \"$3\" --\n> \n> \n> For testing purpose, lets assume this call:\n> # ./vmdiff.sh latest.zip HEAD^1 HEAD\n> \n> # cat latest.zip.files | wc -l\n> 149021\n> \n> # cat latest.zip.files | egrep \"^D\" | wc -l\n> 159\n> \n> # mkdir empty; cd empty; unzip latest.zip ; find * | wc -l\n> 1090\n> \n> My goal is to basically diff (parts of) filesystems against each other\n> and create an archive with all changed files + a file list to know what\n> files were deleted. (I currently do not care about the files\n> permissions+ownership, and it doesnt really matter in the current\n> problem. Also dont ask, why one would store a root-filesystem in git :)\n> \n> What I do not understand: why does the zip file only contains 1090\n> files+dirs if the wc -l shows like 150k files and only like 159 were\n> deleted?\n> There should be like 149k files in that archive.\n> \n> Also only the few files are all from \"var\" and none from etc or srv\n> where definitely files changed in too! (and show up in latest.zip.files)\n> \n> Is there a limit of files git archive can process?\n\nNot explicitly, but there is a limit on the size of command lines and\nxargs will invoke the command multiple times if enough arguments are\ngiven.\n\nWhat happens if you do:\n\n\tgit diff --name-only HEAD^ HEAD | xargs -d'\\n' echo | wc -l\n\n?\n\nWith a small number of items, there should only be one output line, but\nif xargs invokes the command multiple times there will be multiple\nlines.  For example (using -L2 to force a maximum of two arguments per\ninvocation):\n\n\t$ printf '%s\\n' a b c | xargs -d'\\n' echo | wc -l\n\t1\n\t$ printf '%s\\n' a b c | xargs -d'\\n' -L2 echo | wc -l\n\t2\n"},{"id":"266670","messageId":"xmqqwpxqpz7b.fsf@gitster.dls.corp.google.com","threadId":"39925","inReplyTo":"20150723155936.GC14935@serenity.lan","subject":"Re: unexplained behavior/issue with git archive?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-07-23T17:21:12Z","receivedAt":"2015-07-23T17:21:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Keeping <john@keeping.me.uk> writes:\n\n> With a small number of items, there should only be one output line, but\n> if xargs invokes the command multiple times there will be multiple\n> lines.  For example (using -L2 to force a maximum of two arguments per\n> invocation):\n>\n> \t$ printf '%s\\n' a b c | xargs -d'\\n' echo | wc -l\n> \t1\n> \t$ printf '%s\\n' a b c | xargs -d'\\n' -L2 echo | wc -l\n> \t2\n\nYup, I think this thread is mistitled; it looks like an \"unexpected\nbehaviour with xargs\".  Or \"common pitfalls with xargs\", perhaps.\n\nNow, what would be a reasonable workaround.  To work around command\nline length limits (not necessarily for xargs, but the exact same\nissue would arise if you are trying to specify too many pathspecs on\nthe command line), many of our commands take paths from their\nstandard input.  Would it be reasonable to teach \"git archive\" to\nalso do so?\n\nOr would it make sense to teach \"git archive -o\" a new mode to\nappend to an existing archive, so that repeated invocations of \"git\narchive\" via such a use of \"xargs\" would create in the first\ninvocation and then keep appending to the same archive in the\nsubsequent invocations?\n"}]}