{"thread":{"id":"27808","subject":"git-archive and tar options","startedAt":"2011-07-13T23:34:32Z","lastAt":"2011-07-21T16:59:23Z","messageCount":22,"participants":["Neal Kreitzinger","Jeff King","René Scharfe","Andreas Schwab","Jakub Narebski","Junio C Hamano","Sylvain Rabot"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"171299","messageId":"ivla29$liu$1@dough.gmane.org","threadId":"27808","inReplyTo":null,"subject":"git-archive and tar options","fromName":"Neal Kreitzinger","fromEmail":"neal@rsss.com","sentAt":"2011-07-13T23:34:32Z","receivedAt":"2011-07-13T23:34:32Z","isPatch":false,"sender":{"key":"neal@rsss.com","avatar":null},"body":"the git-archive manpage states:\n\n\"git archive [--format=] [--list] [--prefix=/] [] [-o \n| --output=] [--worktree-attributes] [--remote= \n[--exec=]] [path\\u2026]\n\n\n This can be any options that the archiver backend understands. See next \nsection.\"\n\nI have tar 1.23 and want to use the --transform option. How can I feed \ngit-archive additional tar options?\n\nWorking syntax starting points for git-archive and tar:\n\ngit archive --format=tar -o my.tar HEAD Web/Templates/\ntar -cvf my.tar --transform 's,^Web/Templates/,myPath/myWeb/Templates/,' \nWebPortal/Templates/\n\nFailed syntax attempts for feeding tar option to git-archive:\n\ngit archive --format=tar -o my.tar HEAD --transform \n's,^Web/Templates/,myPath/myWeb/Templates/,' WebPortal/Templates/\nerror: unknown option `transform'\n\ngit archive --format=tar -o my.tar --transform \n's,^Web/Templates/,myPath/myWeb/Templates/,' HEAD WebPortal/Templates/\nerror: unknown option `transform'\n\n\nv/r,\nneal \n"},{"id":"171303","messageId":"20110714015656.GA20136@sigill.intra.peff.net","threadId":"27808","inReplyTo":"ivla29$liu$1@dough.gmane.org","subject":"Re: git-archive and tar options","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-07-14T01:56:56Z","receivedAt":"2011-07-14T01:56:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 13, 2011 at 06:34:32PM -0500, Neal Kreitzinger wrote:\n\n> the git-archive manpage states:\n> \n> \"git archive [--format=] [--list] [--prefix=/] [] [-o \n> | --output=] [--worktree-attributes] [--remote= \n> [--exec=]] [path\\u2026]\n> \n> \n> This can be any options that the archiver backend understands. See next \n> section.\"\n>\n> I have tar 1.23 and want to use the --transform option. How can I feed \n> git-archive additional tar options?\n\nRight. And the next section is \"Backend Extra Options\", which has:\n\n zip\n -0\n Store the files instead of deflating them.\n\n -9\n Highest and slowest compression level. You can specify any number from 1 to 9\n to adjust compression speed and ratio.\n\nAnd nothing else. We don't actually call your system \"tar\" to generate\nthe tarball, which is what I assume you thought when you saw \"backend\".\nA patch to make it more clear would be welcome.\n\n> Working syntax starting points for git-archive and tar:\n> \n> git archive --format=tar -o my.tar HEAD Web/Templates/\n> tar -cvf my.tar --transform 's,^Web/Templates/,myPath/myWeb/Templates/,' \n> WebPortal/Templates/\n> \n> Failed syntax attempts for feeding tar option to git-archive:\n> \n> git archive --format=tar -o my.tar HEAD --transform \n> 's,^Web/Templates/,myPath/myWeb/Templates/,' WebPortal/Templates/\n> error: unknown option `transform'\n> \n> git archive --format=tar -o my.tar --transform \n> 's,^Web/Templates/,myPath/myWeb/Templates/,' HEAD WebPortal/Templates/\n> error: unknown option `transform'\n\nYeah, that won't work, because there is no such option. We do have\n\"--prefix\", but I suspect that's not flexible enough for what you want.\n\nSo you're probably stuck with extracting the results of \"git archive\" to\na temporary directory and then using GNU tar to re-archive them (or if\nyou have a checkout, you can just tar that up directly, feeding the list\nfrom \"git ls-files\" into tar). It would be nice if GNU tar could act as\na post-processor, and do something like:\n\n git archive HEAD | tar --pipe-mode --transform=whatever >my.tar\n\nBut AFAIK, nothing like \"--pipe-mode\" exists.\n\nIt would probably not be a very hard feature to add to \"git archive\" if\nyou're interested in doing so.\n\n-Peff\n"},{"id":"171341","messageId":"4E1F2468.6080409@lsrfire.ath.cx","threadId":"27808","inReplyTo":"20110714015656.GA20136@sigill.intra.peff.net","subject":"Re: git-archive and tar options","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-07-14T17:16:24Z","receivedAt":"2011-07-14T17:16:24Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 14.07.2011 03:56, schrieb Jeff King:\n> On Wed, Jul 13, 2011 at 06:34:32PM -0500, Neal Kreitzinger wrote:\n>> Working syntax starting points for git-archive and tar:\n>>\n>> git archive --format=tar -o my.tar HEAD Web/Templates/\n>> tar -cvf my.tar --transform 's,^Web/Templates/,myPath/myWeb/Templates/,' \n>> WebPortal/Templates/\n>>\n>> Failed syntax attempts for feeding tar option to git-archive:\n>>\n>> git archive --format=tar -o my.tar HEAD --transform \n>> 's,^Web/Templates/,myPath/myWeb/Templates/,' WebPortal/Templates/\n>> error: unknown option `transform'\n>>\n>> git archive --format=tar -o my.tar --transform \n>> 's,^Web/Templates/,myPath/myWeb/Templates/,' HEAD WebPortal/Templates/\n>> error: unknown option `transform'\n> \n> Yeah, that won't work, because there is no such option. We do have\n> \"--prefix\", but I suspect that's not flexible enough for what you want.\n\nIf you only need a single subdirectory with a custom prefix you could do\nsomething like this (variables only used to keep the lines short):\n\n\t$ subdir=WebPortal/Templates\n\t$ prefix=myPath/myWeb/Templates/\n\t$ (cd \"$subdir\" && git archive --prefix=\"$prefix\" HEAD) >my.tar\n\nThe output file can be specified with -o as well, of course, but you'd\neither need to use an absolute path or add \"../\" for each directory\nlevel you descend into (-o ../../my.tar in this case).\n\nRené\n"},{"id":"171342","messageId":"20110714172718.GA21341@sigill.intra.peff.net","threadId":"27808","inReplyTo":"4E1F2468.6080409@lsrfire.ath.cx","subject":"Re: git-archive and tar options","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-07-14T17:27:18Z","receivedAt":"2011-07-14T17:27:18Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 14, 2011 at 07:16:24PM +0200, René Scharfe wrote:\n\n> >> git archive --format=tar -o my.tar --transform \n> >> 's,^Web/Templates/,myPath/myWeb/Templates/,' HEAD WebPortal/Templates/\n> >> error: unknown option `transform'\n> > \n> > Yeah, that won't work, because there is no such option. We do have\n> > \"--prefix\", but I suspect that's not flexible enough for what you want.\n> \n> If you only need a single subdirectory with a custom prefix you could do\n> something like this (variables only used to keep the lines short):\n> \n> \t$ subdir=WebPortal/Templates\n> \t$ prefix=myPath/myWeb/Templates/\n> \t$ (cd \"$subdir\" && git archive --prefix=\"$prefix\" HEAD) >my.tar\n> \n> The output file can be specified with -o as well, of course, but you'd\n> either need to use an absolute path or add \"../\" for each directory\n> level you descend into (-o ../../my.tar in this case).\n\nCouldn't you also do:\n\n git archive --prefix=$prefix HEAD:$subdir >my.tar\n\n? I guess that loses the pax header with the commit sha1 in it, though,\nbecause you are feeding a straight tree instead of a commit.\n\nWe didn't when git-archive was written, but these days we have\nget_sha1_with_context to remember incidental things about an object we\nlook up. It should perhaps remember the commit (if any) we used to reach\na treeish, and then the above command line could still insert the pax\nheader.\n\n-Peff\n"},{"id":"171344","messageId":"4E1F2B23.1020908@lsrfire.ath.cx","threadId":"27808","inReplyTo":"20110714172718.GA21341@sigill.intra.peff.net","subject":"Re: git-archive and tar options","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-07-14T17:45:07Z","receivedAt":"2011-07-14T17:45:07Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 14.07.2011 19:27, schrieb Jeff King:\n> On Thu, Jul 14, 2011 at 07:16:24PM +0200, René Scharfe wrote:\n> \n>>>> git archive --format=tar -o my.tar --transform \n>>>> 's,^Web/Templates/,myPath/myWeb/Templates/,' HEAD WebPortal/Templates/\n>>>> error: unknown option `transform'\n>>>\n>>> Yeah, that won't work, because there is no such option. We do have\n>>> \"--prefix\", but I suspect that's not flexible enough for what you want.\n>>\n>> If you only need a single subdirectory with a custom prefix you could do\n>> something like this (variables only used to keep the lines short):\n>>\n>> \t$ subdir=WebPortal/Templates\n>> \t$ prefix=myPath/myWeb/Templates/\n>> \t$ (cd \"$subdir\" && git archive --prefix=\"$prefix\" HEAD) >my.tar\n>>\n>> The output file can be specified with -o as well, of course, but you'd\n>> either need to use an absolute path or add \"../\" for each directory\n>> level you descend into (-o ../../my.tar in this case).\n> \n> Couldn't you also do:\n> \n> git archive --prefix=$prefix HEAD:$subdir >my.tar\n> \n> ? I guess that loses the pax header with the commit sha1 in it, though,\n> because you are feeding a straight tree instead of a commit.\n\nYes, and yes.\n\n> We didn't when git-archive was written, but these days we have\n> get_sha1_with_context to remember incidental things about an object we\n> look up. It should perhaps remember the commit (if any) we used to reach\n> a treeish, and then the above command line could still insert the pax\n> header.\n\nThat's a good idea to increase consistency, as there shouldn't really be\na difference in output between the two subdirectory syntaxes.\n\nI always wondered, however, if the embedded commit ID has really been\nused to identify the corresponding version of an archive that somehow\nlost its filename (due to being piped?).\n\nRené\n"},{"id":"171346","messageId":"m239i8vjlk.fsf@igel.home","threadId":"27808","inReplyTo":"20110714015656.GA20136@sigill.intra.peff.net","subject":"Re: git-archive and tar options","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2011-07-14T17:48:55Z","receivedAt":"2011-07-14T17:48:55Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Jeff King writes:\n\n> So you're probably stuck with extracting the results of \"git archive\" to\n> a temporary directory and then using GNU tar to re-archive them (or if\n> you have a checkout, you can just tar that up directly, feeding the list\n> from \"git ls-files\" into tar).\n\nThat would lose the embedded commit-id, though.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"171350","messageId":"20110714181858.GA25172@sigill.intra.peff.net","threadId":"27808","inReplyTo":"4E1F2B23.1020908@lsrfire.ath.cx","subject":"Re: git-archive and tar options","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-07-14T18:18:58Z","receivedAt":"2011-07-14T18:18:58Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 14, 2011 at 07:45:07PM +0200, René Scharfe wrote:\n\n> > We didn't when git-archive was written, but these days we have\n> > get_sha1_with_context to remember incidental things about an object we\n> > look up. It should perhaps remember the commit (if any) we used to reach\n> > a treeish, and then the above command line could still insert the pax\n> > header.\n> \n> That's a good idea to increase consistency, as there shouldn't really be\n> a difference in output between the two subdirectory syntaxes.\n\nThe patch to do this is pretty tiny. See below.\n\nThere are a few issues, though:\n\n 1. I think this is probably the right thing to do, and most people\n will be happy about it. But I guess I can see an argument that the\n commit-id should not be there, as the subtree does not represent\n that commit.\n\n IOW, if you assume the commit-id in the output means\n \"by the way, this came from commit X\", this change is a good thing.\n If you assume it means \"this is the tree from commit X\", then it's\n not. I have no idea how people use it. I never have, but I always\n assumed the use case was \"I have this random tarball. Where did it\n come from?\".\n\n 2. The object_context already has the sha1 we want, but it is under\n the name \"tree\", which is not an accurate name. It's actually\n \"whatever is on the left side of the :\". Which should be a\n tree-ish, but could be a commit or a tree.\n\n 3. It looks like we fill in object_context whenever we see something\n like \"tree-ish:path\". But we should perhaps also do so when peeling\n something like \"tree-ish^{tree}\".\n\n> I always wondered, however, if the embedded commit ID has really been\n> used to identify the corresponding version of an archive that somehow\n> lost its filename (due to being piped?).\n\nI dunno. I've never used it.\n\n-- >8 --\nSubject: [PATCH] archive: look harder for commit id\n\nWhen \"git archive\" is given a commit, the output will\ncontain the commit sha1 (either as a pax header for tar\nformat, or in a file comment for zip).\n\nWhen it's given a name that resolves to a tree, like:\n\n git archive git-1.7.0:Documentation\n\nthen the archive code never sees the commit, and no\ncommit-id is output. We can use get_sha1_with_context to\nremember the commit that led us to that tree (if any).\n\nSigned-off-by: Jeff King \n---\n archive.c | 5 ++++-\n 1 files changed, 4 insertions(+), 1 deletions(-)\n\ndiff --git a/archive.c b/archive.c\nindex 42f2d2f..d0ba7fb 100644\n--- a/archive.c\n+++ b/archive.c\n@@ -256,11 +256,14 @@ static void parse_treeish_arg(const char **argv,\n \tstruct tree *tree;\n \tconst struct commit *commit;\n \tunsigned char sha1[20];\n+\tstruct object_context oc;\n \n-\tif (get_sha1(name, sha1))\n+\tif (get_sha1_with_context(name, sha1, &oc))\n \t\tdie(\"Not a valid object name\");\n \n \tcommit = lookup_commit_reference_gently(sha1, 1);\n+\tif (!commit)\n+\t\tcommit = lookup_commit_reference_gently(oc.tree, 1);\n \tif (commit) {\n \t\tcommit_sha1 = commit->object.sha1;\n \t\tarchive_time = commit->date;\n-- \n1.7.6.38.ge5b33\n"},{"id":"171366","messageId":"m3oc0wad94.fsf@localhost.localdomain","threadId":"27808","inReplyTo":"20110714181858.GA25172@sigill.intra.peff.net","subject":"Re: git-archive and tar options","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-07-14T19:12:41Z","receivedAt":"2011-07-14T19:12:41Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jeff King writes:\n\n> On Thu, Jul 14, 2011 at 07:45:07PM +0200, René Scharfe wrote:\n> \n> > > We didn't when git-archive was written, but these days we have\n> > > get_sha1_with_context to remember incidental things about an object we\n> > > look up. It should perhaps remember the commit (if any) we used to reach\n> > > a treeish, and then the above command line could still insert the pax\n> > > header.\n> > \n> > That's a good idea to increase consistency, as there shouldn't really be\n> > a difference in output between the two subdirectory syntaxes.\n> \n> The patch to do this is pretty tiny. See below.\n> \n> There are a few issues, though:\n> \n> 1. I think this is probably the right thing to do, and most people\n> will be happy about it. But I guess I can see an argument that the\n> commit-id should not be there, as the subtree does not represent\n> that commit.\n> \n> IOW, if you assume the commit-id in the output means\n> \"by the way, this came from commit X\", this change is a good thing.\n> If you assume it means \"this is the tree from commit X\", then it's\n> not. I have no idea how people use it. I never have, but I always\n> assumed the use case was \"I have this random tarball. Where did it\n> come from?\".\n\nPerhaps we should embed ':' instead in pax header,\nin that case? Or . if ':' is forbidden.\n\n-- \nJakub Narębski\nPoland\n"},{"id":"171384","messageId":"7vei1s36bl.fsf@alter.siamese.dyndns.org","threadId":"27808","inReplyTo":"20110714172718.GA21341@sigill.intra.peff.net","subject":"Re: git-archive and tar options","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-07-14T21:23:10Z","receivedAt":"2011-07-14T21:23:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Couldn't you also do:\n>\n> git archive --prefix=$prefix HEAD:$subdir >my.tar\n>\n> ? I guess that loses the pax header with the commit sha1 in it, though,\n> because you are feeding a straight tree instead of a commit.\n>\n> We didn't when git-archive was written, but these days we have\n> get_sha1_with_context to remember incidental things about an object we\n> look up. It should perhaps remember the commit (if any) we used to reach\n> a treeish, and then the above command line could still insert the pax\n> header.\n\nWhy?\n\nThe tree you are writing out that way look very different from what is\nrecorded in the commit object. What's the point of introducing confusion\nby allowing many tarballs with different contents written from the same\ncommits with such tweaks all labelled with the same pax header?\n"},{"id":"171385","messageId":"20110714212502.GA29848@sigill.intra.peff.net","threadId":"27808","inReplyTo":"7vei1s36bl.fsf@alter.siamese.dyndns.org","subject":"Re: git-archive and tar options","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-07-14T21:25:02Z","receivedAt":"2011-07-14T21:25:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 14, 2011 at 02:23:10PM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > Couldn't you also do:\n> >\n> > git archive --prefix=$prefix HEAD:$subdir >my.tar\n> >\n> > ? I guess that loses the pax header with the commit sha1 in it, though,\n> > because you are feeding a straight tree instead of a commit.\n> >\n> > We didn't when git-archive was written, but these days we have\n> > get_sha1_with_context to remember incidental things about an object we\n> > look up. It should perhaps remember the commit (if any) we used to reach\n> > a treeish, and then the above command line could still insert the pax\n> > header.\n> \n> Why?\n> \n> The tree you are writing out that way look very different from what is\n> recorded in the commit object. What's the point of introducing confusion\n> by allowing many tarballs with different contents written from the same\n> commits with such tweaks all labelled with the same pax header?\n\nSee my later message. I think it depends on how the embedded id is used.\nIs it to say \"this represents the tree of this git commit\"? Or is it to\nhelp people who later have a tarball and have no clue which commit it\nmight have come from?\n\nI don't have a strong opinion either way. I've never used this feature\nat all.\n\n-Peff\n"},{"id":"171387","messageId":"m3k4bka6jb.fsf@localhost.localdomain","threadId":"27808","inReplyTo":"7vei1s36bl.fsf@alter.siamese.dyndns.org","subject":"Re: git-archive and tar options","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-07-14T21:38:05Z","receivedAt":"2011-07-14T21:38:05Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano writes:\n\n> Jeff King writes:\n> \n> > Couldn't you also do:\n> >\n> > git archive --prefix=$prefix HEAD:$subdir >my.tar\n> >\n> > ? I guess that loses the pax header with the commit sha1 in it, though,\n> > because you are feeding a straight tree instead of a commit.\n> >\n> > We didn't when git-archive was written, but these days we have\n> > get_sha1_with_context to remember incidental things about an object we\n> > look up. It should perhaps remember the commit (if any) we used to reach\n> > a treeish, and then the above command line could still insert the pax\n> > header.\n> \n> Why?\n> \n> The tree you are writing out that way look very different from what is\n> recorded in the commit object. What's the point of introducing confusion\n> by allowing many tarballs with different contents written from the same\n> commits with such tweaks all labelled with the same pax header?\n \nPerhaps pax header should contain : then?\nJust a thought.\n\n-- \nJakub Narebski\n"},{"id":"171390","messageId":"7vwrfk1lv3.fsf@alter.siamese.dyndns.org","threadId":"27808","inReplyTo":"20110714212502.GA29848@sigill.intra.peff.net","subject":"Re: git-archive and tar options","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-07-14T23:30:24Z","receivedAt":"2011-07-14T23:30:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> Why?\n>> \n>> The tree you are writing out that way look very different from what is\n>> recorded in the commit object. What's the point of introducing confusion\n>> by allowing many tarballs with different contents written from the same\n>> commits with such tweaks all labelled with the same pax header?\n>\n> See my later message. I think it depends on how the embedded id is used.\n> Is it to say \"this represents the tree of this git commit\"? Or is it to\n> help people who later have a tarball and have no clue which commit it\n> might have come from?\n\nPeople, who have no clue which part of the subtree was extract and what\nleading path was added, would still have to wonder where the tree came\nfrom even with the embedded id. Without your patch, if the tarball has an\nembedded id, wouldn't they at least be able to assume it is the whole\nthing of that commit? If you label a randomly mutated tree with the same\nlabel, you cannot tell the genuine one from manipulated ones.\n\nNot that I have strong opinions on this, either, but that is what I meant\nby \"_introducing_\" confusion.\n"},{"id":"171434","messageId":"4E20AA42.7000003@lsrfire.ath.cx","threadId":"27808","inReplyTo":"7vwrfk1lv3.fsf@alter.siamese.dyndns.org","subject":"Re: git-archive and tar options","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-07-15T20:59:46Z","receivedAt":"2011-07-15T20:59:46Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 15.07.2011 01:30, schrieb Junio C Hamano:\n> Jeff King writes:\n> \n>>> Why?\n>>>\n>>> The tree you are writing out that way look very different from what is\n>>> recorded in the commit object. What's the point of introducing confusion\n>>> by allowing many tarballs with different contents written from the same\n>>> commits with such tweaks all labelled with the same pax header?\n>>\n>> See my later message. I think it depends on how the embedded id is used.\n>> Is it to say \"this represents the tree of this git commit\"? Or is it to\n>> help people who later have a tarball and have no clue which commit it\n>> might have come from?\n> \n> People, who have no clue which part of the subtree was extract and what\n> leading path was added, would still have to wonder where the tree came\n> from even with the embedded id. Without your patch, if the tarball has an\n> embedded id, wouldn't they at least be able to assume it is the whole\n> thing of that commit? If you label a randomly mutated tree with the same\n> label, you cannot tell the genuine one from manipulated ones.\n> \n> Not that I have strong opinions on this, either, but that is what I meant\n> by \"_introducing_\" confusion.\n\nWhen we started to write the ID into generated archives, there was only\ngit-tar-tree and no : syntax. It would write the ID only if\nit was given a commit and not if it got a tree or if the user started it\nfrom a subdirectory. The result was that only the full tree of a commit\nwas branded with the commit ID.\n\nNow we have git archive, a more flexible command line syntax all around,\npath limiting as well as attributes that can affect the contents of the\nfiles in the archive. Back then the commmit ID was sufficient as a\nconcise and canonical label of the archive contents, but now things are\na bit more complicated.\n\nWhich use cases are we aiming for? Do we want to include all of the\ncommand line arguments (with revs resolved to SHA1-IDs)? Only those\nthat modify archive contents? And any applied attributes? Or do we\nwant to get stricter and only write the commit ID if a full unchanged\ntree of a commit is being archived?\n\nRené\n"},{"id":"171596","messageId":"4E2477E1.5090406@gmail.com","threadId":"27808","inReplyTo":"20110714172718.GA21341@sigill.intra.peff.net","subject":"Re: git-archive and tar options","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2011-07-18T18:13:53Z","receivedAt":"2011-07-18T18:13:53Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 7/14/2011 12:27 PM, Jeff King wrote:\n> On Thu, Jul 14, 2011 at 07:16:24PM +0200, René Scharfe wrote:\n>\n>>>> git archive --format=tar -o my.tar --transform\n>>>> 's,^Web/Templates/,myPath/myWeb/Templates/,' HEAD\n>>>> WebPortal/Templates/ error: unknown option `transform'\n>>>\n>>> Yeah, that won't work, because there is no such option. We do\n>>> have \"--prefix\", but I suspect that's not flexible enough for\n>>> what you want.\n>>\n>> If you only need a single subdirectory with a custom prefix you\n>> could do something like this (variables only used to keep the lines\n>> short):\n>>\n>> $ subdir=WebPortal/Templates $ prefix=myPath/myWeb/Templates/ $ (cd\n>> \"$subdir\"&& git archive --prefix=\"$prefix\" HEAD)>my.tar\n>>\n>> The output file can be specified with -o as well, of course, but\n>> you'd either need to use an absolute path or add \"../\" for each\n>> directory level you descend into (-o ../../my.tar in this case).\n>\n> Couldn't you also do:\n>\n> git archive --prefix=$prefix HEAD:$subdir>my.tar\n>\n> ? I guess that loses the pax header with the commit sha1 in it,\n> though, because you are feeding a straight tree instead of a commit.\n>\n> We didn't when git-archive was written, but these days we have\n> get_sha1_with_context to remember incidental things about an object\n> we look up. It should perhaps remember the commit (if any) we used to\n> reach a treeish, and then the above command line could still insert\n> the pax header.\n>\nHEAD:$subdir worked on my bare repo. I ran it for each transformant \npathspec and then combined the archives with tar --catenate:\n\n# git archive --format=tar --prefix=myWeb/myRoot/myAPP/Templates/\nHEAD:WebPortal/Templates/ >myAPP.myTag.tar\n# git archive --format=tar --prefix=opt/mySTUFF/v01/SCRIPTS/\nHEAD:SCRIPTS/ >SCRIPTS.tar\n# tar --file=myAPP.myTag.tar -A SCRIPTS.tar\n\nHowever, the permissions also need to change to 777 and tar --mode would \nnot effect this in combination with --catenation or -x. Is there a way \nI can change the permissions without having to untar->chmod->retar, and \nwithout having to use a non-bare repo as an intermediary?\n\nv/r,\nneal\n"},{"id":"171601","messageId":"4E248A2E.3090902@gmail.com","threadId":"27808","inReplyTo":"4E20AA42.7000003@lsrfire.ath.cx","subject":"Re: git-archive and tar options","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2011-07-18T19:31:58Z","receivedAt":"2011-07-18T19:31:58Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 7/15/2011 3:59 PM, René Scharfe wrote:\n> Am 15.07.2011 01:30, schrieb Junio C Hamano:\n>> Jeff King writes:\n>>\n>>>> Why?\n>>>>\n>>>> The tree you are writing out that way look very different from\n>>>> what is recorded in the commit object. What's the point of\n>>>> introducing confusion by allowing many tarballs with different\n>>>> contents written from the same commits with such tweaks all\n>>>> labelled with the same pax header?\n>>>\n>>> See my later message. I think it depends on how the embedded id\n>>> is used. Is it to say \"this represents the tree of this git\n>>> commit\"? Or is it to help people who later have a tarball and\n>>> have no clue which commit it might have come from?\n>>\n>> People, who have no clue which part of the subtree was extract and\n>> what leading path was added, would still have to wonder where the\n>> tree came from even with the embedded id. Without your patch, if\n>> the tarball has an embedded id, wouldn't they at least be able to\n>> assume it is the whole thing of that commit? If you label a\n>> randomly mutated tree with the same label, you cannot tell the\n>> genuine one from manipulated ones.\n>>\n>> Not that I have strong opinions on this, either, but that is what I\n>> meant by \"_introducing_\" confusion.\n>\n> When we started to write the ID into generated archives, there was\n> only git-tar-tree and no: syntax. It would write the ID\n> only if it was given a commit and not if it got a tree or if the\n> user started it from a subdirectory. The result was that only the\n> full tree of a commit was branded with the commit ID.\n>\n> Now we have git archive, a more flexible command line syntax all\n> around, path limiting as well as attributes that can affect the\n> contents of the files in the archive. Back then the commmit ID was\n> sufficient as a concise and canonical label of the archive contents,\n> but now things are a bit more complicated.\n>\n> Which use cases are we aiming for? Do we want to include all of the\n> command line arguments (with revs resolved to SHA1-IDs)? Only those\n> that modify archive contents? And any applied attributes? Or do we\n> want to get stricter and only write the commit ID if a full unchanged\n> tree of a commit is being archived?\n>\nIn regards to the use cases you enumerated, I think logging the command\nline syntax along with the appropriate ref context (HEAD value, etc)\nwould document exactly what's in the archive.\n\nIn regards to use cases in general, my impression is that git-archive is \nfor producing archives useful for deployment. The target deployed \nstructure may vary so expecting the source git repo to reflect this is \nunfeasable. It seems like utilizing the local tar installation would \neffect the necessary transformations. I'm not sure what the source and \ntarget tar version disparity problems might me.\n\nA practical problem with the pax header is that its only useful if you\nstill have the archive. Archives usually get deleted after being\nextracted. Therefore, an option to also generate (and add to the \narchive) an automatic \"VERSION.TXT\" file of some sort which specifies \nthe context of the archive would be much more useful. It would need its \nown --prefix option because oftentimes it would be dynamically generated \nbased on the git-archive request.\n\nAnother use case is that it seems like there should also be the option \nto only tar the objects changed between a specified range of commits. \nHowever, I'm not sure if tar can handle deletions (moves, deletions, \nrenames) upon extraction in this context.\n\nI can see that my use cases are something that I can script myself, but \nto do so it seems like I would be better off using a non-bare repo \ncheckout as an intermediary. If that is what I am expected to do then I \nam not sure what the usefulness of git-archive is intended to be. Maybe \nI don't understand what others use it for.\n\nv/r,\nneal\n"},{"id":"171609","messageId":"4E249C8D.10107@lsrfire.ath.cx","threadId":"27808","inReplyTo":"4E248A2E.3090902@gmail.com","subject":"Re: git-archive and tar options","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-07-18T20:50:21Z","receivedAt":"2011-07-18T20:50:21Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 18.07.2011 21:31, schrieb Neal Kreitzinger:\n> In regards to use cases in general, my impression is that git-archive is\n> for producing archives useful for deployment. The target deployed\n> structure may vary so expecting the source git repo to reflect this is\n> unfeasable. It seems like utilizing the local tar installation would\n> effect the necessary transformations. I'm not sure what the source and\n> target tar version disparity problems might me.\n\nDirect deployment is not an _intended_ use case, but I see that it might\nbe useful for that, especially with scripting languages.\n\nI'm not sure I like tar's --transform option, though. This seems to be\ntoo heavy a solution. For your example it would be enough to support\nmultiple tree arguments (with their own respective prefixes) in one go.\n\n> A practical problem with the pax header is that its only useful if you\n> still have the archive. Archives usually get deleted after being\n> extracted. Therefore, an option to also generate (and add to the\n> archive) an automatic \"VERSION.TXT\" file of some sort which specifies\n> the context of the archive would be much more useful. It would need its\n> own --prefix option because oftentimes it would be dynamically generated\n> based on the git-archive request.\n\nThe attribute export-subst with its $Format:$ expansion is intended to\nbe used for such version files. It still lacks the ability to produce\ngit-describe-style version strings, but commit hashes can be used instead.\n\n> Another use case is that it seems like there should also be the option\n> to only tar the objects changed between a specified range of commits.\n> However, I'm not sure if tar can handle deletions (moves, deletions,\n> renames) upon extraction in this context.\n\nWell, you could build a list of paths using git log --name-status or\nsimilar and feed that to git archive. If you want to keep a directory\nin sync with a repo, why not use git checkout, though? :)\n\n> I can see that my use cases are something that I can script myself, but\n> to do so it seems like I would be better off using a non-bare repo\n> checkout as an intermediary. If that is what I am expected to do then I\n> am not sure what the usefulness of git-archive is intended to be. Maybe\n> I don't understand what others use it for.\n\nThe primary use case is to create source code archives that people can\ndownload, build and deploy who are not interested in downloading the\nwhole history or in using git at all.\n\nRené\n"},{"id":"171610","messageId":"4E249C94.3040002@lsrfire.ath.cx","threadId":"27808","inReplyTo":"4E2477E1.5090406@gmail.com","subject":"Re: git-archive and tar options","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-07-18T20:50:28Z","receivedAt":"2011-07-18T20:50:28Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 18.07.2011 20:13, schrieb Neal Kreitzinger:\n> HEAD:$subdir worked on my bare repo. I ran it for each transformant\n> pathspec and then combined the archives with tar --catenate:\n> \n> # git archive --format=tar --prefix=myWeb/myRoot/myAPP/Templates/\n> HEAD:WebPortal/Templates/ >myAPP.myTag.tar\n> # git archive --format=tar --prefix=opt/mySTUFF/v01/SCRIPTS/\n> HEAD:SCRIPTS/ >SCRIPTS.tar\n> # tar --file=myAPP.myTag.tar -A SCRIPTS.tar\n> \n> However, the permissions also need to change to 777 and tar --mode would\n> not effect this in combination with --catenation or -x. Is there a way\n> I can change the permissions without having to untar->chmod->retar, and\n> without having to use a non-bare repo as an intermediary?\n\nYou can use the configuration setting tar.umask to affect the\npermissions of the archive entries. Set it to 0 to pass the permission\nbits from the repo unchanged.\n\nRené\n"},{"id":"171634","messageId":"4E24CBFD.9090909@gmail.com","threadId":"27808","inReplyTo":"4E249C94.3040002@lsrfire.ath.cx","subject":"Re: git-archive and tar options","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2011-07-19T00:12:45Z","receivedAt":"2011-07-19T00:12:45Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 7/18/2011 3:50 PM, René Scharfe wrote:\n> Am 18.07.2011 20:13, schrieb Neal Kreitzinger:\n>> HEAD:$subdir worked on my bare repo. I ran it for each transformant\n>> pathspec and then combined the archives with tar --catenate:\n>>\n>> # git archive --format=tar --prefix=myWeb/myRoot/myAPP/Templates/\n>> HEAD:WebPortal/Templates/>myAPP.myTag.tar\n>> # git archive --format=tar --prefix=opt/mySTUFF/v01/SCRIPTS/\n>> HEAD:SCRIPTS/>SCRIPTS.tar\n>> # tar --file=myAPP.myTag.tar -A SCRIPTS.tar\n>>\n>> However, the permissions also need to change to 777 and tar --mode would\n>> not effect this in combination with --catenation or -x. Is there a way\n>> I can change the permissions without having to untar->chmod->retar, and\n>> without having to use a non-bare repo as an intermediary?\n>\n> You can use the configuration setting tar.umask to affect the\n> permissions of the archive entries. Set it to 0 to pass the permission\n> bits from the repo unchanged.\n>\nThe permissions in my repo are 775 and 664 and I want to change them to 777.\n\n-neal\n"},{"id":"171686","messageId":"4E25C54D.2070007@lsrfire.ath.cx","threadId":"27808","inReplyTo":"4E24CBFD.9090909@gmail.com","subject":"Re: git-archive and tar options","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-07-19T17:56:29Z","receivedAt":"2011-07-19T17:56:29Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 19.07.2011 02:12, schrieb Neal Kreitzinger:\n> On 7/18/2011 3:50 PM, René Scharfe wrote:\n>> Am 18.07.2011 20:13, schrieb Neal Kreitzinger:\n>>> However, the permissions also need to change to 777 and tar --mode would\n>>> not effect this in combination with --catenation or -x. Is there a way\n>>> I can change the permissions without having to untar->chmod->retar, and\n>>> without having to use a non-bare repo as an intermediary?\n>>\n>> You can use the configuration setting tar.umask to affect the\n>> permissions of the archive entries. Set it to 0 to pass the permission\n>> bits from the repo unchanged.\n>>\n> The permissions in my repo are 775 and 664 and I want to change them to\n> 777.\n\nGit doesn't store all permission bits. If a file is marked as\nexecutable then you get 777, otherwise 666 -- minus the umask, which is\n0002 by default. So in order to achive rwx permissions for all in the\narchive, you need to A) mark the files as executable in the repository\nand B) set tar.umask to 0 to get allow the world to write.\n\nHowever, what's the reason for requiring this lack of access control?\nWhy o+w?\n\nRené\n"},{"id":"171699","messageId":"1311106207.17539.12.camel@kheops","threadId":"27808","inReplyTo":"m239i8vjlk.fsf@igel.home","subject":"Re: git-archive and tar options","fromName":"Sylvain Rabot","fromEmail":"sylvain@abstraction.fr","sentAt":"2011-07-19T20:10:07Z","receivedAt":"2011-07-19T20:10:07Z","isPatch":false,"sender":{"key":"sylvain@abstraction.fr","avatar":"https://avatars.githubusercontent.com/u/153052?v=4"},"body":"On Thu, 2011-07-14 at 19:48 +0200, Andreas Schwab wrote:\n> Jeff King writes:\n> \n> > So you're probably stuck with extracting the results of \"git archive\" to\n> > a temporary directory and then using GNU tar to re-archive them (or if\n> > you have a checkout, you can just tar that up directly, feeding the list\n> > from \"git ls-files\" into tar).\n> \n> That would lose the embedded commit-id, though.\n> \n> Andreas.\n> \n\nYou can pass the commit id this way.\n\n$ tar --pax-option \"comment=$COMMIT\" ...\n\n-- \nSylvain Rabot \n"},{"id":"171776","messageId":"4E278B34.70300@gmail.com","threadId":"27808","inReplyTo":"4E25C54D.2070007@lsrfire.ath.cx","subject":"Re: git-archive and tar options","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2011-07-21T02:13:08Z","receivedAt":"2011-07-21T02:13:08Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 7/19/2011 12:56 PM, René Scharfe wrote:\n> Am 19.07.2011 02:12, schrieb Neal Kreitzinger:\n>> On 7/18/2011 3:50 PM, René Scharfe wrote:\n>>> Am 18.07.2011 20:13, schrieb Neal Kreitzinger:\n>>>> However, the permissions also need to change to 777 and tar --mode would\n>>>> not effect this in combination with --catenation or -x. Is there a way\n>>>> I can change the permissions without having to untar->chmod->retar, and\n>>>> without having to use a non-bare repo as an intermediary?\n>>> You can use the configuration setting tar.umask to affect the\n>>> permissions of the archive entries. Set it to 0 to pass the permission\n>>> bits from the repo unchanged.\n>>>\n>> The permissions in my repo are 775 and 664 and I want to change them to\n>> 777.\n> Git doesn't store all permission bits. If a file is marked as\n> executable then you get 777, otherwise 666 -- minus the umask, which is\n> 0002 by default. So in order to achive rwx permissions for all in the\n> archive, you need to A) mark the files as executable in the repository\n> and B) set tar.umask to 0 to get allow the world to write.\n>\n> However, what's the reason for requiring this lack of access control?\n> Why o+w?\ntar.umask worked. Thank you for explaining how the permissions work in \nthis context. I now see that 775 and 664 would work for the apache \ncomponent and for executing our binaries. Thanks for pointing this \nout. However, another element of our application is a proprietary \nruntime that runs on top of linux and runs our core binaries. This \nallows us to store our binaries in git and deploy them directly on the \ncustomer server from git (via git-archive). That runtime needs o+w in \norder to update the 'last run date' in the binary which is critical to \nour troubleshooting in the field. o+w is needed because the user's \nruntime instance runs with user permissions when executed from a linux \ncommand line terminal and our users are not setup in the same group as \nthe binaries. Therefore, with tar.umask = 0000 I can deploy 777 and 666 \npermissions and everything will work.\n\nI suppose I could write a script to change the tar.umask entry to 0000 \nonly when running git-archive for the binary portion, and use tar.umask \n0002 when extracting the other portions. I could also change our setup \nto put the users and the runmodules in the same group and use tar.umask \n0002 across the board. These would be more correct than the chmod 777 \nshotgun that we currently use to blast away our permissions problems.\n\ngit-archive is a \"quick\" solution to our immediate deployment needs. \nEventually, I plan on using git on the source and target machines as the \ncore mechanism to \"promote to production\" (ie. deploy to customer \nservers). It looks like others are using git for deployment also. In \nmy previous shops which used other VCS's on minicomputers and \nmainframes, \"promote to production\" meant the universal run path for all \nusers (and especially for productional data transactions) on that \ncentral machine. In my current shop (my first linux shop) we have \nmultiple concurrent versions of production on a multitude of \nproductional machines and even concurrently on an individual \nproductional machine in some cases. The main reason we chose git is \nbecause it is the only VCS that can handle this.\n\nv/r,\nneal\n"},{"id":"171780","messageId":"4E285AEB.8090603@gmail.com","threadId":"27808","inReplyTo":"4E278B34.70300@gmail.com","subject":"Re: git-archive and tar options","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2011-07-21T16:59:23Z","receivedAt":"2011-07-21T16:59:23Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 7/20/2011 9:13 PM, Neal Kreitzinger wrote:\n> On 7/19/2011 12:56 PM, René Scharfe wrote:\n>> Am 19.07.2011 02:12, schrieb Neal Kreitzinger:\n>>> On 7/18/2011 3:50 PM, René Scharfe wrote:\n>>>> Am 18.07.2011 20:13, schrieb Neal Kreitzinger:\n>>>>> However, the permissions also need to change to 777 and tar\n>>>>> --mode would not effect this in combination with --catenation\n>>>>> or -x. Is there a way I can change the permissions without\n>>>>> having to untar->chmod->retar, and without having to use a\n>>>>> non-bare repo as an intermediary?\n>>>> You can use the configuration setting tar.umask to affect the\n>>>> permissions of the archive entries. Set it to 0 to pass the\n>>>> permission bits from the repo unchanged.\n>>>>\n>>> The permissions in my repo are 775 and 664 and I want to change\n>>> them to 777.\n>> Git doesn't store all permission bits. If a file is marked as\n>> executable then you get 777, otherwise 666 -- minus the umask,\n>> which is 0002 by default. So in order to achive rwx permissions for\n>> all in the archive, you need to A) mark the files as executable in\n>> the repository and B) set tar.umask to 0 to get allow the world to\n>> write.\n>>\n>> However, what's the reason for requiring this lack of access\n>> control? Why o+w?\n> tar.umask worked. Thank you for explaining how the permissions work\n> in this context. I now see that 775 and 664 would work for the apache\n> component and for executing our binaries. Thanks for pointing this\n> out. However, another element of our application is a proprietary\n> runtime that runs on top of linux and runs our core binaries. This\n> allows us to store our binaries in git and deploy them directly on\n> the customer server from git (via git-archive). That runtime needs\n> o+w in order to update the 'last run date' in the binary which is\n> critical to our troubleshooting in the field. o+w is needed because\n> the user's runtime instance runs with user permissions when executed\n> from a linux command line terminal and our users are not setup in the\n> same group as the binaries. Therefore, with tar.umask = 0000 I can\n> deploy 777 and 666 permissions and everything will work.\n>\n> I suppose I could write a script to change the tar.umask entry to\n> 0000 only when running git-archive for the binary portion, and use\n> tar.umask 0002 when extracting the other portions. I could also\n> change our setup to put the users and the runmodules in the same\n> group and use tar.umask 0002 across the board. These would be more\n> correct than the chmod 777 shotgun that we currently use to blast\n> away our permissions problems.\n>\n> git-archive is a \"quick\" solution to our immediate deployment needs.\n> Eventually, I plan on using git on the source and target machines as\n> the core mechanism to \"promote to production\" (ie. deploy to customer\n> servers). It looks like others are using git for deployment also. In\n> my previous shops which used other VCS's on minicomputers and\n> mainframes, \"promote to production\" meant the universal run path for\n> all users (and especially for productional data transactions) on that\n> central machine. In my current shop (my first linux shop) we have\n> multiple concurrent versions of production on a multitude of\n> productional machines and even concurrently on an individual\n> productional machine in some cases. The main reason we chose git is\n> because it is the only VCS that can handle this.\n>\nActually, the apache user (web interface) also needs to be able to\nupdate the binaries with 'last run date'. In this context the o+w allows\nthis, also. I don't know enough about apache and permissions to try and \nadd apache to the same group as the binaries at this point.\n\n-neal\n"}]}