{"thread":{"id":"22665","subject":"Running git-archive recursively over submodules","startedAt":"2010-02-15T20:56:16Z","lastAt":"2010-02-16T10:49:48Z","messageCount":4,"participants":["Thomas Rast","Michael J Gruber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"134663","messageId":"201002152156.17132.trast@student.ethz.ch","threadId":"22665","inReplyTo":null,"subject":"Running git-archive recursively over submodules","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-02-15T20:56:16Z","receivedAt":"2010-02-15T20:56:16Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"I'm posting this for posterity.  It came out of an IRC discussion last\nnight.  Google \"git archive submodules\" currently finds two scripts\nfor the same task:\n\n* http://kerneltrap.org/mailarchive/git/2008/3/29/1294614\n  Can pack arbitrary revisions, but seems to be highly specific to the\n  poster's setup.\n\n* http://wiki.github.com/meitar/git-archive-all.sh/\n  Appears to work with normal repos (I haven't really tested) but only\n  packs HEAD.\n\nBoth have in common that they run git-archive in each submodule, and\nthen merge the resulting tars.\n\nThe script below follows a different approach, based on the idea that\nall the objects are already there, you just need to help git find\nthem.  This of course assumes that you have the submodules checked out\n(or at least cloned).  It builds an index that contains a flattened\nversion of the repository and all submodules.  This works for any\nrevision you specify.  The downside is that this does not respect\n.gitattributes.\n\nThe same trick can be used for other commands, so if people find this\nuseful it could be added to git-submodule.\n\nWarning: Error detection and such are left as an exercise to the user.\n\n-- 8< --\n#!/bin/sh\n\nexport revision=\"$1\"\n\nexport GIT_INDEX_FILE=\".git/tmpindex\"\nrm -f \"$GIT_INDEX_FILE\"\n\ngit read-tree $revision\n\nexport up=\"$(pwd)\"\n\nread_one_level () {\n\texport GIT_ALTERNATE_OBJECT_DIRECTORIES=\"$GIT_ALTERNATE_OBJECT_DIRECTORIES\":$(\n\t    git submodule foreach 'echo \"$up/$path/.git/objects\"' |\n\t    grep -E -v '^(Entering|No submodule mapping found)' |\n\t    tr '\\n' : |\n\t    sed 's/:$//'\n\t)\n\n\tgit submodule foreach '\n\t\tcd \"$up\"\n\t\tsubcommit=$(git rev-parse :\"$path\")\n\t\tgit rm --cached \"$path\"\n\t\tgit read-tree -i --prefix=\"$path/\" $subcommit\n\t' >/dev/null\n}\n\nwhile git ls-files -s | grep -q ^160000; do\n    read_one_level\ndone\n\ngit archive --format=tar $(git write-tree)\n\nrm -f \"$GIT_INDEX_FILE\"\n-- >8 --\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"134714","messageId":"4B7A4B60.3090401@drmicha.warpmail.net","threadId":"22665","inReplyTo":"201002152156.17132.trast@student.ethz.ch","subject":"Re: Running git-archive recursively over submodules","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2010-02-16T07:38:08Z","receivedAt":"2010-02-16T07:38:08Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Thomas Rast venit, vidit, dixit 15.02.2010 21:56:\n> I'm posting this for posterity.  It came out of an IRC discussion last\n> night.  Google \"git archive submodules\" currently finds two scripts\n> for the same task:\n> \n> * http://kerneltrap.org/mailarchive/git/2008/3/29/1294614\n>   Can pack arbitrary revisions, but seems to be highly specific to the\n>   poster's setup.\n> \n> * http://wiki.github.com/meitar/git-archive-all.sh/\n>   Appears to work with normal repos (I haven't really tested) but only\n>   packs HEAD.\n> \n> Both have in common that they run git-archive in each submodule, and\n> then merge the resulting tars.\n> \n> The script below follows a different approach, based on the idea that\n> all the objects are already there, you just need to help git find\n> them.  This of course assumes that you have the submodules checked out\n> (or at least cloned).  It builds an index that contains a flattened\n> version of the repository and all submodules.  This works for any\n> revision you specify.  The downside is that this does not respect\n> .gitattributes.\n> \n> The same trick can be used for other commands, so if people find this\n> useful it could be added to git-submodule.\n> \n> Warning: Error detection and such are left as an exercise to the user.\n> \n> -- 8< --\n> #!/bin/sh\n> \n> export revision=\"$1\"\n> \n> export GIT_INDEX_FILE=\".git/tmpindex\"\n> rm -f \"$GIT_INDEX_FILE\"\n> \n> git read-tree $revision\n> \n> export up=\"$(pwd)\"\n> \n> read_one_level () {\n> \texport GIT_ALTERNATE_OBJECT_DIRECTORIES=\"$GIT_ALTERNATE_OBJECT_DIRECTORIES\":$(\n> \t    git submodule foreach 'echo \"$up/$path/.git/objects\"' |\n> \t    grep -E -v '^(Entering|No submodule mapping found)' |\n> \t    tr '\\n' : |\n> \t    sed 's/:$//'\n> \t)\n> \n> \tgit submodule foreach '\n> \t\tcd \"$up\"\n> \t\tsubcommit=$(git rev-parse :\"$path\")\n> \t\tgit rm --cached \"$path\"\n> \t\tgit read-tree -i --prefix=\"$path/\" $subcommit\n> \t' >/dev/null\n> }\n> \n> while git ls-files -s | grep -q ^160000; do\n>     read_one_level\n> done\n\nAm I blind, not seeing a recursion, or is there simply none? I reckon\nyou're reading the super project and one level of submodules into the\ntemporary index - which is fine for most purposes, of course. I would\nsay this is iterating over the submodules of the super project.\n\n> git archive --format=tar $(git write-tree)\n> \n> rm -f \"$GIT_INDEX_FILE\"\n> -- >8 --\n> \n"},{"id":"134719","messageId":"201002161101.42493.trast@student.ethz.ch","threadId":"22665","inReplyTo":"4B7A4B60.3090401@drmicha.warpmail.net","subject":"Re: Running git-archive recursively over submodules","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-02-16T10:01:42Z","receivedAt":"2010-02-16T10:01:42Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"On Tuesday 16 February 2010 08:38:08 Michael J Gruber wrote:\n> Thomas Rast venit, vidit, dixit 15.02.2010 21:56:\n> > while git ls-files -s | grep -q ^160000; do\n> >     read_one_level\n> > done\n> \n> Am I blind, not seeing a recursion, or is there simply none? I reckon\n> you're reading the super project and one level of submodules into the\n> temporary index - which is fine for most purposes, of course. I would\n> say this is iterating over the submodules of the super project.\n\nWell, the while condition reads \"while there are submodule entries in\nmy index\".  The read_one_level function then replaces those submodules\nwith their corresponding trees, which of course may again contain\nsubmodule entries.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"134725","messageId":"4B7A784C.1070405@drmicha.warpmail.net","threadId":"22665","inReplyTo":"201002161101.42493.trast@student.ethz.ch","subject":"Re: Running git-archive recursively over submodules","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2010-02-16T10:49:48Z","receivedAt":"2010-02-16T10:49:48Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Thomas Rast venit, vidit, dixit 16.02.2010 11:01:\n> On Tuesday 16 February 2010 08:38:08 Michael J Gruber wrote:\n>> Thomas Rast venit, vidit, dixit 15.02.2010 21:56:\n>>> while git ls-files -s | grep -q ^160000; do\n>>>     read_one_level\n>>> done\n>>\n>> Am I blind, not seeing a recursion, or is there simply none? I reckon\n>> you're reading the super project and one level of submodules into the\n>> temporary index - which is fine for most purposes, of course. I would\n>> say this is iterating over the submodules of the super project.\n> \n> Well, the while condition reads \"while there are submodule entries in\n> my index\".  The read_one_level function then replaces those submodules\n> with their corresponding trees, which of course may again contain\n> submodule entries.\n> \n\nGotcha, and the rm --cached in foreach makes sure that submodule entries\ndisappear from the index once their contents have been read into it, so\nthat the recursion stops when there's no deeper level. Clever.\n\nThanks for helping my for/while blindness (I guess I was thinking \"for\").\n\nCheers,\nMichael\n"}]}