{"thread":{"id":"31787","subject":"filter-branch IO optimization","startedAt":"2012-10-11T15:39:47Z","lastAt":"2012-10-12T17:20:36Z","messageCount":7,"participants":["Enrico Weigelt","Johannes Sixt","Thomas Rast","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"200990","messageId":"fa1e05a5-54b3-47ff-bd28-dc463ebbc4bd@zcs","threadId":"31787","inReplyTo":"7e000a0f-9e4e-4a4d-a8ce-5d017e17939c@zcs","subject":"filter-branch IO optimization","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-10-11T15:39:47Z","receivedAt":"2012-10-11T15:39:47Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"Hi folks,\n\nfor certain projects, I need to regularily run filter-branch on quite\nlarge repos (>10k commits), and that needs to be run multiple times,\nwhich takes several hours, so I'm looking for optimizations.\n\nThe main goal of this filtering is splitting out many modules from a\nlarge upstream repo into their own downstream repos. This process\nshould be fully deterministic (IOW: running it twice at the same input,\nshould produce exactly same output, so commit IDs stay the same after\nsubsequent runs)\n\nMy current approach is most likely yet a bit too naive:\n\n#1: forkoff new branch from current upstream\n#2: run a tree-filter which:\n    * removes all files not belonging to the wanted module\n    * move the module directory under another subdir (./addons/)\n    * fix author/comitter name/email if empty (because otherwise fails)\n    * fix charater sets and indentions of source files\n#3: loop through `git filter-branch --prune-empty` to get rid of empty\n    merge nodes (which otherwise remain really a lot), until branch\n    remains unchanged\n#4: run plain rebase onto initial commit to linearize the history\n\nAll that is done is on per-module basis (for now only about 10,\nbut soon can become much more).\n\nOne thing I haven't tried yet is using the -d option to move the .git-rewrite\ndir to an tmpfs (have to clarify some operating considerations first) ;-o\n\nThe next step I have in mind is using --subdirectory-filter, but open\nquestsions are:\n\n* does it suffer from the same problems w/ empty username/email like --tree-filter ?\n** if yes: what can I do about it (have an additional pass for fixing that before\n   running the --tree-filter ?\n* can I somehow teach the --subdirectory filter to place the result under some\n  somedir instead of directly to root ?\n* can I use --tree-filter in combination with --subdireectory-filter ? \n  which one is executed first ?\n\n\nthanks\n-- \nMit freundlichen Grüßen / Kind regards \n\nEnrico Weigelt \nVNC - Virtual Network Consult GmbH \nHead Of Development \n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de \n"},{"id":"201012","messageId":"5077119E.4060702@kdbg.org","threadId":"31787","inReplyTo":"fa1e05a5-54b3-47ff-bd28-dc463ebbc4bd@zcs","subject":"Re: filter-branch IO optimization","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2012-10-11T18:36:14Z","receivedAt":"2012-10-11T18:36:14Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 11.10.2012 17:39, schrieb Enrico Weigelt:\n> The main goal of this filtering is splitting out many modules from a\n> large upstream repo into their own downstream repos.\n...\n> The next step I have in mind is using --subdirectory-filter, but open\n> questsions are:\n> \n> * does it suffer from the same problems w/ empty username/email like --tree-filter ?\n\nI think so.\n\n> ** if yes: what can I do about it (have an additional pass for fixing that before\n>    running the --tree-filter ?\n\nUse --env-filter.\n\n> * can I somehow teach the --subdirectory filter to place the result under some\n>   somedir instead of directly to root ?\n\nNo, but see the last example in the man page.\n\n> * can I use --tree-filter in combination with --subdireectory-filter ? \n>   which one is executed first ?\n\nYes. --subdirectory-filter applies first.\n\n-- Hannes\n"},{"id":"201021","messageId":"878vbc21f3.fsf@pctrast.inf.ethz.ch","threadId":"31787","inReplyTo":"fa1e05a5-54b3-47ff-bd28-dc463ebbc4bd@zcs","subject":"Re: filter-branch IO optimization","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-10-11T20:34:24Z","receivedAt":"2012-10-11T20:34:24Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Enrico Weigelt <enrico.weigelt@vnc.biz> writes:\n\n> for certain projects, I need to regularily run filter-branch on quite\n> large repos (>10k commits), and that needs to be run multiple times,\n> which takes several hours, so I'm looking for optimizations.\n[...]\n> #2: run a tree-filter which:\n>     * removes all files not belonging to the wanted module\n>     * move the module directory under another subdir (./addons/)\n>     * fix author/comitter name/email if empty (because otherwise fails)\n\nThe usual advice is \"use an index-filter instead\".  It's *much* faster\nthan a tree filter.  However:\n\n>     * fix charater sets and indentions of source files\n\nThat last step is rather crazy.  At the very least you will want to only\noperate on files that were changed since the parent commit, so as to\navoid scanning the whole tree.  If you do this right, it should also fit\ninto an index-filter.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"201057","messageId":"9de87aac-7e15-48d9-832a-e9a030c5ed43@zcs","threadId":"31787","inReplyTo":"878vbc21f3.fsf@pctrast.inf.ethz.ch","subject":"Re: filter-branch IO optimization","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-10-12T14:49:54Z","receivedAt":"2012-10-12T14:49:54Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"Hi,\n\n> The usual advice is \"use an index-filter instead\".  It's *much*\n> faster\n> than a tree filter.  However:\n\nI've tried the last example from git-filter-branch manpage, but failed.\nSeems like the GIT_INDEX_FILE env variable doesnt get honoured by\ngit-update-index, no index.new file created, and so mv call fails.\n\nMy second try (as index-filter command) was:\n\ngit ls-files -s > ../_INDEX_TMP\ncat ../_INDEX_TMP |\n    sed \"s-\\t\\\"*-&addons/-\" |\n    git update-index --index-info\nrm -f ../_INDEX_TMP\n\nIt works fine in the worktree (i see files renamed in the index),\nbut no success when running it as --index-filter. Seems the index\nfile isn't used at all (or some completely different one).\n\nBy the way, inside the index filter, GIT_INDEX_FILTER here is\n\n/home/devel/vnc/openerp/workspace/pkg/openerp-extra-bundle.git/.git-rewrite/t/../index\n\nObviously a different (temporary) index file, while many examples\non the web, suggesting to use commands like 'git add --cached' or\n'git rm --cached' _without_ passing GIT_INDEX_FILTER variable.\n\nCould there be some bug that this variable isn't honored properly\neverywhere ?\n\n--\nMit freundlichen Grüßen / Kind regards\n\nEnrico Weigelt\nVNC - Virtual Network Consult GmbH\nHead Of Development\n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de\n"},{"id":"201061","messageId":"d4a00074-5134-4314-aa61-f222f41712bb@zcs","threadId":"31787","inReplyTo":"9de87aac-7e15-48d9-832a-e9a030c5ed43@zcs","subject":"Re: filter-branch IO optimization","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-10-12T15:59:44Z","receivedAt":"2012-10-12T15:59:44Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"<snip>\n\nDid some more experiments, and it seems that missing index file\nisn't automatically created.\n\nWhen I instead copy the original index file to the temporary\nlocation, it runs well. But I still have to wait for the final\nresult to check whether it really overwrites the whole index\nor just adds new files.\n\n\ncu\n-- \nMit freundlichen Grüßen / Kind regards \n\nEnrico Weigelt \nVNC - Virtual Network Consult GmbH \nHead Of Development \n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de \n"},{"id":"201069","messageId":"b94baafd-3813-49c6-9848-97bf11960bb9@zcs","threadId":"31787","inReplyTo":"d4a00074-5134-4314-aa61-f222f41712bb@zcs","subject":"Re: filter-branch IO optimization","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-10-12T17:20:23Z","receivedAt":"2012-10-12T17:20:23Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"Hi folks,\n\nnow finally managed the index-filter part.\nThe main problem, IIRC, was that git-update-index didn't\nautomatically create an empty index, so I needed to explicitly\ncopy in (manually created it with an empty repo).\n\nMy current filter code is:\n\nif [ ! \"$GIT_AUTHOR_EMAIL\" ] && [ ! \"$GIT_COMMITTER_EMAIL\" ]; then\n\texport GIT_AUTHOR_EMAIL=\"nobody@none.org\"\n\texport GIT_COMMITTER_NAME=\"nobody@none.org\"\nelif [ ! \"$GIT_AUTHOR_EMAIL\" ]; then\n\texport GIT_AUTHOR_EMAIL=\"$GIT_COMMITTER_EMAIL\"\nelif [ ! \"$GIT_COMITTER_EMAIL\" ]; then\n\texport GIT_COMMITTER_EMAIL=\"$GIT_AUTHOR_NAME\"\nfi\n\nif [ ! \"$GIT_AUTHOR_NAME\" ] && [ ! \"$GIT_COMMITTER_NAME\" ]; then\n\texport GIT_AUTHOR_NAME=\"nobody@none.org\"\n\texport GIT_COMMITTER_NAME=\"nobody@none.org\"\nelif [ ! \"$GIT_AUTHOR_NAME\" ]; then\n\texport GIT_AUTHOR_NAME=\"$GIT_COMMITTER_NAME\"\nelif [ ! \"$GIT_COMITTER_NAME\" ]; then\n\texport GIT_COMMITTER_NAME=\"$GIT_AUTHOR_NAME\"\nfi\n\ncp ../../../../scripts/index.empty $GIT_INDEX_FILE.new\n\ngit ls-files -s |\n    sed \"s-\\t\\\"*-&addons/-\" |\n    grep -e \"\\t*addons/$module\" |\n    ( export GIT_INDEX_FILE=$GIT_INDEX_FILE.new ; git update-index --index-info )\n\nmv $GIT_INDEX_FILE.new $GIT_INDEX_FILE\n\n\nNow another problem: this leaves behind thousands of now empty\nmerge nodes (--prune-empty doesnt seem to catch them all),\nso I loop through additional `git filter-branch --prune-empty`\nruns, until the ref remains unchanged.\n\nThis process is even more time-consuming, as it takes really many\npasses (havent counted them yet).\n\nDoes anyone have an idea, why a single run doesnt catch that all?\n\n\ncu\n"},{"id":"201070","messageId":"20121012172036.GB21409@sigill.intra.peff.net","threadId":"31787","inReplyTo":"9de87aac-7e15-48d9-832a-e9a030c5ed43@zcs","subject":"Re: filter-branch IO optimization","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-10-12T17:20:36Z","receivedAt":"2012-10-12T17:20:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 12, 2012 at 04:49:54PM +0200, Enrico Weigelt wrote:\n\n> > The usual advice is \"use an index-filter instead\".  It's *much*\n> > faster\n> > than a tree filter.  However:\n> \n> I've tried the last example from git-filter-branch manpage, but failed.\n> Seems like the GIT_INDEX_FILE env variable doesnt get honoured by\n> git-update-index, no index.new file created, and so mv call fails.\n> \n> My second try (as index-filter command) was:\n> \n> git ls-files -s > ../_INDEX_TMP\n> cat ../_INDEX_TMP |\n>     sed \"s-\\t\\\"*-&addons/-\" |\n>     git update-index --index-info\n> rm -f ../_INDEX_TMP\n\nI didn't look closely at your individual problem, but that example has\nproven flaky before.  There were some simpler formulations given in this\nthread:\n\n  http://thread.gmane.org/gmane.comp.version-control.git/195492\n\nIn particular, Junio suggested:\n\n  git filter-branch --index-filter '\n    rm -f \"$GIT_INDEX_FILE\"\n    git read-tree --prefix=newsubdir/ \"$GIT_COMMIT\"\n  ' HEAD\n\n-Peff\n"}]}