{"thread":{"id":"17027","subject":"Problems getting rid of large files using git-filter-branch","startedAt":"2009-01-06T21:59:44Z","lastAt":"2009-01-07T18:18:32Z","messageCount":13,"participants":["Øyvind Harboe","Johannes Schindelin","Nicolas Pitre","Stephen R. van den Berg","Sitaram Chamarty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"99513","messageId":"c09652430901061359q7a02291fk656ab23e54b19f5e@mail.gmail.com","threadId":"17027","inReplyTo":null,"subject":"Problems getting rid of large files using git-filter-branch","fromName":"Øyvind Harboe","fromEmail":"oyvind.harboe@zylin.com","sentAt":"2009-01-06T21:59:44Z","receivedAt":"2009-01-06T21:59:44Z","isPatch":false,"sender":{"key":"oyvind.harboe@zylin.com","avatar":null},"body":"I'm trying to get rid of some large objects in my .git repository\nusing git-filter-branch. These are remnants from conversion from\nCVS.\n\n\nQ1: How can I figure out what it is in .git that takes so much space?\n\nQ2: Where can I read more about what to do after running git-filter-branch to\nremoving the offending objects?\n\n\n\n1. I ran this command to get rid of the offending files and that appears to\nhave worked. I can't find any traces of them anymore...\n\ngit filter-branch --tree-filter 'find . -regex \".*toolchain\\..*\" -exec\nrm -f {} \\;' HEAD\n\n2. Running \"git gc\" takes a few seconds. The repository is still\nhuge(it should be\nperhaps 10-20mByte).\n\ndu -skh .git/\n187M    .git/\n\n3. I tried \"git reflog expire --all\" + lots of other tricks in the\nlink below, but no luck.\n\n\n\nI tried the tricks I could find in this thread, but no luck:\n\nhttp://article.gmane.org/gmane.comp.version-control.git/60219/match=trying+use+git+filter+branch+compress\n\n-- \nØyvind Harboe\nhttp://www.zylin.com/zy1000.html\nARM7 ARM9 XScale Cortex\nJTAG debugger and flash programmer\n"},{"id":"99514","messageId":"alpine.DEB.1.00.0901062319070.30769@pacific.mpi-cbg.de","threadId":"17027","inReplyTo":"c09652430901061359q7a02291fk656ab23e54b19f5e@mail.gmail.com","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-01-06T22:20:41Z","receivedAt":"2009-01-06T22:20:41Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 6 Jan 2009, Øyvind Harboe wrote:\n\n> Q1: How can I figure out what it is in .git that takes so much space?\n\nIf it is a pack that is taking so much space:\n\n$ git verify-pack -v $PACK | grep -v \"^chain \" | sort -n -k 4\n\nand then for the last few lines do a\n\n$ git rev-list --all --objects | grep $SHA1\n\nHth,\nDscho\n"},{"id":"99515","messageId":"alpine.LFD.2.00.0901061709510.26118@xanadu.home","threadId":"17027","inReplyTo":"c09652430901061359q7a02291fk656ab23e54b19f5e@mail.gmail.com","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-01-06T22:31:58Z","receivedAt":"2009-01-06T22:31:58Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 6 Jan 2009, Øyvind Harboe wrote:\n\n> Q1: How can I figure out what it is in .git that takes so much space?\n> \n> Q2: Where can I read more about what to do after running git-filter-branch to\n> removing the offending objects?\n> \n> \n> \n> 1. I ran this command to get rid of the offending files and that appears to\n> have worked. I can't find any traces of them anymore...\n> \n> git filter-branch --tree-filter 'find . -regex \".*toolchain\\..*\" -exec\n> rm -f {} \\;' HEAD\n> \n> 2. Running \"git gc\" takes a few seconds. The repository is still\n> huge(it should be\n> perhaps 10-20mByte).\n> \n> du -skh .git/\n> 187M    .git/\n> \n> 3. I tried \"git reflog expire --all\" + lots of other tricks in the\n> link below, but no luck.\n\nOK, try this:\n\n\tcd ..\n\tmv my_repo my_repo.orig\n\tmkdir my_repo\n\tcd my_repo\n\tgit init\n\tgit pull file://$(pwd)/../my_repo.orig\n\nThis is the easiest way to ensure you have only the necessary objects in \nthe new repo, without all the extra stuff tied to reflogs, etc.\n\nThen, if your repo is still seemingly too big, you can get a bit dirty \nwith the sequence Johannes just posted.\n\n\nNicolas\n"},{"id":"99517","messageId":"c09652430901061436w36c013ep938e9cfba43140c9@mail.gmail.com","threadId":"17027","inReplyTo":"alpine.DEB.1.00.0901062319070.30769@pacific.mpi-cbg.de","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Øyvind Harboe","fromEmail":"oyvind.harboe@zylin.com","sentAt":"2009-01-06T22:36:39Z","receivedAt":"2009-01-06T22:36:39Z","isPatch":false,"sender":{"key":"oyvind.harboe@zylin.com","avatar":null},"body":"On Tue, Jan 6, 2009 at 11:20 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n> On Tue, 6 Jan 2009, Øyvind Harboe wrote:\n>\n>> Q1: How can I figure out what it is in .git that takes so much space?\n>\n> If it is a pack that is taking so much space:\n\nit is.\n\n>\n> $ git verify-pack -v $PACK | grep -v \"^chain \" | sort -n -k 4\n\nI have never used the git verify-pack command, but I'm pretty sure the\n\"Terminated\" string isn't the normal output :-)\n\n$ git verify-pack -v\n.git/objects/pack/pack-1e039b82d8ae53ef5ec3614a3021466663cc70a4\nTerminated\n\nThis is running git version 1.6.1. on CentOS on a virtual machine. I'm not quite\nsure how to debug this. I'm sure I've done something wrong when I installed git.\nI'm just a humble user of git trying to convert from cvs/svn.\n\n\n> and then for the last few lines do a\n>\n> $ git rev-list --all --objects | grep $SHA1\n\nI was able to run this procedure on a different machine than the\nserver and I can\nthen tell which objects take up all the space.\n\nHowever, I'm unnerved by git verify-pack \"Terminated\"'ing on me above\nand I'll have\nto sort that out before I can think about using git in production.\n\n\nThanks for the pointers though! They definitely answered my questions!\n\n>\n> Hth,\n> Dscho\n>\n\n\n\n-- \nØyvind Harboe\nhttp://www.zylin.com/zy1000.html\nARM7 ARM9 XScale Cortex\nJTAG debugger and flash programmer\n"},{"id":"99518","messageId":"c09652430901061441xf18d39fw42d54c6c16c3626e@mail.gmail.com","threadId":"17027","inReplyTo":"alpine.LFD.2.00.0901061709510.26118@xanadu.home","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Øyvind Harboe","fromEmail":"oyvind.harboe@zylin.com","sentAt":"2009-01-06T22:41:58Z","receivedAt":"2009-01-06T22:41:58Z","isPatch":false,"sender":{"key":"oyvind.harboe@zylin.com","avatar":null},"body":">> 3. I tried \"git reflog expire --all\" + lots of other tricks in the\n>> link below, but no luck.\n>\n> OK, try this:\n>\n>        cd ..\n>        mv my_repo my_repo.orig\n>        mkdir my_repo\n>        cd my_repo\n>        git init\n>        git pull file://$(pwd)/../my_repo.orig\n>\n> This is the easiest way to ensure you have only the necessary objects in\n> the new repo, without all the extra stuff tied to reflogs, etc.\n\nSuper!\n\nThat worked!\n\n> Then, if your repo is still seemingly too big, you can get a bit dirty\n> with the sequence Johannes just posted.\n\nJohannes procedure had the unexpected side effect of showing that\nmy server setup is flaky somehow though... :-) I'll need his\ntricks for other situations soon enough.\n\n\n-- \nØyvind Harboe\nhttp://www.zylin.com/zy1000.html\nARM7 ARM9 XScale Cortex\nJTAG debugger and flash programmer\n"},{"id":"99522","messageId":"20090106231726.GB13379@cuci.nl","threadId":"17027","inReplyTo":"alpine.LFD.2.00.0901061709510.26118@xanadu.home","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Stephen R. van den Berg","fromEmail":"srb@cuci.nl","sentAt":"2009-01-06T23:17:26Z","receivedAt":"2009-01-06T23:17:26Z","isPatch":false,"sender":{"key":"srb@cuci.nl","avatar":"https://gravatar.com/avatar/f75389059e827634d38e9df2a9b6ecbd50028b5a454442efa1c7205b7ff29c6a?d=mp&s=160"},"body":"Nicolas Pitre wrote:\n>On Tue, 6 Jan 2009, ?yvind Harboe wrote:\n>OK, try this:\n\n>\tgit pull file://$(pwd)/../my_repo.orig\n\nAlternately, try:\n\nrm -rf .git/ORIG_HEAD .git/FETCH_HEAD .git/index .git/logs .git/info/refs \\\n  .git/objects/pack/pack-*.keep .git/refs/original .git/refs/patches \\\n  .git/patches .git/gitk.cache &&\n git prune --expire now &&\n git repack -a -d --window=200 &&\n git gc\n\n-- \nSincerely,\n           Stephen R. van den Berg.\n\n\"Very funny, Mr. Scott. Now beam down my clothes!\"\n"},{"id":"99526","messageId":"alpine.LFD.2.00.0901061948590.26118@xanadu.home","threadId":"17027","inReplyTo":"20090106231726.GB13379@cuci.nl","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-01-07T00:56:07Z","receivedAt":"2009-01-07T00:56:07Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Jan 2009, Stephen R. van den Berg wrote:\n\n> Nicolas Pitre wrote:\n> >On Tue, 6 Jan 2009, ?yvind Harboe wrote:\n> >OK, try this:\n> \n> >\tgit pull file://$(pwd)/../my_repo.orig\n> \n> Alternately, try:\n> \n> rm -rf .git/ORIG_HEAD .git/FETCH_HEAD .git/index .git/logs .git/info/refs \\\n>   .git/objects/pack/pack-*.keep .git/refs/original .git/refs/patches \\\n>   .git/patches .git/gitk.cache &&\n>  git prune --expire now &&\n>  git repack -a -d --window=200 &&\n>  git gc\n\nThis might not be sufficient.  Or at least you better run 'git prune' at \nthe very end, and possibly add -f to 'git repack'.  And if you somehow \ndelete something you shouldn't have deleted then you're really screwed, \nwhereas the pull method in another repository doesn't alter the original \nrepository in case you need to go back to it and try something \ndifferent.\n\n\nNicolas\n"},{"id":"99558","messageId":"c09652430901070026m6ca0ec98ndc7483aac8dfde89@mail.gmail.com","threadId":"17027","inReplyTo":"c09652430901061359q7a02291fk656ab23e54b19f5e@mail.gmail.com","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Øyvind Harboe","fromEmail":"oyvind.harboe@zylin.com","sentAt":"2009-01-07T08:26:10Z","receivedAt":"2009-01-07T08:26:10Z","isPatch":false,"sender":{"key":"oyvind.harboe@zylin.com","avatar":null},"body":"Here is a summary of the solution I used. I'm a beginner in git\nand just summarizing what others told me and what I did. Use at\nyour own risk!\n\n1. Remove anything you know should be removed, e.g.:\n\ngit filter-branch --tree-filter 'find . -regex \".*toolchain\\..*\" -exec\nrm -f {} \\;' HEAD\n\n2. Expire the log:\n\ngit reflog expire --all\n\n3. Delete stuff from .git that should be manually \"verified\" to be\ncorrect. I don't actually\nknow how to \"verify\" that at this point... Use backups Luke!\n\nrm -rf .git/refs/original\n# delete lines w/\"refs/original\" from .git/packed-refs\nvi .git/packed-refs\n# for good measure...\ngit reflog expire --all\ngit gc\n\n4. Your repository is still huge. By creating a new repository and pulling from\nthis one, the garbage will stay in the old one...\n\nmkdir newrep\ncd newrep\ngit init\ngit pull file:///oldrep\n\n5. Check size of .git. If it is still too big, try figuring out which\nfiles that are big by looking at the packs(.git/objects/pack/xxx):\n\n$ git verify-pack -v $PACK | grep -v \"^chain \" | sort -n -k 4\n\nand then for the last few lines do a\n\n$ git rev-list --all --objects | grep $SHA1\n\n6. Go back to #1 until done.\n\nYour repository should now be of reasonable size...\n\nI've found some great scripts for converting from svn/cvs, but really\nthe above procedure\nis necessary to run when converting nasty old repositories...\n\n-- \nØyvind Harboe\nhttp://www.zylin.com/zy1000.html\nARM7 ARM9 XScale Cortex\nJTAG debugger and flash programmer\n"},{"id":"99564","messageId":"alpine.DEB.1.00.0901071101480.7496@intel-tinevez-2-302","threadId":"17027","inReplyTo":"c09652430901061436w36c013ep938e9cfba43140c9@mail.gmail.com","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-01-07T10:07:14Z","receivedAt":"2009-01-07T10:07:14Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 6 Jan 2009, Øyvind Harboe wrote:\n\n> On Tue, Jan 6, 2009 at 11:20 PM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n>\n> > $ git verify-pack -v $PACK | grep -v \"^chain \" | sort -n -k 4\n> \n> I have never used the git verify-pack command, but I'm pretty sure the\n> \"Terminated\" string isn't the normal output :-)\n> \n> $ git verify-pack -v\n> .git/objects/pack/pack-1e039b82d8ae53ef5ec3614a3021466663cc70a4\n> Terminated\n\nI did\n\n\t$ git grep Terminated\n\nand came up empty :-)\n\nSeriously, I guess this could be some OOM thing.  We _should_ handle this \nmore gracefully, but it is possible that some uncatchable condition hits \nyou, such as out-of-stack-space.\n\nI'd try running the command either with strace or with gdb, and I'd look \nat $? after the command returns, to find out what is actually happening.\n\nHth,\nDscho\n"},{"id":"99565","messageId":"c09652430901070215p436db79boae4c56bfa1afbc1a@mail.gmail.com","threadId":"17027","inReplyTo":"alpine.DEB.1.00.0901071101480.7496@intel-tinevez-2-302","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Øyvind Harboe","fromEmail":"oyvind.harboe@zylin.com","sentAt":"2009-01-07T10:15:53Z","receivedAt":"2009-01-07T10:15:53Z","isPatch":false,"sender":{"key":"oyvind.harboe@zylin.com","avatar":null},"body":"On Wed, Jan 7, 2009 at 11:07 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n> On Tue, 6 Jan 2009, Øyvind Harboe wrote:\n>\n>> On Tue, Jan 6, 2009 at 11:20 PM, Johannes Schindelin\n>> <Johannes.Schindelin@gmx.de> wrote:\n>>\n>> > $ git verify-pack -v $PACK | grep -v \"^chain \" | sort -n -k 4\n>>\n>> I have never used the git verify-pack command, but I'm pretty sure the\n>> \"Terminated\" string isn't the normal output :-)\n>>\n>> $ git verify-pack -v\n>> .git/objects/pack/pack-1e039b82d8ae53ef5ec3614a3021466663cc70a4\n>> Terminated\n>\n> I did\n>\n>        $ git grep Terminated\n>\n> and came up empty :-)\n>\n> Seriously, I guess this could be some OOM thing.  We _should_ handle this\n> more gracefully, but it is possible that some uncatchable condition hits\n> you, such as out-of-stack-space.\n>\n> I'd try running the command either with strace or with gdb, and I'd look\n> at $? after the command returns, to find out what is actually happening.\n\nAfter some investigation it turns out that my server has 228mByte of RAM\navailable. It is a virtual server running CentOS, hence the strange number\nand ridiciulously tiny amount of memory(these days).\n\nNow the strange thing is that I'm not getting this error message this morning...\n\nHow would git behave if it ran out of memory?\n\n\n-- \nØyvind Harboe\nhttp://www.zylin.com/zy1000.html\nARM7 ARM9 XScale Cortex\nJTAG debugger and flash programmer\n"},{"id":"99576","messageId":"alpine.DEB.1.00.0901071344360.7496@intel-tinevez-2-302","threadId":"17027","inReplyTo":"c09652430901070215p436db79boae4c56bfa1afbc1a@mail.gmail.com","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-01-07T12:45:11Z","receivedAt":"2009-01-07T12:45:11Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 7 Jan 2009, Øyvind Harboe wrote:\n\n> How would git behave if it ran out of memory?\n\nSomething like\n\n\tfatal: Out of memory, malloc failed\n\nCiao,\nDscho\n"},{"id":"99585","messageId":"alpine.LFD.2.00.0901070955500.26118@xanadu.home","threadId":"17027","inReplyTo":"c09652430901070026m6ca0ec98ndc7483aac8dfde89@mail.gmail.com","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-01-07T15:02:49Z","receivedAt":"2009-01-07T15:02:49Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Jan 2009, Øyvind Harboe wrote:\n\n> Here is a summary of the solution I used. I'm a beginner in git\n> and just summarizing what others told me and what I did. Use at\n> your own risk!\n> \n> 1. Remove anything you know should be removed, e.g.:\n> \n> git filter-branch --tree-filter 'find . -regex \".*toolchain\\..*\" -exec\n> rm -f {} \\;' HEAD\n> \n> 2. Expire the log:\n> \n> git reflog expire --all\n> \n> 3. Delete stuff from .git that should be manually \"verified\" to be\n> correct. I don't actually\n> know how to \"verify\" that at this point... Use backups Luke!\n> \n> rm -rf .git/refs/original\n> # delete lines w/\"refs/original\" from .git/packed-refs\n> vi .git/packed-refs\n> # for good measure...\n> git reflog expire --all\n> git gc\n> \n> 4. Your repository is still huge. By creating a new repository and pulling from\n> this one, the garbage will stay in the old one...\n> \n> mkdir newrep\n> cd newrep\n> git init\n> git pull file:///oldrep\n\nI'd suggest you skip 2 and 3, and do 4 only.  Using 4 makes 2 \nunnecessary, and is far safer than 3.  Manually deleting stuff in .git \nis fine only if you really know what you're doing and have some \nacquaintance with the git internals.\n\n> 5. Check size of .git. If it is still too big, try figuring out which\n> files that are big by looking at the packs(.git/objects/pack/xxx):\n> \n> $ git verify-pack -v $PACK | grep -v \"^chain \" | sort -n -k 4\n> \n> and then for the last few lines do a\n> \n> $ git rev-list --all --objects | grep $SHA1\n> \n> 6. Go back to #1 until done.\n> \n> Your repository should now be of reasonable size...\n> \n> I've found some great scripts for converting from svn/cvs, but really\n> the above procedure\n> is necessary to run when converting nasty old repositories...\n> \n> -- \n> Øyvind Harboe\n> http://www.zylin.com/zy1000.html\n> ARM7 ARM9 XScale Cortex\n> JTAG debugger and flash programmer\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n> \n"},{"id":"99599","messageId":"slrngm9sfo.hl9.sitaramc@sitaramc.homelinux.net","threadId":"17027","inReplyTo":"alpine.DEB.1.00.0901071101480.7496@intel-tinevez-2-302","subject":"Re: Problems getting rid of large files using git-filter-branch","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2009-01-07T18:18:32Z","receivedAt":"2009-01-07T18:18:32Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On 2009-01-07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n>> $ git verify-pack -v\n>> .git/objects/pack/pack-1e039b82d8ae53ef5ec3614a3021466663cc70a4\n>> Terminated\n>\n> I did\n>\n> \t$ git grep Terminated\n>\n> and came up empty :-)\n\nIt comes from libc, afaik.\n"}]}