{"thread":{"id":"36070","subject":"Unable to shrink repository size","startedAt":"2014-03-06T02:55:30Z","lastAt":"2014-03-06T15:25:50Z","messageCount":5,"participants":["Robert Dailey","Elijah Newren","Fredrik Gustafsson","Duy Nguyen","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"236146","messageId":"CAHd499AW6nev81iVVhuoYfT0us28SSBDwbHCBa3teYB=cJR99g@mail.gmail.com","threadId":"36070","inReplyTo":null,"subject":"Unable to shrink repository size","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2014-03-06T02:55:30Z","receivedAt":"2014-03-06T02:55:30Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"I have a git-svn clone that I've been working on which is a full and\ncomplete conversion of our SVN repository at work.\n\nIt started out as 1.4GB (git count-objects -v, looking at\n'size-pack'). I have run the following script to clean up a directory\nin the repo history that I suspect are huge (we had a third party\nlibrary checked in that, uncompressed, was about 1.2GB in size):\n\n-------------------------------\nfiles=$@\necho \"Removing: $files...\"\ngit filter-branch --index-filter \"git rm -rf --cached --ignore-unmatch\n$files\" -- --all\n\n# remove the temporary history git-filter-branch otherwise leaves\nbehind for a long time\nrm -rf .git/refs/original/ && git reflog expire --expire=now --all &&\ngit gc --aggressive --prune=now\n-------------------------------\n\nEven though I seem to have removed it, the repository size (looking at\n'size-pack' again) only went down about 200MB, so it's at 1.2GB now.\nThere is about 3-5 years of commit history in this repository.\n\nWhat I'd like to do is somehow hunt down the largest commit (*not*\nblob) in the entire history of the repository to hopefully find out\nwhere huge directories have been checked in.\n\nI can't do a search for largest file (which most google results seem\nto show to do) since the culprit is really thousands of unnecessary\nfiles checked into a single subdirectory somewhere in history.\n\nCan anyone offer me some advice to help me reduce the size of my repo\nfurther? Thanks.\n"},{"id":"236147","messageId":"CABPp-BGPPj63zLjjp5R6+vpJr3nLX_aSCz34qCf2E8tm8J7EDQ@mail.gmail.com","threadId":"36070","inReplyTo":"CAHd499AW6nev81iVVhuoYfT0us28SSBDwbHCBa3teYB=cJR99g@mail.gmail.com","subject":"Re: Unable to shrink repository size","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2014-03-06T05:21:23Z","receivedAt":"2014-03-06T05:21:23Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Mar 5, 2014 at 6:55 PM, Robert Dailey wrote:\n> What I'd like to do is somehow hunt down the largest commit (*not*\n> blob) in the entire history of the repository to hopefully find out\n> where huge directories have been checked in.\n>\n> I can't do a search for largest file (which most google results seem\n> to show to do) since the culprit is really thousands of unnecessary\n> files checked into a single subdirectory somewhere in history.\n>\n> Can anyone offer me some advice to help me reduce the size of my repo\n> further? Thanks.\n\nIf you are trying to see which commits changed the most files (and/or\nlines), you may find\n git log --oneline --shortstat\nhelpful. A quick search through that output may help you decide where\nto dig further.\n\n\nAlso, I'm sure someone else here probably has a better idea, but\nhere's a quick hack I came up with to look at \"commit sizes\":\n\nmkdir tmp && for i in $(git rev-list --all); do git branch -f dummy $i\n&& git bundle create tmp/$i.bundle dummy --not dummy^@; done\n\nFollow that up with a quick \"ls -alrSh tmp/\" and you can see the\n\"size\" of each commit.\n\n\nHope that helps,\nElijah\n"},{"id":"236149","messageId":"20140306074610.GO24892@paksenarrion.iveqy.com","threadId":"36070","inReplyTo":"CAHd499AW6nev81iVVhuoYfT0us28SSBDwbHCBa3teYB=cJR99g@mail.gmail.com","subject":"Re: Unable to shrink repository size","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2014-03-06T07:46:10Z","receivedAt":"2014-03-06T07:46:10Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Wed, Mar 05, 2014 at 08:55:30PM -0600, Robert Dailey wrote:\n> What I'd like to do is somehow hunt down the largest commit (*not*\n> blob) in the entire history of the repository to hopefully find out\n> where huge directories have been checked in.\n> \n> I can't do a search for largest file (which most google results seem\n> to show to do) since the culprit is really thousands of unnecessary\n> files checked into a single subdirectory somewhere in history.\n> \n> Can anyone offer me some advice to help me reduce the size of my repo\n> further? Thanks.\n\nI'm not sure if you can do this with git. However since git is a command\nline application it's pretty easy to script it with sh. The negative\npart beeing the lack of speed, but since this is a one-time thing I\ndon't think that it matters.\n\nSince you told us that it is a commit with a huge number of files that\nyou're looking for, I took that approach instead of calculating the size\nof each commit, since that would be more expensive to do.\n\nfor commit in $(git log --pretty=oneline | cut -d \" \" -f 1)\ndo\n\tnbr=$(git show --pretty=\"format:\" --name-only $commit | wc -l)\n\techo \"$nbr: $commit\"\ndone | sort | tail -1\n\nThis will give you the commit with most files changed. (Although, there\nwill be a +1 error on the number of files edited).\n\n-- \nMed vänlig hälsning\nFredrik Gustafsson\n\ntel: 0733-608274\ne-post: iveqy@iveqy.com\n"},{"id":"236158","messageId":"CACsJy8AcQAYVTOr5u-VFJ4VkOGNMuOK39gJc-EPZvK+W0OFYsw@mail.gmail.com","threadId":"36070","inReplyTo":"CAHd499AW6nev81iVVhuoYfT0us28SSBDwbHCBa3teYB=cJR99g@mail.gmail.com","subject":"Re: Unable to shrink repository size","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-03-06T12:56:32Z","receivedAt":"2014-03-06T12:56:32Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 6, 2014 at 9:55 AM, Robert Dailey wrote:\n> What I'd like to do is somehow hunt down the largest commit (*not*\n> blob) in the entire history of the repository to hopefully find out\n> where huge directories have been checked in.\n\nAnother try\n\ngit rev-list --all|while read i;do echo -n \"$i \"; git ls-tree -lr\n$i|awk '{a+=$4;} END {print a;}';done\n\nYou may need to feed it to gnuplot or something to see steep slopes.\n-- \nDuy\n"},{"id":"236166","messageId":"20140306152550.GA18519@sigill.intra.peff.net","threadId":"36070","inReplyTo":"CAHd499AW6nev81iVVhuoYfT0us28SSBDwbHCBa3teYB=cJR99g@mail.gmail.com","subject":"Re: Unable to shrink repository size","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-03-06T15:25:50Z","receivedAt":"2014-03-06T15:25:50Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 05, 2014 at 08:55:30PM -0600, Robert Dailey wrote:\n\n> What I'd like to do is somehow hunt down the largest commit (*not*\n> blob) in the entire history of the repository to hopefully find out\n> where huge directories have been checked in.\n> \n> I can't do a search for largest file (which most google results seem\n> to show to do) since the culprit is really thousands of unnecessary\n> files checked into a single subdirectory somewhere in history.\n\nOther people have offered scripts to look at commit sizes. But it might\nalso be useful to see sizes by subdirectory. Sort of a \"du\" across all\nof history. Script is below.\n\nNote that this script also uses cat-file's \"%(objectsize:disk)\". So it\nis finding the actual on-disk storage, taking into account delta\nstorage. You will need git v1.8.5 or later to use this feature.\n\n git rev-list --objects --all |\n git cat-file --batch-check=\"%(objectsize:disk) %(rest)\" |\n perl -lne '\n my ($size, $path) = split / /, $_, 2;\n next unless defined $path; # commit obj\n do {\n $sizes{$path} += $size;\n } while ($path =~ s{/[^/]+$}{});\n\n END { print \"$sizes{$_} $_\" for (keys %sizes) }\n ' |\n sort -rn\n\n-Peff\n"}]}