# Unable to shrink repository size 5 messages from 2014-03-06 to 2014-03-06. Participants: Robert Dailey, Elijah Newren, Fredrik Gustafsson, Duy Nguyen, Jeff King. Thread: https://gitlist.dev/t/36070 ## Robert Dailey, 2014-03-06 02:55 Subject: Unable to shrink repository size Message-ID: URL: https://gitlist.dev/e/CAHd499AW6nev81iVVhuoYfT0us28SSBDwbHCBa3teYB%3DcJR99g%40mail.gmail.com ``` I have a git-svn clone that I've been working on which is a full and complete conversion of our SVN repository at work. It started out as 1.4GB (git count-objects -v, looking at 'size-pack'). I have run the following script to clean up a directory in the repo history that I suspect are huge (we had a third party library checked in that, uncompressed, was about 1.2GB in size): ------------------------------- files=$@ echo "Removing: $files..." git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch $files" -- --all # remove the temporary history git-filter-branch otherwise leaves behind for a long time rm -rf .git/refs/original/ && git reflog expire --expire=now --all && git gc --aggressive --prune=now ------------------------------- Even though I seem to have removed it, the repository size (looking at 'size-pack' again) only went down about 200MB, so it's at 1.2GB now. There is about 3-5 years of commit history in this repository. What I'd like to do is somehow hunt down the largest commit (*not* blob) in the entire history of the repository to hopefully find out where huge directories have been checked in. I can't do a search for largest file (which most google results seem to show to do) since the culprit is really thousands of unnecessary files checked into a single subdirectory somewhere in history. Can anyone offer me some advice to help me reduce the size of my repo further? Thanks. ``` ## Elijah Newren, 2014-03-06 05:21 Subject: Re: Unable to shrink repository size Message-ID: URL: https://gitlist.dev/e/CABPp-BGPPj63zLjjp5R6%2BvpJr3nLX_aSCz34qCf2E8tm8J7EDQ%40mail.gmail.com In-Reply-To: ``` On Wed, Mar 5, 2014 at 6:55 PM, Robert Dailey wrote: > What I'd like to do is somehow hunt down the largest commit (*not* > blob) in the entire history of the repository to hopefully find out > where huge directories have been checked in. > > I can't do a search for largest file (which most google results seem > to show to do) since the culprit is really thousands of unnecessary > files checked into a single subdirectory somewhere in history. > > Can anyone offer me some advice to help me reduce the size of my repo > further? Thanks. If you are trying to see which commits changed the most files (and/or lines), you may find git log --oneline --shortstat helpful. A quick search through that output may help you decide where to dig further. Also, I'm sure someone else here probably has a better idea, but here's a quick hack I came up with to look at "commit sizes": mkdir tmp && for i in $(git rev-list --all); do git branch -f dummy $i && git bundle create tmp/$i.bundle dummy --not dummy^@; done Follow that up with a quick "ls -alrSh tmp/" and you can see the "size" of each commit. Hope that helps, Elijah ``` ## Fredrik Gustafsson, 2014-03-06 07:46 Subject: Re: Unable to shrink repository size Message-ID: <20140306074610.GO24892@paksenarrion.iveqy.com> URL: https://gitlist.dev/e/20140306074610.GO24892%40paksenarrion.iveqy.com In-Reply-To: ``` On Wed, Mar 05, 2014 at 08:55:30PM -0600, Robert Dailey wrote: > What I'd like to do is somehow hunt down the largest commit (*not* > blob) in the entire history of the repository to hopefully find out > where huge directories have been checked in. > > I can't do a search for largest file (which most google results seem > to show to do) since the culprit is really thousands of unnecessary > files checked into a single subdirectory somewhere in history. > > Can anyone offer me some advice to help me reduce the size of my repo > further? Thanks. I'm not sure if you can do this with git. However since git is a command line application it's pretty easy to script it with sh. The negative part beeing the lack of speed, but since this is a one-time thing I don't think that it matters. Since you told us that it is a commit with a huge number of files that you're looking for, I took that approach instead of calculating the size of each commit, since that would be more expensive to do. for commit in $(git log --pretty=oneline | cut -d " " -f 1) do nbr=$(git show --pretty="format:" --name-only $commit | wc -l) echo "$nbr: $commit" done | sort | tail -1 This will give you the commit with most files changed. (Although, there will be a +1 error on the number of files edited). -- Med vänlig hälsning Fredrik Gustafsson tel: 0733-608274 e-post: iveqy@iveqy.com ``` ## Duy Nguyen, 2014-03-06 12:56 Subject: Re: Unable to shrink repository size Message-ID: URL: https://gitlist.dev/e/CACsJy8AcQAYVTOr5u-VFJ4VkOGNMuOK39gJc-EPZvK%2BW0OFYsw%40mail.gmail.com In-Reply-To: ``` On Thu, Mar 6, 2014 at 9:55 AM, Robert Dailey wrote: > What I'd like to do is somehow hunt down the largest commit (*not* > blob) in the entire history of the repository to hopefully find out > where huge directories have been checked in. Another try git rev-list --all|while read i;do echo -n "$i "; git ls-tree -lr $i|awk '{a+=$4;} END {print a;}';done You may need to feed it to gnuplot or something to see steep slopes. -- Duy ``` ## Jeff King, 2014-03-06 15:25 Subject: Re: Unable to shrink repository size Message-ID: <20140306152550.GA18519@sigill.intra.peff.net> URL: https://gitlist.dev/e/20140306152550.GA18519%40sigill.intra.peff.net In-Reply-To: ``` On Wed, Mar 05, 2014 at 08:55:30PM -0600, Robert Dailey wrote: > What I'd like to do is somehow hunt down the largest commit (*not* > blob) in the entire history of the repository to hopefully find out > where huge directories have been checked in. > > I can't do a search for largest file (which most google results seem > to show to do) since the culprit is really thousands of unnecessary > files checked into a single subdirectory somewhere in history. Other people have offered scripts to look at commit sizes. But it might also be useful to see sizes by subdirectory. Sort of a "du" across all of history. Script is below. Note that this script also uses cat-file's "%(objectsize:disk)". So it is finding the actual on-disk storage, taking into account delta storage. You will need git v1.8.5 or later to use this feature. git rev-list --objects --all | git cat-file --batch-check="%(objectsize:disk) %(rest)" | perl -lne ' my ($size, $path) = split / /, $_, 2; next unless defined $path; # commit obj do { $sizes{$path} += $size; } while ($path =~ s{/[^/]+$}{}); END { print "$sizes{$_} $_" for (keys %sizes) } ' | sort -rn -Peff ```