Unable to shrink repository size

I have a git-svn clone that I've been working on which is a full and complete conversion of our SVN repository at work.

It started out as 1.4GB (git count-objects -v, looking at 'size-pack'). I have run the following script to clean up a directory in the repo history that I suspect are huge (we had a third party library checked in that, uncompressed, was about 1.2GB in size):

------------------------------- files=$@ echo "Removing: $files..." git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch $files" -- --all

# remove the temporary history git-filter-branch otherwise leaves behind for a long time rm -rf .git/refs/original/ && git reflog expire --expire=now --all && git gc --aggressive --prune=now -------------------------------

Even though I seem to have removed it, the repository size (looking at 'size-pack' again) only went down about 200MB, so it's at 1.2GB now. There is about 3-5 years of commit history in this repository.

What I'd like to do is somehow hunt down the largest commit (*not* blob) in the entire history of the repository to hopefully find out where huge directories have been checked in.

I can't do a search for largest file (which most google results seem to show to do) since the culprit is really thousands of unnecessary files checked into a single subdirectory somewhere in history.

Can anyone offer me some advice to help me reduce the size of my repo further? Thanks.

On Wed, Mar 5, 2014 at 6:55 PM, Robert Dailey <rcdailey.lists@gmail.com> wrote:

If you are trying to see which commits changed the most files (and/or
lines), you may find
   git log --oneline --shortstat
helpful.  A quick search through that output may help you decide where
to dig further.

Also, I'm sure someone else here probably has a better idea, but here's a quick hack I came up with to look at "commit sizes":

mkdir tmp && for i in $(git rev-list --all); do git branch -f dummy $i && git bundle create tmp/$i.bundle dummy --not dummy^@; done

Follow that up with a quick "ls -alrSh tmp/" and you can see the "size" of each commit.

Hope that helps, Elijah

On Wed, Mar 05, 2014 at 08:55:30PM -0600, Robert Dailey wrote:

I'm not sure if you can do this with git. However since git is a command line application it's pretty easy to script it with sh. The negative part beeing the lack of speed, but since this is a one-time thing I don't think that it matters.

Since you told us that it is a commit with a huge number of files that you're looking for, I took that approach instead of calculating the size of each commit, since that would be more expensive to do.

for commit in $(git log --pretty=oneline | cut -d " " -f 1)
do
	nbr=$(git show --pretty="format:" --name-only $commit | wc -l)
	echo "$nbr: $commit"
done | sort | tail -1

This will give you the commit with most files changed. (Although, there will be a +1 error on the number of files edited).

On Thu, Mar 6, 2014 at 9:55 AM, Robert Dailey <rcdailey.lists@gmail.com> wrote:

Another try

git rev-list --all|while read i;do echo -n "$i "; git ls-tree -lr $i|awk '{a+=$4;} END {print a;}';done

You may need to feed it to gnuplot or something to see steep slopes.

On Wed, Mar 05, 2014 at 08:55:30PM -0600, Robert Dailey wrote:

Other people have offered scripts to look at commit sizes. But it might also be useful to see sizes by subdirectory. Sort of a "du" across all of history. Script is below.

Note that this script also uses cat-file's "%(objectsize:disk)". So it is finding the actual on-disk storage, taking into account delta storage. You will need git v1.8.5 or later to use this feature.

  git rev-list --objects --all |
  git cat-file --batch-check="%(objectsize:disk) %(rest)" |
  perl -lne '
    my ($size, $path) = split / /, $_, 2;
    next unless defined $path; # commit obj
    do {
      $sizes{$path} += $size;
    } while ($path =~ s{/[^/]+$}{});

    END { print "$sizes{$_} $_" for (keys %sizes) }
  ' |
  sort -rn

-Peff