threads / discuss / 21112

How to delete large files

Subject: How to delete large files

## tl;dr

5 messages between Oct 2, 2009 and Oct 2, 2009.

replies: 4people: 4as markdown or json

Nils Homer· Oct 2, 2009, 10:03 UTC · lore

I wish to delete some large offending files (10MB to >100MB each) from my git repository (git://bfast.git.sourceforge.net/gitroot/bfast/bfast). The current size of the repo is 656MB.

I created a backup of my repository and then searched for such offending files using a script found here: http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-ob jects-and-trim-your-waist-line/ where I modified the script to output in MB instead of KB.

This gave me a list of files that I wanted to delete: -e size pack SHA location 113 113 f9f2faab597d4f8ccbfac2864347dbc256353fbf tests.long/save/save.tar.gz 113 113 926b1ba880a26354c4a6b9391985f57fbc9a1174 tests.long/save/save.tar.gz 113 113 e568480bcb8239e6d1ed8d2da86c309c0d3d101b tests.long/save/save.tar.gz 113 113 e3c0ee53f20e8ebfb60eaefcd7b405168c26a565 tests.long/save/save.tar.gz 103 103 ee2ee50c5075d05d29764c8d4b9acc2acedda919 tests.long/save/save.tar.gz 35 35 319c75945c27096093dbab5a0bf6a9a08089bc2d tests.long/data/data.tar.gz 11 11 805193c74ceeffca9da3a2788545e701d77e1caf tests/save/save.tar.gz 11 11 658e4a78c1028875ab597d6bde5823cd6a1694b9 tests/save/save.tar.gz

So I decided to remove these files using: git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch tests.long/save/save.tar.gz tests.long/data/data.tar.gz tests/save/save.tar.gz" HEAD

I then ran: rm -rf .git/refs/original git reflog expire --expire=now all git gc --prune=now

Still (using du h) the repository is 656MB and I can see the above files in the revision list: git rev-list --all --objects | grep tests.long/save/save.tar.gz ee2ee50c5075d05d29764c8d4b9acc2acedda919 tests.long/save/save.tar.gz e568480bcb8239e6d1ed8d2da86c309c0d3d101b tests.long/save/save.tar.gz f9f2faab597d4f8ccbfac2864347dbc256353fbf tests.long/save/save.tar.gz 926b1ba880a26354c4a6b9391985f57fbc9a1174 tests.long/save/save.tar.gz e3c0ee53f20e8ebfb60eaefcd7b405168c26a565 tests.long/save/save.tar.gz

Could this be because of tags that I had previously created?

I am running git version 1.6.3.3.  I appreciate any help in advance,

Nils Homer

Matthieu Moy· Oct 2, 2009, 11:41 UTC · re: Nils Homer · lore

Re: How to delete large files

Nils Homer <nilshomer@gmail.com> writes:

Show 10 quoted lines

> Still (using du h) the repository is 656MB and I can see the above files in
> the revision list:
> git rev-list --all --objects | grep tests.long/save/save.tar.gz
> ee2ee50c5075d05d29764c8d4b9acc2acedda919 tests.long/save/save.tar.gz
> e568480bcb8239e6d1ed8d2da86c309c0d3d101b tests.long/save/save.tar.gz
> f9f2faab597d4f8ccbfac2864347dbc256353fbf tests.long/save/save.tar.gz
> 926b1ba880a26354c4a6b9391985f57fbc9a1174 tests.long/save/save.tar.gz
> e3c0ee53f20e8ebfb60eaefcd7b405168c26a565 tests.long/save/save.tar.gz
>
> Could this be because of tags that I had previously created?

I guess so. Anything pointing to the old history will prevent Git from deleting it.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Mikael Magnusson· Oct 2, 2009, 12:26 UTC · re: Nils Homer · lore

Re: How to delete large files

2009/10/2 Nils Homer <nilshomer@gmail.com>:

Show 50 quoted lines

> I wish to delete some large offending files (10MB to >100MB each) from my
> git repository (git://bfast.git.sourceforge.net/gitroot/bfast/bfast).  The
> current size of the repo is 656MB.
>
> I created a backup of my repository and then searched for such offending
> files using a script found here:
> http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-ob
> jects-and-trim-your-waist-line/
> where I modified the script to output in MB instead of KB.
>
> This gave me a list of files that I wanted to delete:
> -e   size  pack                                      SHA
> location
> 113  113   f9f2faab597d4f8ccbfac2864347dbc256353fbf
> tests.long/save/save.tar.gz
> 113  113   926b1ba880a26354c4a6b9391985f57fbc9a1174
> tests.long/save/save.tar.gz
> 113  113   e568480bcb8239e6d1ed8d2da86c309c0d3d101b
> tests.long/save/save.tar.gz
> 113  113   e3c0ee53f20e8ebfb60eaefcd7b405168c26a565
> tests.long/save/save.tar.gz
> 103  103   ee2ee50c5075d05d29764c8d4b9acc2acedda919
> tests.long/save/save.tar.gz
> 35   35    319c75945c27096093dbab5a0bf6a9a08089bc2d
> tests.long/data/data.tar.gz
> 11   11    805193c74ceeffca9da3a2788545e701d77e1caf  tests/save/save.tar.gz
> 11   11    658e4a78c1028875ab597d6bde5823cd6a1694b9  tests/save/save.tar.gz
>
> So I decided to remove these files using:
> git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch
> tests.long/save/save.tar.gz tests.long/data/data.tar.gz
> tests/save/save.tar.gz" HEAD
>
> I then ran:
> rm -rf .git/refs/original
> git reflog expire --expire=now 卟ll
> git gc --prune=now
>
> Still (using du 虐) the repository is 656MB and I can see the above files in
> the revision list:
> git rev-list --all --objects | grep tests.long/save/save.tar.gz
> ee2ee50c5075d05d29764c8d4b9acc2acedda919 tests.long/save/save.tar.gz
> e568480bcb8239e6d1ed8d2da86c309c0d3d101b tests.long/save/save.tar.gz
> f9f2faab597d4f8ccbfac2864347dbc256353fbf tests.long/save/save.tar.gz
> 926b1ba880a26354c4a6b9391985f57fbc9a1174 tests.long/save/save.tar.gz
> e3c0ee53f20e8ebfb60eaefcd7b405168c26a565 tests.long/save/save.tar.gz
>
> Could this be because of tags that I had previously created?
>
> I am running git version 1.6.3.3.  I appreciate any help in advance,

Well, you just gave "HEAD" to git filter-branch to rewrite, i think you want --all to rewrite all refs you have.

-- 
Mikael Magnusson

Johannes Sixt· Oct 2, 2009, 13:41 UTC · re: Mikael Magnusson · lore

Re: How to delete large files

Mikael Magnusson schrieb:

> Well, you just gave "HEAD" to git filter-branch to rewrite, i think
> you want --all to rewrite all refs you have.

... and '--tag-filter cat' to rewrite tags as well.

-- Hannes

Nils Homer· Oct 2, 2009, 19:20 UTC · re: Johannes Sixt · lore

Re: How to delete large files

Thank-you for all of your insightful help. Combining all the advice, the commands that worked are:

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch $files" --tag-name-filter cat -- --all

git -rf .git/refs/original

git reflog expire --expire=now

git gc --prune=now

I then cloned the repository to a different location and replaced my centralized version with the cloned copy.

Thanks,

Nils

On 10/2/09 6:41 AM, "Johannes Sixt" <j.sixt@viscovery.net> wrote:

Show 7 quoted lines

> Mikael Magnusson schrieb:
>> Well, you just gave "HEAD" to git filter-branch to rewrite, i think
>> you want --all to rewrite all refs you have.
> 
> ... and '--tag-filter cat' to rewrite tags as well.
> 
> -- Hannes