Trying to use git-filter-branch to compress history by removing large, obsolete binary files

Hi,

I'm using git-cvsimport to import some CVS repos, which unfortunately included dozens of large regression test output files in their ancient history...some of which measure hundreds of megabytes in size. I'd like to prune them out of the git history (I don't have access to prune them out of the CVS history), but I'm running into problems.

The following set of instructions will duplicate my problem with a smaller repo; why is the local git repository bigger after running git-filter-branch rather than smaller as I'd expect? I'm probably missing something obvious, but I have no idea what it is.

The steps:

# Make a small repo mkdir test cd test git init echo hi > there git add there git commit -m 'Small repo'

# Add a random 10M binary file dd if=/dev/urandom of=testme.txt count=10 bs=1M git add testme.txt git commit -m 'Add big binary file'

# Remove the 10M binary file git rm testme.txt git commit -m 'Remove big binary file'

# Compress the repo, see how big the repo is git gc --aggressive --prune du -ks . # 10548K du -ks .git # 10532K

# Try to rewrite history to remove the binary file git-filter-branch --tree-filter 'rm -f testme.txt' HEAD git reset --hard

# Try to recompress and clean up, then check the new size git gc --aggressive --prune du -ks . # 10580K !?!?!? du -ks .git # 10564K

Thanks, Elijah

On Sun, Oct 07, 2007 at 03:23:59PM -0600, Elijah Newren wrote:

The usual suspect would be the reflog.

Gruesse,

On 10/7/07, Frank Lichtenheld <frank@lichtenheld.de> wrote:

The git-filter-branch documentation mentions creating refs/original under .git. Unfortunately, it doesn't contain any links or documentation on how I'd clean those out and I haven't been able to figure it out. I asked on #git how to clean these out and got some answers that didn't work (git branch -d and something else I don't remember). So...how do I fix the reflog, and then repack to have a pack under 11MB in size?

Thanks, Elijah

Elijah Newren, Mon, Oct 08, 2007 00:00:51 +0200:

rm -rf .git/refs/original/refs/heads/<the branch where HEAD pointed to> (assuming you haven't repacked yet)

or just edit .git/packed-refs and remove everything "refs/original" which fits the criteria

git reflog expire --all (it is a bit to much. You can just edit .git/logs/* in any text editor)

On 10/7/07, Alex Riesen <raa.lkml@gmail.com> wrote: <snip>

So...

$ du -hs . 11M . $ rm -rf .git/refs/original/ $ vi .git/packed-refs # Remove the line referring to refs/original... $ git reflog expire --all $ git gc --aggressive --prune $ du -hs . 11M .

It's still 11MB.

Any other ideas?

Elijah

Elijah Newren, Mon, Oct 08, 2007 00:24:49 +0200:

you missed something. Your example compresses to about 124k.

On 10/7/07, Alex Riesen <raa.lkml@gmail.com> wrote:

What version of git are you running? I reran all the steps to which you responded (repeated below for clarity) with git-1.5.3.3 and still get 11MB. Also, you must have different filesystem extents than me since an empty git repo takes 196k here[1], so I don't think any repo is going to get down to 124k.

My understanding of the steps you suggest would work:

# Make a small repo mkdir test cd test git init echo hi > there git add there git commit -m 'Small repo'

# Add a random 10M binary file dd if=/dev/urandom of=testme.txt count=10 bs=1M git add testme.txt git commit -m 'Add big binary file'

# Remove the 10M binary file git rm testme.txt git commit -m 'Remove big binary file'

# Compress the repo, see how big the repo is git gc --aggressive --prune du -ks . # 10548K du -ks .git # 10532K

# Try to rewrite history to remove the binary file git-filter-branch --tree-filter 'rm -f testme.txt' HEAD git reset --hard

# Try to recompress and clean up, then check the new size git gc --aggressive --prune du -ks . # 10580K !?!?!? du -ks .git # 10564K

# Do the stuff Alex suggests to trim the history rm -rf .git/refs/original/ vi .git/packed-refs # Use vi to remove the line referring to refs/original... git reflog expire --all git gc --aggressive --prune du -ks . # Still 10564K

Thanks, Elijah

[1] An empty git repo takes 196k for me, as can be seen by: $mkdir tmp $cd tmp $git init $du -hs . 196K .

Elijah Newren, Mon, Oct 08, 2007 02:09:50 +0200:

git version 1.5.3.4.225.g31b973 (irrelevant custom modifications)

it is ext3. I do not install the hooks (~8k apparent, ~32k fs blocks) and never activate logs by default.

another part of the suggestion re reflogs was to look into the logs, to check if expire actually removed anything. It seems to have been the culprit.

Alex Riesen wrote:

On my system, running git version 1.5.3.3.131.g34c6d,

	git reflog expire --all

does absolutely nothing.

	git reflog expire --expire=0 --all

truncates all the reflogs. I'm not sure if this is intended or not.

On Sun, Oct 07, 2007 at 04:24:49PM -0600, Elijah Newren wrote:

I believe this should work:

git reflog expire --all --expire-unreachable=0 git gc --prune

Warning: all unreachable references will be removed!

Dmitry

On 10/7/07, Dmitry Potapov <dpotapov@gmail.com> wrote:

Yes, this seems to work.  So the history-rewriting steps are

git-filter-branch --tree-filter 'rm -f testme.txt' HEAD git reset --hard rm -rf .git/refs/original/ vi .git/packed-refs # Use vi to remove the line referring to refs/original... git reflog expire --all --expire-unreachable=0 git gc --prune

Seems like a wrapper is needed.  :-)

What other scenarios could lead to unreachable references? I don't know how to determine whether this is safe or not (except that these were test repositories anyway, so I don't care what happens to them).

Thanks! Elijah

On Sun, Oct 07, 2007 at 06:22:28PM -0600, Elijah Newren wrote:

Actually, I would rather not, because you rarely need to remove anything immediately, and 30 days delay is reasonable time to give you a chance to recover that you removed accidentally. You can reduce it by setting appropriate value for gc.reflogExpireUnreachable in your configuration. The only thing you need to do is to remove .git/refs/original/heads/something after you are sure that git-filter-branch did exactly what you wanted.

Any re-writing of history leads to that.

Git logs all your action, so even re-writing history would not be so disastrous if you suddenly realized that you did something wrong. The history is stored for 30 days by default. Usually, you do not need to mess with Git internals like you did above. Your useless files still will disappear after being unreachable for 30 days.

OTOH, if you want to have a clean repository immediately, I believe 'git clone' is a better option. After you made a local clone using it, 'git gc' should remove old garbage.

Dmitry

Dmitry Potapov wrote:

git-rebase being the most common culprit, right alongside 'git commit --amend'.

A clone only fetches revs reachable from a ref, so pruning immediately after a clone is completely pointless.

On 2007-10-08 11:27:33 +0200, Andreas Ericsson wrote:

StGit (and presumably guilt, and any other similar tool) are just glorified rebase wrappers, so they'll generate tons of unreachable objects too.

On Mon, Oct 08, 2007 at 11:27:33AM +0200, Andreas Ericsson wrote:

Not true. git-clone copies the whole pack, so it can contain unreachable objects. Here is a simple script that demonstrates that without garbage collection the size of the cloned repository will be the same as the original one.

=========================================== # Make a small repo mkdir test cd test git init echo hi > there git add there git commit -m 'Small repo'

# Add a random 10M binary file dd if=/dev/urandom of=testme.txt count=10 bs=1M git add testme.txt git commit -m 'Add big binary file'

# Remove the 10M binary file git rm testme.txt git commit -m 'Remove big binary file'

# Compress the repo, see how big the repo is git gc --aggressive --prune du -ks . # 10348 du -ks .git # 10344

git-whatchanged

# Try to rewrite history to remove the binary file git-filter-branch --tree-filter 'rm -f testme.txt' HEAD git reset --hard

# Remove original refs rm .git/refs/original/refs/heads/master

# Remove back cd ..

# Clone repository git-clone -l test/.git test2

cd test2 du -ks .git # 10360

# Now run garbage collection git gc du -ks .git # 96

===========================================

Dmitry

On 2007-10-08 16:40:17 +0400, Dmitry Potapov wrote:

[...]

Try without the -l option and with a file:// URL:

  git clone file:///path/to/test/.git test2

From the git-clone man page:

--local::
-l::
        When the repository to clone from is on a local machine, this
        flag bypasses normal "git aware" transport mechanism and
        clones the repository by making a copy of HEAD and everything
        under objects and refs directories. The files under
        `.git/objects/` directory are hardlinked to save space when
        possible. This is now the default when the source repository
        is specified with `/path/to/repo` syntax, so it essentially is
        a no-op option. To force copying instead of hardlinking (which
        may be desirable if you are trying to make a back-up of your
        repository), but still avoid the usual "git aware" transport
        mechanism, `--no-hardlinks` can be used.

Hi,

On Sun, 7 Oct 2007, Elijah Newren wrote:

Just clone it.  The clone will be much smaller.

Ciao, Dscho

On 10/7/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

$ git clone test test2 <snip> $ du -hs test 11M test $ du -hs test2 11M test2

Any other ideas?

Hi,

On Sun, 7 Oct 2007, Elijah Newren wrote:

Yep.  Maybe it is necessary to run "git gc" in test2.

Ciao, Dscho

Hi,

On 10/7/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

<snip>

Sweet, finally solved!  That brings test2 down to 340K.

However, the solution seems somewhat involved...it requires running git-filter-branch, git reset, removing the .git/refs/original/ directory, editing .git/packed-refs in some editor, running git reflog expire, cloning the resulting repository, and running git gc yet again. It seems like there has to be an easier way. (Anyone have one?)

Oh, and git-filter-branch could really use some explanatory note about how to actually complete rewriting the history.

Thanks, Elijah

Hi,

On Sun, 7 Oct 2007, Elijah Newren wrote:

It should be as easy as git filter-branch and git clone.

It does what it should do. It is _your_ task to look at refs/original/* if everything went alright. Then you just delete the checked refs.

What made your case so cumbersome was that you wanted the big objects out _now_, instead of having them in for a grace period. BTW this grace period is in place to help _you_, not the program. (In case you fscked up and need those objects back.)

Ciao, Dscho

On 10/7/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Yes, a git filter-branch, git clone, AND git gc in the clone avoids all those funny ref editing commands. However, cloning a 5.6GB repo (the size of one of the real repos I'm dealing with) will likely take a long time (and may push me past the limits of disk space), so using other steps to avoid the need to clone actually seems nicer.

Sure, I think that's a sane default. And I think it's fine that it should be my task to look at the refs to check that everything worked okay and delete them. But it's nearly impossible to figure out how to do that! _That_ is my complaint. I got multiple misleading or incomplete answers (both on this list and in #git) before getting some working solutions, so this task is obviously far from trivial. I really think that adding instructions about how to check and delete the relevant refs would be a very useful addition to the documentation.

Thanks everyone for the help!

Elijah

Elijah Newren wrote:

You can just delete the logs and references that you don't want and run git gc --prune.

However.

git gc creates a new pack before deleting the old one. Garbage collection usually does this; make a copy of everything to a new place and then free all of the old space. If *that* is a problem, ie you don't have enough space for two copies of the repository and the junk, you'll have to do a partial import, leave the junk you don't want unpacked, cleanup and prune, then finish the import. Which sounds like a lot of hassle when you should really just find a place with more space to work with!

Sam.

On Mon, Oct 08, 2007 at 01:34:07AM +0100, Johannes Schindelin wrote:

It seems odd to me, by the way, that filter-branch has its own home-grown backup mechanism. Lots of other commands can "lose" commits, but none of them keep an extra backup like this.

And I find it tedious for quicker jobs which it might otherwise be useful for (e.g. rewrites of commits in my tree not yet in upstream), unless I wrap it in a script that cleans up after itself.

--b.

Hi,

On Sun, 7 Oct 2007, J. Bruce Fields wrote:

The rationale was this: filter-branch recently learnt how to rewrite many branches, and it might be tedious to find out which ones. But then, there is git log --no-walk --all, so maybe I really should get rid of refs/original/*?

I'd like to have some comments from the heavier filter-branch users on that...

Ciao, Dscho

Johannes Schindelin schrieb:

IMHO, a backup of the original refs is needed. However, it may be wise to store them in the refs/heads namespace so that 'git branch -d' can delete them and 'git branch -m' can move them back if something went wrong.

-- Hannes

On Mon, Oct 08, 2007 at 08:22:42AM +0200, Johannes Sixt wrote:

And we can't rely instead on reflogs or some other existing mechanism?

If people want backups like this it'd seem easier to turn this on optionally with commandline switches, like patch's --backup, --prefix, --suffix options.

Having it by default leave these backups around, even when everything succeeds, makes for unnecessary cleanup work in the normal case, and is inconsistent with the behavior of other git commands that destroy or rewrite history.

--b.

On Mon, Oct 08, 2007 at 10:36:50AM -0400, J. Bruce Fields wrote:

I think what makes git-filter-branch different is that you can change a large amount of history with git-filter-branch, including large numbers of tags, etc. The reflog is quite sufficient to recover from a screwed up "git commit --amend". But I don't think the reflog is going to be sufficient given the kinds of changes that git-filter-branch can potentially do to your repository. Maybe default of --backup vs --no-backup could be changed via a config parameter, but I think the default is of backing up refs is a good think....

Perhaps a solution would be to add "git-filter-branch --cleanup" that that clears the reflog and wipes the backed up tags; perhaps first asking interactively if the user is really sure he/she wants to do this.

						- Ted

On Mon, Oct 08, 2007 at 12:37:01PM -0400, Theodore Tso wrote:

Yeah, it's clearly designed with rewriting a whole repo in mind.

It might also be handy, though, as a quick way to rewrite a single branch. (E.g., "add 'Acked-by: Joe' to everything in 'for-upstream' not in 'origin'", or "rename foo to bar in every commit in 'topic' not in 'origin'".).

I find the current defaults awkward for that case. Maybe it'd make sense to treat the two cases differently.

Maybe.

--b.

Hi,

On Mon, 8 Oct 2007, Theodore Tso wrote:

FWIW after reading Bruce's reasoning, I tend towards having no "backups" by default (I say "backups", since they are _only_ written when the respective branch has changed).

And I do not think that the config variable is a good approach; if you want backups or not is a per-case decision. So your proposal would only result in even more confusion.

My preference ATM is to write nothing per default, but only when --original <namespace> was given.

Ciao, Dscho

Elijah Newren, Sun, Oct 07, 2007 23:23:59 +0200:

git-filter-branch makes a backup of your original references:

$ git filter-branch --help
...
       Always verify that the rewritten version is correct: The original refs,
       if different from the rewritten ones, will be stored in the namespace
       refs/original/.
...

These will keep your big files in repository.