git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Trying to use git-filter-branch to compress history by removing large, obsolete binary files

From
Elijah Newren <newren@gmail.com>
Date
Oct 7, 2007, 21:23 UTC
Message-ID
<51419b2c0710071423y1b194f22gb6ccaa57303029d1@mail.gmail.com>
Hi,

I'm using git-cvsimport to import some CVS repos, which unfortunately included dozens of large regression test output files in their ancient history...some of which measure hundreds of megabytes in size. I'd like to prune them out of the git history (I don't have access to prune them out of the CVS history), but I'm running into problems.

The following set of instructions will duplicate my problem with a smaller repo; why is the local git repository bigger after running git-filter-branch rather than smaller as I'd expect? I'm probably missing something obvious, but I have no idea what it is.

The steps:

# Make a small repo mkdir test cd test git init echo hi > there git add there git commit -m 'Small repo'

# Add a random 10M binary file dd if=/dev/urandom of=testme.txt count=10 bs=1M git add testme.txt git commit -m 'Add big binary file'

# Remove the 10M binary file git rm testme.txt git commit -m 'Remove big binary file'

# Compress the repo, see how big the repo is git gc --aggressive --prune du -ks . # 10548K du -ks .git # 10532K

# Try to rewrite history to remove the binary file git-filter-branch --tree-filter 'rm -f testme.txt' HEAD git reset --hard

# Try to recompress and clean up, then check the new size git gc --aggressive --prune du -ks . # 10580K !?!?!? du -ks .git # 10564K

Thanks, Elijah

Next: Frank Lichtenheld
Message 1 of 31 in “Trying to use git-filter-branch to compress history by removing large, obsolete binary files”
  1. Elijah NewrenOct 7, 2007
  2. Frank LichtenheldOct 7, 2007
  3. Elijah NewrenOct 7, 2007
  4. Alex RiesenOct 7, 2007
  5. Elijah NewrenOct 7, 2007
  6. Alex RiesenOct 7, 2007
  7. Elijah NewrenOct 8, 2007
  8. Alex RiesenOct 8, 2007
  9. Andreas EricssonOct 8, 2007
  10. Dmitry PotapovOct 7, 2007
  11. Elijah NewrenOct 8, 2007
  12. Dmitry PotapovOct 8, 2007
  13. Andreas EricssonOct 8, 2007
  14. Karl HasselströmOct 8, 2007
  15. Dmitry PotapovOct 8, 2007
  16. Karl HasselströmOct 8, 2007
  17. Johannes SchindelinOct 7, 2007
  18. Elijah NewrenOct 7, 2007
  19. Johannes SchindelinOct 7, 2007
  20. Elijah NewrenOct 7, 2007
  21. Johannes SchindelinOct 8, 2007
  22. Elijah NewrenOct 8, 2007
  23. Sam VilainOct 8, 2007
  24. J. Bruce FieldsOct 8, 2007
  25. Johannes SchindelinOct 8, 2007
  26. Johannes SixtOct 8, 2007
  27. J. Bruce FieldsOct 8, 2007
  28. Theodore TsoOct 8, 2007
  29. J. Bruce FieldsOct 8, 2007
  30. Johannes SchindelinOct 9, 2007
  31. Alex RiesenOct 7, 2007

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.