Configuring git to for forget removed files

Hello world I have a project that I store in a git repository. It's a bunch of source tarballs and some bash scripts to compile it all. Git makes it easy to distribute any changes I make across the computers I run. The problem I have is that over time the repository gets ever larger. When I update to a newer version of something I git rm the old tarball but git still keeps a copy and the folder grows ever larger. At the moment the only solution I have is to periodically rm -rf .git and start again. This works but is less than ideal because I lose all the history for my build scripts. What I would like is to be able to tell git to not keep a copy of anything that has been git rm. The build scripts never get removed, only altered so their history would be preserved. Is it possible to make git delete its backup copies of removed files?

Andy