{"thread":{"id":"22737","subject":"Configuring git to for forget removed files","startedAt":"2010-02-20T10:37:39Z","lastAt":"2010-02-21T21:14:02Z","messageCount":8,"participants":["Andrew Benton","Tim Visher","Avery Pennarun","Junio C Hamano","Jonathan Nieder","Larry D'Anna","Jacob Helwig"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"135152","messageId":"4B7FBB73.70004@gmail.com","threadId":"22737","inReplyTo":null,"subject":"Configuring git to for forget removed files","fromName":"Andrew Benton","fromEmail":"b3nton@gmail.com","sentAt":"2010-02-20T10:37:39Z","receivedAt":"2010-02-20T10:37:39Z","isPatch":false,"sender":{"key":"b3nton@gmail.com","avatar":null},"body":"Hello world\nI have a project that I store in a git repository. It's a bunch of source tarballs and\nsome bash scripts to compile it all. Git makes it easy to distribute any changes I make\nacross the computers I run. The problem I have is that over time the repository gets ever\nlarger. When I update to a newer version of something I git rm the old tarball but git\nstill keeps a copy and the folder grows ever larger. At the moment the only solution I\nhave is to periodically rm -rf .git and start again. This works but is less than ideal\nbecause I lose all the history for my build scripts.\nWhat I would like is to be able to tell git to not keep a copy of anything that has been\ngit rm. The build scripts never get removed, only altered so their history would be\npreserved. Is it possible to make git delete its backup copies of removed files?\n\nAndy\n"},{"id":"135161","messageId":"c115fd3c1002200741l25f32685t998a19e76922a2d2@mail.gmail.com","threadId":"22737","inReplyTo":"4B7FBB73.70004@gmail.com","subject":"Re: Configuring git to for forget removed files","fromName":"Tim Visher","fromEmail":"tim.visher@gmail.com","sentAt":"2010-02-20T15:41:02Z","receivedAt":"2010-02-20T15:41:02Z","isPatch":false,"sender":{"key":"tim.visher@gmail.com","avatar":"https://gravatar.com/avatar/98307ce54bcac1a46b5d845827d5386e2a27895ff08790cdd3877ad527760575?d=mp&s=160"},"body":"Hi Andy,\n\nOn Sat, Feb 20, 2010 at 5:37 AM, Andrew Benton <b3nton@gmail.com> wrote:\n> I have a project that I store in a git repository. It's a bunch of source\n> tarballs and some bash scripts to compile it all. Git makes it easy to\n> distribute any changes I make across the computers I run. The problem I have\n> is that over time the repository gets ever larger. When I update to a newer\n> version of something I git rm the old tarball but git still keeps a copy and\n> the folder grows ever larger. At the moment the only solution I have is to\n> periodically rm -rf .git and start again. This works but is less than ideal\n> because I lose all the history for my build scripts.\n>\n> What I would like is to be able to tell git to not keep a copy of anything\n> that has been git rm. The build scripts never get removed, only altered so\n> their history would be preserved. Is it possible to make git delete its backup\n> copies of removed files?\n\nI don't know if I can really speak to your hoped for conclusion\nalthough `git filter-branch` is where you want to look for rewriting\nhistory.  However, that's also an entirely impractical solution if\nyour repo is at all public because it would completely break sharing.\n\nThat being said, have you thought of changing your repo strategy?\nIMHO, storing binary blobs that change at all regularly in _any_ SCMS\nis a problem waiting to happen.  It's different if you have assets\nthat are fairly stable like images for a system's UI or dependencies\nthat have been stabilized, but that doesn't sound like your situation.\n\nAs a thought, why not try to do something along the lines of\nmaintaining a symlink to whatever tarballs your project currently\ndepends on as a 'foolib-latest' and then having a separate directory\nthat has a file that you can change.  You could maintain backups of\nthat using a tool like rsync (since you obviously aren't concerned\nwith maintaining history there) rather than git.  Then you could\ndecide arbitrarily how many backups you want to make and try to\nmaintain what version of the file went with which commit in your repo.\n The main problem I see with that is that you loose a lot of the\nadvantages of having a SCMS because you can't reliably checkout a\nprevious commit and build it; at least not without some very serious\neffort.\n\nAnother possible solution if you maintain the sources that are\ngenerating the tarballs is to treat the tarballs as artifacts of the\nbuild rather than as assets that should be managed by the SCMS.  In\nthat way, you might spend more time during each build but your repo\nwould be much cleaner and would have the added advantage of being able\nto completely build itself at every commit point.\n\nAnyway, just food for thought.\n\n\n-- \n\nIn Christ,\n\nTimmy V.\n\nhttp://burningones.com/\nhttp://five.sentenc.es/ - Spend less time on e-mail\n"},{"id":"135171","messageId":"32541b131002201050l35c095a0i1b525e8be7812e59@mail.gmail.com","threadId":"22737","inReplyTo":"4B7FBB73.70004@gmail.com","subject":"Re: Configuring git to for forget removed files","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-02-20T18:50:25Z","receivedAt":"2010-02-20T18:50:25Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Sat, Feb 20, 2010 at 5:37 AM, Andrew Benton <b3nton@gmail.com> wrote:\n> I have a project that I store in a git repository. It's a bunch of source\n> tarballs and\n> some bash scripts to compile it all. Git makes it easy to distribute any\n> changes I make\n> across the computers I run. The problem I have is that over time the\n> repository gets ever\n> larger. When I update to a newer version of something I git rm the old\n> tarball but git\n> still keeps a copy and the folder grows ever larger.\n\nYou can use 'git filter-branch', as Tim already mentioned, or use a\ngit 'shallow clone' to only get the most recent versions of things.\n\nAlternatively, have you thought about storing *uncompressed* tarballs\nin git instead of compressed ones?  Then when you update to a newer\nversion, git can compute an xdelta from one to the other and store\nonly the changes.  That means you can have full history *and* not\nwaste too much disk space.  Git compresses the objects anyway when it\nstores them in the repository.\n\nHave fun,\n\nAvery\n"},{"id":"135179","messageId":"7v8wann2fi.fsf@alter.siamese.dyndns.org","threadId":"22737","inReplyTo":"4B7FBB73.70004@gmail.com","subject":"Re: Configuring git to for forget removed files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-20T19:16:01Z","receivedAt":"2010-02-20T19:16:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andrew Benton <b3nton@gmail.com> writes:\n\n> I have a project that I store in a git repository. It's a bunch of source tarballs and\n> some bash scripts to compile it all. Git makes it easy to distribute any changes I make\n> across the computers I run. The problem I have is that over time the repository gets ever\n> larger. When I update to a newer version of something I git rm the old tarball but git\n> still keeps a copy and the folder grows ever larger. At the moment the only solution I\n> have is to periodically rm -rf .git and start again. This works but is less than ideal\n> because I lose all the history for my build scripts.\n> What I would like is to be able to tell git to not keep a copy of anything that has been\n> git rm. The build scripts never get removed, only altered so their history would be\n> preserved. Is it possible to make git delete its backup copies of removed files?\n\nYou are either being unreasonable, or haven't thought things through.\n\nLet's say you have your build script with a tarball of frotz-1.42.tar.gz\nin the initial revision.  The script extracts from tarball and builds.\n\nNow you update your build script once, and make a commit.\n\nThen you add frotz-1.43.tar.gz and remove frotz-1.42.tar.gz.  You may\nadjust the build script to extract frotz-1.43 instead of frotz-1.42 in the\nsame commit, or your script may be written loosely and extract any tarball\nthat matches frotz-*.tar.gz wildcard in which case the build script may\nnot change.\n\nYou now have three commits:\n\n - initial one: ships frotz-1.42 and builds it;\n - second one: ships frotz-1.42 and builds it better;\n - third one: ships frotz-1.43 and builds it in some way.\n\nYou clone it to some other machine and build the tip; everything goes well\nand you are happy.  What should happen if you do:\n\n $ git checkout HEAD^\n $ make\n\nShould it build frotz-1.43, or should it fail?\n\nIf you somehow obliterate frotz-1.42.tar.gz out of the history with some\nmagic you described, there should not be any frotz-1.42.tar.gz in the\nhistory, so there is no way you can build frotz-1.42 out of this checkout.\nYour \"second\" tree can only have one or two shapes:\n\n - It can record only build script and nothing else, in which case the\n   above \"make\" will have to fail.\n\n - With some magic you described, it records your build script and\n   frotz-1.43.tar.gz, and frotz-1.43 is built.\n\nYou need to realize that the magic have to adjust your build script so\nthat it does not require the exact version of frotz-1.42.  Namely, the\nbuild script you wrote not only knew that the next version of tarball will\nmatch frotz-*.tar.gz (and that is why you can extract the contents from\nit), but also somehow anticipated the build infrastructure change the\nupstream will make when they update from 1.42 to 1.43 and was magically\ncapable of building either versions.  And you did that back when you\ndidn't have the source to frotz-1.43 and how it would look like.\n\nYou also need to realize that nowhere in your set-up up to the point you\nmade three commits, you never told anybody that frotz-1.43.tar.gz replaces\nfrotz-1.42.tar.gz. The only thing you said was to remove frotz-1.42.tar.gz.\n\nIf you make the checkout of second one to fail to build because your\n\"obliterate\" is not to include any tarball in the second version, then you\nare being unreasonable.\n\nIf you are asking for the magic to include frotz-1.43 instead of\nfrotz-1.42, and further adjust your old build script to anticipate\nthe change between 1.42 and 1.43, you haven't described how that magic\nshould happen, so you haven't thought things through.\n\nOne way out would be to do it like this instead:\n\n - initial one: your build script, and frotz-1.42 extracted in frotz/\n   directory already.  Do not ship a tarball.\n\n - second one: your improved build script, and the same frotz/ directory\n   without any change.\n\n - third one: your build script, either improved or the same from the\n   previous one, and frotz-1.43 extracted in frotz/ directory.\n\nThis way, the checkout from the second one will build frotz-1.42.  Also\nyou could see if your build scripts from the second version would build\nfrotz-1.43 as well by doing something like:\n\n    $ git checkout HEAD^\n    $ git checkout master -- frotz/\n    $ make\n\nYou will ship both versions of frotz, but between 1.42 and 1.43 there will\nbe a lot of similarities, so packed result will be far smaller than\nstoring two compressed tarballs.  In fact, I wouldn't be surprised if it\nwere smaller than storing one compressed tarball.\n"},{"id":"135208","messageId":"20100221024746.GA9628@progeny.tock","threadId":"22737","inReplyTo":"4B7FBB73.70004@gmail.com","subject":"Re: Configuring git to for forget removed files","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-02-21T02:47:46Z","receivedAt":"2010-02-21T02:47:46Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Andy,\n\nAndrew Benton wrote:\n\n> I have a project that I store in a git repository. It's a bunch of\n> source tarballs and some bash scripts to compile it all. Git makes\n> it easy to distribute any changes I make across the computers I run.\n\nThis is not really what git is intended to do.\n\n - git generally works better with files that are easy to diff; see\n   the “filter” attribute in gitattributes(5) for one way this is\n   sometimes achieved with meaningful binary files (e.g., the\n   compressed files OpenOffice produces)\n\n - Though git can cope with large projects, it generally works best\n   when track the smallest meaningful unit that can be tested alone.\n   Submodules can be used to stitch them together.\n\nThus if it is important to you to track the history of this project, I\nwould suggest giving each source tree its own repository and stitching\nthem together with a “supermodule” that tracks your scripts and\nincludes references to the appropriate versions of each source\npackage.\n\nSee http://who-t.blogspot.com/2009/04/big-fat-xorg-supermodule.html\nfor an example of this kind of thing.\n\nOn the other hand, I don’t get the impression it is so important here\nto track the history from the beginning, so:\n\n> The problem I have is that over time the repository gets ever\n> larger. When I update to a newer version of something I git rm the\n> old tarball but git still keeps a copy and the folder grows ever\n> larger. At the moment the only solution I have is to periodically rm\n> -rf .git and start again. This works but is less than ideal because\n> I lose all the history for my build scripts.\n\nMaybe you could keep the build scripts in a git repository and\nsynchronizing the tarballs out of line with some other tool, such as\nrsync or unison.\n\n> Is it possible to make git delete its\n> backup copies of removed files?\n\ngit is not intended to be a backup tool; as you’ve noticed, the older\nversions gradually accumulate and it becomes apparent over time that\nit would be really nice for older commits to expire.\n\nI am guessing, but it sounds to me like what you are looking for is\nsomething that is distributed like git but is a backup system.  Or in\nother words, a way to record a few snapshots like LVM or btrfs, but\nsuch that new snapshots can be easily transfered to another computer.\nAt least I would be glad to learn of such a tool. ;-)\n\nHope that helps,\nJonathan\n"},{"id":"135246","messageId":"4B8135F2.3000709@gmail.com","threadId":"22737","inReplyTo":"20100221024746.GA9628@progeny.tock","subject":"Re: Configuring git to for forget removed files","fromName":"Andrew Benton","fromEmail":"b3nton@gmail.com","sentAt":"2010-02-21T13:32:34Z","receivedAt":"2010-02-21T13:32:34Z","isPatch":false,"sender":{"key":"b3nton@gmail.com","avatar":null},"body":"On 21/02/10 02:47, Jonathan Nieder wrote:\n> Maybe you could keep the build scripts in a git repository and\n> synchronizing the tarballs out of line with some other tool, such as\n> rsync or unison.\n\nThanks, that seems to do what I want. I'll use git to keep track of the bash scripts and\nrsync -r --delete --ignore-existing --progress --exclude '.git'\nto synchronise the source tarballs\n\nAndy\n"},{"id":"299158","messageId":"20100221203212.GA10876@cthulhu","threadId":"22737","inReplyTo":"4B7FBB73.70004@gmail.com","subject":"Re: Configuring git to for forget removed files","fromName":"Larry D'Anna","fromEmail":"larry@elder-gods.org","sentAt":"2010-02-21T20:32:12Z","receivedAt":"2010-02-21T20:32:12Z","isPatch":false,"sender":{"key":"larry@elder-gods.org","avatar":"https://avatars.githubusercontent.com/u/3013304?v=4"},"body":"* Andrew Benton (b3nton@gmail.com) [100220 05:37]:\n> Hello world\n> I have a project that I store in a git repository. It's a bunch of source tarballs and\n> some bash scripts to compile it all. Git makes it easy to distribute any changes I make\n> across the computers I run. The problem I have is that over time the repository gets ever\n> larger. When I update to a newer version of something I git rm the old tarball but git\n> still keeps a copy and the folder grows ever larger. At the moment the only solution I\n> have is to periodically rm -rf .git and start again. This works but is less than ideal\n> because I lose all the history for my build scripts.\n> What I would like is to be able to tell git to not keep a copy of anything that has been\n> git rm. The build scripts never get removed, only altered so their history would be\n> preserved. Is it possible to make git delete its backup copies of removed files?\n\nThis reminds me of a scenario I wish git had some way of supporting: I have a\nlarge collection of mp3s that I have duplicated across several computers.  I\nwould love to be able to use git to sync changes between the copies, but there\nare several problems: \n\n1) git is really slow when dealing with thousands of multi-megabyte blobs.\n\n2) commiting it to git is going to double the size of the directory, and I don't\nreally have space for that on one of the computers that the directory lives on.\n\n3) there's no way to discard old history without breaking push and pull.\n\nI'm not sure exactly what it would take to address 1, but 2 could be addressed\npretty easily using btrfs file clones (once btrfs is stable), and 3 could be\ndealt with by improving support for shallow clones.\n\n     --larry\n\n"},{"id":"299159","messageId":"20100221211402.GA1493@vfb-9.home","threadId":"22737","inReplyTo":"20100221203212.GA10876@cthulhu","subject":"Re: Configuring git to for forget removed files","fromName":"Jacob Helwig","fromEmail":"jacob.helwig@gmail.com","sentAt":"2010-02-21T21:14:02Z","receivedAt":"2010-02-21T21:14:02Z","isPatch":false,"sender":{"key":"jacob.helwig@gmail.com","avatar":"https://avatars.githubusercontent.com/u/14557?v=4"},"body":"On 15:32 Sun 21 Feb     , Larry D'Anna wrote:\n> * Andrew Benton (b3nton@gmail.com) [100220 05:37]:\n> > Hello world\n> > I have a project that I store in a git repository. It's a bunch of source tarballs and\n> > some bash scripts to compile it all. Git makes it easy to distribute any changes I make\n> > across the computers I run. The problem I have is that over time the repository gets ever\n> > larger. When I update to a newer version of something I git rm the old tarball but git\n> > still keeps a copy and the folder grows ever larger. At the moment the only solution I\n> > have is to periodically rm -rf .git and start again. This works but is less than ideal\n> > because I lose all the history for my build scripts.\n> > What I would like is to be able to tell git to not keep a copy of anything that has been\n> > git rm. The build scripts never get removed, only altered so their history would be\n> > preserved. Is it possible to make git delete its backup copies of removed files?\n> \n> This reminds me of a scenario I wish git had some way of supporting: I have a\n> large collection of mp3s that I have duplicated across several computers.  I\n> would love to be able to use git to sync changes between the copies, but there\n> are several problems: \n> \n> 1) git is really slow when dealing with thousands of multi-megabyte blobs.\n> \n> 2) commiting it to git is going to double the size of the directory, and I don't\n> really have space for that on one of the computers that the directory lives on.\n> \n> 3) there's no way to discard old history without breaking push and pull.\n> \n> I'm not sure exactly what it would take to address 1, but 2 could be addressed\n> pretty easily using btrfs file clones (once btrfs is stable), and 3 could be\n> dealt with by improving support for shallow clones.\n> \n>      --larry\n\nIn all seriousness: Why not use a tool that was actually designed for\nwhat you're trying to do? (Sync a music collection across computers.)\nSomething like syrep[0]?\n\n[0] http://0pointer.de/lennart/projects/syrep\n\n-- \nJacob Helwig\n"}]}