{"thread":{"id":"24356","subject":"Cutting history","startedAt":"2010-07-10T03:25:53Z","lastAt":"2010-07-10T20:12:01Z","messageCount":7,"participants":["Enrico Weigelt","Joshua Jensen","Chris Frey","Jakub Narebski","Martin Pettersson","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"145259","messageId":"20100710032553.GB554@nibiru.local","threadId":"24356","inReplyTo":null,"subject":"Cutting history","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-07-10T03:25:53Z","receivedAt":"2010-07-10T03:25:53Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"\nHi folks,\n\n\nI'm using git for automatic backups (eg. database dumps). This \nworks quite well, but as time goes, the history (and so the repo)\ngets larger and larger. It would be really nice to allow cutting\noff old stuff (eg. after N commits in the past). \n\nMaybe that could be done by introducing \"stopper\" tags: commits\nthat have an stopper-tag may have missing parents, and git-gc\ncan be told to ignore those parents and throw away everything\nbehind the stopper (if not referenced otherwise).\n\nA probably cleaner, but more invasive way could be making refs\nto vectors, which may contain stop points (multiple ones in case\nof merges) additionally to the start point. Remote transmits only\ncontain the commits within this range, and GC also just scans\nthe range (instead of following all parents).\n\n\nWhat do you think about this ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"145260","messageId":"4C37F24E.30407@workspacewhiz.com","threadId":"24356","inReplyTo":"20100710032553.GB554@nibiru.local","subject":"Re: Cutting history","fromName":"Joshua Jensen","fromEmail":"jjensen@workspacewhiz.com","sentAt":"2010-07-10T04:08:46Z","receivedAt":"2010-07-10T04:08:46Z","isPatch":false,"sender":{"key":"jjensen@workspacewhiz.com","avatar":"https://avatars.githubusercontent.com/u/111687?v=4"},"body":" ----- Original Message -----\nFrom: Enrico Weigelt\nDate: 7/9/2010 9:25 PM\n> I'm using git for automatic backups (eg. database dumps). This\n> works quite well, but as time goes, the history (and so the repo)\n> gets larger and larger. It would be really nice to allow cutting\n> off old stuff (eg. after N commits in the past).\n>\n> Maybe that could be done by introducing \"stopper\" tags: commits\n> that have an stopper-tag may have missing parents, and git-gc\n> can be told to ignore those parents and throw away everything\n> behind the stopper (if not referenced otherwise).\n>\n> A probably cleaner, but more invasive way could be making refs\n> to vectors, which may contain stop points (multiple ones in case\n> of merges) additionally to the start point. Remote transmits only\n> contain the commits within this range, and GC also just scans\n> the range (instead of following all parents).\nYour post reminded me of this: http://progit.org/2010/03/17/replace.html\n\nJosh\n"},{"id":"145262","messageId":"20100710064304.GA15600@foursquare.net","threadId":"24356","inReplyTo":"4C37F24E.30407@workspacewhiz.com","subject":"Re: Cutting history","fromName":"Chris Frey","fromEmail":"cdfrey@foursquare.net","sentAt":"2010-07-10T06:43:04Z","receivedAt":"2010-07-10T06:43:04Z","isPatch":false,"sender":{"key":"cdfrey@foursquare.net","avatar":null},"body":"On Fri, Jul 09, 2010 at 10:08:46PM -0600, Joshua Jensen wrote:\n> Your post reminded me of this: http://progit.org/2010/03/17/replace.html\n\nWow. This is what I get for not following git development more closely. :-)\n\nDoesn't this open a potential security problem? Suppose you want to pull\nfrom another developer's repo, and he's replaced some of the history\nin your own tree. According to the above article, it is possible to\nshare replacements, presumably like any other ref.\n\nIs it possible to have your own branch history \"replaced\" by fetching\nsomeone else's repo into a remote branch?\n\n- Chris\n"},{"id":"145266","messageId":"m3tyo7lo6n.fsf@localhost.localdomain","threadId":"24356","inReplyTo":"4C37F24E.30407@workspacewhiz.com","subject":"Re: Cutting history","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-07-10T08:47:14Z","receivedAt":"2010-07-10T08:47:14Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Joshua Jensen writes:\n\n> ----- Original Message -----\n> From: Enrico Weigelt\n> Date: 7/9/2010 9:25 PM\n>\n> > I'm using git for automatic backups (eg. database dumps). This\n> > works quite well, but as time goes, the history (and so the repo)\n> > gets larger and larger. It would be really nice to allow cutting\n> > off old stuff (eg. after N commits in the past).\n\nThis is certainly Using Git For What It Was Not Intended...\n\n> >\n> > Maybe that could be done by introducing \"stopper\" tags: commits\n> > that have an stopper-tag may have missing parents, and git-gc\n> > can be told to ignore those parents and throw away everything\n> > behind the stopper (if not referenced otherwise).\n> >\n> > A probably cleaner, but more invasive way could be making refs\n> > to vectors, which may contain stop points (multiple ones in case\n> > of merges) additionally to the start point. Remote transmits only\n> > contain the commits within this range, and GC also just scans\n> > the range (instead of following all parents).\n>\n> Your post reminded me of this: http://progit.org/2010/03/17/replace.html\n\nAnother solution would be to make history shallower like shallow clone\n(\"git clone --depth \") does it[1], and then prune history. Or\nyou can use grafts to cauterize history.\n\nBoth of those solutions have disadvantages wrt pushing and pulling to\nother repositories (shallow clone less so), but I don't think that\nwould be a problem for your situation.\n\n[1] Documentation/technical/shallow.txt \n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"145269","messageId":"201007101740.20854.martin@siamect.com","threadId":"24356","inReplyTo":"m3tyo7lo6n.fsf@localhost.localdomain","subject":"Re: Cutting history","fromName":"Martin Pettersson","fromEmail":"martin@siamect.com","sentAt":"2010-07-10T10:40:20Z","receivedAt":"2010-07-10T10:40:20Z","isPatch":false,"sender":{"key":"martin@siamect.com","avatar":"https://gravatar.com/avatar/97ed731b9c70a7342247447388d2d3581a1316ad8b69979ab88efd37b9de5f1a?d=mp&s=160"},"body":"On Saturday, July 10, 2010 03:47:14 pm you wrote:\n> Joshua Jensen writes:\n> > ----- Original Message -----\n> > \n> > From: Enrico Weigelt\n> > Date: 7/9/2010 9:25 PM\n> > \n> > > I'm using git for automatic backups (eg. database dumps). This\n> > > works quite well, but as time goes, the history (and so the repo)\n> > > gets larger and larger. It would be really nice to allow cutting\n> > > off old stuff (eg. after N commits in the past).\n> \n> This is certainly Using Git For What It Was Not Intended...\n> \n> > > Maybe that could be done by introducing \"stopper\" tags: commits\n> > > that have an stopper-tag may have missing parents, and git-gc\n> > > can be told to ignore those parents and throw away everything\n> > > behind the stopper (if not referenced otherwise).\n> > > \n> > > A probably cleaner, but more invasive way could be making refs\n> > > to vectors, which may contain stop points (multiple ones in case\n> > > of merges) additionally to the start point. Remote transmits only\n> > > contain the commits within this range, and GC also just scans\n> > > the range (instead of following all parents).\n> > \n> > Your post reminded me of this: http://progit.org/2010/03/17/replace.html\n> \n> Another solution would be to make history shallower like shallow clone\n> (\"git clone --depth \") does it[1], and then prune history. Or\n> you can use grafts to cauterize history.\n> \n> Both of those solutions have disadvantages wrt pushing and pulling to\n> other repositories (shallow clone less so), but I don't think that\n> would be a problem for your situation.\n> \n> [1] Documentation/technical/shallow.txt\n\nDon't complicate things, just make new repo when the old one is too large. \nThat is what I do and it is for me the best backup system I ever had.\nMartin\n"},{"id":"145272","messageId":"AANLkTikY8RKseD8K4RVrLHnSdW_Su8hVRPRFkzzz1rGv@mail.gmail.com","threadId":"24356","inReplyTo":"m3tyo7lo6n.fsf@localhost.localdomain","subject":"Re: Cutting history","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-10T11:58:50Z","receivedAt":"2010-07-10T11:58:50Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, Jul 10, 2010 at 08:47, Jakub Narebski wrote:\n> Joshua Jensen writes:\n>\n>>   ----- Original Message -----\n>> From: Enrico Weigelt\n>> Date: 7/9/2010 9:25 PM\n>>\n>> > I'm using git for automatic backups (eg. database dumps). This\n>> > works quite well, but as time goes, the history (and so the repo)\n>> > gets larger and larger. It would be really nice to allow cutting\n>> > off old stuff (eg. after N commits in the past).\n>\n> This is certainly Using Git For What It Was Not Intended...\n\nIt actually works very well though. I use Git to back up MySQL\ndatabases like this.\n\nHere's the script I use to dump MySQL databases:\n\n http://github.com/avar/linode-etc/blob/master/bin/cron/mysqldump-to-git\n\nAnd a small wrapper to dump them all:\n\n http://github.com/avar/linode-etc/blob/master/bin/cron/mysqldump-to-git-all\n\nI make dumps every 6 hours:\n\n http://github.com/avar/linode-etc/blob/master/cron.d/v-mysql-git-backup\n\nAnd after each dump I repack & prune (some of this is probably\nredundant given the linear history) the repository:\n\n http://github.com/avar/linode-etc/blob/master/bin/cron/git-repack-and-gc-dir\n\nAnd here's graph showing how big the dumps get:\n\n http://munin.nix.is/nix.is/v.nix.is/dirs_var_backup_mysql.html\n\nThe climbing charts before the size cutoff were before I started\nrepacking them.\n\nAs for pruning old history, I thought this *should* work for pruning\nhistory older than 7 days (given that you dump daily):\n\n git rebase --strategy=base --onto master~8 master~7\n\nBut of course that deletes new commits. I need to freshen up on my\nrebase understanding. Maybe someone else on list knows how to do\nthat. I thought git rebase --interactive might work, but I can't get\nit to display the root commit. Maybe you need git-filter-branch.\n"},{"id":"145280","messageId":"AANLkTilcw6gpGI1TO_iRmComID3n0w2biox0ac1uzNyU@mail.gmail.com","threadId":"24356","inReplyTo":"AANLkTikY8RKseD8K4RVrLHnSdW_Su8hVRPRFkzzz1rGv@mail.gmail.com","subject":"Re: Cutting history","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2010-07-10T20:12:01Z","receivedAt":"2010-07-10T20:12:01Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, Jul 10, 2010 at 11:58, Ævar Arnfjörð Bjarmason wrote:\n\n> As for pruning old history, I thought this *should* work for pruning\n> history older than 7 days (given that you dump daily):\n>\n>    git rebase --strategy=base --onto master~8 master~7\n>\n> But of course that deletes new commits. I need to freshen up on my\n> rebase understanding. Maybe someone else on list knows how to do\n> that. I thought git rebase --interactive might work, but I can't get\n> it to display the root commit. Maybe you need git-filter-branch.\n\nThiago Macieira on #git provided the answer. You can do that with\ngrafts and git filter-branch. E.g. rewriting the history so that you\nonly have the 7 latest commits:\n\n git rev-list HEAD | sed '7q;d' > .git/info/grafts &&\n test -s .git/info/grafts &&\n git filter-branch -f HEAD\n"}]}