threads / discuss / 24356

Cutting history

Subject: Cutting history

## tl;dr

7 messages between Jul 10, 2010 and Jul 10, 2010.

replies: 6people: 6as markdown or json

Enrico Weigelt· Jul 10, 2010, 03:25 UTC · lore
Hi folks,

I'm using git for automatic backups (eg. database dumps). This works quite well, but as time goes, the history (and so the repo) gets larger and larger. It would be really nice to allow cutting off old stuff (eg. after N commits in the past).

Maybe that could be done by introducing "stopper" tags: commits that have an stopper-tag may have missing parents, and git-gc can be told to ignore those parents and throw away everything behind the stopper (if not referenced otherwise).

A probably cleaner, but more invasive way could be making refs to vectors, which may contain stop points (multiple ones in case of merges) additionally to the start point. Remote transmits only contain the commits within this range, and GC also just scans the range (instead of following all parents).

What do you think about this ?
cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------
Joshua Jensen· Jul 10, 2010, 04:08 UTC · re: Enrico Weigelt · lore

Re: Cutting history

  ----- Original Message -----
From: Enrico Weigelt
Date: 7/9/2010 9:25 PM
Show 15 quoted lines
> I'm using git for automatic backups (eg. database dumps). This
> works quite well, but as time goes, the history (and so the repo)
> gets larger and larger. It would be really nice to allow cutting
> off old stuff (eg. after N commits in the past).
>
> Maybe that could be done by introducing "stopper" tags: commits
> that have an stopper-tag may have missing parents, and git-gc
> can be told to ignore those parents and throw away everything
> behind the stopper (if not referenced otherwise).
>
> A probably cleaner, but more invasive way could be making refs
> to vectors, which may contain stop points (multiple ones in case
> of merges) additionally to the start point. Remote transmits only
> contain the commits within this range, and GC also just scans
> the range (instead of following all parents).
Your post reminded me of this: http://progit.org/2010/03/17/replace.html
Josh
Chris Frey· Jul 10, 2010, 06:43 UTC · re: Joshua Jensen · lore

Re: Cutting history

On Fri, Jul 09, 2010 at 10:08:46PM -0600, Joshua Jensen wrote:
> Your post reminded me of this: http://progit.org/2010/03/17/replace.html
Wow.  This is what I get for not following git development more closely. :-)

Doesn't this open a potential security problem? Suppose you want to pull from another developer's repo, and he's replaced some of the history in your own tree. According to the above article, it is possible to share replacements, presumably like any other ref.

Is it possible to have your own branch history "replaced" by fetching someone else's repo into a remote branch?

- Chris
Jakub Narebski· Jul 10, 2010, 08:47 UTC · re: Joshua Jensen · lore

Re: Cutting history

Joshua Jensen <jjensen@workspacewhiz.com> writes:
Show 8 quoted lines
>   ----- Original Message -----
> From: Enrico Weigelt
> Date: 7/9/2010 9:25 PM
>
> > I'm using git for automatic backups (eg. database dumps). This
> > works quite well, but as time goes, the history (and so the repo)
> > gets larger and larger. It would be really nice to allow cutting
> > off old stuff (eg. after N commits in the past).
This is certainly Using Git For What It Was Not Intended...
Show 13 quoted lines
> >
> > Maybe that could be done by introducing "stopper" tags: commits
> > that have an stopper-tag may have missing parents, and git-gc
> > can be told to ignore those parents and throw away everything
> > behind the stopper (if not referenced otherwise).
> >
> > A probably cleaner, but more invasive way could be making refs
> > to vectors, which may contain stop points (multiple ones in case
> > of merges) additionally to the start point. Remote transmits only
> > contain the commits within this range, and GC also just scans
> > the range (instead of following all parents).
>
> Your post reminded me of this: http://progit.org/2010/03/17/replace.html

Another solution would be to make history shallower like shallow clone ("git clone --depth <depth>") does it[1], and then prune history. Or you can use grafts to cauterize history.

Both of those solutions have disadvantages wrt pushing and pulling to other repositories (shallow clone less so), but I don't think that would be a problem for your situation.

[1] Documentation/technical/shallow.txt 
-- 
Jakub Narebski
Poland
ShadeHawk on #git
Martin Pettersson· Jul 10, 2010, 10:40 UTC · re: Jakub Narebski · lore

Re: Cutting history

On Saturday, July 10, 2010 03:47:14 pm you wrote:
Show 35 quoted lines
> Joshua Jensen <jjensen@workspacewhiz.com> writes:
> >   ----- Original Message -----
> > 
> > From: Enrico Weigelt
> > Date: 7/9/2010 9:25 PM
> > 
> > > I'm using git for automatic backups (eg. database dumps). This
> > > works quite well, but as time goes, the history (and so the repo)
> > > gets larger and larger. It would be really nice to allow cutting
> > > off old stuff (eg. after N commits in the past).
> 
> This is certainly Using Git For What It Was Not Intended...
> 
> > > Maybe that could be done by introducing "stopper" tags: commits
> > > that have an stopper-tag may have missing parents, and git-gc
> > > can be told to ignore those parents and throw away everything
> > > behind the stopper (if not referenced otherwise).
> > > 
> > > A probably cleaner, but more invasive way could be making refs
> > > to vectors, which may contain stop points (multiple ones in case
> > > of merges) additionally to the start point. Remote transmits only
> > > contain the commits within this range, and GC also just scans
> > > the range (instead of following all parents).
> > 
> > Your post reminded me of this: http://progit.org/2010/03/17/replace.html
> 
> Another solution would be to make history shallower like shallow clone
> ("git clone --depth <depth>") does it[1], and then prune history.  Or
> you can use grafts to cauterize history.
> 
> Both of those solutions have disadvantages wrt pushing and pulling to
> other repositories (shallow clone less so), but I don't think that
> would be a problem for your situation.
> 
> [1] Documentation/technical/shallow.txt

Don't complicate things, just make new repo when the old one is too large. That is what I do and it is for me the best backup system I ever had. Martin

Ævar Arnfjörð Bjarmason· Jul 10, 2010, 11:58 UTC · re: Jakub Narebski · lore

Re: Cutting history

On Sat, Jul 10, 2010 at 08:47, Jakub Narebski <jnareb@gmail.com> wrote:
Show 12 quoted lines
> Joshua Jensen <jjensen@workspacewhiz.com> writes:
>
>>   ----- Original Message -----
>> From: Enrico Weigelt
>> Date: 7/9/2010 9:25 PM
>>
>> > I'm using git for automatic backups (eg. database dumps). This
>> > works quite well, but as time goes, the history (and so the repo)
>> > gets larger and larger. It would be really nice to allow cutting
>> > off old stuff (eg. after N commits in the past).
>
> This is certainly Using Git For What It Was Not Intended...

It actually works very well though. I use Git to back up MySQL databases like this.

Here's the script I use to dump MySQL databases:
    http://github.com/avar/linode-etc/blob/master/bin/cron/mysqldump-to-git
And a small wrapper to dump them all:
    http://github.com/avar/linode-etc/blob/master/bin/cron/mysqldump-to-git-all
I make dumps every 6 hours:
    http://github.com/avar/linode-etc/blob/master/cron.d/v-mysql-git-backup

And after each dump I repack & prune (some of this is probably redundant given the linear history) the repository:

    http://github.com/avar/linode-etc/blob/master/bin/cron/git-repack-and-gc-dir
And here's graph showing how big the dumps get:
    http://munin.nix.is/nix.is/v.nix.is/dirs_var_backup_mysql.html

The climbing charts before the size cutoff were before I started repacking them.

As for pruning old history, I thought this *should* work for pruning history older than 7 days (given that you dump daily):

    git rebase --strategy=base --onto master~8 master~7

But of course that deletes new commits. I need to freshen up on my rebase understanding. Maybe someone else on list knows how to do that. I thought git rebase --interactive might work, but I can't get it to display the root commit. Maybe you need git-filter-branch.

Ævar Arnfjörð Bjarmason· Jul 10, 2010, 20:12 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: Cutting history

On Sat, Jul 10, 2010 at 11:58, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
Show 9 quoted lines
> As for pruning old history, I thought this *should* work for pruning
> history older than 7 days (given that you dump daily):
>
>    git rebase --strategy=base --onto master~8 master~7
>
> But of course that deletes new commits. I need to freshen up on my
> rebase understanding. Maybe someone else on list knows how to do
> that. I thought git rebase --interactive might work, but I can't get
> it to display the root commit. Maybe you need git-filter-branch.

Thiago Macieira on #git provided the answer. You can do that with grafts and git filter-branch. E.g. rewriting the history so that you only have the 7 latest commits:

    git rev-list HEAD | sed '7q;d' > .git/info/grafts &&
    test -s .git/info/grafts &&
    git filter-branch -f HEAD

← back to recent threads