{"thread":{"id":"26254","subject":"git-repack & big files","startedAt":"2011-01-11T07:37:18Z","lastAt":"2011-01-13T08:00:40Z","messageCount":6,"participants":["Pietro Battiston","Phillip Susi","Stephen Bash"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"159308","messageId":"1294731438.3300.973.camel@voubian.casa","threadId":"26254","inReplyTo":null,"subject":"git-repack & big files","fromName":"Pietro Battiston","fromEmail":"me@pietrobattiston.it","sentAt":"2011-01-11T07:37:18Z","receivedAt":"2011-01-11T07:37:18Z","isPatch":false,"sender":{"key":"me@pietrobattiston.it","avatar":"https://gravatar.com/avatar/6de10c7b53eb76c5b2c03b58c9feee9291ce8e953ca759281075323ed022014e?d=mp&s=160"},"body":"Hello,\n\nfirst, I do know git is not optimized for big files, and that's fine.\nBut it is able, on my machine with 3 GB of RAM, to succesfully backup my\nhome directory¹, which contains, among others, several files of several\nhundreds of megabytes each. And I like that a lot.\n\nSince it perfectly does what it is not optimized to do... I then wonder\nwhen it does not do what it declares: if I run git-repack² with the\nparameter --window-memory set to, for instance, \"100m\", it takes\nhundreds and hundreds of MB of memory until it runs out of memory, fails\na malloc and aborts.\nSo, two questions:\n\n1) is there a bug, is the documentation about that parameter a bit too\noptimistic or did I just not understand it?\n\n2) do I have any hope that in one way or another my 500+ MB mailboxes\nwith relatively small changes over time are archived smartly (=diffs) by\ngit at the current state of development? If I understand correctly, the\nproject git-bigfiles³ would just \"solve\" my problems by not making\ndifferences of big files.\n\nthanks for the clarifications\n\nPietro\n\n\n¹ Just for the records: through gibak:\nhttp://eigenclass.org/hiki/gibak-0.3.0\n\n² git version 1:1.7.2.3-2.2 on Debian\n\n³ http://caca.zoy.org/wiki/git-bigfiles\n"},{"id":"159346","messageId":"4D2C7AA9.1020506@cfl.rr.com","threadId":"26254","inReplyTo":"1294731438.3300.973.camel@voubian.casa","subject":"Re: git-repack & big files","fromName":"Phillip Susi","fromEmail":"psusi@cfl.rr.com","sentAt":"2011-01-11T15:43:37Z","receivedAt":"2011-01-11T15:43:37Z","isPatch":false,"sender":{"key":"psusi@cfl.rr.com","avatar":null},"body":"On 1/11/2011 2:37 AM, Pietro Battiston wrote:\n> Since it perfectly does what it is not optimized to do... I then wonder\n> when it does not do what it declares: if I run git-repack² with the\n> parameter --window-memory set to, for instance, \"100m\", it takes\n> hundreds and hundreds of MB of memory until it runs out of memory, fails\n> a malloc and aborts.\n> So, two questions:\n\n--window-memory reduces the window size to try and stay under the limit,\n but the window size can not be reduced below 1.\n\n> 2) do I have any hope that in one way or another my 500+ MB mailboxes\n> with relatively small changes over time are archived smartly (=diffs) by\n> git at the current state of development? If I understand correctly, the\n> project git-bigfiles³ would just \"solve\" my problems by not making\n> differences of big files.\n\nGit is not a backup tool.  You should use rsync rdiff-backup instead.\n"},{"id":"159365","messageId":"1294772603.3435.67.camel@voubian.casa","threadId":"26254","inReplyTo":"4D2C7AA9.1020506@cfl.rr.com","subject":"Re: git-repack & big files","fromName":"Pietro Battiston","fromEmail":"me@pietrobattiston.it","sentAt":"2011-01-11T19:03:23Z","receivedAt":"2011-01-11T19:03:23Z","isPatch":false,"sender":{"key":"me@pietrobattiston.it","avatar":"https://gravatar.com/avatar/6de10c7b53eb76c5b2c03b58c9feee9291ce8e953ca759281075323ed022014e?d=mp&s=160"},"body":"Il giorno mar, 11/01/2011 alle 10.43 -0500, Phillip Susi ha scritto:\n> On 1/11/2011 2:37 AM, Pietro Battiston wrote:\n> > Since it perfectly does what it is not optimized to do... I then wonder\n> > when it does not do what it declares: if I run git-repack² with the\n> > parameter --window-memory set to, for instance, \"100m\", it takes\n> > hundreds and hundreds of MB of memory until it runs out of memory, fails\n> > a malloc and aborts.\n> > So, two questions:\n> \n> --window-memory reduces the window size to try and stay under the limit,\n>  but the window size can not be reduced below 1.\n\nOK, I think I understood. Still, I don't think there are many doubts\nthat the documentation is misleading when it says \"the window size will\ndynamically scale down so as to not take up more than N bytes in\nmemory\". That's all.\n\n> > 2) do I have any hope that in one way or another my 500+ MB mailboxes\n> > with relatively small changes over time are archived smartly (=diffs) by\n> > git at the current state of development? If I understand correctly, the\n> > project git-bigfiles³ would just \"solve\" my problems by not making\n> > differences of big files.\n> \n> Git is not a backup tool.  You should use rsync rdiff-backup instead.\n\nThat's unfortunate - I think I prefer to split my mailboxes than to\nloose many of the nice features git provides. But thanks a lot for the\nsuggestion.\n\nPietro\n"},{"id":"159366","messageId":"19558451.124576.1294773160655.JavaMail.root@mail.hq.genarts.com","threadId":"26254","inReplyTo":"1294772603.3435.67.camel@voubian.casa","subject":"Re: git-repack & big files","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2011-01-11T19:12:40Z","receivedAt":"2011-01-11T19:12:40Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"----- Original Message -----\n> From: \"Pietro Battiston\" <me@pietrobattiston.it>\n> To: \"Phillip Susi\" <psusi@cfl.rr.com>\n> Sent: Tuesday, January 11, 2011 2:03:23 PM\n> Subject: Re: git-repack & big files\n>\n> > > 2) do I have any hope that in one way or another my 500+ MB\n> > > mailboxes\n> > > with relatively small changes over time are archived smartly\n> > > (=diffs) by\n> > > git at the current state of development? If I understand\n> > > correctly, the\n> > > project git-bigfiles³ would just \"solve\" my problems by not making\n> > > differences of big files.\n> >\n> > Git is not a backup tool. You should use rsync rdiff-backup instead.\n> \n> That's unfortunate - I think I prefer to split my mailboxes than to\n> loose many of the nice features git provides. But thanks a lot for the\n> suggestion.\n\nOff topic: Do you have the option to change to a solution that uses MailDir as the storage format rather than mbox?\n   \n   http://en.wikipedia.org/wiki/Maildir\n\nStephen\n"},{"id":"159372","messageId":"4D2CB3F5.106@cfl.rr.com","threadId":"26254","inReplyTo":"1294772603.3435.67.camel@voubian.casa","subject":"Re: git-repack & big files","fromName":"Phillip Susi","fromEmail":"psusi@cfl.rr.com","sentAt":"2011-01-11T19:48:05Z","receivedAt":"2011-01-11T19:48:05Z","isPatch":false,"sender":{"key":"psusi@cfl.rr.com","avatar":null},"body":"On 1/11/2011 2:03 PM, Pietro Battiston wrote:\n> That's unfortunate - I think I prefer to split my mailboxes than to\n> loose many of the nice features git provides. But thanks a lot for the\n> suggestion.\n\nI'm curious what features of git you find helpful for this purpose.  The\nhistory log doesn't seem useful at all.  Generally mail is only added,\nand sometimes deleted, never changed, so it also does not seem useful to\nkeep multiple revisions.  If you really want that though, rdiff-backup\nwill do that and keep the old revisions delta compressed.\n\nI use Maildir instead of mbox and do a nightly incremental backup of the\nwhole system with dump, so any mail I might loose by accident I can pull\nfrom the backup if it is older than a day, and I never delete mail.\n"},{"id":"159429","messageId":"1294905640.4033.920.camel@voubian.casa","threadId":"26254","inReplyTo":"4D2CB3F5.106@cfl.rr.com","subject":"Re: git-repack & big files","fromName":"Pietro Battiston","fromEmail":"me@pietrobattiston.it","sentAt":"2011-01-13T08:00:40Z","receivedAt":"2011-01-13T08:00:40Z","isPatch":false,"sender":{"key":"me@pietrobattiston.it","avatar":"https://gravatar.com/avatar/6de10c7b53eb76c5b2c03b58c9feee9291ce8e953ca759281075323ed022014e?d=mp&s=160"},"body":"Il giorno mar, 11/01/2011 alle 14.48 -0500, Phillip Susi ha scritto: \n> On 1/11/2011 2:03 PM, Pietro Battiston wrote:\n> > That's unfortunate - I think I prefer to split my mailboxes than to\n> > loose many of the nice features git provides. But thanks a lot for the\n> > suggestion.\n> \n> I'm curious what features of git you find helpful for this purpose.  \n\nMany more, I guess, that the ones I'll be able to remember now. But for\ninstance some features that make it better than rdiff-backup are:\n\n1) I like to see how a given file changed at a given point in time with\na comfortable interface - not just \"restore this there\", or search for\nthe right diff gzipped somewhere\n\n2) I like to delete some given files/folders that I forgot to\n(.git)ignore from all the backups with a single command\n\n3) I love the fact that if I move/rename a file/folder, git notices it\n(and doesn't think I just deleted some files and created some others).\nSince I often move/rename files/folders, when I knew git I really though\n\"after years of waiting, finally backups will be smart\".\n\n4) in principle - though I admit I still never tried - I like the idea\nthat if I have two copies of the git repo, I can backup once on each\n(think of one staying home and one following me when I travel) and then\nrebase one on the other\n\n5) to backup my home the first time, rdiff-backup takes slightly less\nthan 5 hours and uses 32 GB , git takes around 2 hours and uses 17 GB\n\n6) in general, just having a powerful interface I'm used to\n\n> The\n> history log doesn't seem useful at all. \n\nI also like the fact that my commits have comments such as \"before\nchanging PC\", \"after system upgrade\", \"before reordering mail\"...\n\n\n> Generally mail is only added,\n> and sometimes deleted, never changed, so it also does not seem useful to\n> keep multiple revisions.  \n\nI'm not sure I get what you mean - mail is added and deleted, hence the\nmailbox is changed, hence I find it useful to keep multiple revisions.\n\n\n> If you really want that though, rdiff-backup\n> will do that and keep the old revisions delta compressed.\n> \n\nYes, I think I will live with rdiff-backup. And miss git, but I\nperfectly understand that git simply doesn't aim at solving my problem,\nand that's fair.\n\n\n> I use Maildir instead of mbox and do a nightly incremental backup of the\n> whole system with dump, so any mail I might loose by accident I can pull\n> from the backup if it is older than a day, and I never delete mail.\n\nYep, Maildir is nice from this point of view. But nope, it is not\npractical for me to change now.\n\nthanks to all for suggestions (and sorry for the OT)\n\nPietro\n"}]}