git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Using git as a general backup mechanism (was Re: Using GIT to store /etc)

From
Steven Grimm <koreth@midwinter.com>
Date
Dec 12, 2006, 22:49 UTC
Message-ID
<457F31E6.8090701@midwinter.com>
In-Reply-To
<200612121553.37499.andyparkins@gmail.com>

This discussion reminds me of a use of git I've had in the back of my head to try out for a while. Right now I'm doing my local snapshot backups using the rsync-with-hard-links scheme (http://www.mikerubel.org/computers/rsync_snapshots/ if you're not familiar with it). This is nice in that the contents of files that don't change are only stored once on the backup disk. But it is less than optimal in that a file that changes even a little bit is stored from scratch.

What would be great for this would be to store each day's backup as a git revision; with a periodic repack, this would be much more space-efficient than the rsync hard links.

The problem is that while that would give me a very efficient backup scheme, the repository would still grow over time. In rsync land, I solve the disk space issue by keeping two weeks' worth of daily snapshots, then six months' worth of weekly snapshots, then two years' worth of monthly snapshots; files that change daily have a constant number of revisions stored in my backups, and older files drop off the backup disk as they age.

Given that there's no way (or is there?) to delete revisions from the *beginning* of a git revision history, right now it seems like the only approach that comes close is to give up on the "daily then weekly then monthly" thing -- probably fine given the space savings of delta compression -- and periodically make shallow clones of the backup repository that fetch all but the first N revisions; once a shallow clone is made, the original gets deleted and the clone is the new backup repo.

But it would sure be more efficient to be able to "shallow-ize" an existing repository. That would be useful for things other than backups, too, e.g. the recent request for some way to track just the current version of the kernel code rather than its revision history. If there were a shallowize command, you could do something like "git pull; git shallowize --depth 1" to track the latest revision without keeping the history locally.

Anyone think that sounds like an interesting thing to explore?
-Steve
Previous: Andy ParkinsNext: Johannes Schindelin
Message 24 of 34 in “Using GIT to store /etc (Or: How to make GIT store all file permission bits)”
  1. Kyle MoffettDec 10, 2006
  2. Jeff GarzikDec 10, 2006
  3. Jakub NarebskiDec 10, 2006
  4. Kyle MoffettDec 10, 2006
  5. Jakub NarebskiDec 10, 2006
  6. Jakub NarebskiDec 10, 2006
  7. Kyle MoffettDec 10, 2006
  8. Andreas EricssonDec 11, 2006
  9. Jeff GarzikDec 11, 2006
  10. Josef WeidendorferDec 11, 2006
  11. Johannes SchindelinDec 11, 2006
  12. Josef WeidendorferDec 11, 2006
  13. Santi BéjarDec 10, 2006
  14. Kyle MoffettDec 10, 2006
  15. Jakub NarebskiDec 10, 2006
  16. David LangJan 10, 2007
  17. Shawn O. PearceJan 10, 2007
  18. David LangJan 10, 2007
  19. Shawn O. PearceJan 12, 2007
  20. Nikolai WeibullDec 11, 2006
  21. Daniel BarkalowDec 12, 2006
  22. Kyle MoffettDec 12, 2006
  23. Andy ParkinsDec 12, 2006
  24. Using git as a general backup mechanism (was Re: Using GIT to store /etc)Steven Grimm, Dec 12, 2006
  25. Johannes SchindelinDec 12, 2006
  26. Steven GrimmDec 12, 2006
  27. Johannes SchindelinDec 13, 2006
  28. Martin LanghoffDec 12, 2006
  29. Martin LanghoffDec 12, 2006
  30. Junio C HamanoDec 12, 2006
  31. Steven GrimmDec 14, 2006
  32. Junio C HamanoDec 15, 2006
  33. Daniel BarkalowDec 13, 2006
  34. Chris RiddochDec 14, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.