{"thread":{"id":"49885","subject":"How to efficiently backup a bare repository?","startedAt":"2018-11-23T10:23:34Z","lastAt":"2018-11-25T01:16:34Z","messageCount":3,"participants":["Guilhem Bonnefille","Ævar Arnfjörð Bjarmason","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"363960","messageId":"CA+BUw6gjfpiHhy+jYzxeO4NDOKiMUH0XZ3-c5o7ygdKBCKWm2Q@mail.gmail.com","threadId":"49885","inReplyTo":null,"subject":"How to efficiently backup a bare repository?","fromName":"Guilhem Bonnefille","fromEmail":"guilhem.bonnefille@gmail.com","sentAt":"2018-11-23T10:23:20Z","receivedAt":"2018-11-23T10:23:34Z","isPatch":false,"sender":{"key":"guilhem.bonnefille@gmail.com","avatar":"https://gravatar.com/avatar/375364bfee1f61197c540e37465abe3619fc24eb3a36b0edcea7f15b124036b0?d=mp&s=160"},"body":"Hi,\n\nI'm managing many bare repositories for development teams.\n\nOne service we want to offer is to let developers retrieve old state\nof the repository up to 30 days. For example, one developer\n(accidently) removed (push -f) a branch/tag and realize few days later\n(after vacations) that it was an error.\n\nWhat is the best approach to do this?\n\nCurrently, we use a classical approach, backuping all the repo every\nday. But this is far from efficient as:\n- we accumulate 30th copies of the repository\n- due to packing logic of Git, even if the content is mostly similar,\nfrom one backup to another, there is no way to deduplicate.\n\nIs there any tricks based on reflog? Even for deleted refs (branch/tags)?\nIs there any tooling playing with the internal of git to offer such\nfeature, like copying all refs in a timestamped refs directory to\nretain objects?\n\nThanks in advance for any tips letting improve the backup.\n-- \nGuilhem BONNEFILLE\n-=- JID: guyou@im.apinc.org MSN: guilhem_bonnefille@hotmail.com\n-=- mailto:guilhem.bonnefille@gmail.com\n-=- http://nathguil.free.fr/\n"},{"id":"364015","messageId":"8736rq14fe.fsf@evledraar.gmail.com","threadId":"49885","inReplyTo":"CA+BUw6gjfpiHhy+jYzxeO4NDOKiMUH0XZ3-c5o7ygdKBCKWm2Q@mail.gmail.com","subject":"Re: How to efficiently backup a bare repository?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-24T22:44:37Z","receivedAt":"2018-11-24T22:44:45Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Nov 23 2018, Guilhem Bonnefille wrote:\n\n> I'm managing many bare repositories for development teams.\n>\n> One service we want to offer is to let developers retrieve old state\n> of the repository up to 30 days. For example, one developer\n> (accidently) removed (push -f) a branch/tag and realize few days later\n> (after vacations) that it was an error.\n>\n> What is the best approach to do this?\n>\n> Currently, we use a classical approach, backuping all the repo every\n> day. But this is far from efficient as:\n> - we accumulate 30th copies of the repository\n> - due to packing logic of Git, even if the content is mostly similar,\n> from one backup to another, there is no way to deduplicate.\n>\n> Is there any tricks based on reflog? Even for deleted refs (branch/tags)?\n> Is there any tooling playing with the internal of git to offer such\n> feature, like copying all refs in a timestamped refs directory to\n> retain objects?\n>\n> Thanks in advance for any tips letting improve the backup.\n\nThere's no easy out of the box way to do exactly what you've\ndescribed. A few things come to mind:\n\na) If you can simply deny non-fast-forwards that's ideal. E.g. for some\n branches you care about, or tags. This is how most of us deal with\n this issue in practice. I.e. have some \"blessed\" refs that matter,\n and if someone rewinds their own topic branch that's their own\n problem.\n\nb) You could as you touched upon have a post-receive hook that detects\n non-fast-forwards, and e.g. pushes a clobberd \"master\" or \"v1.0\" to\n some backup repo's 2018-11-24-23-39-04-master or whatever. Then users\n could grab old versions of refs from that repo. I do a similar thing\n at work to archive certain refs (old tags), but without renaming\n them.\n\n The advantage is that you get all refs ever, the disadvantage is that\n you're not going to get a copy of the repo as it was N days ago,\n it'll need to be manually pieced together.\n\nc) Git could be made block-level de-duplication friendly. I was planning\n to work on it, but it's a small enough itch that I didn't care, but\n initial results look promising:\n https://public-inbox.org/git/20180125002942.GA21184@sigill.intra.peff.net/\n\nd) Note that if you're e.g. rsyncing repos that are actively being\n pushed into you're likely to sometimes end up with corrupt repos\n unless you're very careful about what you grab and in what\n order. Best to backup repos with \"git fetch\".\n\ne) If you're burned by one-off cases like this dev going away for 30\n days you could bump the default expiry that comes with git from 2\n weeks to e.g. 6 weeks. It's still a manual process to recover data\n (with fsck etc), but at least it's there.\n"},{"id":"364017","messageId":"xmqqy39i0xee.fsf@gitster-ct.c.googlers.com","threadId":"49885","inReplyTo":"8736rq14fe.fsf@evledraar.gmail.com","subject":"Re: How to efficiently backup a bare repository?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-25T01:16:25Z","receivedAt":"2018-11-25T01:16:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> There's no easy out of the box way to do exactly what you've\n> described. A few things come to mind:\n> ...\n\nWouldn't it suffice to have a cron job that runs something like\n\n\tD=$(date +\"%Y-%m-%d\")\n\tgit fetch $serving \"refs/*:refs/backup-$D/*\"\n\non the back-up box to fetch from the repository on the box the\nend-users push into once a day? In the back-up repository, the\nrefs/backup-2018-11-25/heads/master reference would be today's tip\nof the master branch of the serving repository. You can set the\nexpiry timeout to \"now\" (i.e. \"gc\" will immediately drop unreachable\nobjects, and that is fine because you expicitly have refs to pin\nthese objects anyway), get the dedup from \"git fetch\" for free,\nrepack the backup repository as a whole, and dropping the whole\nrefs/backup-2018-10-25/* hierarcy on 2018-11-25 is all you need to\nexpire the refs.\n\nYou may want to play with the ref-advertisement limiting options in\nthe recent Git, if it is too much to grow the amount of \"have\"s by\n30x for the common ancestry negotiation. But that is a small\nimplementation detail.\n\n"}]}