{"thread":{"id":"41268","subject":"[ANNOUNCE] Git-backup - Backup set of Git repositories & just files; efficiently","startedAt":"2016-01-27T15:09:01Z","lastAt":"2016-01-27T15:09:01Z","messageCount":1,"participants":["Kirill Smelkov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"276890","messageId":"20160127150900.GB32669@teco.navytux.spb.ru","threadId":"41268","inReplyTo":null,"subject":"[ANNOUNCE] Git-backup - Backup set of Git repositories & just files; efficiently","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-01-27T15:09:01Z","receivedAt":"2016-01-27T15:09:01Z","isPatch":false,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"https://lab.nexedi.com/kirr/git-backup.git\n\nThis program backups files and set of bare Git repositories into one Git repository.\nFiles are copied to blobs and then added to tree under certain place, and for\nGit repositories, all reachable objects are pulled in with maintaining index\nwhich remembers reference -> sha1 for all pulled repositories.\n\nThis allows to leverage Git's good data deduplication ability, especially for\ncases when there are many hosted repositories which are forks of each other,\nand for backup to have history and be otherwise managed as a usual Git\nrepository.  In particular it is possible to use standard git pull/push to\nsynchronize backups in several places.\n\nBackup workflow is:\n\n1. create backup repository::\n\n     $ mkdir backup\n     $ cd backup\n     $ git init         # both bare and non-bare possible\n\n2. pull files and Git repositories into backup repository::\n\n     $ git-backup pull dir1:prefix1 dir2:prefix2 ...\n\n   This will pull bare Git repositories & just files from `dir1` into backup\n   under `prefix1`, from `dir2` into backup prefix `prefix2`, etc...\n\n3. restore files and Git repositories from backup::\n\n     $ git-backup restore <backup-state-sha1> prefix1:dir1\n\n   Restore Git repositories & just files from backup `prefix1` into `dir1`,\n   from backup `prefix2` into `dir2`, etc...\n\n   Backup state to restore is taken from <backup-state-sha1> which is sha1 or\n   ref pointing to backup repository state.\n\n4. backup repository itself can be managed with Git. In particular it can be\n   synchronized between several places with standard git pull/push, be\n   repacked, etc::\n\n     $ git push ...\n     $ git pull ...\n\n\nTechnical overview of how it works\n----------------------------------\n\nAfter objects from backuped Git repositories are pulled in, we create new\ncommit which references tree with changed backup index and files, and also has\nall head objects from pulled-in repositories in its parents(*). This way backup\nhas history and all pulled objects become reachable from single head commit in\nbackup repository. In particular that means that the whole state of backup can\nbe described with only single sha1, and that backup repository itself could be\nsynchronized via standard git pull/push, be repacked, etc.\n\nRestoration process is the opposite - from a particular backup state, files are\nextracted at a proper place, and for Git repositories a pack with all objects\nreachable from that repository heads is prepared and extracted from backup\nrepository object database.\n\nData for all backuped files and repositories can be accessed if one has access\nto backup repository, so either they all should be in the same security domain,\nor extra care has to be taken to protect access to backup repository.\n\nFile permissions are not managed with strict details due to inherent\nnature of Git. This aspect can be improved with e.g. etckeeper-like\n(http://etckeeper.branchable.com/) approach if needed.\n\nNOTE the idea of pulling all refs together is similar to git-namespaces\n     http://git-scm.com/docs/gitnamespaces\n\n(*) Tag objects are handled specially - because in a lot of places Git insists and\n    assumes commit parents can only be commit objects. We encode tag objects in\n    specially-crafted commit object on pull, and decode back on backup restore.\n\n    We do likewise if a ref points to tree or blob, which is valid in Git.\n"}]}