{"thread":{"id":"63399","subject":"Incremental Backup of repositories using Git","startedAt":"2025-05-05T14:35:48Z","lastAt":"2025-05-27T22:21:15Z","messageCount":14,"participants":["Akash S","Justin Tobler","Abhishek Dalmia","Jeff King","Michal Suchánek","rsbecker@nexbridge.com"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"517251","messageId":"PH7PR19MB70252D42F5D04FFC0331AB63C08E2@PH7PR19MB7025.namprd19.prod.outlook.com","threadId":"63399","inReplyTo":null,"subject":"Incremental Backup of repositories using Git","fromName":"Akash S","fromEmail":"akashs@commvault.com","sentAt":"2025-05-05T14:35:43Z","receivedAt":"2025-05-05T14:35:48Z","isPatch":false,"sender":{"key":"akashs@commvault.com","avatar":null},"body":"Hi,\n\nCurrently we are backing up repositories using the \"git clone -bare\" command and save it to disk. If we want to restore, we just run git push -mirror from the repo that was saved during the backup.\n\nCurrently we are running full backups (run git clone -bare) everyday, which is taking a lot of disk space and time.\n\nAre there any possible ways to backup only the incremental changes of a repository? And somehow construct the whole repository when we want to do a restore from the incremental backups?\n\nThanks,\nAkash\n"},{"id":"517266","messageId":"2dz3cema2mr5mrlvuroemnyeqyrglxfmusfdz2kaghv6rvj3ro@ti2dhu45fdmr","threadId":"63399","inReplyTo":"PH7PR19MB70252D42F5D04FFC0331AB63C08E2@PH7PR19MB7025.namprd19.prod.outlook.com","subject":"Re: Incremental Backup of repositories using Git","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-05-05T16:18:36Z","receivedAt":"2025-05-05T16:22:59Z","isPatch":false,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/05/05 02:35PM, Akash S wrote:\n> Hi,\n> \n> Currently we are backing up repositories using the \"git clone -bare\" command and save it to disk. If we want to restore, we just run git push -mirror from the repo that was saved during the backup.\n> \n> Currently we are running full backups (run git clone -bare) everyday, which is taking a lot of disk space and time.\n> \n> Are there any possible ways to backup only the incremental changes of a repository? And somehow construct the whole repository when we want to do a restore from the incremental backups?\n\nYou could look into using git-bundle(1) to create incremental bundles\nusing exclusions. Examples:\n\n # Creates a bundle containing the last 10 commits for main.\n $ git bundle create inc-backup main~10..main\n\n # Creates incremental bundle based on time for all references.\n $ git bundle create inc-backup --all --since=7.days\n\nThese bundles can then be \"unbundled\" into a repository as long as the\nrepo contains the required prerequisite objects.\n\n-Justin\n\n> \n> Thanks,\n> Akash\n> \n"},{"id":"517333","messageId":"SJ1PR19MB6401D7734B73C453E491D54DAE89A@SJ1PR19MB6401.namprd19.prod.outlook.com","threadId":"63399","inReplyTo":"2dz3cema2mr5mrlvuroemnyeqyrglxfmusfdz2kaghv6rvj3ro@ti2dhu45fdmr","subject":"RE: Incremental Backup of repositories using Git","fromName":"Abhishek Dalmia","fromEmail":"adalmia@commvault.com","sentAt":"2025-05-06T12:44:34Z","receivedAt":"2025-05-06T12:44:40Z","isPatch":false,"sender":{"key":"adalmia@commvault.com","avatar":null},"body":"Hi Justin\n\n(My previous email got blocked due to HTML content)\n\nThanks for the recommendation. We want to backup all the repo contents, so could you please comment if the following steps will help us backup and restore everything, or we might miss some tags/references?\n\nDuring backup:\n- Create full bundle first time using: git bundle create --all\n- Create further incremental bundles using: git bundle create --since=\"\" -all\n\t- making sure we don't miss out any time\n\nDuring restore:\n- Create the initial repo with: git clone -bare - using the full bundle we created earlier\n- For restoring further incremental bundle files\n\t- git fetch 'refs/*:refs/*'\n\t- I can't use --all here, that works only with remote repos\n\nWill using 'refs/*:refs/*' restore everything, or is it possible any git data might get missed out?\n\nRegards,\nAbhishek\n\n-----Original Message-----\nFrom: Justin Tobler \nSent: 05 May 2025 21:49\nTo: Akash S \nCc: git@vger.kernel.org; Adithya Urugudige ; Abhishek Dalmia \nSubject: Re: Incremental Backup of repositories using Git\n\n[Some people who received this message don't often get email from jltobler@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]\n\nExternal email. Inspect before opening.\n\n\n\nOn 25/05/05 02:35PM, Akash S wrote:\n> Hi,\n>\n> Currently we are backing up repositories using the \"git clone -bare\" command and save it to disk. If we want to restore, we just run git push -mirror from the repo that was saved during the backup.\n>\n> Currently we are running full backups (run git clone -bare) everyday, which is taking a lot of disk space and time.\n>\n> Are there any possible ways to backup only the incremental changes of a repository? And somehow construct the whole repository when we want to do a restore from the incremental backups?\n\nYou could look into using git-bundle(1) to create incremental bundles using exclusions. Examples:\n\n # Creates a bundle containing the last 10 commits for main.\n $ git bundle create inc-backup main~10..main\n\n # Creates incremental bundle based on time for all references.\n $ git bundle create inc-backup --all --since=7.days\n\nThese bundles can then be \"unbundled\" into a repository as long as the repo contains the required prerequisite objects.\n\n-Justin\n\n>\n> Thanks,\n> Akash\n>\n"},{"id":"517392","messageId":"hanlqq5mma3dvbfq4j4u2zgz5mjegejjg3gjrhyggg2e6ozd5t@354nrc4nq6gn","threadId":"63399","inReplyTo":"SJ1PR19MB6401D7734B73C453E491D54DAE89A@SJ1PR19MB6401.namprd19.prod.outlook.com","subject":"Re: Incremental Backup of repositories using Git","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-05-06T20:46:42Z","receivedAt":"2025-05-06T20:51:07Z","isPatch":false,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/05/06 12:44PM, Abhishek Dalmia wrote:\n> Hi Justin\n> \n> (My previous email got blocked due to HTML content)\n> \n> Thanks for the recommendation. We want to backup all the repo contents, so could you please comment if the following steps will help us backup and restore everything, or we might miss some tags/references?\n> \n> During backup:\n> - Create full bundle first time using: git bundle create --all\n> - Create further incremental bundles using: git bundle create --since=\"\" -all\n> \t- making sure we don't miss out any time\n\nJust something to note, it's ok if a bundle contains objects that\nalready exist in the repository. So some overlap with the previous\nbackup would be fine.\n\n> During restore:\n> - Create the initial repo with: git clone -bare - using the full bundle we created earlier\n> - For restoring further incremental bundle files\n> \t- git fetch 'refs/*:refs/*'\n> \t- I can't use --all here, that works only with remote repos\n\nThis seems reasonable to me. It may be worth validating that the bundles\nwould apply to a fresh repository. If an incremental bundle depends on\nsome prerequistite objects that are not in a repository it cannot be\napplied. This means if you have a series of incremental backups, they\nall would depend on each other and one missing in the middle could\nprevent subsequent bundles from being applied.\n\n> Will using 'refs/*:refs/*' restore everything, or is it possible any git data might get missed out?\n\nThat refspec captures all references and mirrors them. All branches and\ntags, along with all reachable objects from them, would be fetched.\n\n-Justin\n"},{"id":"517537","messageId":"SJ1PR19MB64010CDB3F21FE91C97E566BAE8BA@SJ1PR19MB6401.namprd19.prod.outlook.com","threadId":"63399","inReplyTo":"hanlqq5mma3dvbfq4j4u2zgz5mjegejjg3gjrhyggg2e6ozd5t@354nrc4nq6gn","subject":"RE: Incremental Backup of repositories using Git","fromName":"Abhishek Dalmia","fromEmail":"adalmia@commvault.com","sentAt":"2025-05-08T10:24:55Z","receivedAt":"2025-05-08T10:25:04Z","isPatch":false,"sender":{"key":"adalmia@commvault.com","avatar":null},"body":"Hi Justin,\n\nI ran into an edge case while testing incremental backups with git bundle. If a commit is created with a timestamp earlier than the latest full or incremental backup, it can be excluded from the next bundle due to the --since parameter even if there is a buffer.\nGiven this, do you think git bundle is still the most reliable approach for incremental backups, or is there a better alternative worth exploring?\n\nRegards,\nAbhishek \n\n-----Original Message-----\nFrom: Justin Tobler \nSent: 07 May 2025 02:17\nTo: Abhishek Dalmia \nCc: Akash S ; git@vger.kernel.org; Adithya Urugudige \nSubject: Re: Incremental Backup of repositories using Git\n\nExternal email. Inspect before opening.\n\n\n\nOn 25/05/06 12:44PM, Abhishek Dalmia wrote:\n> Hi Justin\n>\n> (My previous email got blocked due to HTML content)\n>\n> Thanks for the recommendation. We want to backup all the repo contents, so could you please comment if the following steps will help us backup and restore everything, or we might miss some tags/references?\n>\n> During backup:\n> - Create full bundle first time using: git bundle create \n> --all\n> - Create further incremental bundles using: git bundle create --since=\"\" -all\n> - making sure we don't miss out any time\n\nJust something to note, it's ok if a bundle contains objects that already exist in the repository. So some overlap with the previous backup would be fine.\n\n> During restore:\n> - Create the initial repo with: git clone -bare \n> - using the full bundle we created earlier\n> - For restoring further incremental bundle files\n> - git fetch 'refs/*:refs/*'\n> - I can't use --all here, that works only with remote repos\n\nThis seems reasonable to me. It may be worth validating that the bundles would apply to a fresh repository. If an incremental bundle depends on some prerequistite objects that are not in a repository it cannot be applied. This means if you have a series of incremental backups, they all would depend on each other and one missing in the middle could prevent subsequent bundles from being applied.\n\n> Will using 'refs/*:refs/*' restore everything, or is it possible any git data might get missed out?\n\nThat refspec captures all references and mirrors them. All branches and tags, along with all reachable objects from them, would be fetched.\n\n-Justin\n"},{"id":"517589","messageId":"20250508183910.GB10864@coredump.intra.peff.net","threadId":"63399","inReplyTo":"SJ1PR19MB64010CDB3F21FE91C97E566BAE8BA@SJ1PR19MB6401.namprd19.prod.outlook.com","subject":"Re: Incremental Backup of repositories using Git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-05-08T18:39:10Z","receivedAt":"2025-05-08T18:39:18Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2025 at 10:24:55AM +0000, Abhishek Dalmia wrote:\n\n> I ran into an edge case while testing incremental backups with git\n> bundle. If a commit is created with a timestamp earlier than the\n> latest full or incremental backup, it can be excluded from the next\n> bundle due to the --since parameter even if there is a buffer.\n\nYeah, I don't think you want to use \"--since\" here, since it is about\ncommit timestamps. You care about the state of the refs at a particular\ntime. Or more accurately, you care that you have captured a particular\nref state previously.\n\nSo ideally you'd snapshot that state in an atomic way, feed it as the\n\"current\" state when doing a bundle, and then save it for later. You can\neasily create such a snapshot with for-each-ref, but I don't think\ngit-bundle has a way to provide the exact set of ref tips and their\nvalues (it just takes rev-list arguments, and wants to resolve the refs\nthemselves).\n\nYou could probably get away with just creating a bundle with the current\nstate, and then pulling the snapshot values from the created bundle.\nSomething like this:\n\n # for initial backup\n if ! test -e last-bundle-snapshot; then\n >last-bundle-snapshot\n fi\n\n # mark everything from last as seen, so we do not include it,\n # along with --all (or your choice of refs) to pick up everything\n # we have currently\n sed -e 's/^/^/' last-bundle-snapshot\n\nOr alternatively, instead of using git-bundle at all, you could just\nstore a collection of ref snapshots (from \"for-each-ref\") and thin packs\n(from \"pack-objects --thin --stdout\", fed from the old snapshot and the\nnew). Which is really all that bundles are anyway.\n\n-Peff\n"},{"id":"517590","messageId":"aBz8U35YX7UH-PTW@kitsune.suse.cz","threadId":"63399","inReplyTo":"PH7PR19MB70252D42F5D04FFC0331AB63C08E2@PH7PR19MB7025.namprd19.prod.outlook.com","subject":"Re: Incremental Backup of repositories using Git","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2025-05-08T18:47:47Z","receivedAt":"2025-05-08T18:47:50Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"Hello,\n\nOn Mon, May 05, 2025 at 02:35:43PM +0000, Akash S wrote:\n> Hi,\n> \n> Currently we are backing up repositories using the \"git clone -bare\" command and save it to disk. If we want to restore, we just run git push -mirror from the repo that was saved during the backup.\n> \n> Currently we are running full backups (run git clone -bare) everyday, which is taking a lot of disk space and time.\n> \n> Are there any possible ways to backup only the incremental changes of a repository? And somehow construct the whole repository when we want to do a restore from the incremental backups?\n\nIf you have one of those filesystems that support deduplication on\nfilesystem level you could make each snapshot as a full repository with\nall objects unpacked, and the filesystem would deduplicate the objects\nfor you.\n\nThe downside is that you have no way to do multiple full backups this\nway, and you would have to use something else for that (such as those\nbundles, or plain archiving the repository as files in a tar archive or\nsuch.\n\nThanks\n\nMichal\n"},{"id":"517591","messageId":"20250508194731.GA13108@coredump.intra.peff.net","threadId":"63399","inReplyTo":"aBz8U35YX7UH-PTW@kitsune.suse.cz","subject":"Re: Incremental Backup of repositories using Git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-05-08T19:47:31Z","receivedAt":"2025-05-08T19:47:32Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2025 at 08:47:47PM +0200, Michal Suchánek wrote:\n\n> If you have one of those filesystems that support deduplication on\n> filesystem level you could make each snapshot as a full repository with\n> all objects unpacked, and the filesystem would deduplicate the objects\n> for you.\n> \n> The downside is that you have no way to do multiple full backups this\n> way, and you would have to use something else for that (such as those\n> bundles, or plain archiving the repository as files in a tar archive or\n> such.\n\nThis is tempting, but I suspect that storing the objects unpacked will\nbecome unfeasibly large, because you are missing out on delta\ncompression in the packfiles. You can compare the on-disk and\nuncompressed sizes of objects in a repo like this:\n\n git cat-file --batch-all-objects --unordered \\\n --batch-check='%(objectsize:disk) %(objectsize)' |\n perl -alne '\n $disk += $F[0];\n $true += $F[1];\n END {\n print \"$true / $disk = \", int($true / $disk);\n }\n '\n\nIt's not entirely fair because the \"true\" size is missing out on zlib\ncompression that loose objects would get. But that's at best going to be\nabout 4:1 (and in practice worse, since trees are full of sha1 hashes\nthat don't compress very well).\n\nIn my copy of linux.git, that yields ~135G versus ~2.4G, for a factor of\n56. Even if we grant 4:1 compression from zlib, that's still inflating\nyour on-disk repository by a factor of 14.\n\nIf you have the patience, you can run:\n\n git cat-file --batch-all-objects --unordered --batch | gzip | wc -c\n\nto get a better sense of what it looks like with the extra deflate (this\nis cheating a bit, because it will find cross-object compression\nopportunities which would not be there in loose objects storage, but\nshould get you in the right ballpark).\n\nYou're probably also paying some inode costs with loose objects (1K\ntrees at the root of linux.git all pay 4K or whatever as individual\nloose objects).\n\nSo you're probably much better off with some strategy .keep files. I.e.,\nmake a good big pack and mark it with .keep, so that it is retained\nforever.\n\n-Peff\n"},{"id":"517594","messageId":"029701dbc054$a6e9af00$f4bd0d00$@nexbridge.com","threadId":"63399","inReplyTo":"20250508194731.GA13108@coredump.intra.peff.net","subject":"RE: Incremental Backup of repositories using Git","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2025-05-08T20:06:08Z","receivedAt":"2025-05-08T20:06:29Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On May 8, 2025 3:48 PM, Jeff King wrote:\n>On Thu, May 08, 2025 at 08:47:47PM +0200, Michal Suchánek wrote:\n>\n>> If you have one of those filesystems that support deduplication on\n>> filesystem level you could make each snapshot as a full repository\n>> with all objects unpacked, and the filesystem would deduplicate the\n>> objects for you.\n>>\n>> The downside is that you have no way to do multiple full backups this\n>> way, and you would have to use something else for that (such as those\n>> bundles, or plain archiving the repository as files in a tar archive\n>> or such.\n>\n>This is tempting, but I suspect that storing the objects unpacked will become\n>unfeasibly large, because you are missing out on delta compression in the packfiles.\n>You can compare the on-disk and uncompressed sizes of objects in a repo like this:\n>\n> git cat-file --batch-all-objects --unordered \\\n> --batch-check='%(objectsize:disk) %(objectsize)' |\n> perl -alne '\n> $disk += $F[0];\n> $true += $F[1];\n> END {\n> print \"$true / $disk = \", int($true / $disk);\n> }\n> '\n>\n>It's not entirely fair because the \"true\" size is missing out on zlib compression that\n>loose objects would get. But that's at best going to be about 4:1 (and in practice\n>worse, since trees are full of sha1 hashes that don't compress very well).\n>\n>In my copy of linux.git, that yields ~135G versus ~2.4G, for a factor of 56. Even if we\n>grant 4:1 compression from zlib, that's still inflating your on-disk repository by a\n>factor of 14.\n>\n>If you have the patience, you can run:\n>\n> git cat-file --batch-all-objects --unordered --batch | gzip | wc -c\n>\n>to get a better sense of what it looks like with the extra deflate (this is cheating a bit,\n>because it will find cross-object compression opportunities which would not be\n>there in loose objects storage, but should get you in the right ballpark).\n>\n>You're probably also paying some inode costs with loose objects (1K trees at the\n>root of linux.git all pay 4K or whatever as individual loose objects).\n>\n>So you're probably much better off with some strategy .keep files. I.e., make a good\n>big pack and mark it with .keep, so that it is retained forever.\n\nAs a possible alternative, would some kind of information presented via the proposed\ngit blame-tree series (or call it git annotate-tree perhaps) be useful for this enhancement?\nI am not sure what the results will look like, but it might be useful and then cached by\nthe backup strategy. I'm grasping at straws, though.\n\n--Randall\n\n"},{"id":"517597","messageId":"20250508202033.GE18229@coredump.intra.peff.net","threadId":"63399","inReplyTo":"029701dbc054$a6e9af00$f4bd0d00$@nexbridge.com","subject":"Re: Incremental Backup of repositories using Git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-05-08T20:20:33Z","receivedAt":"2025-05-08T20:20:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 08, 2025 at 04:06:08PM -0400, rsbecker@nexbridge.com wrote:\n\n> As a possible alternative, would some kind of information presented via the proposed\n> git blame-tree series (or call it git annotate-tree perhaps) be useful for this enhancement?\n> I am not sure what the results will look like, but it might be useful and then cached by\n> the backup strategy. I'm grasping at straws, though.\n\nI don't think so. From an efficiency perspective, your best git-aware\nbackup really is going to be packfiles representing slices of history,\ndepending on each other. I.e., bundles or something approximating them.\n\n-Peff\n"},{"id":"517637","messageId":"aB3GGe09Mf7mVExx@kitsune.suse.cz","threadId":"63399","inReplyTo":"20250508194731.GA13108@coredump.intra.peff.net","subject":"Re: Incremental Backup of repositories using Git","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2025-05-09T09:08:41Z","receivedAt":"2025-05-09T09:08:44Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"On Thu, May 08, 2025 at 03:47:31PM -0400, Jeff King wrote:\n> On Thu, May 08, 2025 at 08:47:47PM +0200, Michal Suchánek wrote:\n> \n> > If you have one of those filesystems that support deduplication on\n> > filesystem level you could make each snapshot as a full repository with\n> > all objects unpacked, and the filesystem would deduplicate the objects\n> > for you.\n> > \n> > The downside is that you have no way to do multiple full backups this\n> > way, and you would have to use something else for that (such as those\n> > bundles, or plain archiving the repository as files in a tar archive or\n> > such.\n> \n> This is tempting, but I suspect that storing the objects unpacked will\n> become unfeasibly large, because you are missing out on delta\n> compression in the packfiles. You can compare the on-disk and\n> uncompressed sizes of objects in a repo like this:\n> \n> git cat-file --batch-all-objects --unordered \\\n> --batch-check='%(objectsize:disk) %(objectsize)' |\n> perl -alne '\n> $disk += $F[0];\n> $true += $F[1];\n> END {\n> print \"$true / $disk = \", int($true / $disk);\n> }\n> '\n> \n> It's not entirely fair because the \"true\" size is missing out on zlib\n> compression that loose objects would get. But that's at best going to be\n> about 4:1 (and in practice worse, since trees are full of sha1 hashes\n> that don't compress very well).\n> \n> In my copy of linux.git, that yields ~135G versus ~2.4G, for a factor of\n> 56. Even if we grant 4:1 compression from zlib, that's still inflating\n> your on-disk repository by a factor of 14.\n\nSo with this estimate you recoup that size inflation after 14\nincremental backups.\n\nSince no other working incremantal backup strategy was proposed so far\nthis is the best one ;-)\n\nThanks\n\nMichal\n"},{"id":"517652","messageId":"aB3jYNAJAbKjSWAS@kitsune.suse.cz","threadId":"63399","inReplyTo":"PH7PR19MB70252D42F5D04FFC0331AB63C08E2@PH7PR19MB7025.namprd19.prod.outlook.com","subject":"Re: Incremental Backup of repositories using Git","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2025-05-09T11:13:36Z","receivedAt":"2025-05-09T11:13:39Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"On Mon, May 05, 2025 at 02:35:43PM +0000, Akash S wrote:\n> Hi,\n> \n> Currently we are backing up repositories using the \"git clone -bare\" command and save it to disk. If we want to restore, we just run git push -mirror from the repo that was saved during the backup.\n> \n> Currently we are running full backups (run git clone -bare) everyday, which is taking a lot of disk space and time.\n> \n> Are there any possible ways to backup only the incremental changes of a repository? And somehow construct the whole repository when we want to do a restore from the incremental backups?\n\nHello,\n\nfirst, to make it easier to update the backup the clone should be done\nwith --bare --mirror.\n\nIf your clone ends up having multiple packs and loose objects you\ncan reduce its size with\n\ngit --git-dir=/path/to/clone repack -adk\n\nThis should give you a repository with a single pack and no loose\nobjects.\n\nThe -k (or --cruft) option is required, using only -ad seems to corrupt\nrepositories quite reliably.\n\nTo speed up the clone next time around you can make a copy of the\nprevious backup and fetch from the remote repository but because\nthere is no safe way I am aware of to eliminate no longer referenced\nobjects you will accumulate cruft this way.\n\nThis is now a complete backup, and should be made readonly to not get\ncorrupted with further operations.\n\nThe inrementeal backups are somewhat speculative, I have not tested this\nat all.\n\nYou can create a shared clone of the full backup, update the origin URL\nof the shared clone to the remote repository to backup, and do a fetch\n-p (which now should do the right thing because the initial clone was\nset up as mirror).\n\nTo repack you need to use the --local option in addition.\n\nWith this you should have a valid repository for each backup with the\nincremental backups sharing most objects with the full backup.\n\nThese can be inspected with git commands, exported over gitweb, or\nwhatever.\n\nThanks\n\nMichal\n"},{"id":"517653","messageId":"aB3lYsiJQ5n7awXE@kitsune.suse.cz","threadId":"63399","inReplyTo":"aB3jYNAJAbKjSWAS@kitsune.suse.cz","subject":"Re: Incremental Backup of repositories using Git","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2025-05-09T11:22:10Z","receivedAt":"2025-05-09T11:22:13Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"On Fri, May 09, 2025 at 01:13:36PM +0200, Michal Suchánek wrote:\n> On Mon, May 05, 2025 at 02:35:43PM +0000, Akash S wrote:\n> > Hi,\n> > \n> > Currently we are backing up repositories using the \"git clone -bare\" command and save it to disk. If we want to restore, we just run git push -mirror from the repo that was saved during the backup.\n> > \n> > Currently we are running full backups (run git clone -bare) everyday, which is taking a lot of disk space and time.\n> > \n> > Are there any possible ways to backup only the incremental changes of a repository? And somehow construct the whole repository when we want to do a restore from the incremental backups?\n> \n> Hello,\n> \n> first, to make it easier to update the backup the clone should be done\n> with --bare --mirror.\n> \n> If your clone ends up having multiple packs and loose objects you\n> can reduce its size with\n> \n> git --git-dir=/path/to/clone repack -adk\n> \n> This should give you a repository with a single pack and no loose\n> objects.\n> \n> The -k (or --cruft) option is required, using only -ad seems to corrupt\n> repositories quite reliably.\n> \n> To speed up the clone next time around you can make a copy of the\n> previous backup and fetch from the remote repository but because\n> there is no safe way I am aware of to eliminate no longer referenced\n> objects you will accumulate cruft this way.\n> \n> This is now a complete backup, and should be made readonly to not get\n> corrupted with further operations.\n> \n> The inrementeal backups are somewhat speculative, I have not tested this\n> at all.\n> \n> You can create a shared clone of the full backup, update the origin URL\n\nAlso with the --mirror option, it's not transferred.\n\n> of the shared clone to the remote repository to backup, and do a fetch\n> -p (which now should do the right thing because the initial clone was\n> set up as mirror).\n> \n> To repack you need to use the --local option in addition.\n> \n> With this you should have a valid repository for each backup with the\n> incremental backups sharing most objects with the full backup.\n> \n> These can be inspected with git commands, exported over gitweb, or\n> whatever.\n> \n> Thanks\n> \n> Michal\n> \n"},{"id":"519029","messageId":"SJ1PR19MB6401A2664CD337F9DF482317AE64A@SJ1PR19MB6401.namprd19.prod.outlook.com","threadId":"63399","inReplyTo":"20250508183910.GB10864@coredump.intra.peff.net","subject":"RE: Incremental Backup of repositories using Git","fromName":"Abhishek Dalmia","fromEmail":"adalmia@commvault.com","sentAt":"2025-05-27T22:21:11Z","receivedAt":"2025-05-27T22:21:15Z","isPatch":false,"sender":{"key":"adalmia@commvault.com","avatar":null},"body":"Hi Justin/Jeff\n(prev. email got rejected due to HTML content)\nI tried researching more, if we have the previous state of repo then we can use git fetch --all and have storage level incremental backups, by using the changed objects under .git/objects (.pack by preventing auto gc). But it will not be feasible, to keep around the repo clone for the incremental backups.\n\nI researched about git fetch-pack in a git init --bare repo, which might have helped here, but it is not working as expected:\n1. It doesn't work with https ->\n$ git fetch-pack --thin --shallow-exclude=28307688f7344018cad46c310826a82041b39b8d https://github.com/elastic/elasticsearch refs/heads/main\nfatal: protocol 'https' is not supported\n2. With ssh it says fatal: the remote end hung up unexpectedly ->\n$ git fetch-pack --thin --shallow-exclude=28307688f7344018cad46c310826a82041b39b8d mailto:git@github.com:elastic/elasticsearch.git refs/heads/main\nfatal: the remote end hung up unexpectedly\nIs what I require here (fetch new objects, without requiring previous objects present) technically possible with git-cli/ libgit2 library?\nWe can have some metadata to tell us what commits were backed up for each ref in the previous backup if that can help us.\n\nAs an alternative I tried out with API requests to download commit blobs, but that just hits rate limits too often and is far slower than git protocol.\n\n-----Original Message-----\nFrom: Jeff King \nSent: 09 May 2025 00:09\nTo: Abhishek Dalmia \nCc: Justin Tobler ; Akash S ; git@vger.kernel.org; Adithya Urugudige \nSubject: Re: Incremental Backup of repositories using Git\n\n[You don't often get email from mailto:peff@peff.net. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]\n\nExternal email. Inspect before opening.\n\n\n\nOn Thu, May 08, 2025 at 10:24:55AM +0000, Abhishek Dalmia wrote:\n\n> I ran into an edge case while testing incremental backups with git \n> bundle. If a commit is created with a timestamp earlier than the \n> latest full or incremental backup, it can be excluded from the next \n> bundle due to the --since parameter even if there is a buffer.\n\nYeah, I don't think you want to use \"--since\" here, since it is about commit timestamps. You care about the state of the refs at a particular time. Or more accurately, you care that you have captured a particular ref state previously.\n\nSo ideally you'd snapshot that state in an atomic way, feed it as the \"current\" state when doing a bundle, and then save it for later. You can easily create such a snapshot with for-each-ref, but I don't think git-bundle has a way to provide the exact set of ref tips and their values (it just takes rev-list arguments, and wants to resolve the refs themselves).\n\nYou could probably get away with just creating a bundle with the current state, and then pulling the snapshot values from the created bundle.\nSomething like this:\n\n # for initial backup\n if ! test -e last-bundle-snapshot; then\n >last-bundle-snapshot\n fi\n\n # mark everything from last as seen, so we do not include it,\n # along with --all (or your choice of refs) to pick up everything\n # we have currently\n sed -e 's/^/^/' last-bundle-snapshot\n\nOr alternatively, instead of using git-bundle at all, you could just store a collection of ref snapshots (from \"for-each-ref\") and thin packs (from \"pack-objects --thin --stdout\", fed from the old snapshot and the new). Which is really all that bundles are anyway.\n\n-Peff\n"}]}