RE: Incremental Backup of repositories using Git

From: Abhishek Dalmia <adalmia@commvault.com>
Date: May 8, 2025, 10:24 UTC
Message-ID: <SJ1PR19MB64010CDB3F21FE91C97E566BAE8BA@SJ1PR19MB6401.namprd19.prod.outlook.com>
In-Reply-To: <hanlqq5mma3dvbfq4j4u2zgz5mjegejjg3gjrhyggg2e6ozd5t@354nrc4nq6gn>

Hi Justin,

I ran into an edge case while testing incremental backups with git bundle. If a commit is created with a timestamp earlier than the latest full or incremental backup, it can be excluded from the next bundle due to the --since parameter even if there is a buffer. Given this, do you think git bundle is still the most reliable approach for incremental backups, or is there a better alternative worth exploring?

Regards, Abhishek

-----Original Message-----
From: Justin Tobler <jltobler@gmail.com> 
Sent: 07 May 2025 02:17
To: Abhishek Dalmia <adalmia@commvault.com>
Cc: Akash S <akashs@commvault.com>; git@vger.kernel.org; Adithya Urugudige <aurugudige@commvault.com>
Subject: Re: Incremental Backup of repositories using Git

External email. Inspect before opening.

On 25/05/06 12:44PM, Abhishek Dalmia wrote:

Show 11 quoted lines

> Hi Justin
>
> (My previous email got blocked due to HTML content)
>
> Thanks for the recommendation. We want to backup all the repo contents, so could you please comment if the following steps will help us backup and restore everything, or we might miss some tags/references?
>
> During backup:
> - Create full bundle first time using: git bundle create 
> <full-bundle-file-path> --all
> - Create further incremental bundles using: git bundle create <inc-bundle-file-path> --since="<last-backup-time>" -all
>       - making sure we don't miss out any time

Just something to note, it's ok if a bundle contains objects that already exist in the repository. So some overlap with the previous backup would be fine.

Show 6 quoted lines

> During restore:
> - Create the initial repo with: git clone -bare 
> <full-bundle-file-path> - using the full bundle we created earlier
> - For restoring further incremental bundle files
>       - git fetch <inc-bundle-file-path> 'refs/*:refs/*'
>       - I can't use --all here, that works only with remote repos

This seems reasonable to me. It may be worth validating that the bundles would apply to a fresh repository. If an incremental bundle depends on some prerequistite objects that are not in a repository it cannot be applied. This means if you have a series of incremental backups, they all would depend on each other and one missing in the middle could prevent subsequent bundles from being applied.

> Will using 'refs/*:refs/*' restore everything, or is it possible any git data might get missed out?

That refspec captures all references and mirrors them. All branches and tags, along with all reachable objects from them, would be fetched.

-Justin