threads / discuss / 23260

mirror git repos: rsync vs. --mirror

Subject: mirror git repos: rsync vs. --mirror

## tl;dr

4 messages between Mar 30, 2010 and Mar 31, 2010.

replies: 3people: 3as markdown or json

Robin H. Johnson· Mar 30, 2010, 20:54 UTC · lore

I'm trying to see about the state of automated mirroring git repositories between different systems, eg. a private backend and a public frontend, and ran into a catch between two possible options, each with a downside. If there are other means besides these, I'm all ears for them.

rsync:
------
How:
- Simply rsync the git repository, atomically, and use --delete-after.
Benefits:
- Guaranteed to match between master/slaves.
Caveats:
- Need to know that the master repo is in a good state.
- If a repo is repacked, there is a lot more traffic in the next sync.
git clone --mirror && git fetch:
--------------------------------
How:
- Setup with: "git clone --mirror URL"
- Cronjob with: "git fetch"
Benefits:
- Extremely fast
- Light on traffic
Caveats:
- Branches and tags deleted in master persist in slaves :-(
- Seperate gc/repack needed on slaves.

The traffic vs. deleted branches/tags is the main tradeoff here, esp. as repos get larger.

Can git-fetch perhaps be made to delete the branches/tags?
-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85
Julian Phillips· Mar 30, 2010, 22:00 UTC · re: Robin H. Johnson · lore

Re: mirror git repos: rsync vs. --mirror

On Tue, 30 Mar 2010 20:54:09 +0000, "Robin H. Johnson" <robbat2@gentoo.org> wrote:

Show 5 quoted lines
> I'm trying to see about the state of automated mirroring git
> repositories between different systems, eg. a private backend and a
> public frontend, and ran into a catch between two possible options, each
> with a downside. If there are other means besides these, I'm all ears
> for them.
You can mitigate the git caveats somewhat ...
Show 16 quoted lines
> git clone --mirror && git fetch:
> --------------------------------
> How:
> - Setup with: "git clone --mirror URL"
> - Cronjob with: "git fetch"
> Benefits:
> - Extremely fast
> - Light on traffic
> Caveats:
> - Branches and tags deleted in master persist in slaves :-(
> - Seperate gc/repack needed on slaves.
> 
> The traffic vs. deleted branches/tags is the main tradeoff here, esp. as
> repos get larger.
> 
> Can git-fetch perhaps be made to delete the branches/tags?

I don't think that fetch can do it, but "git remote prune origin" will. You could also add "git gc --auto" to your cron script to deal with the repacking - I don't think fetch already does that ...

-- 
Julian
Sitaram Chamarty· Mar 31, 2010, 01:44 UTC · re: Julian Phillips · lore

Re: mirror git repos: rsync vs. --mirror

On Wed, Mar 31, 2010 at 3:30 AM, Julian Phillips <julian@quantumfyre.co.uk> wrote:

> On Tue, 30 Mar 2010 20:54:09 +0000, "Robin H. Johnson"
> <robbat2@gentoo.org>
>> git clone --mirror && git fetch:
>> Caveats:
>> - Branches and tags deleted in master persist in slaves :-(
> I don't think that fetch can do it, but "git remote prune origin" will.

as of 1.7 at least, fetch has --prune, and I tested it yesterday when someone (same person?) asked this on #git

Robin H. Johnson· Mar 31, 2010, 03:05 UTC · re: Sitaram Chamarty · lore

Re: mirror git repos: rsync vs. --mirror

On Wed, Mar 31, 2010 at 07:14:46AM +0530, Sitaram Chamarty wrote:
Show 13 quoted lines
> On Wed, Mar 31, 2010 at 3:30 AM, Julian Phillips
> <julian@quantumfyre.co.uk> wrote:
> > On Tue, 30 Mar 2010 20:54:09 +0000, "Robin H. Johnson"
> > <robbat2@gentoo.org>
> 
> >> git clone --mirror && git fetch:
> 
> >> Caveats:
> >> - Branches and tags deleted in master persist in slaves :-(
> 
> > I don't think that fetch can do it, but "git remote prune origin" will.
> as of 1.7 at least, fetch has --prune, and I tested it yesterday when
> someone (same person?) asked this on #git
Wasn't me asking, but thanks, that helps lots!
Looks like it was added for v1.6.6.
-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

← back to recent threads