Re: How to mirror and augment a git repository

From: Sebastian Tennant <sdt@sebyte.me>
Date: Mar 8, 2023, 16:54 UTC
Message-ID: <87fsaf9oa8.fsf@sebyte.me>
In-Reply-To: <ZAWfkkmrPUqNmrtu@coredump.intra.peff.net>

Hello Jeff,

Thanks for your explanation.  It's been really helpful.

Quoth Jeff King <peff@peff.net> on Mon, 6 Mar 2023 03:08:50 -0500:

Show 16 quoted lines

> On Sat, Mar 04, 2023 at 12:19:16PM +0000, Sebastian Tennant wrote:
>
>>  Augmented mirror:
>>
>>    $ git clone --mirror <upstream> upstream
>>    $ cd upstream
>>    $ git remote update  # regular cron job
>
> The problem here is that your refspec for "origin" in the mirror
> will be "+refs/*:refs/*". So it claims to have responsibility for
> the whole refs namespace. And because of the "+", it will happily
> overwrite local refs when fetching, including branches pushed up by
> the client. You noticed it most with "prune", because that deletes
> local branches not present in upstream repo. But a similar problem
> would happen if both a client and the upstream had a branch named
> "foo".

Understood.

Show 10 quoted lines

>> I've tried running the augmented mirror as a plain bare repo, i.e.
>>
>>    $ git config --unset remote.origin.fetch
>>    $ git config --unset remote.origin.mirror
>>
>> but then the cron job (git remote update) is no longer sufficient in
>> making all upstream activity available downstream.
>
> Right. If you drop the fetch refspec entirely, then it will only fetch
> HEAD during your cron jobs, which is not what you want.

Precisely.

Show 7 quoted lines

> You want a refspec that tells Git to fetch everything, but you need
> to divide up the "refs/" namespace into local stuff and mirrored
> stuff.
>
> You could use the normal "+refs/heads/*:refs/remotes/origin/*" refspec,
> but it's awkward for the clients to access "refs/remotes/" on the
> mirror.

Indeed. To fetch a known ref, a client (also with the normal fetch refspec) would have to do something like this, for example:

 $ git fetch origin\
       refs/remotes/origin/<ref>:refs/remotes/upstream/<ref>

Alternatively, they could add an additional fetch refspec to their config:

[remote="origin"]
 ...
 fetch = +refs/heads/*:refs/remotes/origin/*             # normal
 fetch = +refs/remotes/origin/*:refs/remotes/upstream/*  # additional

This would have the advantage of fetching all the upstream refs on the next update giving them a better idea of what's happening upstream.

Is my understanding more or less correct?

Show 8 quoted lines

> So you probably want to keep the upstream branches in "refs/heads/",
> but mark a special part of the namespace. Like:
>
>   cd augmented-mirror
>   git config remote.origin.fetch '+refs/heads/*:refs/heads/upstream/*'
>
> And then "git fetch" will pull all of the remote branches into the
> "upstream/" namespace.

I see. And creating the 'upstream' namespace under 'heads' (instead of under 'remotes') is crucial, since any client (with a normal fetch refspec) will then receive those refs automatically, i.e. without the additional fetch refspec I've described above.

> And it's safe to prune, because it will only delete branches in
> refs/heads/upstream/ (and you may want to just "git fetch --prune"
> as you fetch via cron, which is a little more efficient than a
> separate "git remote prune").

Understood and noted.

> Clients can name their branches whatever they like, as long as they
> don't start with "upstream/", and that won't interfere with the
> mirror.

Got it.  Thanks again.

Sebastian