How to mirror and augment a git repository

Hello list,

I wish to mirror _and augment_ an upstream git repository.

              .--------.
              |Upstream|
              '--------'
                   |
          .----------------.
          |Augmented mirror|
          '----------------'
           /       |       \
   .--------. .--------. .--------.
   |Client#1| |Client#2| |Client#3|
   '--------' '--------' '--------'

Clients of the augmented mirror must have access to everything available from upstream but must also be able to collaborate on additional development branches not available from upstream.

Initial approach:

 Augmented mirror:

   $ git clone --mirror <upstream> upstream
   $ cd upstream
   $ git remote update  # regular cron job

 Clients (bare repo & worktrees preferred):

   $ git clone --bare <mirror> mirror
   $ cd mirror
   $ git config remote.origin.fetch\
         "+refs/heads/*:refs/remotes/origin/*"
   $ git remote update

This arrangement worked fine until I decded to run:

   $ git remote prune origin

on the augmented mirror and lost all the additional development branches the clients had added and shared amongst themselves.

I've tried running the augmented mirror as a plain bare repo, i.e.

   $ git config --unset remote.origin.fetch
   $ git config --unset remote.origin.mirror

but then the cron job (git remote update) is no longer sufficient in making all upstream activity available downstream.

So, how best to run an augmented mirror such as this?

If my initial approach was correct, is there a way to protect the additional branches so that ‘git remote prune origin’ may be run safely on the augmented mirror from time to time?

Any help/tips/pointers/suggestions much appreciated.

Sebastian

On 3/4/23 19:19, Sebastian Tennant wrote:

I guess the augmented mirror is integration tree before changes from clients going upstream, right?

Quoth Bagas Sanjaya <bagasdotme@gmail.com> on Sun, 5 Mar 2023 16:02:05 +0700:

Some of the changes introduced by the clients may end up being accepted upstream. The augmented mirror is primarily a way of collaborating on and distributing customisations between clients.

On Sat, Mar 04, 2023 at 12:19:16PM +0000, Sebastian Tennant wrote:

The problem here is that your refspec for "origin" in the mirror will be "+refs/*:refs/*". So it claims to have responsibility for the whole refs namespace. And because of the "+", it will happily overwrite local refs when fetching, including branches pushed up by the client. You noticed it most with "prune", because that deletes local branches not present in upstream repo. But a similar problem would happen if both a client and the upstream had a branch named "foo".

Right. If you drop the fetch refspec entirely, then it will only fetch HEAD during your cron jobs, which is not what you want. You want a refspec that tells Git to fetch everything, but you need to divide up the "refs/" namespace into local stuff and mirrored stuff.

You could use the normal "+refs/heads/*:refs/remotes/origin/*" refspec, but it's awkward for the clients to access "refs/remotes/" on the mirror.

So you probably want to keep the upstream branches in "refs/heads/", but mark a special part of the namespace. Like:

  cd augmented-mirror
  git config remote.origin.fetch '+refs/heads/*:refs/heads/upstream/*'

And then "git fetch" will pull all of the remote branches into the "upstream/" namespace. And it's safe to prune, because it will only delete branches in refs/heads/upstream/ (and you may want to just "git fetch --prune" as you fetch via cron, which is a little more efficient than a separate "git remote prune").

Clients can name their branches whatever they like, as long as they don't start with "upstream/", and that won't interfere with the mirror.

-Peff

Hello Jeff,

Thanks for your explanation.  It's been really helpful.

Quoth Jeff King <peff@peff.net> on Mon, 6 Mar 2023 03:08:50 -0500:

Understood.

Precisely.

Indeed. To fetch a known ref, a client (also with the normal fetch refspec) would have to do something like this, for example:

 $ git fetch origin\
       refs/remotes/origin/<ref>:refs/remotes/upstream/<ref>

Alternatively, they could add an additional fetch refspec to their config:

[remote="origin"]
 ...
 fetch = +refs/heads/*:refs/remotes/origin/*             # normal
 fetch = +refs/remotes/origin/*:refs/remotes/upstream/*  # additional

This would have the advantage of fetching all the upstream refs on the next update giving them a better idea of what's happening upstream.

Is my understanding more or less correct?

I see. And creating the 'upstream' namespace under 'heads' (instead of under 'remotes') is crucial, since any client (with a normal fetch refspec) will then receive those refs automatically, i.e. without the additional fetch refspec I've described above.

Understood and noted.

Got it.  Thanks again.

Sebastian

On Wed, Mar 08, 2023 at 04:54:39PM +0000, Sebastian Tennant wrote:

Yes, that's exactly correct. In some ways it is cleaner than using "refs/heads/upstream" because it leaves the whole "refs/heads/" namespace intact for local client branches. It depends on how much of a pain it is to configure the extra refspec in each client. :)

-Peff

Hello Jeff,

Alas, I thought I understood fetch refspecs, but it appears not.

Quoth Jeff King <peff@peff.net> on Wed, 8 Mar 2023 22:12:29 -0500:

I decided to go with the cleaner approach.

Here are the actions I'm taking to configure the mirror:

 $ git clone --bare https://url.of/project.git
 $ cd project.git
 $ git remote rename origin upstream
 $ git config remote.upstream.fetch\
       '+refs/heads/*:refs/remotes/upstream/*'
 $ git fetch upstream --prune

At this point there are no refs under refs/heads (and very many under refs/remotes/upstream).

Here are the actions I'm then taking on a client:

 $ git clone --bare mirror:path/to/project.git
 $ cd project.git
 $ git remote rename origin mirror
 $ git config remote.mirror.fetch\
       '+refs/heads/*:refs/remotes/mirror/*'

At this point, both mirror and client have a normal fetch refspec, i.e. no additional refspec has been added to the client, yet when I run:

 $ git fetch mirror --prune

on the client, all the refs on mirror under refs/remotes/upstream are fetched and placed under refs/remotes/mirror on the client.

My understanding of refspec:

 +refs/heads/*:refs/remotes/mirror/*

is "fetch only those refs under refs/heads and place them under refs/remotes/mirror", which in this case should mean that no refs are fetched (since there are none on mirror under refs/heads).

What's going on here that I'm just not getting?

If I add the additional refspec to the client:

 $ git config --add remote.mirror.fetch\
   '+refs/remotes/upstream/*:refs/remotes/upstream/*'

and fetch once more, I end up with all the refs already under refs/remotes/mirror duplicated under refs/remotes/upstream.

Sebastian

On Sat, Mar 11, 2023 at 10:47:40AM +0000, Sebastian Tennant wrote:

You'd have refs under refs/heads at this point. They were created when you did the original bare clone (since bare clones fetch all heads to start with, though they don't set up a refspec).

And they won't be deleted by the pruning fetch, of course, because you configured the refspec to limit itself to refs/remotes/upstream on the local side.

If you don't want them (and I think you don't), you can just initialize the repository directly, and then fetch, like:

  git init --bare project.git
  cd project.git
  git config remote.upstream.url https://url.of/project.git
  [and then configure refspec and fetch --prune as before]

This bare clone will do the same thing. So you'll end up with a copy of all of the heads created in the earlier step. Worse, they won't be the current state of those branches, but stale ones left from when you created the mirror repo.

I think you want _two_ refspecs in the clients:

  - one to fetch the client-local branches stored on the mirror. That
    is:

      +refs/heads/*:refs/remotes/origin/*

    and those branches just appear as normal.

  - one to fetch the mirrored upstream branches from the special
    namespace on the mirror. That one is:

      +refs/remotes/upstream/*:refs/remotes/upstream/*

The refspec you showed above for the client is fetching from refs/heads/* on the remote side. So it will never look at refs/remotes/upstream from the mirror.

Your understanding is right, but the gotcha of how "clone --bare" works is making things more confusing.

Right. Those are the copies you actually _want_ to have. The ones pulled from refs/heads/* aren't. You'd expect that to be empty to start, and then eventually it would be populated by pushes from local clients.

-Peff

Quoth Jeff King <peff@peff.net> on Sat, 11 Mar 2023 09:55:13 -0500:

I see the refs under refs/heads at last!

My mistake was looking for them in the file system (instead of in file ‘packed-refs’).

This is precisely the behaviour I've been wanting (and mistakenly expecting).

Yup, this is what I want, and actually what I already have in place, configuration wise.

The problem was in my partial understanding of the consequences of passing option --bare to ‘git clone’ (and in my not thinking to look in file ‘packed-refs’).

Thanks again Jeff.

Sebastian

On Sun, Mar 12, 2023 at 06:11:22PM +0000, Sebastian Tennant wrote:

The best way to check is "git for-each-ref", which handles the storage details (including if we ever move to a new storage mechanism like reftable).

To be fair, I usually have to double-check the rules for "--bare" myself. :)

Glad everything is working now.

-Peff