Re: Synchronous replication on push

From: Taylor R Campbell <git@campbell.mumble.net>
Date: Nov 4, 2024, 13:35 UTC
Message-ID: <20241104133544.A04D760A95@jupiter.mumble.net>
In-Reply-To: <ZyY74N_NjmaJ2677@tapette.crustytoothpaste.net>

Show 12 quoted lines

> Date: Sat, 2 Nov 2024 14:49:04 +0000
> From: "brian m. carlson" <sandals@crustytoothpaste.net>
> 
> On 2024-11-02 at 13:35:11, Taylor R Campbell wrote:
> > I'm asking about how to configure a _single_ frontend remote, from the
> > perspective of developers who are pushing from their development
> > workstations, so that it replicates to one or many backend stores.
> > This is, for example, the usage model of Github's proprietary
> > implementation.
> 
> I don't think there's built-in functionality for this and I'm not sure
> that it can be done without additional software.

I'm happy to write some additional software.

But I would like to understand what constraints there are on, e.g., pre-receive hooks and the ref updates of git push that make them collide in the ways I discovered, so that I can understand how to make that additional software reliable.

For example:

- Can I suppress the local ref updates of the remote in git push, just
  like git send-pack doesn't attempt any local ref updates of the
  remote?  Or can I defer them to the post-receive hook?

  (By `local ref updates of the remote', I mean updates of the refs
  that live in the local repository for the remote.backend.fetch or
  remote.backend.push refspecs, rather than refs that exist in the
  remote repository which obviously I do want to update.)

- Can I use git send-pack with a custom remote transport?

- When I git clone --mirror, explicitly disable the mirror flag, and
  then git push in the pre-receive hook, why is there an error message
  printed even though the push exits with status zero and appears to
  have had all the effects I want?

- What undesirable side effects can git push have in a mirror cloned
  with git clone --mirror, but with the mirror flag subsequently
  disabled?

- What undesirable side effects can git push have in a pre-receive
  hook if I explicitly disable the quarantine environment by unsetting
  GIT_QUARANTINE_PATH in the environment?

Show 7 quoted lines

> If you really wanted to try to do this with out of the box Git, you
> could create a `pre-receive` hook that did policy controls and then on
> success, took all of the objects from the quarantine and rsynced them
> (without overwriting) to the remote store, and then use the
> `reference-transaction` hook to replicate the reference transaction to
> the remote side via SSH or something.  I haven't tested this, so it
> might or might not work, but you could try it.

Thanks, can you expand on how this would work with the constraints I listed in my question? Recapitulating:

   One option, of course, is to use a replicated file system like
   glusterfs, or replicated block store like DRBD.  But that

   (a) likely requires a lot more round-trips than git push/send-pack,
   (b) can't be used for replication to other git hosts like Github, and
   (c) can't be used for other remote transports like git-cinnabar.

It sounds like rsyncing over ssh is incompatible with (b) and (c), but perhaps I misunderstood what you're getting at. I tried to see if there is some way that reference-transaction hooks help me here but there wasn't anything obvious to me.