{"thread":{"id":"62439","subject":"Synchronous replication on push","startedAt":"2024-11-02T02:14:56Z","lastAt":"2024-11-05T01:34:34Z","messageCount":10,"participants":["Taylor R Campbell","Matěj Cepl","brian m. carlson","Konstantin Ryabitsev","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"506477","messageId":"20241102020653.766D1609AC@jupiter.mumble.net","threadId":"62439","inReplyTo":null,"subject":"Synchronous replication on push","fromName":"Taylor R Campbell","fromEmail":"git@campbell.mumble.net","sentAt":"2024-11-02T02:06:53Z","receivedAt":"2024-11-02T02:14:56Z","isPatch":false,"sender":{"key":"git@campbell.mumble.net","avatar":null},"body":"Suppose I have a front end repository:\n\nuser@frontend.example.com:/repo.git\n\nWhenever I push anything to it, I want the push -- that is, all the\nobjects, and all the ref updates -- to be synchronously replicated to\nanother remote repository, the back end:\n\ngit@backend.example.com:/repo.git\n\nIf this replication fails -- whether because the back end is down, or\nbecause the front end crashed and rolled back to an earlier state, or\nbecause the back end has been updated independently and rejects a\nforce push, or whatever -- I want the push to fail.  But, absent these\nfailures, I want frontend and backend to store the same set of objects\nand refs.\n\n(Actually, I want to replicate it to a quorum of multiple back ends\nwith a three-phase commit protocol -- but I'll start with the\nsingle-replica case for simplicity.)\n\nHow can I do this with git?\n\n\nOne option, of course, is to use a replicated file system like\nglusterfs, or replicated block store like DRBD.  But that\n\n(a) likely requires a lot more round-trips than git push/send-pack,\n(b) can't be used for replication to other git hosts like Github, and\n(c) can't be used for other remote transports like git-cinnabar.\n\nSo I'd like to do this at the git level, not at the file system or\nblock store level.\n\n\nHere are some approaches I've tried:\n\n1. `git clone --mirror -o backend git@backend.example.com:/repo.git'\n   to create the front end repository, plus the following pre-receive\n   hook in the front end:\n\n\t#!/bin/sh\n\texec git push backend\n\n   This doesn't work because the pre-receive hook runs in the\n   quarantine environment, and `git push' wants to update\n   `refs/heads/main', which is forbidden in the quarantine\n   environment.\n\n   (However, git push to frontend doesn't actually fail with nonzero\n   exit status -- it prints an error message, `ref updates forbidden\n   inside quarantine environment', but exits wtih status 0.)\n\n   But maybe the ref update is harmless in this environment.\n\n2. Same as (1), but the pre-receive hook is:\n\n\t#!/bin/sh\n\tunset GIT_QUARANTINE_PATH\n\texec git push backend\n\n   This doesn't work because `git push' in the pre-receive hook\n   doesn't find anything it needs to push -- the ref update hasn't\n   happened yet.\n\n3. Same as (1), but the pre-receive hook assembles a command line of\n\n\texec git push backend ${new0}:${ref0} ${new1}:${ref1} ...,\n\n   with all the ref updates passed on stdin (ignoring the old values).\n\n   This fails because `--mirror can't be combined with refspecs'.\n\n4. Same as (3), but remote.backend.mirror is explicitly disabled after\n   `git clone --mirror' finishes.\n\n   On push to the primary, this prints an error message\n\n\tremote: error: update_ref failed for ref 'refs/heads/main': ref updates forbidden inside quarantine environment\n\n   but somehow the push succeeds in spite of this message, and the\n   primary and replica both get updated.\n\n   And if I inject an error on push to the replica, by making the\n   replica's pre-receive hook fail with nonzero exit status, neither\n   primary nor replica is updated and the push fails with an error\n   message (`pre-receive hook declined') _and_ nonzero exit status --\n   as desired.\n\n   So maybe this actually works, but the error message on _successful_\n   pushes is unsettling!\n\n5. Same as (1), but the pre-receive hook assembles a command line of\n\n\texec git send-pack git@backend.example.com:/repo.git \\\n\t\t${new0}:${ref0} ${new1}:${ref1} ...\n\n   with all the ref updates passed on stdin (ignoring the old values).\n\n   This seems to work, and it propagates errors injected on push to\n   the replica, but it is limited to local or ssh remotes, as far as I\n   can tell -- it does not appear that git-send-pack works with custom\n   remote transports.\n\nPerhaps using mirror clones is the wrong approach here, and perhaps I\nshould instead explicitly create tracking branches in the primary that\nare only updated if the push succeeds -- but this will still require\ngetting around the quarantine restrictions on git push in the\npre-receive hook.\n\n\nIs there a way to achieve this (ideally, with plausible extension to a\nthree-phase commit protocol) that doesn't trigger unsettling nonfatal\nerror messages and that works with custom remote transports?\n"},{"id":"506479","messageId":"D5BM0CBSPT9I.97E2CAX9DE17@cepl.eu","threadId":"62439","inReplyTo":"20241102020653.766D1609AC@jupiter.mumble.net","subject":"Re: Synchronous replication on push","fromName":"Matěj Cepl","fromEmail":"mcepl@cepl.eu","sentAt":"2024-11-02T10:09:52Z","receivedAt":"2024-11-02T10:18:59Z","isPatch":false,"sender":{"key":"mcepl@cepl.eu","avatar":"https://avatars.githubusercontent.com/u/198999?v=4"},"body":"On Sat Nov 2, 2024 at 3:06 AM CET, Taylor R Campbell wrote:\n> Suppose I have a front end repository:\n>\n> user@frontend.example.com:/repo.git\n>\n> Whenever I push anything to it, I want the push -- that is, all the\n> objects, and all the ref updates -- to be synchronously replicated to\n> another remote repository, the back end:\n>\n> git@backend.example.com:/repo.git\n\nhttps://stackoverflow.com/q/14290113/164233\n\n-- \nhttp://matej.ceplovi.cz/blog/, @mcepl@floss.social\nGPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8\n \nWe understand our competition isn’t with Caldera or SuSE--our\ncompetition is with Microsoft.\n    -- Bob Young of Red Hat\n       http://www.linuxjournal.com/article/3553\n\n"},{"id":"506483","messageId":"20241102133511.6375B609AC@jupiter.mumble.net","threadId":"62439","inReplyTo":"D5BM0CBSPT9I.97E2CAX9DE17@cepl.eu","subject":"Re: Synchronous replication on push","fromName":"Taylor R Campbell","fromEmail":"git@campbell.mumble.net","sentAt":"2024-11-02T13:35:11Z","receivedAt":"2024-11-02T13:35:12Z","isPatch":false,"sender":{"key":"git@campbell.mumble.net","avatar":null},"body":"> Date: Sat, 02 Nov 2024 11:09:52 +0100\n> From: Matěj Cepl <mcepl@cepl.eu>\n> \n> On Sat Nov 2, 2024 at 3:06 AM CET, Taylor R Campbell wrote:\n> > Suppose I have a front end repository:\n> >\n> > user@frontend.example.com:/repo.git\n> >\n> > Whenever I push anything to it, I want the push -- that is, all the\n> > objects, and all the ref updates -- to be synchronously replicated to\n> > another remote repository, the back end:\n> >\n> > git@backend.example.com:/repo.git\n> \n> https://stackoverflow.com/q/14290113/164233\n\nThanks, but that is about how to configure my local repository to use\nmultiple remotes for a single git push command, which is not what I'm\nasking about.\n\nI'm asking about how to configure a _single_ frontend remote, from the\nperspective of developers who are pushing from their development\nworkstations, so that it replicates to one or many backend stores.\nThis is, for example, the usage model of Github's proprietary\nimplementation.\n"},{"id":"506485","messageId":"ZyY74N_NjmaJ2677@tapette.crustytoothpaste.net","threadId":"62439","inReplyTo":"20241102133511.6375B609AC@jupiter.mumble.net","subject":"Re: Synchronous replication on push","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-11-02T14:49:04Z","receivedAt":"2024-11-02T14:49:12Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-11-02 at 13:35:11, Taylor R Campbell wrote:\n> I'm asking about how to configure a _single_ frontend remote, from the\n> perspective of developers who are pushing from their development\n> workstations, so that it replicates to one or many backend stores.\n> This is, for example, the usage model of Github's proprietary\n> implementation.\n\nI don't think there's built-in functionality for this and I'm not sure\nthat it can be done without additional software.\n\nIf you really wanted to try to do this with out of the box Git, you\ncould create a `pre-receive` hook that did policy controls and then on\nsuccess, took all of the objects from the quarantine and rsynced them\n(without overwriting) to the remote store, and then use the\n`reference-transaction` hook to replicate the reference transaction to\nthe remote side via SSH or something.  I haven't tested this, so it\nmight or might not work, but you could try it.\n\nNote that GitHub has a separate service that does the replication and\nintercepts the ref update to send it through the three-phase commit, so\nthey don't rely on features of core Git to implement this functionality.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"506526","messageId":"20241104133544.A04D760A95@jupiter.mumble.net","threadId":"62439","inReplyTo":"ZyY74N_NjmaJ2677@tapette.crustytoothpaste.net","subject":"Re: Synchronous replication on push","fromName":"Taylor R Campbell","fromEmail":"git@campbell.mumble.net","sentAt":"2024-11-04T13:35:44Z","receivedAt":"2024-11-04T13:35:50Z","isPatch":false,"sender":{"key":"git@campbell.mumble.net","avatar":null},"body":"> Date: Sat, 2 Nov 2024 14:49:04 +0000\n> From: \"brian m. carlson\" <sandals@crustytoothpaste.net>\n> \n> On 2024-11-02 at 13:35:11, Taylor R Campbell wrote:\n> > I'm asking about how to configure a _single_ frontend remote, from the\n> > perspective of developers who are pushing from their development\n> > workstations, so that it replicates to one or many backend stores.\n> > This is, for example, the usage model of Github's proprietary\n> > implementation.\n> \n> I don't think there's built-in functionality for this and I'm not sure\n> that it can be done without additional software.\n\nI'm happy to write some additional software.\n\nBut I would like to understand what constraints there are on, e.g.,\npre-receive hooks and the ref updates of git push that make them\ncollide in the ways I discovered, so that I can understand how to make\nthat additional software reliable.\n\nFor example:\n\n- Can I suppress the local ref updates of the remote in git push, just\n  like git send-pack doesn't attempt any local ref updates of the\n  remote?  Or can I defer them to the post-receive hook?\n\n  (By `local ref updates of the remote', I mean updates of the refs\n  that live in the local repository for the remote.backend.fetch or\n  remote.backend.push refspecs, rather than refs that exist in the\n  remote repository which obviously I do want to update.)\n\n- Can I use git send-pack with a custom remote transport?\n\n- When I git clone --mirror, explicitly disable the mirror flag, and\n  then git push in the pre-receive hook, why is there an error message\n  printed even though the push exits with status zero and appears to\n  have had all the effects I want?\n\n- What undesirable side effects can git push have in a mirror cloned\n  with git clone --mirror, but with the mirror flag subsequently\n  disabled?\n\n- What undesirable side effects can git push have in a pre-receive\n  hook if I explicitly disable the quarantine environment by unsetting\n  GIT_QUARANTINE_PATH in the environment?\n\n> If you really wanted to try to do this with out of the box Git, you\n> could create a `pre-receive` hook that did policy controls and then on\n> success, took all of the objects from the quarantine and rsynced them\n> (without overwriting) to the remote store, and then use the\n> `reference-transaction` hook to replicate the reference transaction to\n> the remote side via SSH or something.  I haven't tested this, so it\n> might or might not work, but you could try it.\n\nThanks, can you expand on how this would work with the constraints I\nlisted in my question?  Recapitulating:\n\n   One option, of course, is to use a replicated file system like\n   glusterfs, or replicated block store like DRBD.  But that\n\n   (a) likely requires a lot more round-trips than git push/send-pack,\n   (b) can't be used for replication to other git hosts like Github, and\n   (c) can't be used for other remote transports like git-cinnabar.\n\nIt sounds like rsyncing over ssh is incompatible with (b) and (c), but\nperhaps I misunderstood what you're getting at.  I tried to see if\nthere is some way that reference-transaction hooks help me here but\nthere wasn't anything obvious to me.\n"},{"id":"506529","messageId":"20241104-real-marmoset-of-success-9a6c8e@meerkat","threadId":"62439","inReplyTo":"20241104133544.A04D760A95@jupiter.mumble.net","subject":"Re: Synchronous replication on push","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2024-11-04T14:40:06Z","receivedAt":"2024-11-04T14:40:08Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Mon, Nov 04, 2024 at 01:35:44PM +0000, Taylor R Campbell wrote:\n> > > I'm asking about how to configure a _single_ frontend remote, from the\n> > > perspective of developers who are pushing from their development\n> > > workstations, so that it replicates to one or many backend stores.\n> > > This is, for example, the usage model of Github's proprietary\n> > > implementation.\n> > \n> > I don't think there's built-in functionality for this and I'm not sure\n> > that it can be done without additional software.\n> \n> I'm happy to write some additional software.\n\nAlternatively, you can take a look at grokmirror, which is what kernel.org\nuses:\nhttps://pypi.org/project/grokmirror/\n\nIt's pull-based instead of push-based, for several reasons:\n\n1. We replicate to multiple worldwide frontends, and we expect that some of\n   them may be unreachable at the time when we attempt a push\n2. This allows us to propagate repository deletes\n3. This allows us to propagate details like descriptions and authors\n\nGrokmirror also has a listener daemon that can trigger a pull, so it's\npossible to have near-instantaneous replication by notifying the remote node\nthat a repository has been updated and should be pulled.\n\n-K\n"},{"id":"506544","messageId":"20241104155041.235E260AB3@jupiter.mumble.net","threadId":"62439","inReplyTo":"20241104-real-marmoset-of-success-9a6c8e@meerkat","subject":"Re: Synchronous replication on push","fromName":"Taylor R Campbell","fromEmail":"git@campbell.mumble.net","sentAt":"2024-11-04T15:50:40Z","receivedAt":"2024-11-04T15:50:42Z","isPatch":false,"sender":{"key":"git@campbell.mumble.net","avatar":null},"body":"> Date: Mon, 4 Nov 2024 09:40:06 -0500\n> From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>\n> \n> Alternatively, you can take a look at grokmirror, which is what kernel.org\n> uses:\n> https://pypi.org/project/grokmirror/\n> \n> It's pull-based instead of push-based, for several reasons:\n> \n> 1. We replicate to multiple worldwide frontends, and we expect that some of\n>    them may be unreachable at the time when we attempt a push\n> 2. This allows us to propagate repository deletes\n> 3. This allows us to propagate details like descriptions and authors\n> \n> Grokmirror also has a listener daemon that can trigger a pull, so it's\n> possible to have near-instantaneous replication by notifying the remote node\n> that a repository has been updated and should be pulled.\n\nThanks, that looks useful, but it's not quite what I'm looking for.\n\nPart of the goal is essentially the same (qualitative) types of\nservice guarantee that Github advertises:[*] once the user's `git\npush' command has succeeded with nonzero exit status, the objects and\nref updates have been written to multiple backing stores so it would\ntake a failure of a quorum of those backing stores to lose the data.\n\nIn particular, a backend may reject an update, and when this happens\n(in the multi-backend case, by enough backends that no quorum is\nreached), the user who ran `git push' needs to know that it failed so\nthey don't, e.g., delete their branch, run git gc, and go on their\nmerry way having silently lost data.\n\n\n[*] https://github.blog/engineering/infrastructure/stretching-spokes/\n"},{"id":"506573","messageId":"ZylMZGLdwVDFcAwF@tapette.crustytoothpaste.net","threadId":"62439","inReplyTo":"20241104133544.A04D760A95@jupiter.mumble.net","subject":"Re: Synchronous replication on push","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-11-04T22:36:20Z","receivedAt":"2024-11-04T22:36:28Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-11-04 at 13:35:44, Taylor R Campbell wrote:\n> Thanks, can you expand on how this would work with the constraints I\n> listed in my question?  Recapitulating:\n> \n>    One option, of course, is to use a replicated file system like\n>    glusterfs, or replicated block store like DRBD.  But that\n> \n>    (a) likely requires a lot more round-trips than git push/send-pack,\n>    (b) can't be used for replication to other git hosts like Github, and\n>    (c) can't be used for other remote transports like git-cinnabar.\n> \n> It sounds like rsyncing over ssh is incompatible with (b) and (c), but\n> perhaps I misunderstood what you're getting at.  I tried to see if\n> there is some way that reference-transaction hooks help me here but\n> there wasn't anything obvious to me.\n\nIt should be noted that you cannot do what GitHub does with the\nthree-phase commit with arbitrary remotes.  A three-phase commit\nprovides a prepared-to-commit stage where the backends agree that they\n(or at least a majority of them) will make the change.  The Git protocol\ndoesn't offer such functionality, so you can't use arbitrary remotes for\nthis purpose.  You'll need to either replicate to only hosts you control\n(as GitHub does), or you'll need to give up on having your three-phase\ncommit operation.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"506579","messageId":"20241104234705.GA3017597@coredump.intra.peff.net","threadId":"62439","inReplyTo":"20241102020653.766D1609AC@jupiter.mumble.net","subject":"Re: Synchronous replication on push","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-04T23:47:05Z","receivedAt":"2024-11-04T23:47:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Nov 02, 2024 at 02:06:53AM +0000, Taylor R Campbell wrote:\n\n> Whenever I push anything to it, I want the push -- that is, all the\n> objects, and all the ref updates -- to be synchronously replicated to\n> another remote repository, the back end:\n\nThis isn't quite how replication works at, say, GitHub. But let me first\nexplain some of what you're seeing, and then I'll give some higher level\ncomments at the end.\n\n> Here are some approaches I've tried:\n> \n> 1. `git clone --mirror -o backend git@backend.example.com:/repo.git'\n>    to create the front end repository, plus the following pre-receive\n>    hook in the front end:\n> \n> \t#!/bin/sh\n> \texec git push backend\n> \n>    This doesn't work because the pre-receive hook runs in the\n>    quarantine environment, and `git push' wants to update\n>    `refs/heads/main', which is forbidden in the quarantine\n>    environment.\n> \n>    (However, git push to frontend doesn't actually fail with nonzero\n>    exit status -- it prints an error message, `ref updates forbidden\n>    inside quarantine environment', but exits wtih status 0.)\n> \n>    But maybe the ref update is harmless in this environment.\n\nI think the quarantine error is working as designed. If your push\nupdates local refs in the frontend repo, any object-existence checks it\ndoes from the quarantine area are not necessarily valid if the\nquarantine environment goes away without migrating the objects (e.g., if\nyou reject the push).\n\nSo this:\n\n> 2. Same as (1), but the pre-receive hook is:\n> \n> \t#!/bin/sh\n> \tunset GIT_QUARANTINE_PATH\n> \texec git push backend\n\nis potentially dangerous. Instead, you should disable push's attempt to\nupdate the local tracking refs. There isn't an option to do that, but\nif you don't have a \"fetch\" config line, then there are no tracking\nrefs. I.e., rather than using \"clone --mirror\", create your frontend\nrepo like this:\n\n  git init --bare\n  git config remote.backend.url git@backend.example.com:/repo.git\n  git fetch backend refs/*:refs/*\n\nAnd then push won't try to update anything in the frontend repo.\n\n  Side note: there's a small maybe-bug here that I noticed if the\n  backend is on the same local filesystem. In that case\n  GIT_QUARANTINE_PATH remains set for the receive-pack process running\n  on the backend repo, and will refuse to update refs (where it should\n  be safe to do so!). In your example that doesn't happen because\n  GIT_QUARANTINE_PATH does not make it across the ssh connection. But\n  arguably we should be clearing GIT_QUARANTINE_PATH in local_repo_env\n  like we do for GIT_DIR, etc. I don't think you ran into this, but just\n  another hiccup I found while trying to reproduce your situation.\n\nMoving on...\n\n>    This doesn't work because `git push' in the pre-receive hook\n>    doesn't find anything it needs to push -- the ref update hasn't\n>    happened yet.\n\nRight. You could do it from a post-receive, but if the point is to be\nable to reject the push to the frontend, it must happen before the refs\nhave been updated! So...\n\n> 3. Same as (1), but the pre-receive hook assembles a command line of\n> \n> \texec git push backend ${new0}:${ref0} ${new1}:${ref1} ...,\n> \n>    with all the ref updates passed on stdin (ignoring the old values).\n\n...yes, this is the correct approach. You're not _quite_ passing all of\nthe relevant info, though, because you're ignoring the old value of each\nref. And ideally you'd make sure you were moving backend's ref0 from\n\"old0\" to \"new0\"; otherwise you risk overwriting something that happened\nindependently on the backend. Of course that creates new questions,\nlike what happens when the frontend and backend get out of sync.\n\n>    This fails because `--mirror can't be combined with refspecs'.\n\nYes. I don't think you really want \"--mirror\" in the first place, since\nyou won't be fetching from the backend (or will you? If you are, that\ncreates new questions about atomicity and syncing). If you do the\ninit+fetch above, it won't be set.\n\n> 4. Same as (3), but remote.backend.mirror is explicitly disabled after\n>    `git clone --mirror' finishes.\n> \n>    On push to the primary, this prints an error message\n> \n> \tremote: error: update_ref failed for ref 'refs/heads/main': ref updates forbidden inside quarantine environment\n> \n>    but somehow the push succeeds in spite of this message, and the\n>    primary and replica both get updated.\n\nThis is again the quarantine issue updating local tracking branches.\nHowever, we don't consider that a hard error, as updating them is\nopportunistic (we'd get the new values on the next fetch anyway).\n\nIf you drop the refspec as above, you shouldn't see that any more.\n\n> 5. Same as (1), but the pre-receive hook assembles a command line of\n> \n> \texec git send-pack git@backend.example.com:/repo.git \\\n> \t\t${new0}:${ref0} ${new1}:${ref1} ...\n> \n>    with all the ref updates passed on stdin (ignoring the old values).\n> \n>    This seems to work, and it propagates errors injected on push to\n>    the replica, but it is limited to local or ssh remotes, as far as I\n>    can tell -- it does not appear that git-send-pack works with custom\n>    remote transports.\n\nI don't remember all of the limitations of send-pack anymore. Even\nthough \"push\" is more porcelain than plumbing, I'd probably still\nrecommend it for a script, just because I think direct use of send-pack\nisn't going to be all that exercised, so you are likely to find missing\nbits of functionality and so forth. I think just dropping the refspecs\nand using push would be following the more well-trodden path.\n\n\nNow back to the main point: is this a good way to do replication? I\ndon't think it's _terrible_, but there are two flaws I can see:\n\n  1. You're not kicking off the backend push until the frontend has\n     received and processed the whole pack. So you're doubling the\n     end-to-end latency of the push. In an ideal world you'd actually\n     stream the incoming packfile to the backend, which would doing its\n     own quarantined index-pack[*] on it in real-time. And then when you\n     get to the pre-receive hook, all that's left is for all of the\n     replicas to agree to commit to the ref update.\n\n     [*] That would fix the latency, but of course you'd be spending a\n     bunch of CPU on each replica to do the same indexing computation.\n     You _could_ do that once, streaming the result out to the replicas,\n     and then sending them just the resulting index. But there is some\n     safety in repeating the computation on each replica (they _should_\n     all have the same objects, but if that isn't the case, you'd notice\n     if one of them was missing, say, a delta base that the others\n     have). GitHub's original replication design did repeat the\n     computation, and AFAIK that is still the case today.\n\n  2. Using \"push\" isn't a very atomic way of updating refs. The backends\n     will either accept the push or not, and then the frontend will try\n     to update its refs. What if it fails? What if another push comes in\n     simultaneously? Can they overwrite each other or lose pushed data?\n     Or get the frontend and backends out of sync?\n\n     Git's ref atomicity strategy is generally to take a lock on a ref,\n     then check that its current value is the expected \"old\" value, and\n     then update it to the \"new\" value and release the lock atomically.\n     So you probably want to ask each backend replica to take the ref\n     locks and check the old values, then respond \"yes, I'm ready to\n     commit\", and then you send back \"OK, commit\" at which point they do\n     the update.\n\n     But \"push\" doesn't give you that kind of granularity (neither for\n     the backends or on the frontend). Back when GitHub's replication\n     system was designed, nothing did, and we had to use custom code.\n     These days the reference-transaction lets you act in that stage\n     where the ref lock is held (and my understanding is that GitLab\n     implemented it to do the same kind of three-phase commit).\n\n     But I don't have much experience with it myself. It might be\n     enough if the frontend transaction hook talked to the backends,\n     initiating an update-ref there with a transaction hook to pause and\n     wait for the three-phase agreement.\n\nMaybe some of that points you in the right direction.\n\n-Peff\n"},{"id":"506587","messageId":"20241105013433.4E52260A64@jupiter.mumble.net","threadId":"62439","inReplyTo":"20241104234705.GA3017597@coredump.intra.peff.net","subject":"Re: Synchronous replication on push","fromName":"Taylor R Campbell","fromEmail":"git@campbell.mumble.net","sentAt":"2024-11-05T01:34:32Z","receivedAt":"2024-11-05T01:34:34Z","isPatch":false,"sender":{"key":"git@campbell.mumble.net","avatar":null},"body":"> Date: Mon, 4 Nov 2024 18:47:05 -0500\n> From: Jeff King <peff@peff.net>\n> \n> On Sat, Nov 02, 2024 at 02:06:53AM +0000, Taylor R Campbell wrote:\n> \n> > Whenever I push anything to it, I want the push -- that is, all the\n> > objects, and all the ref updates -- to be synchronously replicated to\n> > another remote repository, the back end:\n> \n> This isn't quite how replication works at, say, GitHub. But let me first\n> explain some of what you're seeing, and then I'll give some higher level\n> comments at the end.\n\nGreat, thanks!  I understand Github works differently, and I'm not\ntrying to replicate everything about Github's architecture, which I\nexpect to take substantial novel software engineering effort.  But I\nam trying to make sure I understand how the parts fit together well\nenough provide qualitatively similar types of guarantees about\ndurability when the user's `git push' exits nonzero.\n\nI really have two different goals here, which have similar needs for\nrelaying pushes but which I'm sure will diverge at some point:\n\n1. provide a synchronous push/pull git frontend to an hg backend with\n   git-cinnabar (so to ordinary git clients it looks just like an\n   ordinary git remote, without needing git-cinnabar), and\n\n2. provide a git frontend that replicates to one or many git backends\n   for better resilience to server loss.\n\n>                           Instead, you should disable push's attempt to\n> update the local tracking refs. There isn't an option to do that, but\n> if you don't have a \"fetch\" config line, then there are no tracking\n> refs. I.e., rather than using \"clone --mirror\", create your frontend\n> repo like this:\n> \n>   git init --bare\n>   git config remote.backend.url git@backend.example.com:/repo.git\n>   git fetch backend refs/*:refs/*\n> \n> And then push won't try to update anything in the frontend repo.\n\nThanks, that hadn't occurred to me as an option.\n\n>   Side note: there's a small maybe-bug here that I noticed if the\n>   backend is on the same local filesystem. In that case\n>   GIT_QUARANTINE_PATH remains set for the receive-pack process running\n>   on the backend repo, and will refuse to update refs (where it should\n>   be safe to do so!). In your example that doesn't happen because\n>   GIT_QUARANTINE_PATH does not make it across the ssh connection. But\n>   arguably we should be clearing GIT_QUARANTINE_PATH in local_repo_env\n>   like we do for GIT_DIR, etc. I don't think you ran into this, but just\n>   another hiccup I found while trying to reproduce your situation.\n\n(I did actually run into this, so in my test scripts I have been using\n\ngit {clone,config,...} ext::\"env -i PATH=$PATH git %s /path/to/backend.git\" ...\n\ninstead of just\n\ngit {clone,config,...} /path/to/backend.git ...\n\nin order to nix GIT_QUARANTINE_PATH from the environment -- and\nanything else I might not have thought of -- while running\ngit-receive-pack on the backend.  But it didn't seem germane to the\nproblem at hand so I didn't want to clutter up my already somewhat\nlong question with such details unless someone asked me to share my\nreproducer!)\n\n> > 3. Same as (1), but the pre-receive hook assembles a command line of\n> > \n> > \texec git push backend ${new0}:${ref0} ${new1}:${ref1} ...,\n> > \n> >    with all the ref updates passed on stdin (ignoring the old values).\n> \n> ...yes, this is the correct approach. You're not _quite_ passing all of\n> the relevant info, though, because you're ignoring the old value of each\n> ref. And ideally you'd make sure you were moving backend's ref0 from\n> \"old0\" to \"new0\"; otherwise you risk overwriting something that happened\n> independently on the backend. Of course that creates new questions,\n> like what happens when the frontend and backend get out of sync.\n\nRight -- there will be some combination of --force-with-lease or\npre-receive tests at the other end to handle this.  But for now my\nfocus is on making git push work in pre-receive at all.\n\nAs long as anything out-of-sync leads to noisy failure, possibly\nrequiring manual intervention, that's good enough for now (and I'm not\n(yet) concerned with .\n\n> > \tremote: error: update_ref failed for ref 'refs/heads/main': ref updates forbidden inside quarantine environment\n> > \n> >    but somehow the push succeeds in spite of this message, and the\n> >    primary and replica both get updated.\n> \n> This is again the quarantine issue updating local tracking branches.\n> However, we don't consider that a hard error, as updating them is\n> opportunistic (we'd get the new values on the next fetch anyway).\n> \n> If you drop the refspec as above, you shouldn't see that any more.\n\nYes, thanks!\n\n> Now back to the main point: is this a good way to do replication? I\n> don't think it's _terrible_, but there are two flaws I can see:\n\nThese are all good points that I will consider once I get to them now\nthat I can make progress past the obstacle of local tracking ref\nupdates in pre-receive git push, thanks.\n\n>   1. You're not kicking off the backend push until the frontend has\n>      received and processed the whole pack. So you're doubling the\n>      end-to-end latency of the push. In an ideal world you'd actually\n>      stream the incoming packfile to the backend, which would doing its\n>      own quarantined index-pack[*] on it in real-time. And then when you\n>      get to the pre-receive hook, all that's left is for all of the\n>      replicas to agree to commit to the ref update.\n\nGit doesn't currently have any hooks for doing this, right?  So\npresumably this will require a custom git-receive-pack replacement\nthat understands the git wire protocol to stream the packfile to\nbackends (which is what I assume Github's spokes proxies do).\n\n>   2. Using \"push\" isn't a very atomic way of updating refs. The backends\n>      will either accept the push or not, and then the frontend will try\n>      to update its refs. What if it fails? What if another push comes in\n>      simultaneously? Can they overwrite each other or lose pushed data?\n>      Or get the frontend and backends out of sync?\n\nRight -- there's a lot to work out for the three-phase commit part.\nOne simplification for now is to reject non-fast-forward pushes (and\nref deletion), and to not worry too much about ordering of independent\nref updates or whether I even want serializable isolation or just\nread-repeatable or -committed for that.\n\nThat said, regarding push atomicity: Suppose users concurrently do\n\nalice$ git push frontend X Y\nbob$ git push frontend Y X\n\nThat is, there are overlapping ref updates, and suppose Alice and Bob\nhave incompatible referents for X and Y (non-fast-forward, or they're\nusing --force-with-lease but not --atomic, or whatever).\n\nWhen are the locks on X and Y taken relative to pre-receive in the\nfrontend?  Can the pre-receive hooks for Alice's push and Bob's push\nrun concurrently or are they serialized by locks on the common refs X\nand Y?  This can't deadlock, can it?  (I assume the locks on refs are\ntaken in a consistent order.)\n\nIt's unclear to me from the githooks(5), git-push(1), and\ngit-receive-pack(1) man pages what the ordering of hooks and ref\nlocking is, or what serialization guarantees hooks have -- if any.\n"}]}