{"thread":{"id":"51537","subject":"Push force-with-lease with multi-URL remote","startedAt":"2019-07-27T17:14:04Z","lastAt":"2019-07-29T22:29:03Z","messageCount":12,"participants":["Christopher Head","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"379420","messageId":"20190727095440.1aac3b3c@amdahl.home.chead.ca","threadId":"51537","inReplyTo":null,"subject":"Push force-with-lease with multi-URL remote","fromName":"Christopher Head","fromEmail":"bugs@chead.ca","sentAt":"2019-07-27T16:54:40Z","receivedAt":"2019-07-27T17:14:04Z","isPatch":false,"sender":{"key":"bugs@chead.ca","avatar":null},"body":"Hi folks,\nWhen a single remote has multiple push URLs, Git’s force-with-lease\nlogic appears to be:\n\nFor each URL:\n1. Read refs/heads/mybranch (call this commit X)\n2. Read refs/remotes/myremote/mybranch (call this commit Y)\n3. Send to the URL an atomic compare-and-swap, replacing Y with X.\n4. If step 3 succeeded, change refs/remotes/myremote/mybranch to X.\n\nThis means that, assuming both URLs start out identical, the second URL\nwill always fail because refs/remots/myremote/mybranch has been updated\nfrom Y to X, and therefore the second compare-and-swap fails. I can’t\nimagine any situation in which this behaviour is actually useful.\n\nThis is what I would expect:\n\n1. Read refs/heads/mybranch (call this commit X)\n2. Read refs/remotes/myremote/mybranch (call this commit Y)\n3. For each URL:\n3a. Send to the URL an atomic compare-and-swap, replacing Y with X.\n4. If any (or maybe all) of the CAS operations in 3a succeeded, change\nrefs/remotes/myremote/mybranch to X.\n\nThoughts? Does anyone have a use case for the existing behaviour? I\nhave attached a shell script which constructs some repos and\ndemonstrates the situation.\n\nThanks!\n-- \nChristopher Head\n"},{"id":"379421","messageId":"xmqqzhkzl69o.fsf@gitster-ct.c.googlers.com","threadId":"51537","inReplyTo":"20190727095440.1aac3b3c@amdahl.home.chead.ca","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-07-27T17:46:43Z","receivedAt":"2019-07-27T17:46:50Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christopher Head writes:\n\n> Hi folks,\n> When a single remote has multiple push URLs, Git’s force-with-lease\n> logic appears to be:\n>\n> For each URL:\n> 1. Read refs/heads/mybranch (call this commit X)\n> 2. Read refs/remotes/myremote/mybranch (call this commit Y)\n> 3....\n> from Y to X, and therefore the second compare-and-swap fails. I can’t\n> imagine any situation in which this behaviour is actually useful.\n\nQuite honestly, the true culprit of the above story is that you are\nletting multiple logically different remote repositories [*1*] use\nthe same single remote-tracking refes/remotes/myremote/ hierarchy.\n\nIf your previous \"git push myremote\" (with or without lease does not\nmatter, as this discussion is to illustrate that your setup is\nfundamentally wrong) updated X but for some reason failed to update\nY (perhaps the network to Y was unreachable back then), and\nrefs/remotes/myremote/mybrach got updated to reflect the update to\nX, what happens to your next \"git push myremote\" (or more\nspecifically, \"git push Y\")? The repository on your local side\nthinks that the other party has already took the previous push but\nin reality that is the state of X, and Y hasn't seen that previous\npush.\n\n"},{"id":"379422","messageId":"0DC4FE1B-90A9-414B-AFAC-E328608C4F98@chead.ca","threadId":"51537","inReplyTo":"xmqqzhkzl69o.fsf@gitster-ct.c.googlers.com","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Christopher Head","fromEmail":"bugs@chead.ca","sentAt":"2019-07-27T18:15:37Z","receivedAt":"2019-07-27T18:15:44Z","isPatch":false,"sender":{"key":"bugs@chead.ca","avatar":null},"body":"On July 27, 2019 10:46:43 AM PDT, Junio C Hamano wrote:\n>Christopher Head writes:\n>\n>Quite honestly, the true culprit of the above story is that you are\n>letting multiple logically different remote repositories [*1*] use\n>the same single remote-tracking refes/remotes/myremote/ hierarchy.\n\nThey weren’t supposed to be logically different (if I understand what you mean by that phrase). My intention was for the different URLs to be mirrors of each other, and multiple push URLs seemed to be the easiest way to update all the mirrors without having to mess around with making them trust each other and sending notifications and such.\n\nIf not this, then what are multiple push URLs on a single remote meant for?\n\n>If your previous \"git push myremote\" (with or without lease does not\n>matter, as this discussion is to illustrate that your setup is\n>fundamentally wrong) updated X but for some reason failed to update\n>Y (perhaps the network to Y was unreachable back then), and\n>refs/remotes/myremote/mybrach got updated to reflect the update to\n>X, what happens to your next \"git push myremote\" (or more\n>specifically, \"git push Y\")? The repository on your local side\n>thinks that the other party has already took the previous push but\n>in reality that is the state of X, and Y hasn't seen that previous\n>push.\n\nOf course it wouldn’t be perfect even with my proposed behaviour. It just seemed more useful than the existing behaviour, which will essentially *never* do anything useful as far as I can tell.\n\n-- \nChristopher Head\n"},{"id":"379429","messageId":"xmqqv9vnkxg1.fsf@gitster-ct.c.googlers.com","threadId":"51537","inReplyTo":"0DC4FE1B-90A9-414B-AFAC-E328608C4F98@chead.ca","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-07-27T20:57:18Z","receivedAt":"2019-07-27T20:57:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christopher Head writes:\n\n> On July 27, 2019 10:46:43 AM PDT, Junio C Hamano wrote:\n>>Christopher Head writes:\n>>\n>>Quite honestly, the true culprit of the above story is that you are\n>>letting multiple logically different remote repositories [*1*] use\n>>the same single remote-tracking refes/remotes/myremote/ hierarchy.\n>\n> They weren’t supposed to be logically different (if I understand\n> what you mean by that phrase). My intention was for the different\n> URLs to be mirrors of each other,...\n\nWhat I would call \"logically the same\" is the set of repositories\nthat are synchronized at the server side, which may or may not have\nmultiple URLs to reach it, but behave as if it is just a single one\nwithout your doing anything special. Your wanting to actively make\nthem in sync by the above definition makes them logically different\nset of repositories. But the phrasing does not matter much.\n\nOne repository at a hosting service (which may iternally be\nreplicated, but that is not even observable from outside) may be\nreached over https:// or ssh://, and the result of pushing to either\none of the URLs would be observable by immediately fetching back\nfrom either one. Having both URLs and trying to use either one that\nworks may help under flaky proxy situation, for example.\n\nIn the reverse direction, I think \"git fetch\" supports the notion of\n of repositories, so that fetch from multiple remotes can be\ninitiated with a single command, but I am not sure if we added the\nsame concept to the pushing side. I personally want to have\nfiner control, so when I push my work to multiple repositories, each\nof them are treated as totally different push targets, and a script\ncontrols multiple \"git push\" processes to each of them in parallel,\nwith retries and all. I think having multiple pushURL and pushing\nto them is sort-of OK, but what is broken in your configuration is\nthat you have a single remote-tracking branch hierarchy---if you get\nrid of it, so that refs/remotes/myremote/ does not exist and does\nnot get updated, I think things would work fine.\n\n\n\n\n\n\n"},{"id":"379433","messageId":"C021D654-ECD4-4A23-9DE0-D272C3A3D901@chead.ca","threadId":"51537","inReplyTo":"xmqqv9vnkxg1.fsf@gitster-ct.c.googlers.com","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Christopher Head","fromEmail":"bugs@chead.ca","sentAt":"2019-07-27T21:43:06Z","receivedAt":"2019-07-27T21:43:13Z","isPatch":false,"sender":{"key":"bugs@chead.ca","avatar":null},"body":"On July 27, 2019 1:57:18 PM PDT, Junio C Hamano wrote:\n>What I would call \"logically the same\" is the set of repositories\n>that are synchronized at the server side, which may or may not have\n>multiple URLs to reach it, but behave as if it is just a single one\n>without your doing anything special. Your wanting to actively make\n>them in sync by the above definition makes them logically different\n>set of repositories. But the phrasing does not matter much.\n\nOK, I would probably have just used one push URL in this scenario.\n\n>One repository at a hosting service (which may iternally be\n>replicated, but that is not even observable from outside) may be\n>reached over https:// or ssh://, and the result of pushing to either\n>one of the URLs would be observable by immediately fetching back\n>from either one. Having both URLs and trying to use either one that\n>works may help under flaky proxy situation, for example.\n\nThat makes sense, I guess, if the unusable URLs can be expected to fail fast.\n\n>In the reverse direction, I think \"git fetch\" supports the notion of\n> of repositories, so that fetch from multiple remotes can be\n>initiated with a single command, but I am not sure if we added the\n>same concept to the pushing side. I personally want to have\n>finer control, so when I push my work to multiple repositories, each\n>of them are treated as totally different push targets, and a script\n>controls multiple \"git push\" processes to each of them in parallel,\n>with retries and all. I think having multiple pushURL and pushing\n>to them is sort-of OK, but what is broken in your configuration is\n>that you have a single remote-tracking branch hierarchy---if you get\n>rid of it, so that refs/remotes/myremote/ does not exist and does\n>not get updated, I think things would work fine.\n\nYes, I agree, the presence of only a single remote tracking ref is what makes the use of a single remote with multiple URLs suboptimal here—it was just a better than the other options. I tend to have pretty reliable Internet connectivity, and I don’t particularly want to go writing custom scripts. I just want to use the normal push and fetch commands. Using multiple URLs on one remote is OK, though the single remote tracking ref is annoying. Using separate remotes works, but is more annoying due to having to remember to push to all of them. If I understand what remote groups are (separate remotes but you can act on all of them with one command?) then they should be perfect. However it does not look like they work for pushing. Would it make sense to add?\n-- \nChristopher Head\n"},{"id":"379471","messageId":"xmqqa7cxl8od.fsf@gitster-ct.c.googlers.com","threadId":"51537","inReplyTo":"C021D654-ECD4-4A23-9DE0-D272C3A3D901@chead.ca","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-07-29T05:19:14Z","receivedAt":"2019-07-29T05:19:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christopher Head writes:\n\n> On July 27, 2019 1:57:18 PM PDT, Junio C Hamano wrote:\n> ...\n>>In the reverse direction, I think \"git fetch\" supports the notion of\n>> of repositories, so that fetch from multiple remotes can be\n>>initiated with a single command, but I am not sure if we added the\n>>same concept to the pushing side.\n>>...\n\n> If I understand what remote groups are (separate remotes but you\n> can act on all of them with one command?) then they should be\n> perfect. However it does not look like they work for\n> pushing.\n\nYup, you are confirming what I already said ;-).\n\nI do not offhand think of a fundamental reason why the \nconcept should only apply to the fetching direction. I am not sure\nabout a few design issues if we were to have \"push\" groups, though,\nand somebody who wants to have the feature must think hard about.\n\nShould the same be usable for both fetching and pushing, or\nshould there be two separate and independent namespaces, one for\nfetch groups and the other for push groups, so that the set of\nrepositories \"git fetch groupA\" fetches from could be configured to\nbe different from the set of repositories \"git push groupA\"? It can\nbe argued for both ways, but I am unsure about the pros and cons.\n\nHow should the feature interact with push atomicity? We obviously\nwould not want (and probably cannot afford) to arrange two or more\nrepositories coodinate by participating in two-phase commit etc., so\nthe best we could do may be to not even initiate push after seeing\na push to one destination fail, but there may be a better definition\npeople can come up with.\n\n\n"},{"id":"379490","messageId":"20190729102009.GC2755@sigill.intra.peff.net","threadId":"51537","inReplyTo":"20190727095440.1aac3b3c@amdahl.home.chead.ca","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-07-29T10:20:10Z","receivedAt":"2019-07-29T10:20:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jul 27, 2019 at 09:54:40AM -0700, Christopher Head wrote:\n\n> For each URL:\n> 1. Read refs/heads/mybranch (call this commit X)\n> 2. Read refs/remotes/myremote/mybranch (call this commit Y)\n> 3. Send to the URL an atomic compare-and-swap, replacing Y with X.\n> 4. If step 3 succeeded, change refs/remotes/myremote/mybranch to X.\n> \n> This means that, assuming both URLs start out identical, the second URL\n> will always fail because refs/remots/myremote/mybranch has been updated\n> from Y to X, and therefore the second compare-and-swap fails. I can’t\n> imagine any situation in which this behaviour is actually useful.\n\nMy general feeling is that having multiple push URLs for a remote is a\npoorly designed feature in Git (and I think the discussion elsewhere in\nthis thread went there, as well).\n\nBut since we do have it, and if we are not going to deprecate it[1], it\nseems like this case should pick the X value of myremote/mybranch ahead\nof time, and then use it consistently for each push. There are questions\nof partial push failures, etc, but as you note the current behavior\nisn't ever useful. I think it just a case where two features do not\ninteract well (and since neither is used all that frequently, nobody has\nnoticed).\n\n-Peff\n\n[1] I would not be at all sad to see multiple URLs like this get\n deprecated in favor of multiple remotes with convenient grouping\n options. If that happens, then your original problem goes away. ;)\n"},{"id":"379497","messageId":"xmqq7e81vuc3.fsf@gitster-ct.c.googlers.com","threadId":"51537","inReplyTo":"20190729102009.GC2755@sigill.intra.peff.net","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-07-29T13:33:32Z","receivedAt":"2019-07-29T13:33:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> My general feeling is that having multiple push URLs for a remote is a\n> poorly designed feature in Git (and I think the discussion elsewhere in\n> this thread went there, as well).\n\nThat's being generous. I do not think it was even designed---at\nleast, the interaction with remote-tracking is ill thought out,\nbut I think the updating of remote-tracking by pretending to have\nturned around and fetched immediately after it has done its thing\ncame much later than multiple URLs for push. A remote with multiple\nURLs without any remote-tracking (i.e. \"push only remote\") behaves\nsemi-sensibly.\n\n> But since we do have it, and if we are not going to deprecate it[1], it\n> seems like this case should pick the X value of myremote/mybranch ahead\n> of time, and then use it consistently for each push.\n\nI agree but only if the listed ones are separate ones. If the URLs\nare separate paths to reach the same remote (e.g. https:// and ssh://\ngoing to the same place), the current definition would make more sense.\n"},{"id":"379500","messageId":"20190729064741.005bc16a@amdahl.home.chead.ca","threadId":"51537","inReplyTo":"xmqq7e81vuc3.fsf@gitster-ct.c.googlers.com","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Christopher Head","fromEmail":"chead@chead.ca","sentAt":"2019-07-29T13:47:41Z","receivedAt":"2019-07-29T14:14:04Z","isPatch":false,"sender":{"key":"chead@chead.ca","avatar":null},"body":"On Mon, 29 Jul 2019 06:33:32 -0700\nJunio C Hamano wrote:\n\n> I agree but only if the listed ones are separate ones. If the URLs\n> are separate paths to reach the same remote (e.g. https:// and ssh://\n> going to the same place), the current definition would make more\n> sense.\n\nI realize I’m a bit biased towards my personal use case, but I wonder\nif it would make sense to consider how frequently each case occurs?\n\nCase 1: someone wants to keep multiple repos mirrored, by always\npushing to all of them (my use case).\n\nCase 2: someone wants to push to one repo, but vagaries of Internet\nconnectivity mean that sometimes they can’t use SSH and other times\nthey can’t use HTTP (or they prefer one protocol but sometimes that\none doesn’t work), therefore they want both URLs so that when one fails\nthe other may work. I suppose the most common situation in this case is\nthat you want to use SSH so that you don’t have to type a password, but\nsometimes you’re in a site which only allows HTTP connections and\ntyping a password as a fallback is preferable to failing altogether?\n\nFor me, case 1 happens quite frequently but case 2 pretty much never—I\ndon’t think I’ve ever been somewhere that blocks port 22 outbound, so I\nalways just use SSH. But I realize other people’s experience varies.\n-- \nChristopher Head\n"},{"id":"379514","messageId":"20190729192040.GD14943@sigill.intra.peff.net","threadId":"51537","inReplyTo":"xmqq7e81vuc3.fsf@gitster-ct.c.googlers.com","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-07-29T19:20:40Z","receivedAt":"2019-07-29T19:20:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 29, 2019 at 06:33:32AM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > My general feeling is that having multiple push URLs for a remote is a\n> > poorly designed feature in Git (and I think the discussion elsewhere in\n> > this thread went there, as well).\n> \n> That's being generous. I do not think it was even designed---at\n> least, the interaction with remote-tracking is ill thought out,\n> but I think the updating of remote-tracking by pretending to have\n> turned around and fetched immediately after it has done its thing\n> came much later than multiple URLs for push. A remote with multiple\n> URLs without any remote-tracking (i.e. \"push only remote\") behaves\n> semi-sensibly.\n\nYeah, the auto-update of the tracking refs came later (so I think you\ncould argue the bad interaction is my fault!).\n\n> > But since we do have it, and if we are not going to deprecate it[1], it\n> > seems like this case should pick the X value of myremote/mybranch ahead\n> > of time, and then use it consistently for each push.\n> \n> I agree but only if the listed ones are separate ones. If the URLs\n> are separate paths to reach the same remote (e.g. https:// and ssh://\n> going to the same place), the current definition would make more sense.\n\nHmm, true. I'd almost argue that --force-with-lease, at least in its\ndefault mode with no explicit lease source specified, should allow an\nupdate from X to Y to be a successful noop if the remote \"somehow\"\nalready moved to Y.\n\nThis multi-URL push is one such \"somehow\", but I could imagine a case\nwhere two other independent processes are racing. And we do not care at\nall how we get to \"Y\", only that we get there.\n\nBut I haven't thought it through carefully, and I wonder if some users\nwould be unhappy not to find out that somebody had moved to \"Y\" already.\n\n-Peff\n"},{"id":"379553","messageId":"xmqqwog0tt27.fsf@gitster-ct.c.googlers.com","threadId":"51537","inReplyTo":"20190729192040.GD14943@sigill.intra.peff.net","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-07-29T21:44:00Z","receivedAt":"2019-07-29T21:44:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Yeah, the auto-update of the tracking refs came later (so I think you\n> could argue the bad interaction is my fault!).\n\nHeh, I somehow thought it was somebody else.\n\n> Hmm, true. I'd almost argue that --force-with-lease, at least in its\n> default mode with no explicit lease source specified, should allow an\n> update from X to Y to be a successful noop if the remote \"somehow\"\n> already moved to Y.\n\nI've already written the --force-with-lease that expects what you\nhave on your remote-tracking branch off as a gross misdesign that\nshould be deprecated in the longer term; I do not have a strong\nopinion on the tweaks to be done to the feature until it gets\ndropped ;-)\n\n"},{"id":"379560","messageId":"20190729222900.GA8665@sigill.intra.peff.net","threadId":"51537","inReplyTo":"xmqqwog0tt27.fsf@gitster-ct.c.googlers.com","subject":"Re: Push force-with-lease with multi-URL remote","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-07-29T22:29:01Z","receivedAt":"2019-07-29T22:29:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 29, 2019 at 02:44:00PM -0700, Junio C Hamano wrote:\n\n> > Hmm, true. I'd almost argue that --force-with-lease, at least in its\n> > default mode with no explicit lease source specified, should allow an\n> > update from X to Y to be a successful noop if the remote \"somehow\"\n> > already moved to Y.\n> \n> I've already written the --force-with-lease that expects what you\n> have on your remote-tracking branch off as a gross misdesign that\n> should be deprecated in the longer term; I do not have a strong\n> opinion on the tweaks to be done to the feature until it gets\n> dropped ;-)\n\nWell, that part I certainly agree with. ;)\n\n-Peff\n"}]}