{"thread":{"id":"53588","subject":"Git multiple remotes push stop at first failed connection","startedAt":"2020-06-01T00:28:55Z","lastAt":"2020-06-02T16:54:25Z","messageCount":5,"participants":["John Siu","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"398965","messageId":"CAGKX4vGhTbEqZS9+iYA2wZWRRaddQC6O4KV+zLaNYKkZgN36Xg@mail.gmail.com","threadId":"53588","inReplyTo":null,"subject":"Git multiple remotes push stop at first failed connection","fromName":"John Siu","fromEmail":"john.sd.siu@gmail.com","sentAt":"2020-06-01T00:28:38Z","receivedAt":"2020-06-01T00:28:55Z","isPatch":false,"sender":{"key":"john.sd.siu@gmail.com","avatar":"https://gravatar.com/avatar/3270ac639147206b23fe4f3273c8b6436a4b21830a3fde065ebab4210972cfdf?d=mp&s=160"},"body":"Hi,\n\ngit version 2.26.2\nOS: MacOS 10.15.5\nInstall via brew\n\nLet say my project has following remotes:\n\n$ git remote -v\ngit.all \"server A git url\" (fetch)\ngit.all \"server A git url\" (push)\ngit.all \"server B git url\" (push)\ngit.all \"server C git ur\" (push)\n\nWhen all serverA/B/C are online, \"git push\" works.\n\nHowever \"git push\" will stop at the first server it failed to connect.\nSo if git cannot connect to server A, it will not continue with server\nB/C.\n\nIn the past I have server C turn off from time to time, so failing the\nlast push is expected. However recently server A went offline\ncompletely and we notice git is not pushing to the remaining 2\nremotes.\n\nNot sure if this is intended behavior or can be improved.\n\nJohn Siu\n"},{"id":"399001","messageId":"20200601214003.GA3309882@coredump.intra.peff.net","threadId":"53588","inReplyTo":"CAGKX4vGhTbEqZS9+iYA2wZWRRaddQC6O4KV+zLaNYKkZgN36Xg@mail.gmail.com","subject":"Re: Git multiple remotes push stop at first failed connection","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-06-01T21:40:03Z","receivedAt":"2020-06-01T21:40:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, May 31, 2020 at 08:28:38PM -0400, John Siu wrote:\n\n> Let say my project has following remotes:\n> \n> $ git remote -v\n> git.all \"server A git url\" (fetch)\n> git.all \"server A git url\" (push)\n> git.all \"server B git url\" (push)\n> git.all \"server C git ur\" (push)\n> \n> When all serverA/B/C are online, \"git push\" works.\n\nA slight nomenclature nit, but that's _one_ remote that has several\npush urls.\n\n> However \"git push\" will stop at the first server it failed to connect.\n> So if git cannot connect to server A, it will not continue with server\n> B/C.\n> \n> In the past I have server C turn off from time to time, so failing the\n> last push is expected. However recently server A went offline\n> completely and we notice git is not pushing to the remaining 2\n> remotes.\n> \n> Not sure if this is intended behavior or can be improved.\n\nI don't think we've ever documented the error-handling semantics.\nLooking at the relevant code in builtin/push.c:do_push():\n\n url_nr = push_url_of_remote(remote, &url);\n if (url_nr) {\n for (i = 0; i < url_nr; i++) {\n struct transport *transport =\n transport_get(remote, url[i]);\n if (flags & TRANSPORT_PUSH_OPTIONS)\n transport->push_options = push_options;\n if (push_with_options(transport, push_refspec, flags))\n errs++;\n }\n } else {\n struct transport *transport =\n transport_get(remote, NULL);\n if (flags & TRANSPORT_PUSH_OPTIONS)\n transport->push_options = push_options;\n if (push_with_options(transport, push_refspec, flags))\n errs++;\n }\n return !!errs;\n\nit does seem to try each one and collect the errors. But the underlying\ntransport code is so ready to die() on errors, taking down the whole\nprocess, that I suspect it rarely manages to do so. You're probably much\nbetter off defining a separate remote for each push destination, then\nrunning your own shell loop:\n\n err=0\n for dst in serverA serverB serverC; do\n git push $dst || err=1\n done\n exit $err\n\nThere's really no benefit to doing it all in a single Git process, as\nwe'd connect to each independently, run a separate independent\npack-objects for each, etc.\n\nI'd even suggest that Git implement such a loop itself, as we did for\n\"git fetch --all\", but sadly \"push --all\" is already taken for a\ndifferent meaning (but it might still be worth doing under a different\noption name).\n\n-Peff\n"},{"id":"399012","messageId":"CAGKX4vH42x7nhCgtJ80fiNeaj1ROVxc-R8gO6xV21-6D3GbQSg@mail.gmail.com","threadId":"53588","inReplyTo":"20200601214003.GA3309882@coredump.intra.peff.net","subject":"Re: Git multiple remotes push stop at first failed connection","fromName":"John Siu","fromEmail":"john.sd.siu@gmail.com","sentAt":"2020-06-02T01:32:55Z","receivedAt":"2020-06-02T01:33:12Z","isPatch":false,"sender":{"key":"john.sd.siu@gmail.com","avatar":"https://gravatar.com/avatar/3270ac639147206b23fe4f3273c8b6436a4b21830a3fde065ebab4210972cfdf?d=mp&s=160"},"body":"Thank Jeff for replying.\n\nYes, one route we are looking into is use fetch -all, push -all and\nwrap push tag with a loop, or may do what you suggest and wrap all 3\nin a loop for consistency.\n\nOn Mon, Jun 1, 2020 at 5:40 PM Jeff King wrote:\n>\n> On Sun, May 31, 2020 at 08:28:38PM -0400, John Siu wrote:\n>\n> > Let say my project has following remotes:\n> >\n> > $ git remote -v\n> > git.all \"server A git url\" (fetch)\n> > git.all \"server A git url\" (push)\n> > git.all \"server B git url\" (push)\n> > git.all \"server C git ur\" (push)\n> >\n> > When all serverA/B/C are online, \"git push\" works.\n>\n> A slight nomenclature nit, but that's _one_ remote that has several\n> push urls.\n>\n> > However \"git push\" will stop at the first server it failed to connect.\n> > So if git cannot connect to server A, it will not continue with server\n> > B/C.\n> >\n> > In the past I have server C turn off from time to time, so failing the\n> > last push is expected. However recently server A went offline\n> > completely and we notice git is not pushing to the remaining 2\n> > remotes.\n> >\n> > Not sure if this is intended behavior or can be improved.\n>\n> I don't think we've ever documented the error-handling semantics.\n> Looking at the relevant code in builtin/push.c:do_push():\n>\n> url_nr = push_url_of_remote(remote, &url);\n> if (url_nr) {\n> for (i = 0; i < url_nr; i++) {\n> struct transport *transport =\n> transport_get(remote, url[i]);\n> if (flags & TRANSPORT_PUSH_OPTIONS)\n> transport->push_options = push_options;\n> if (push_with_options(transport, push_refspec, flags))\n> errs++;\n> }\n> } else {\n> struct transport *transport =\n> transport_get(remote, NULL);\n> if (flags & TRANSPORT_PUSH_OPTIONS)\n> transport->push_options = push_options;\n> if (push_with_options(transport, push_refspec, flags))\n> errs++;\n> }\n> return !!errs;\n>\n> it does seem to try each one and collect the errors. But the underlying\n> transport code is so ready to die() on errors, taking down the whole\n> process, that I suspect it rarely manages to do so. You're probably much\n> better off defining a separate remote for each push destination, then\n> running your own shell loop:\n>\n> err=0\n> for dst in serverA serverB serverC; do\n> git push $dst || err=1\n> done\n> exit $err\n>\n> There's really no benefit to doing it all in a single Git process, as\n> we'd connect to each independently, run a separate independent\n> pack-objects for each, etc.\n>\n> I'd even suggest that Git implement such a loop itself, as we did for\n> \"git fetch --all\", but sadly \"push --all\" is already taken for a\n> different meaning (but it might still be worth doing under a different\n> option name).\n>\n> -Peff\n"},{"id":"399031","messageId":"xmqqr1uxtow4.fsf@gitster.c.googlers.com","threadId":"53588","inReplyTo":"20200601214003.GA3309882@coredump.intra.peff.net","subject":"Re: Git multiple remotes push stop at first failed connection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-06-02T16:26:35Z","receivedAt":"2020-06-02T16:26:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> There's really no benefit to doing it all in a single Git process, as\n> we'd connect to each independently, run a separate independent\n> pack-objects for each, etc.\n>\n> I'd even suggest that Git implement such a loop itself, as we did for\n> \"git fetch --all\", but sadly \"push --all\" is already taken for a\n> different meaning (but it might still be worth doing under a different\n> option name).\n\nI wonder if it is possible to update the implementation to do so\nwithout changing the UI at all, though.\n\nThe presence of the \"--all\" option in \"fetch\" command is tied\nclosely to the fact that it makes no sense to have multiple URLs\nthat are used to download from at the same time under a single\nremote name (e.g. what should \"remotes/origin/master\" point at if\ntwo URLs say different things if such an arrangement were allowed?).\n\nOn the other hand, the pushURL for a single remote can be multiple\nplaces for redundancy (a possible #leftoverbits here is that we\nshould probably disable the \"pretend that we immediately turned\naround and fetched from them after pushing\" optimization when\npushing to a remote that has multiple pushURLs defined) does not\nneed an extra option. If the way we currently push is suboptimal\nand it is better to spawn a separate \"git push\" instance via the\nrun_command() API, that can safely be done as a bugfix without\naffecting any UI elements, no?\n\n"},{"id":"399034","messageId":"CAGKX4vFcqQ_0XFb5qOku9wAxF3+fj-fByrm+zmSXHr3k60yjKw@mail.gmail.com","threadId":"53588","inReplyTo":"xmqqr1uxtow4.fsf@gitster.c.googlers.com","subject":"Re: Git multiple remotes push stop at first failed connection","fromName":"John Siu","fromEmail":"john.sd.siu@gmail.com","sentAt":"2020-06-02T16:54:10Z","receivedAt":"2020-06-02T16:54:25Z","isPatch":false,"sender":{"key":"john.sd.siu@gmail.com","avatar":"https://gravatar.com/avatar/3270ac639147206b23fe4f3273c8b6436a4b21830a3fde065ebab4210972cfdf?d=mp&s=160"},"body":"On Tue, Jun 2, 2020 at 12:26 PM Junio C Hamano wrote:\n>\n> Jeff King writes:\n>\n> > There's really no benefit to doing it all in a single Git process, as\n> > we'd connect to each independently, run a separate independent\n> > pack-objects for each, etc.\n> >\n> > I'd even suggest that Git implement such a loop itself, as we did for\n> > \"git fetch --all\", but sadly \"push --all\" is already taken for a\n> > different meaning (but it might still be worth doing under a different\n> > option name).\n>\n\nYes. We notice the fetch/push --all is for branches.\n\n> I wonder if it is possible to update the implementation to do so\n> without changing the UI at all, though.\n>\n> The presence of the \"--all\" option in \"fetch\" command is tied\n> closely to the fact that it makes no sense to have multiple URLs\n> that are used to download from at the same time under a single\n> remote name (e.g. what should \"remotes/origin/master\" point at if\n> two URLs say different things if such an arrangement were allowed?).\n>\n> On the other hand, the pushURL for a single remote can be multiple\n> places for redundancy (a possible #leftoverbits here is that we\n> should probably disable the \"pretend that we immediately turned\n> around and fetched from them after pushing\" optimization when\n> pushing to a remote that has multiple pushURLs defined) does not\n> need an extra option. If the way we currently push is suboptimal\n> and it is better to spawn a separate \"git push\" instance via the\n> run_command() API, that can safely be done as a bugfix without\n> affecting any UI elements, no?\n>\n\nI agree a \"bugfix\" for push only is good enough and safe. As the\ncurrent behavior is already pushing to all pushURLs of a single\nremote. We are not trying to change behavior or do anything extra.\n"}]}