threads / discuss / 53588

Git multiple remotes push stop at first failed connection

Subject: Git multiple remotes push stop at first failed connection

## tl;dr

5 messages between Jun 1, 2020 and Jun 2, 2020.

replies: 4people: 3as markdown or json

John Siu· Jun 1, 2020, 00:28 UTC · lore
Hi,
git version 2.26.2
OS: MacOS 10.15.5
Install via brew
Let say my project has following remotes:

$ git remote -v git.all "server A git url" (fetch) git.all "server A git url" (push) git.all "server B git url" (push) git.all "server C git ur" (push)

When all serverA/B/C are online, "git push" works.

However "git push" will stop at the first server it failed to connect. So if git cannot connect to server A, it will not continue with server B/C.

In the past I have server C turn off from time to time, so failing the last push is expected. However recently server A went offline completely and we notice git is not pushing to the remaining 2 remotes.

Not sure if this is intended behavior or can be improved.
John Siu
Jeff King· Jun 1, 2020, 21:40 UTC · re: John Siu · lore

Re: Git multiple remotes push stop at first failed connection

On Sun, May 31, 2020 at 08:28:38PM -0400, John Siu wrote:
Show 9 quoted lines
> Let say my project has following remotes:
> 
> $ git remote -v
> git.all "server A git url" (fetch)
> git.all "server A git url" (push)
> git.all "server B git url" (push)
> git.all "server C git ur" (push)
> 
> When all serverA/B/C are online, "git push" works.

A slight nomenclature nit, but that's _one_ remote that has several push urls.

Show 10 quoted lines
> However "git push" will stop at the first server it failed to connect.
> So if git cannot connect to server A, it will not continue with server
> B/C.
> 
> In the past I have server C turn off from time to time, so failing the
> last push is expected. However recently server A went offline
> completely and we notice git is not pushing to the remaining 2
> remotes.
> 
> Not sure if this is intended behavior or can be improved.

I don't think we've ever documented the error-handling semantics. Looking at the relevant code in builtin/push.c:do_push():

          url_nr = push_url_of_remote(remote, &url);
          if (url_nr) {
                  for (i = 0; i < url_nr; i++) {
                          struct transport *transport =
                                  transport_get(remote, url[i]);
                          if (flags & TRANSPORT_PUSH_OPTIONS)
                                  transport->push_options = push_options;
                          if (push_with_options(transport, push_refspec, flags))
                                  errs++;
                  }
          } else {
                  struct transport *transport =
                          transport_get(remote, NULL);
                  if (flags & TRANSPORT_PUSH_OPTIONS)
                          transport->push_options = push_options;
                  if (push_with_options(transport, push_refspec, flags))
                          errs++;
          }
          return !!errs;

it does seem to try each one and collect the errors. But the underlying transport code is so ready to die() on errors, taking down the whole process, that I suspect it rarely manages to do so. You're probably much better off defining a separate remote for each push destination, then running your own shell loop:

  err=0
  for dst in serverA serverB serverC; do
    git push $dst || err=1
  done
  exit $err

There's really no benefit to doing it all in a single Git process, as we'd connect to each independently, run a separate independent pack-objects for each, etc.

I'd even suggest that Git implement such a loop itself, as we did for "git fetch --all", but sadly "push --all" is already taken for a different meaning (but it might still be worth doing under a different option name).

-Peff
John Siu· Jun 2, 2020, 01:32 UTC · re: Jeff King · lore

Re: Git multiple remotes push stop at first failed connection

Thank Jeff for replying.

Yes, one route we are looking into is use fetch -all, push -all and wrap push tag with a loop, or may do what you suggest and wrap all 3 in a loop for consistency.

On Mon, Jun 1, 2020 at 5:40 PM Jeff King <peff@peff.net> wrote:
Show 72 quoted lines
>
> On Sun, May 31, 2020 at 08:28:38PM -0400, John Siu wrote:
>
> > Let say my project has following remotes:
> >
> > $ git remote -v
> > git.all "server A git url" (fetch)
> > git.all "server A git url" (push)
> > git.all "server B git url" (push)
> > git.all "server C git ur" (push)
> >
> > When all serverA/B/C are online, "git push" works.
>
> A slight nomenclature nit, but that's _one_ remote that has several
> push urls.
>
> > However "git push" will stop at the first server it failed to connect.
> > So if git cannot connect to server A, it will not continue with server
> > B/C.
> >
> > In the past I have server C turn off from time to time, so failing the
> > last push is expected. However recently server A went offline
> > completely and we notice git is not pushing to the remaining 2
> > remotes.
> >
> > Not sure if this is intended behavior or can be improved.
>
> I don't think we've ever documented the error-handling semantics.
> Looking at the relevant code in builtin/push.c:do_push():
>
>           url_nr = push_url_of_remote(remote, &url);
>           if (url_nr) {
>                   for (i = 0; i < url_nr; i++) {
>                           struct transport *transport =
>                                   transport_get(remote, url[i]);
>                           if (flags & TRANSPORT_PUSH_OPTIONS)
>                                   transport->push_options = push_options;
>                           if (push_with_options(transport, push_refspec, flags))
>                                   errs++;
>                   }
>           } else {
>                   struct transport *transport =
>                           transport_get(remote, NULL);
>                   if (flags & TRANSPORT_PUSH_OPTIONS)
>                           transport->push_options = push_options;
>                   if (push_with_options(transport, push_refspec, flags))
>                           errs++;
>           }
>           return !!errs;
>
> it does seem to try each one and collect the errors. But the underlying
> transport code is so ready to die() on errors, taking down the whole
> process, that I suspect it rarely manages to do so. You're probably much
> better off defining a separate remote for each push destination, then
> running your own shell loop:
>
>   err=0
>   for dst in serverA serverB serverC; do
>     git push $dst || err=1
>   done
>   exit $err
>
> There's really no benefit to doing it all in a single Git process, as
> we'd connect to each independently, run a separate independent
> pack-objects for each, etc.
>
> I'd even suggest that Git implement such a loop itself, as we did for
> "git fetch --all", but sadly "push --all" is already taken for a
> different meaning (but it might still be worth doing under a different
> option name).
>
> -Peff
Junio C Hamano· Jun 2, 2020, 16:26 UTC · re: Jeff King · lore

Re: Git multiple remotes push stop at first failed connection

Jeff King <peff@peff.net> writes:
Show 8 quoted lines
> There's really no benefit to doing it all in a single Git process, as
> we'd connect to each independently, run a separate independent
> pack-objects for each, etc.
>
> I'd even suggest that Git implement such a loop itself, as we did for
> "git fetch --all", but sadly "push --all" is already taken for a
> different meaning (but it might still be worth doing under a different
> option name).

I wonder if it is possible to update the implementation to do so without changing the UI at all, though.

The presence of the "--all" option in "fetch" command is tied closely to the fact that it makes no sense to have multiple URLs that are used to download from at the same time under a single remote name (e.g. what should "remotes/origin/master" point at if two URLs say different things if such an arrangement were allowed?).

On the other hand, the pushURL for a single remote can be multiple places for redundancy (a possible #leftoverbits here is that we should probably disable the "pretend that we immediately turned around and fetched from them after pushing" optimization when pushing to a remote that has multiple pushURLs defined) does not need an extra option. If the way we currently push is suboptimal and it is better to spawn a separate "git push" instance via the run_command() API, that can safely be done as a bugfix without affecting any UI elements, no?

John Siu· Jun 2, 2020, 16:54 UTC · re: Junio C Hamano · lore

Re: Git multiple remotes push stop at first failed connection

On Tue, Jun 2, 2020 at 12:26 PM Junio C Hamano <gitster@pobox.com> wrote:
Show 12 quoted lines
>
> Jeff King <peff@peff.net> writes:
>
> > There's really no benefit to doing it all in a single Git process, as
> > we'd connect to each independently, run a separate independent
> > pack-objects for each, etc.
> >
> > I'd even suggest that Git implement such a loop itself, as we did for
> > "git fetch --all", but sadly "push --all" is already taken for a
> > different meaning (but it might still be worth doing under a different
> > option name).
>
Yes. We notice the fetch/push --all is for branches.
Show 19 quoted lines
> I wonder if it is possible to update the implementation to do so
> without changing the UI at all, though.
>
> The presence of the "--all" option in "fetch" command is tied
> closely to the fact that it makes no sense to have multiple URLs
> that are used to download from at the same time under a single
> remote name (e.g. what should "remotes/origin/master" point at if
> two URLs say different things if such an arrangement were allowed?).
>
> On the other hand, the pushURL for a single remote can be multiple
> places for redundancy (a possible #leftoverbits here is that we
> should probably disable the "pretend that we immediately turned
> around and fetched from them after pushing" optimization when
> pushing to a remote that has multiple pushURLs defined) does not
> need an extra option.  If the way we currently push is suboptimal
> and it is better to spawn a separate "git push" instance via the
> run_command() API, that can safely be done as a bugfix without
> affecting any UI elements, no?
>

I agree a "bugfix" for push only is good enough and safe. As the current behavior is already pushing to all pushURLs of a single remote. We are not trying to change behavior or do anything extra.

← back to recent threads