# [PATCH] Fix potential local deadlock during fetch-pack 6 messages from 2011-03-29 to 2011-03-31. Participants: Junio C Hamano, Shawn Pearce, Erik Faye-Lund. Thread: https://gitlist.dev/t/26911 ## Junio C Hamano, 2011-03-29 17:06 Subject: [PATCH] Fix potential local deadlock during fetch-pack Message-ID: <7vtyelsvp0.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vtyelsvp0.fsf%40alter.siamese.dyndns.org ``` The fetch-pack/upload-pack protocol relies on the underlying transport (local pipe or TCP socket) to have enough slack to allow one window worth of data in flight without blocking the writer. Traditionally we always relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k bytes) to stream. The recent "progressive-stride" change allows "fetch-pack" to send up to 1024 "have"s without reading any response from "upload-pack". The outgoing pipe of "upload-pack" can be clogged with many ACK and NAK that are unread, while "fetch-pack" is still stuffing its outgoing pike with more "have"s, leading to a deadlock. Revert the change unless we are in stateless rpc (aka smart-http) mode, as using a large window full of "have"s is still a good way to help reduce the number of back-and-forth, and there is no buffering issue there (it is strictly "ping-pong" without an overlap). Signed-off-by: Junio C Hamano --- builtin/fetch-pack.c | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c index 1724b76..e5fdacd 100644 --- a/builtin/fetch-pack.c +++ b/builtin/fetch-pack.c @@ -229,16 +229,17 @@ static void insert_alternate_refs(void) } #define INITIAL_FLUSH 16 +#define PIPESAFE_FLUSH 32 #define LARGE_FLUSH 1024 static int next_flush(int count) { - if (count < INITIAL_FLUSH * 2) - count += INITIAL_FLUSH; - else if (count < LARGE_FLUSH) + int flush_limit = args.stateless_rpc ? LARGE_FLUSH : PIPESAFE_FLUSH; + + if (count < flush_limit) count <<= 1; else - count += LARGE_FLUSH; + count += flush_limit; return count; } ``` ## Shawn Pearce, 2011-03-29 17:22 Subject: Re: [PATCH] Fix potential local deadlock during fetch-pack Message-ID: URL: https://gitlist.dev/e/AANLkTikZ%3DPOeQi5nnt8D9v4Z-_OaXNrXDjP2awWUtKzA%40mail.gmail.com In-Reply-To: <7vtyelsvp0.fsf@alter.siamese.dyndns.org> ``` On Tue, Mar 29, 2011 at 10:06, Junio C Hamano wrote: > The fetch-pack/upload-pack protocol relies on the underlying transport > (local pipe or TCP socket) to have enough slack to allow one window worth > of data in flight without blocking the writer.  Traditionally we always > relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k Its 64. Because the client "races ahead" one window before ever reading. Which is closer to 3K of data in flight. > The recent "progressive-stride" change allows "fetch-pack" to send up to > 1024 "have"s without reading any response from "upload-pack".  The > outgoing pipe of "upload-pack" can be clogged with many ACK and NAK that > are unread, while "fetch-pack" is still stuffing its outgoing pike with s/pike/pipe/ > @@ -229,16 +229,17 @@ static void insert_alternate_refs(void) >  } > >  #define INITIAL_FLUSH 16 > +#define PIPESAFE_FLUSH 32 >  #define LARGE_FLUSH 1024 > >  static int next_flush(int count) >  { > -       if (count < INITIAL_FLUSH * 2) > -               count += INITIAL_FLUSH; > -       else if (count < LARGE_FLUSH) > +       int flush_limit = args.stateless_rpc ? LARGE_FLUSH : PIPESAFE_FLUSH; > + > +       if (count < flush_limit) >                count <<= 1; >        else > -               count += LARGE_FLUSH; > +               count += flush_limit; Nak. You still deadlock because when count reaches PIPESAFE_FLUSH you still double it to 2*PIPESAFE_FLUSH here. Instead I think you mean: if (args.stateless_rpc) { if (count < LARGE_FLUSH) count <<= 1; else count += LARGE_FLUSH; } else { if (count * 2 < PIPESAFE_FLUSH) count <<= 1; } -- Shawn. ``` ## Shawn Pearce, 2011-03-29 17:30 Subject: Re: [PATCH] Fix potential local deadlock during fetch-pack Message-ID: URL: https://gitlist.dev/e/AANLkTi%3DUDPFkox9Gsgv5Bbfsb-VTRweY9D2SNAAhptAc%40mail.gmail.com In-Reply-To: ``` On Tue, Mar 29, 2011 at 10:22, Shawn Pearce wrote: > On Tue, Mar 29, 2011 at 10:06, Junio C Hamano wrote: >> The fetch-pack/upload-pack protocol relies on the underlying transport >> (local pipe or TCP socket) to have enough slack to allow one window worth >> of data in flight without blocking the writer.  Traditionally we always >> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k > >> +               count += flush_limit; > > Nak. You still deadlock because when count reaches PIPESAFE_FLUSH you > still double it to 2*PIPESAFE_FLUSH here. Instead I think you mean: I take this comment back. Re-reading fetch-pack.c the next_flush() method is accepting as input a running counter of how many have lines have already been sent to the remote peer, and is never reset to 0. Therefore it is necessary to add the next round size to count and return it. -- Shawn. ``` ## Erik Faye-Lund, 2011-03-30 09:42 Subject: Re: [PATCH] Fix potential local deadlock during fetch-pack Message-ID: URL: https://gitlist.dev/e/AANLkTikCRRftjExdhMgFGFMy-Dc1UCMdW1nTYV2bK1-T%40mail.gmail.com In-Reply-To: <7vtyelsvp0.fsf@alter.siamese.dyndns.org> ``` On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano wrote: > The fetch-pack/upload-pack protocol relies on the underlying transport > (local pipe or TCP socket) to have enough slack to allow one window worth > of data in flight without blocking the writer.  Traditionally we always > relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k > bytes) to stream. > Hmm, this explanation makes me wonder: Could this be related to the deadlock we're experiencing with git-push over the git-protocol on Windows when side-band-64k is enabled? ``` ## Erik Faye-Lund, 2011-03-30 09:50 Subject: Re: [PATCH] Fix potential local deadlock during fetch-pack Message-ID: URL: https://gitlist.dev/e/AANLkTinyok%2BUnR8pOpp43v5EHEN8_YDCC0MFcBfC1LRY%40mail.gmail.com In-Reply-To: ``` On Wed, Mar 30, 2011 at 11:42 AM, Erik Faye-Lund wrote: > On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano wrote: >> The fetch-pack/upload-pack protocol relies on the underlying transport >> (local pipe or TCP socket) to have enough slack to allow one window worth >> of data in flight without blocking the writer.  Traditionally we always >> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k >> bytes) to stream. >> > > Hmm, this explanation makes me wonder: Could this be related to the > deadlock we're experiencing with git-push over the git-protocol on > Windows when side-band-64k is enabled? > No, It doesn't seem like that's it. The socket buffers appears to be 8k by default on Windows, which should be plenty, right? ---8<--- diff --git a/compat/mingw.c b/compat/mingw.c index d527562..985c271 100644 --- a/compat/mingw.c +++ b/compat/mingw.c @@ -1404,7 +1404,7 @@ int mingw_getnameinfo(const struct sockaddr *sa, socklen_t salen, int mingw_socket(int domain, int type, int protocol) { - int sockfd; + int sockfd, val, len; SOCKET s; ensure_socket_initialization(); @@ -1428,6 +1428,12 @@ int mingw_socket(int domain, int type, int protocol) return error("unable to make a socket file descriptor: %s", strerror(errno)); } + + len = sizeof(val); + if (!getsockopt(s, SOL_SOCKET, SO_RCVBUF, (char *)&val, &len)) + fprintf(stderr, "SO_RCVBUF: %d\n", val); + len = sizeof(val); + if (!getsockopt(s, SOL_SOCKET, SO_SNDBUF, (char *)&val, &len)) + fprintf(stderr, "SO_SNDBUF: %d\n", val); return sockfd; } ---8<--- $ git push git://localhost SO_RCVBUF: 8192 SO_SNDBUF: 8192 ... ``` ## Shawn Pearce, 2011-03-31 16:24 Subject: Re: [PATCH] Fix potential local deadlock during fetch-pack Message-ID: URL: https://gitlist.dev/e/AANLkTiki2__qAqgDnB6yu8kzxNW-bMj2qJcx50yawCWM%40mail.gmail.com In-Reply-To: ``` On Wed, Mar 30, 2011 at 02:42, Erik Faye-Lund wrote: > On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano wrote: >> The fetch-pack/upload-pack protocol relies on the underlying transport >> (local pipe or TCP socket) to have enough slack to allow one window worth >> of data in flight without blocking the writer.  Traditionally we always >> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k >> bytes) to stream. > > Hmm, this explanation makes me wonder: Could this be related to the > deadlock we're experiencing with git-push over the git-protocol on > Windows when side-band-64k is enabled? I think its unrelated. It might also be a deadlock, but the push protocol is quite a bit different. As far as I remember, there is no risk of deadlock in the push protocol. -- Shawn. ```