threads / patch / 26911

patchFix potential local deadlock during fetch-pack

Subject: [PATCH] Fix potential local deadlock during fetch-pack

## tl;dr

6 messages between Mar 29, 2011 and Mar 31, 2011. Diffs are folded; open one to read it.

replies: 5people: 3as markdown or json

Junio C Hamano· Mar 29, 2011, 17:06 UTC · lore

The fetch-pack/upload-pack protocol relies on the underlying transport (local pipe or TCP socket) to have enough slack to allow one window worth of data in flight without blocking the writer. Traditionally we always relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k bytes) to stream.

The recent "progressive-stride" change allows "fetch-pack" to send up to 1024 "have"s without reading any response from "upload-pack". The outgoing pipe of "upload-pack" can be clogged with many ACK and NAK that are unread, while "fetch-pack" is still stuffing its outgoing pike with more "have"s, leading to a deadlock.

Revert the change unless we are in stateless rpc (aka smart-http) mode, as using a large window full of "have"s is still a good way to help reduce the number of back-and-forth, and there is no buffering issue there (it is strictly "ping-pong" without an overlap).

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 builtin/fetch-pack.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

Show changes to builtin/fetch-pack.c +5 −4

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 1724b76..e5fdacd 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -229,16 +229,17 @@ static void insert_alternate_refs(void)
 }
 
 #define INITIAL_FLUSH 16
+#define PIPESAFE_FLUSH 32
 #define LARGE_FLUSH 1024
 
 static int next_flush(int count)
 {
-	if (count < INITIAL_FLUSH * 2)
-		count += INITIAL_FLUSH;
-	else if (count < LARGE_FLUSH)
+	int flush_limit = args.stateless_rpc ? LARGE_FLUSH : PIPESAFE_FLUSH;
+
+	if (count < flush_limit)
 		count <<= 1;
 	else
-		count += LARGE_FLUSH;
+		count += flush_limit;
 	return count;
 }

Shawn Pearce· Mar 29, 2011, 17:22 UTC · re: Junio C Hamano · lore

Re: [PATCH] Fix potential local deadlock during fetch-pack

On Tue, Mar 29, 2011 at 10:06, Junio C Hamano <gitster@pobox.com> wrote:

> The fetch-pack/upload-pack protocol relies on the underlying transport
> (local pipe or TCP socket) to have enough slack to allow one window worth
> of data in flight without blocking the writer.  Traditionally we always
> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k

Its 64. Because the client "races ahead" one window before ever reading. Which is closer to 3K of data in flight.

> The recent "progressive-stride" change allows "fetch-pack" to send up to
> 1024 "have"s without reading any response from "upload-pack".  The
> outgoing pipe of "upload-pack" can be clogged with many ACK and NAK that
> are unread, while "fetch-pack" is still stuffing its outgoing pike with

s/pike/pipe/

Show 19 quoted lines

> @@ -229,16 +229,17 @@ static void insert_alternate_refs(void)
>  }
>
>  #define INITIAL_FLUSH 16
> +#define PIPESAFE_FLUSH 32
>  #define LARGE_FLUSH 1024
>
>  static int next_flush(int count)
>  {
> -       if (count < INITIAL_FLUSH * 2)
> -               count += INITIAL_FLUSH;
> -       else if (count < LARGE_FLUSH)
> +       int flush_limit = args.stateless_rpc ? LARGE_FLUSH : PIPESAFE_FLUSH;
> +
> +       if (count < flush_limit)
>                count <<= 1;
>        else
> -               count += LARGE_FLUSH;
> +               count += flush_limit;

Nak. You still deadlock because when count reaches PIPESAFE_FLUSH you still double it to 2*PIPESAFE_FLUSH here. Instead I think you mean:

  if (args.stateless_rpc) {
    if (count < LARGE_FLUSH)
      count <<= 1;
    else
      count += LARGE_FLUSH;
  } else {
    if (count * 2 < PIPESAFE_FLUSH)
      count <<= 1;
  }

-- 
Shawn.

Shawn Pearce· Mar 29, 2011, 17:30 UTC · re: Shawn Pearce · lore

Re: [PATCH] Fix potential local deadlock during fetch-pack

On Tue, Mar 29, 2011 at 10:22, Shawn Pearce <spearce@spearce.org> wrote:

Show 10 quoted lines

> On Tue, Mar 29, 2011 at 10:06, Junio C Hamano <gitster@pobox.com> wrote:
>> The fetch-pack/upload-pack protocol relies on the underlying transport
>> (local pipe or TCP socket) to have enough slack to allow one window worth
>> of data in flight without blocking the writer.  Traditionally we always
>> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
>
>> +               count += flush_limit;
>
> Nak. You still deadlock because when count reaches PIPESAFE_FLUSH you
> still double it to 2*PIPESAFE_FLUSH here. Instead I think you mean:

I take this comment back. Re-reading fetch-pack.c the next_flush() method is accepting as input a running counter of how many have lines have already been sent to the remote peer, and is never reset to 0. Therefore it is necessary to add the next round size to count and return it.

-- 
Shawn.

Erik Faye-Lund· Mar 30, 2011, 09:42 UTC · re: Junio C Hamano · lore

Re: [PATCH] Fix potential local deadlock during fetch-pack

On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 6 quoted lines

> The fetch-pack/upload-pack protocol relies on the underlying transport
> (local pipe or TCP socket) to have enough slack to allow one window worth
> of data in flight without blocking the writer.  Traditionally we always
> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
> bytes) to stream.
>

Hmm, this explanation makes me wonder: Could this be related to the deadlock we're experiencing with git-push over the git-protocol on Windows when side-band-64k is enabled?

Erik Faye-Lund· Mar 30, 2011, 09:50 UTC · re: Erik Faye-Lund · lore

Re: [PATCH] Fix potential local deadlock during fetch-pack

On Wed, Mar 30, 2011 at 11:42 AM, Erik Faye-Lund <kusmabite@gmail.com> wrote:

Show 12 quoted lines

> On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> The fetch-pack/upload-pack protocol relies on the underlying transport
>> (local pipe or TCP socket) to have enough slack to allow one window worth
>> of data in flight without blocking the writer.  Traditionally we always
>> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
>> bytes) to stream.
>>
>
> Hmm, this explanation makes me wonder: Could this be related to the
> deadlock we're experiencing with git-push over the git-protocol on
> Windows when side-band-64k is enabled?
>

No, It doesn't seem like that's it. The socket buffers appears to be 8k by default on Windows, which should be plenty, right?

---8<---

Show changes to compat/mingw.c +8 −1

diff --git a/compat/mingw.c b/compat/mingw.c
index d527562..985c271 100644
--- a/compat/mingw.c
+++ b/compat/mingw.c
@@ -1404,7 +1404,7 @@ int mingw_getnameinfo(const struct sockaddr *sa,
socklen_t salen,

 int mingw_socket(int domain, int type, int protocol)
 {
-	int sockfd;
+	int sockfd, val, len;
 	SOCKET s;

 	ensure_socket_initialization();
@@ -1428,6 +1428,12 @@ int mingw_socket(int domain, int type, int protocol)
 		return error("unable to make a socket file descriptor: %s",
 			strerror(errno));
 	}
+
+	len = sizeof(val);
+	if (!getsockopt(s, SOL_SOCKET, SO_RCVBUF, (char *)&val, &len))
+		fprintf(stderr, "SO_RCVBUF: %d\n", val);
+	len = sizeof(val);
+	if (!getsockopt(s, SOL_SOCKET, SO_SNDBUF, (char *)&val, &len))
+		fprintf(stderr, "SO_SNDBUF: %d\n", val);
 	return sockfd;
 }

---8<---

$ git push git://localhost
SO_RCVBUF: 8192
SO_SNDBUF: 8192
...

Shawn Pearce· Mar 31, 2011, 16:24 UTC · re: Erik Faye-Lund · lore

Re: [PATCH] Fix potential local deadlock during fetch-pack

On Wed, Mar 30, 2011 at 02:42, Erik Faye-Lund <kusmabite@gmail.com> wrote:

Show 10 quoted lines

> On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> The fetch-pack/upload-pack protocol relies on the underlying transport
>> (local pipe or TCP socket) to have enough slack to allow one window worth
>> of data in flight without blocking the writer.  Traditionally we always
>> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
>> bytes) to stream.
>
> Hmm, this explanation makes me wonder: Could this be related to the
> deadlock we're experiencing with git-push over the git-protocol on
> Windows when side-band-64k is enabled?

I think its unrelated. It might also be a deadlock, but the push protocol is quite a bit different. As far as I remember, there is no risk of deadlock in the push protocol.

-- 
Shawn.