# [PATCH] Limit the number of requests outstanding in ssh-fetch. 3 messages from 2005-10-04 to 2005-10-04. Participants: Daniel Barkalow, Dan Aloni, Stefan Roese. Thread: https://gitlist.dev/t/2008 ## Daniel Barkalow, 2005-10-04 04:24 Subject: [PATCH] Limit the number of requests outstanding in ssh-fetch. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0510040016110.23242%40iabervon.org ``` This completes fetches if there are more than 100 outstanding requests and there are more to prefetch. Signed-off-by: Daniel Barkalow --- Lightly tested; I reduced the limit to 5, and pulled a small tree successfully with some of the requests being completed early. I didn't have the test case to verify that a limit of 100 is sufficiently low, but handwaving suggests that it should be. ssh-fetch.c | 28 +++++++++++++++++++++++++++- 1 files changed, 27 insertions(+), 1 deletions(-) 528f784890df88a5c054913136bbc53b164ebb94 diff --git a/ssh-fetch.c b/ssh-fetch.c --- a/ssh-fetch.c +++ b/ssh-fetch.c @@ -36,12 +36,26 @@ static ssize_t force_write(int fd, void return ret; } +static int prefetches = 0; + +static struct object_list *in_transit = NULL; +static struct object_list **end_of_transit = &in_transit; + void prefetch(unsigned char *sha1) { char type = 'o'; + struct object_list *node; + if (prefetches > 100) { + fetch(in_transit->item->sha1); + } + node = xmalloc(sizeof(struct object_list)); + node->next = NULL; + node->item = lookup_unknown_object(sha1); + *end_of_transit = node; + end_of_transit = &node->next; force_write(fd_out, &type, 1); force_write(fd_out, sha1, 20); - //memcpy(requested + 20 * prefetches++, sha1, 20); + prefetches++; } static char conn_buf[4096]; @@ -51,6 +65,18 @@ int fetch(unsigned char *sha1) { int ret; signed char remote; + struct object_list *temp; + + if (memcmp(sha1, in_transit->item->sha1, 20)) { + // we must have already fetched it to clean the queue + return has_sha1_file(sha1) ? 0 : -1; + } + prefetches--; + temp = in_transit; + in_transit = in_transit->next; + if (!in_transit) + end_of_transit = &in_transit; + free(temp); if (conn_buf_posn) { remote = conn_buf[0]; ``` ## Dan Aloni, 2005-10-04 07:16 Subject: Re: [PATCH] Limit the number of requests outstanding in ssh-fetch. Message-ID: <20051004071638.GA23725@localdomain> URL: https://gitlist.dev/e/20051004071638.GA23725%40localdomain In-Reply-To: ``` On Tue, Oct 04, 2005 at 12:24:55AM -0400, Daniel Barkalow wrote: > This completes fetches if there are more than 100 outstanding requests > and there are more to prefetch. > > Signed-off-by: Daniel Barkalow > --- > > Lightly tested; I reduced the limit to 5, and pulled a small tree > successfully with some of the requests being completed early. I didn't > have the test case to verify that a limit of 100 is sufficiently low, but > handwaving suggests that it should be. Good, it seems that your patch works alright with the local Linux complete database fetch. -- Dan Aloni da-x@monatomic.org, da-x@colinux.org, da-x@gmx.net ``` ## Stefan Roese, 2005-10-04 07:46 Subject: Re: [PATCH] Limit the number of requests outstanding in ssh-fetch. Message-ID: <200510040946.49509.sr@denx.de> URL: https://gitlist.dev/e/200510040946.49509.sr%40denx.de In-Reply-To: <20051004071638.GA23725@localdomain> ``` On Tuesday 04 October 2005 09:16, Dan Aloni wrote: > > Lightly tested; I reduced the limit to 5, and pulled a small tree > > successfully with some of the requests being completed early. I didn't > > have the test case to verify that a limit of 100 is sufficiently low, but > > handwaving suggests that it should be. > > Good, it seems that your patch works alright with the local Linux complete > database fetch. Works for me too (on a big ssh fetch, which stalled always without this patch). Best regards, Stefan ```