{"thread":{"id":"1950","subject":"[PATCH 0/3] http-fetch enhancements","startedAt":"2005-09-26T17:51:49Z","lastAt":"2005-09-28T03:37:11Z","messageCount":4,"participants":["Nick Hengeveld","Daniel Barkalow"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"9307","messageId":"20050926175148.GA9410@reactrix.com","threadId":"1950","inReplyTo":null,"subject":"[PATCH 0/3] http-fetch enhancements","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-09-26T17:51:49Z","receivedAt":"2005-09-26T17:51:49Z","isPatch":true,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"The following series contains some http-fetch enhancements, based on\nour requirements for use of SSL client certificates and partial HTTP\ntransfers.\n\n-- \nFor a successful technology, reality must take precedence over public\nrelations, for nature cannot be fooled.\n"},{"id":"9341","messageId":"Pine.LNX.4.63.0509261823590.23242@iabervon.org","threadId":"1950","inReplyTo":"20050926175148.GA9410@reactrix.com","subject":"Re: [PATCH 0/3] http-fetch enhancements","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-09-26T22:29:02Z","receivedAt":"2005-09-26T22:29:02Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 26 Sep 2005, Nick Hengeveld wrote:\n\n> The following series contains some http-fetch enhancements, based on\n> our requirements for use of SSL client certificates and partial HTTP\n> transfers.\n\nIf you happen to know how to have curl do multiple simultaneous downloads, \nthat would be a big performance win, and I should be able to explain how \nto get this to work. I haven't gotten around to learning libcurl well \nenough to do the flow control.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"9413","messageId":"20050928023947.GA385@reactrix.com","threadId":"1950","inReplyTo":"Pine.LNX.4.63.0509261823590.23242@iabervon.org","subject":"Re: [PATCH 0/3] http-fetch enhancements","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-09-28T02:39:47Z","receivedAt":"2005-09-28T02:39:47Z","isPatch":true,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"On Mon, Sep 26, 2005 at 03:29:02PM -0700, Daniel Barkalow wrote:\n\n>    If you happen to know how to have curl do multiple simultaneous downloads,\n>    that would be a big performance win, and I should be able to explain how\n>    to get this to work. I haven't gotten around to learning libcurl well\n>    enough to do the flow control.\n\nThe curl multi interface looks pretty straightforward.  What did you have\nin mind as far as which requests would be running concurrently and how they\nwould need to be limited?\n\n-- \nFor a successful technology, reality must take precedence over public\nrelations, for nature cannot be fooled.\n"},{"id":"9415","messageId":"Pine.LNX.4.63.0509272258230.23242@iabervon.org","threadId":"1950","inReplyTo":"20050928023947.GA385@reactrix.com","subject":"Re: [PATCH 0/3] http-fetch enhancements","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-09-28T03:37:11Z","receivedAt":"2005-09-28T03:37:11Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 27 Sep 2005, Nick Hengeveld wrote:\n\n> On Mon, Sep 26, 2005 at 03:29:02PM -0700, Daniel Barkalow wrote:\n> \n> >    If you happen to know how to have curl do multiple simultaneous downloads,\n> >    that would be a big performance win, and I should be able to explain how\n> >    to get this to work. I haven't gotten around to learning libcurl well\n> >    enough to do the flow control.\n> \n> The curl multi interface looks pretty straightforward.  What did you have\n> in mind as far as which requests would be running concurrently and how they\n> would need to be limited?\n\nThe way fetch.c calls the functions, there's a prefetch() that indicates \nthat a given object is needed, and a fetch() that is responsible for \nmaking sure the object is available when it returns (or returning an \nerror). It is arranged such that the same list of hashes is given to each \nof the functions in the same order.\n\nOne method is to send requests in prefetch() and accept responses in \nfetch(); this is what git-ssh-pull now does. This seems in practice to \nlead to ~100 outstanding requests at the high point, which is great for \nthroughput, but I'm not sure how polite it is. IIRC, browsers tend to do \n~4 simultaneous connections, or at least used to.\n\nThe other method is to keep track of what you're fetching, and block in \nprefetch() if too many connections are in use until some connections are \nfree, or in fetch() if that is called before the download is complete.\n\nNote that it should theoretically be possible to make additional requests \non the same connection, provided it's \"keep alive\", even before reading \nthe response, so long as the code is able to figure out what happened if \nthe server actually closes it (after the first request) instead of serving \nthe later ones.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"}]}