threads / patch / 4915

patchupload-pack: fix timeout in create_pack_file

Subject: [PATCH] upload-pack: fix timeout in create_pack_file

## tl;dr

4 messages between Jul 18, 2006 and Jul 24, 2006. Diffs are folded; open one to read it.

replies: 3people: 2as markdown or json

Matthias Lederhofer· Jul 18, 2006, 17:14 UTC · lore

Signed-off-by: Matthias Lederhofer <matled@gmx.net>
---
<yacc> fatal: packfile '../linux-2.6/.git/objects/pack/tmp-7iPJo5'
       SHA1 mismatch
<yacc> error: git-fetch-pack: unable to read from git-index-pack
<yacc> error: git-index-pack died with error code 128
<yacc> Any idea what this means?
This happens after ~12 minutes.  The problem is that the loop in
upload-pack.c actually sending the pack does not reset the timeout.
I'd guess --timeout is 600 or a bit more on git.kernel.org :)

This does not help for low timeouts with slow clients. If a client is slow enough so the server is blocked for more time than specified by timeout the connection will be closed too (e.g. 15kb/s with a timeout of 30 (git adds 10 extra) is not enough). We should either add a warning to the man page or try to fix this. I don't know if this can be fixed not using non-blocking sockets.

Perhaps support for resume would be quite useful too but I've no idea how hard this is to implement.

A workaround for is to pull a part of the repository first and the
rest later.  For example using this:
$ git init-db
$ git fetch URL refs/tags/old-tag:refs/tags/old-tag
$ git fetch URL refs/tags/newer-tag:refs/tags/newer-tag
..
---
 upload-pack.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

Show changes to upload-pack.c +2 −0

diff --git a/upload-pack.c b/upload-pack.c
index f6f5a7e..07ecdb4 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -182,6 +182,8 @@ static void create_pack_file(void)
 		ssize_t sz;
 		int pe, pu, pollsize;
 
+		reset_timeout();
+
 		pollsize = 0;
 		pe = pu = -1;

-- 
1.4.2.rc1.ge7a0

Junio C Hamano· Jul 24, 2006, 06:23 UTC · re: Matthias Lederhofer · lore

Re: [PATCH] upload-pack: fix timeout in create_pack_file

Matthias Lederhofer <matled@gmx.net> writes:

Show 15 quoted lines

> Signed-off-by: Matthias Lederhofer <matled@gmx.net>
> ---
> <yacc> fatal: packfile '../linux-2.6/.git/objects/pack/tmp-7iPJo5'
>        SHA1 mismatch
> <yacc> error: git-fetch-pack: unable to read from git-index-pack
> <yacc> error: git-index-pack died with error code 128
> <yacc> Any idea what this means?
> This happens after ~12 minutes.  The problem is that the loop in
> upload-pack.c actually sending the pack does not reset the timeout.
> I'd guess --timeout is 600 or a bit more on git.kernel.org :)
>
> This does not help for low timeouts with slow clients.  If a client is
> slow enough so the server is blocked for more time than specified by
> timeout the connection will be closed too (e.g. 15kb/s with a timeout
> of 30 (git adds 10 extra) is not enough).

Where do we add 10 extra?

> We should either add a
> warning to the man page or try to fix this.  I don't know if this can
> be fixed not using non-blocking sockets.

I think the intent of "timeout" was to protect us from funny clients by avoiding talking with the ones that take too much time doing something that should not take too long. When we are in create_pack_file(), we are already committed to the heaviest operation anyway, so one possibility might be to stop doing timeout at that point. I am not sure if that is acceptable, though -- it opens up the daemon to even easier DoS than it currently is.

My gut feeling is that your patch would be fine as is (have you tried and confirmed that it helps cases other than slow clients?)

> Perhaps support for resume would be quite useful too but I've no idea
> how hard this is to implement.

That would be _very_ hard.

Matthias Lederhofer· Jul 24, 2006, 07:10 UTC · re: Junio C Hamano · lore

Re: [PATCH] upload-pack: fix timeout in create_pack_file

Junio C Hamano <junkio@cox.net> wrote:

Show 6 quoted lines

> > This does not help for low timeouts with slow clients.  If a client is
> > slow enough so the server is blocked for more time than specified by
> > timeout the connection will be closed too (e.g. 15kb/s with a timeout
> > of 30 (git adds 10 extra) is not enough).
> 
> Where do we add 10 extra?

Oops, never mind, I misread the source.

Show 16 quoted lines

> > We should either add a
> > warning to the man page or try to fix this.  I don't know if this can
> > be fixed not using non-blocking sockets.
> 
> I think the intent of "timeout" was to protect us from funny
> clients by avoiding talking with the ones that take too much
> time doing something that should not take too long.  When we are
> in create_pack_file(), we are already committed to the heaviest
> operation anyway, so one possibility might be to stop doing
> timeout at that point.  I am not sure if that is acceptable,
> though -- it opens up the daemon to even easier DoS than it
> currently is.
> 
> My gut feeling is that your patch would be fine as is (have you
> tried and confirmed that it helps cases other than slow
> clients?)

What else? The problem solved by this patch is the missing reset_timeout while downloading pack files and this affects all clients that are too slow (cannot download the packfile within --timeout seconds). I tried a server with --timeout=600 and a client that cannot download the whole pack file in 10 minutes. Without the patch the server closes the connection, with this patch it works. Additionally I tried it with a client reading so slow that the server was blocked for more than --timeout seconds, then the server closes the connection too (this is more or less the DoS case).

> > Perhaps support for resume would be quite useful too but I've no idea
> > how hard this is to implement.
> 
> That would be _very_ hard.

I don't know how much demand there is for resuming but there is always rsync (at least on kernel.org and other repositories that need it can install it too) to get the new objects with support for resuming.

Junio C Hamano· Jul 24, 2006, 07:55 UTC · re: Matthias Lederhofer · lore

Re: [PATCH] upload-pack: fix timeout in create_pack_file

Matthias Lederhofer <matled@gmx.net> writes:

Show 11 quoted lines

> Junio C Hamano <junkio@cox.net> wrote:
>
>> My gut feeling is that your patch would be fine as is (have you
>> tried and confirmed that it helps cases other than slow
>> clients?)
> ... I tried a server with --timeout=600 and a client
> that cannot download the whole pack file in 10 minutes.  Without the
> patch the server closes the connection, with this patch it works.
> Additionally I tried it with a client reading so slow that the server
> was blocked for more than --timeout seconds, then the server closes
> the connection too (this is more or less the DoS case).

Thanks, that's all I wanted to know.  Let's have it.