Re: Cannot clone redirecting stdout

From: Jeff King <peff@peff.net>
Date: Sep 11, 2009, 16:05 UTC
Message-ID: <20090911160510.GA10848@coredump.intra.peff.net>
In-Reply-To: <9e0f31700909110846h54959ae6u466ceda40799ba37@mail.gmail.com>

On Fri, Sep 11, 2009 at 05:46:23PM +0200, Johan Sørensen wrote:

Show 13 quoted lines

> Some quick tests seem to indicate it's related to the fact that our
> wonderful little fork+exec git-daemon[1] (which is different from the
> one distributed with git) exec's to "git-upload-pack --strict
> --timeout=30 /path/to/repo". Now, why exactly that'll trigger when the
> no-progress flag is given I'm not sure of. The daemon itself execs as
> soon as it figures out what repo the client requested, so apart from
> the timeout the only thing it reacts to is the header (the
> "${headersize}git-upload-pack project/repo.git\0host=gitorious.org\0"
> part).
> 
> We also do redirect stderr to /dev/null for reasons I cannot remember
> (so probably not good ones), that may be related as well. Well run
> some more tests...

Ah. I have a theory. If I do a clone of git://gitorious.org/qt/qt.git, the counting/compressing stages take a long time (I timed it at 1m40 before it actually sends any data). And looking at upload-pack.c, we leave the 30-second alarm set while creating the pack. Meaning we die 30 seconds into creating the pack.

When progress is being displayed, however, the progress timer actually uses SIGALRM, as well. So we are constantly resetting the timer and it never goes off.

And we should be able to test this theory. How long does it take for the failure case to fail:

  $ time git clone -n git://gitorious.org/qt/qt.git repo >log
  fatal: The remote end hung up unexpectedly
  fatal: early EOF
  fatal: index-pack failed

  real    0m31.106s
  user    0m0.000s
  sys     0m0.012s

Hmm. Suspicious. :)

So that implies to me a few things:

  1. You guys should really pack your repos, as you are wasting over a
     minute of CPU time every time somebody clones this repo.

  2. Upload-pack has what I consider a bug. The --timeout should be
     suspended while we are actually crunching numbers to create the
     pack. We probably don't want it when sending the pack, either, as
     people with slow connections (or big repos) will get timed out
     during the send. Probably we just want to apply it to times when we
     are waiting to get the list of refs from the client.

  3. Upload-pack and the progress code are both using the global alarm
     timer and signal, and that is papering over the bug in (2) when
     progress is enabled. I'm not sure of the simplest way of having
     those interact. But maybe we can just ignore it, because we
     probably don't want to using the --timeout alarm during the packing
     phase, anyway.

-Peff