Re: Slow git pushes: sitting 1 minute in pack-objects

From: Stephen Morton <stephen.c.morton@gmail.com>
Date: Mar 19, 2015, 20:31 UTC
Message-ID: <CAH8BJxErCMk1JQMMKm9_j4bckB7bEVB+r28_zYovE=pe1nUa_Q@mail.gmail.com>
In-Reply-To: <20150316221532.GA16679@peff.net>

On Mon, Mar 16, 2015 at 6:15 PM, Jeff King <peff@peff.net> wrote:

Show 19 quoted lines

> On Mon, Mar 09, 2015 at 09:37:25PM -0400, Stephen Morton wrote:
>
>> 3. Not sure how long this part takes. It takes 1/3 - 1/2 of the time
>> when straced, but I think it's much less, as little as 10s when not
>> straced.
>> It then reads a bunch of what look like objects from filehandle 0
>> (presumably stdin, read from the remote end?)
>> It then tries to lstat() these filenames in .git/<sha1>
>> ./git/refs/<sha1>, .git/heads/<sha>, etc. It always fails ENOENT.
>> It fails some 120,000 times. This could be a problem. Though I haven't
>> checked to see if this happens on a fast push on another machine.
>
> Hmm. The "push" process must feed the set of object boundaries to
> "pack-objects" so it knows what to pack (i.e., what we want to send, and
> what the other side has).
>
> 120,000 is an awfully large number of objects to be pass there, though.
> Does the repo you are pushing to by any chance have an extremely large
> number of refs (e.g., on the order of 120K tags)?

No. There are on the order of 9,500 refs (mostly tags) but nowhere near 120k.

>
> Can you try building git with this patch which would tell us more about
> where those objects are coming from?

    <patch snipped>

> Those are all rather blunt debugging methods, but hopefully it can get
> us a sense of where the time is going.
>
> -Peff

Thanks Peff,

I haven't tried your patch, but I tried to backtrack a bit and double-check that the problem always happened in a fresh clone with no other modifications.

It did _not_ happen in a new clone --a push took just 5s -- and I think the culprit could have been "repack.writebitmaps=true". Although I had thought writebitmaps was not originally enabled, I now suspect that it was. Let me follow up on that first, before I recompile git with your changes.

Thanks again, I really appreciate the help.

Steve