From: Shawn Pearce <spearce@spearce.org>
Date: Wed, 10 Jun 2015 03:46:24 GMT
Subject: Re: [PATCH] index-pack: avoid excessive re-reading of pack directory
Message-ID: <CAJo=hJv6O66yFC_O_4aeL2JxBOtk2f+k1wGt3VW7dk71Q1ov3A@mail.gmail.com>
In-Reply-To: <20150605122921.GA7586@peff.net>

On Fri, Jun 5, 2015 at 5:29 AM, Jeff King <peff@peff.net> wrote:
>
> However, some code paths make a large number of
> has_sha1_file checks which are _not_ expected to return 1.
> The collision test in index-pack.c is such a case. On a
> local system, this can cause a performance slowdown of
> around 5%. But on a system with high-latency system calls
> (like NFS), it can be much worse.
>
> This patch introduces a "quick" flag to has_sha1_file which
> callers can use when they would prefer high performance at
> the cost of false negatives during repacks. There may be
> other code paths that can use this, but the index-pack one
> is the most obviously critical, so we'll start with
> switching that one.

Hilarious. We did this in JGit back in ... uhm before 2009. :)

But its Java. So of course we had to do optimizations.


> @@ -3169,6 +3169,8 @@ int has_sha1_file(const unsigned char *sha1)
>                 return 1;
>         if (has_loose_object(sha1))
>                 return 1;
> +       if (flags & HAS_SHA1_QUICK)
> +               return 0;
>         reprepare_packed_git();
>         return find_pack_entry(sha1, &e);

Something else we do is readdir() over the loose objects and store
them in a map in memory. That way we avoid stat() calls during that
has_loose_object() path. This is apparently a win enough of the time
that we always do that when receiving a pack over the wire (client or
server).