Re: [PATCH] index-pack: avoid excessive re-reading of pack directory
- From
Shawn Pearce <spearce@spearce.org>
- Date
- Jun 10, 2015, 03:46 UTC
- Message-ID
- <CAJo=hJv6O66yFC_O_4aeL2JxBOtk2f+k1wGt3VW7dk71Q1ov3A@mail.gmail.com>
- In-Reply-To
- <20150605122921.GA7586@peff.net>
On Fri, Jun 5, 2015 at 5:29 AM, Jeff King <peff@peff.net> wrote:
Show 14 quoted lines
> > However, some code paths make a large number of > has_sha1_file checks which are _not_ expected to return 1. > The collision test in index-pack.c is such a case. On a > local system, this can cause a performance slowdown of > around 5%. But on a system with high-latency system calls > (like NFS), it can be much worse. > > This patch introduces a "quick" flag to has_sha1_file which > callers can use when they would prefer high performance at > the cost of false negatives during repacks. There may be > other code paths that can use this, but the index-pack one > is the most obviously critical, so we'll start with > switching that one.
Hilarious. We did this in JGit back in ... uhm before 2009. :)
But its Java. So of course we had to do optimizations.
Show 8 quoted lines
> @@ -3169,6 +3169,8 @@ int has_sha1_file(const unsigned char *sha1) > return 1; > if (has_loose_object(sha1)) > return 1; > + if (flags & HAS_SHA1_QUICK) > + return 0; > reprepare_packed_git(); > return find_pack_entry(sha1, &e);
Something else we do is readdir() over the loose objects and store them in a map in memory. That way we avoid stat() calls during that has_loose_object() path. This is apparently a win enough of the time that we always do that when receiving a pack over the wire (client or server).