Re: [PATCH 09/16] documentation: add documentation for the bitmap format
- From
Shawn Pearce <spearce@spearce.org>
- Date
- Jul 1, 2013, 19:13 UTC
- Message-ID
- <CAJo=hJv=rb2S1DqERQ9GJRQkix=Z5pG2jEyghVDcCHPSF7AKDA@mail.gmail.com>
- In-Reply-To
- <CAFFbUKKm89n0HG6xUhYMLs_yjRJ8n0jFtOEEN=vXxJfWKLx5FA@mail.gmail.com>
On Mon, Jul 1, 2013 at 11:47 AM, Colby Ranger <cranger@google.com> wrote:
Show 10 quoted lines
>> But I think we are comparing >> apples to steaks here, Vincent is (rightfully) concerned about process >> startup performance, whereas our timings were assuming the process was >> already running. >> > > I did some timing on loading the reverse index for the kernel and it > is pretty slow (~1200ms). I just submitted a fix to do a bucket sort > and reduced that to ~450ms, which is still slow but much better: > https://eclipse.googlesource.com/jgit/jgit/+/6cc532a43cf28403cb623d3df8600a2542a40a43%5E%21/
A reverse index that is hot in RAM would obviously load in about 0ms. But a cold load of a reverse index that uses only 4 bytes per object (as Colby did here) for 3.1M objects could take ~590ms to read from disk, assuming spinning media moving 20 MiB/s. If 8 byte offsets were also stored this could be more like 1700ms.
Numbers obviously get better if the spinning media can transfer at 40 MiB/s, now its more like 295ms for 4 bytes/object and 885ms for 12 bytes/object.
I think its still reasonable to compute the reverse index on the fly. But JGit certainly does have the benefit of reusing it across requests by relying on process memory based caches. C Git needs to rely on the kernel buffer cache, which requires this data be written out to a file to be shared.