Re: reftable: new ref storage format

From: Shawn Pearce <spearce@spearce.org>
Date: Jul 16, 2017, 21:31 UTC
Message-ID: <CAJo=hJue2MYcE-xc9qjRXfMCf5f5+QPzmZQwHtqgr_zALeQziA@mail.gmail.com>
In-Reply-To: <CAD0k6qSpNTWkn-97nQQ1DJrh=sd3dppTXytfbafqj-eVsWDTFg@mail.gmail.com>

On Sun, Jul 16, 2017 at 2:13 PM, Dave Borowitz <dborowitz@google.com> wrote:

Show 18 quoted lines

> On Sun, Jul 16, 2017 at 3:43 PM, Shawn Pearce <spearce@spearce.org> wrote:
>> True... but... in my "android" example repository we have 866,456 live
>> refs. A block size of 64k needs only 443 blocks, and a 12k index, to
>> get the file to compress to 28M (vs. 62M packed-refs).
>>
>> Index records are averaging 28 bytes per block. That gives us room for
>> about 1955 blocks, or 4,574,700 refs before the index block exceeds
>> 64k.
>
> That's only a 5x increase over the current number of refs in this
> android repo. I would not be so sure this repo doesn't grow another 5x
> in the next few years. Especially as the other optimizations for
> working with large repos start to be applied, so it won't be
> prohibitively painful to work with such a repo.
>
> Are we ok with increasing the block size when this eventually happens?
> (At least I think that's what we would have to do, I haven't been
> following closely the discussion on scaling limits.)

I think I'd try letting the index grow to 4 blocks (256k) before I considered increasing the block size. Remember pack idx files are much larger, and are loaded wholesale into memory by JGit. A ref idx at 256k might not be problematic.