Re: reftable [v6]: new ref storage format

From: Shawn Pearce <spearce@spearce.org>
Date: Aug 8, 2017, 22:27 UTC
Message-ID: <CAJo=hJsyoFeCQbeJ=2XCRcE1U0zYaRr8VvzXHwkPwisdfUm71Q@mail.gmail.com>
In-Reply-To: <xmqqtw1hc28z.fsf@gitster.mtv.corp.google.com>

On Tue, Aug 8, 2017 at 12:01 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 17 quoted lines

> I notice that you raised the location of restart table within a
> block in this iteration (or maybe it happened in v5).
>
> This forces you to hold all contents in core before the first byte
> is written out.  You start from the first entry (which will become
> the first restart entry), emit a handful as prefix compressed
> entries, emit a full entry (which will become the next restart
> entry), ... until you have enough to fill both the data and the
> restart table, then start writing from the header (which needs the
> length of the block), restart table and then data.
>
> I think it is OK to do so for the blocks whose size is limited to
> 16M, but I wonder if it is sensible to do the same for the index
> block whose limit is 2G.  If you keep the restart table after the
> data, then you could stream out the entries as you emit, write the
> restart table, and then seek back to fix the length in the header,
> without holding the 2G in core, no?

Yes. I'm torn on the ordering: restart table first or restart table last.

The advantage of it first is the reader can immediately work with it, without necessarily touching the rest of the block. The disadvantage is a writer can only stream at block sizes, as the writer is forced to buffer the entire block. As it happens my implementation in JGit buffers the entire block anyway, so this didn't really factor as an issue for me.

Given that the index can now also be multi-level, I don't expect to see a 2G index. A 2G index forces the reader to load the entire 2G to take advantage of the restart table. It may be more efficient for such a reader to have had the writer make a mutli-level index, instead of a single monster index block. And so perhaps the writer shouldn't make a 2G index block that she is forced to buffer. :)

Perhaps I'll move it back to the tail of the block. I can see the streaming writer code is maybe more straightforward that way.