Re: reftable [v7]: new ref storage format

From: Stefan Beller <sbeller@google.com>
Date: Aug 16, 2017, 06:15 UTC
Message-ID: <CAGZ79kZ4m0-KBFs1pbOvRqkR=0vn-Jbn1FATL_KzW+km0K-S2A@mail.gmail.com>
In-Reply-To: <CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com>

On Tue, Aug 15, 2017 at 7:48 PM, Shawn Pearce <spearce@spearce.org> wrote:

Show 19 quoted lines

> 7th iteration of the reftable storage format.
>
> You can read a rendered version of this here:
> https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md
>
> Changes from v6:
> - Blocks are variable sized, and alignment is optional.
> - ref index is required on variable sized multi-block files.
>
> - restart_count/offsets are again at the end of the block.
> - value_type = 0x3 is only for symbolic references.
> - "other" files cannot be stored in reftable.
>
> - object blocks are explicitly optional.
> - object blocks use position (offset in bytes), not block id.
> - removed complex log_chained format for log blocks
>
> - Layout uses log, ref file extensions
> - Described reader algorithm to obtain a snapshot

- back to the old "intra-block index is last"
  for all block types. ok.
- changed (only ref?) indexes to start char + 3 byte size:
  Which starting char do object/log indexes have?

"Unaligned files must include the ref index to support fast lookup."

Why this? I would imagine the client (which has ~5 branches), would not need this, but only a ref block, that's it.

Ctrl-F for 'block_size' reveals nothing is computed relative to the block_size in this format, yet we can set it to an arbitrary number. If following the spec, the reader at $DAY_JOB needs to be able to read both aligned and unaligned reftables, despite our plan to ever write aligned ref tables, what would the reader use the block_size for? (I think we can omit that field from the header/footer now, no?)