git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: expanding pack idx fanout table

From
Junio C Hamano <gitster@pobox.com>
Date
Jul 8, 2013, 17:37 UTC
Message-ID
<7vsizpymuy.fsf@alter.siamese.dyndns.org>
In-Reply-To
<CAJo=hJsto1ik=GTC8c3+2_jBuUqcAPL0UWp-1uoYYMpgbLB+qg@mail.gmail.com>
Shawn Pearce <spearce@spearce.org> writes:
Show 11 quoted lines
> Has anyone studied the impact of converting the pack idx fanout table
> from 256 entries to 65536 entries?
>
> Back of the envelope estimates for 3.1M objects in linux.git suggests
> a 2^16 fanout table would decrease the number of binary search
> iterations from ~14 to ~6. The increased table costs an extra 255 KiB
> of disk. On a 70M idx file this is noise.
>
> I'm starting to wonder if increasing the fanout table once the object
> count is above a certain threshold is a reasonable optimization for
> larger repositories.

Yeah, and I do not think we have to be worried too much about backward compatibility for .idx files, as they are local and can be regenerated if an older version cannot read it.

I also wonder if we can generate a finer-grained fan-out table on the fly, perhaps lazily, without changing the on-disk format ;-)

Previous: Shawn PearceNext: Shawn Pearce
Message 2 of 4 in “expanding pack idx fanout table”
  1. Shawn PearceJul 8, 2013
  2. Junio C HamanoJul 8, 2013
  3. Shawn PearceJul 8, 2013
  4. Jeff KingJul 9, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.