git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [WIP v2 0/2] Modifying pack objects to support --blob-max-bytes

From
Jeff King <peff@peff.net>
Date
Jun 7, 2017, 09:46 UTC
Message-ID
<20170607094641.pibxlk5nbyii5p63@sigill.intra.peff.net>
In-Reply-To
<20170605103523.190c9f16@twelve2.svl.corp.google.com>
On Mon, Jun 05, 2017 at 10:35:23AM -0700, Jonathan Tan wrote:
Show 22 quoted lines
> > The rest of the pack code uses a varint encoding which is generally
> > much smaller than a uint64 for most files, but can handle arbitrary
> > sizes.
> > 
> > The one thing it loses is that you wouldn't have a fixed-size record, so
> > if you were planning to dump this directly to disk and binary-search it,
> > that won't work. OTOH, you could make pseudo-pack-entries and just
> > index them along with the rest of the objects in the pack .idx.
> > 
> > The one subtle thing there is that the pseudo-entry would have to say
> > "this is my sha1". And then we'd end up repeating that sha1 in the .idx
> > file. So it's optimal on the network but wastes 20 bytes on disk (unless
> > index-pack throws away the in-pack sha1s as it indexes, which is
> > certainly an option).
> 
> If we end up going with the varint approach (which seems reasonable),
> maybe the client could just expand the varints into uint64s so that it
> has a binary-searchable file. I think it's better to keep this list
> separate from the pack .idx file (there has been some discussion on this
> - [1] and its replies).
> 
> [1] https://public-inbox.org/git/777ab8f2-c31a-d07b-ffe3-f8333f408ea1@jeffhostetler.com/

OK. If we're keeping it separate anyway, then I agree that just expanding the varints is a good solution. And we don't have to care too much about the local storage, because it can be changed out later without touching the on-the-wire protocol.

-Peff
Previous: Jonathan Tan
Message 17 of 17 in “[WIP 0/2] Modifying pack-objects to support --blob-size-limit”
  1. Jonathan TanJun 2, 2017
  2. 1/2 pack-objects: rename want_.* to ignore_.*Jonathan Tan, Jun 2, 2017
  3. Junio C HamanoJun 2, 2017
  4. 2/2 pack-objects: support --blob-size-limitJonathan Tan, Jun 2, 2017
  5. Junio C HamanoJun 2, 2017
  6. 0/2 Modifying pack objects to support --blob-max-bytesJonathan Tan, Jun 2, 2017
  7. 1/2 pack-objects: rename want_.* to ignore_.*Jonathan Tan, Jun 2, 2017
  8. 2/2 pack-objects: support --blob-max-bytesJonathan Tan, Jun 2, 2017
  9. Jeff KingJun 2, 2017
  10. Jonathan NiederJun 2, 2017
  11. Jeff KingJun 7, 2017
  12. Junio C HamanoJun 3, 2017
  13. Jeff HostetlerJun 15, 2017
  14. Jonathan TanJun 15, 2017
  15. Jeff KingJun 2, 2017
  16. Jonathan TanJun 5, 2017
  17. Jeff KingJun 7, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.