git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCHv4 09/10] pack-objects: Estimate pack size; abort early if pack size limit is exceeded

From
Johan Herland <johan@herland.net>
Date
May 23, 2011, 17:07 UTC
Message-ID
<201105231907.27639.johan@herland.net>
In-Reply-To
<BANLkTimQURetDxYeAJr5sVRB-ja9yuqT7Q@mail.gmail.com>
On Monday 23. May 2011, Shawn Pearce wrote:
Show 6 quoted lines
> We can still get a tighter estimate if we wanted to. I wouldn't mix
> it into this patch, but make a new one on top of it. During delta
> compression we hold onto deltas, or at least compute and retain the
> size of the chosen delta. We could re-check the pack size after the
> Compressing phase by including the delta sizes in the estimate, and
> if we are over, abort before writing.

Ok. Not sure when I'll have the time/courage to dive into this, but I'll at least give it a try.

Show 6 quoted lines
> For non-delta, non-reuse we may be able to guess by just using the
> loose object size. The loose object is most likely compressed at the
> same compression ratio as the outgoing pack stream will use, so a
> deflate(inflate(loose)) cycle is going to be very close in total
> bytes used. If we over shoot the limit by more than some fudge
> factor (say 8K in 1M limit or 0.7%), abort before writing.

I already have an unsubmitted patch on top of the series that includes the on-disk/compressed size of loose objects in the estimate. However, it's quite intrusive (need to extend sha1_object_info() to return compressed size of loose objects). Also, since I don't yet take the delta compression into account, these numbers are obviously unreliable.

That said, in the cases where loose objects are not deltified it seems the compressed/loose versions are about 3 to 7 bytes larger than the corresponding compressed/packed versions. I guess this is due to the loose files using a "<type> SP <size> NUL" text header (deflated), whereas the pack uses a more compact binary format (not deflated).

We could test a large corpus (e.g. linux-kernel) to find the average difference between compressed/loose size and compressed/packed size, and then multiply this with the number of non-delta, non-reuse object to determine the fudge factor you describe above.

Have fun! :)
...Johan
-- 
Johan Herland, <johan@herland.net>
www.herland.net
Previous: Shawn PearceNext: Junio C Hamano
Message 23 of 26 in “[PATCHv4 00/10] Push limits”
  1. Johan HerlandMay 23, 2011
  2. 01/10 Update technical docs to reflect side-band-64k capability in receive-packJohan Herland, May 23, 2011
  3. 02/10 send-pack: Attempt to retrieve remote status even if pack-objects failsJohan Herland, May 23, 2011
  4. Junio C HamanoMay 23, 2011
  5. Johan HerlandMay 23, 2011
  6. 03/10 Tighten rules for matching server capabilities in server_supports()Johan Herland, May 23, 2011
  7. 04/10 receive-pack: Prepare for addition of the new 'limit-*' family of capabilitiesJohan Herland, May 23, 2011
  8. Junio C HamanoMay 23, 2011
  9. Johan HerlandMay 24, 2011
  10. 05/10 pack-objects: Teach new option --max-commit-count, limiting #commits in packJohan Herland, May 23, 2011
  11. Junio C HamanoMay 23, 2011
  12. Johan HerlandMay 24, 2011
  13. 06/10 send-pack/receive-pack: Allow server to refuse pushes with too many commitsJohan Herland, May 23, 2011
  14. Junio C HamanoMay 23, 2011
  15. Johan HerlandMay 24, 2011
  16. 07/10 pack-objects: Allow --max-pack-size to be used together with --stdoutJohan Herland, May 23, 2011
  17. Junio C HamanoMay 24, 2011
  18. Johan HerlandMay 24, 2011
  19. 08/10 send-pack/receive-pack: Allow server to refuse pushing too large packsJohan Herland, May 23, 2011
  20. Junio C HamanoMay 24, 2011
  21. 09/10 pack-objects: Estimate pack size; abort early if pack size limit is exceededJohan Herland, May 23, 2011
  22. Shawn PearceMay 23, 2011
  23. Johan HerlandMay 23, 2011
  24. Junio C HamanoMay 24, 2011
  25. Johan HerlandMay 24, 2011
  26. 10/10 receive-pack: Allow server to refuse pushes with too many objectsJohan Herland, May 23, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.