git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Huge win, compressing a window of delta runs as a unit

From
Nicolas Pitre <nico@cam.org>
Date
Aug 17, 2006, 18:24 UTC
Message-ID
<Pine.LNX.4.64.0608171416030.11359@localhost.localdomain>
In-Reply-To
<9e4733910608171103j545a7fd5x2b61b97376de04be@mail.gmail.com>
On Thu, 17 Aug 2006, Jon Smirl wrote:
Show 15 quoted lines
> On 8/17/06, Nicolas Pitre <nico@cam.org> wrote:
> > We're streaving for optimal data storage here so don't be afraid to use
> > one of the available types for an "object stream" object.  Because when
> > you think of it, the deflating of multiple objects into a single zlib
> > stream can be applied to all object types not only deltas.  If ever
> > deflating many blobs into one zlib stream is dimmed worth it then the
> > encoding will already be ready for it.  Also you can leverage existing
> > code to write headers, etc.
> 
> Here are two more case that need to be accounted for in the packs.
> 
> 1) If you zip something and it gets bigger. We need an entry that just
> stores the object without it being zipped. Zipping jpegs or mpegs will
> likely make them significantly bigger. Or does zlib like already
> detect this case and do a null compression?
zlib appears to do the right thing here.
Show 5 quoted lines
> 2) If you delta something and the delta is bigger than the object
> being deltaed. The delta code should detect this and store the full
> object instead of the delta. Again jpegs and mpegs will trigger this.
> You may even want to say that the delta has to be smaller than 80% of
> the full object.

I'd invite you to read te try_delta() code in [builtin-]builtin-pack-objects.c. You might find that we're already way more agressive than that with rather simple heuristics that have been tweaked and tuned for over a year now. The commit logs for that file (before the rename to builtin-pack-objects.c) is also quite enlightening.

Nicolas
Previous: Jon SmirlNext: Nicolas Pitre
Message 16 of 36 in “Huge win, compressing a window of delta runs as a unit”
  1. Jon SmirlAug 16, 2006
  2. Shawn PearceAug 17, 2006
  3. Johannes SchindelinAug 17, 2006
  4. Johannes SchindelinAug 17, 2006
  5. Jon SmirlAug 17, 2006
  6. Johannes SchindelinAug 17, 2006
  7. Nicolas PitreAug 17, 2006
  8. Johannes SchindelinAug 17, 2006
  9. Jon SmirlAug 17, 2006
  10. Nicolas PitreAug 17, 2006
  11. Jon SmirlAug 17, 2006
  12. Nicolas PitreAug 17, 2006
  13. Jon SmirlAug 17, 2006
  14. Nicolas PitreAug 17, 2006
  15. Jon SmirlAug 17, 2006
  16. Nicolas PitreAug 17, 2006
  17. Nicolas PitreAug 18, 2006
  18. Jon SmirlAug 18, 2006
  19. Nicolas PitreAug 18, 2006
  20. Jon SmirlAug 18, 2006
  21. Nicolas PitreAug 21, 2006
  22. Shawn PearceAug 21, 2006
  23. Jakub NarebskiAug 21, 2006
  24. Jon SmirlAug 21, 2006
  25. Jon SmirlAug 18, 2006
  26. Johannes SchindelinAug 18, 2006
  27. Jon SmirlAug 18, 2006
  28. Linus TorvaldsAug 19, 2006
  29. Nicolas PitreAug 18, 2006
  30. Shawn PearceAug 21, 2006
  31. Jon SmirlAug 21, 2006
  32. Nicolas PitreAug 21, 2006
  33. Jon SmirlAug 21, 2006
  34. Nicolas PitreAug 21, 2006
  35. Nicolas PitreAug 21, 2006
  36. Nicolas PitreAug 21, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.