Re: Huge win, compressing a window of delta runs as a unit
- From
Nicolas Pitre <nico@cam.org>
- Date
- Aug 17, 2006, 18:24 UTC
- Message-ID
- <Pine.LNX.4.64.0608171416030.11359@localhost.localdomain>
- In-Reply-To
- <9e4733910608171103j545a7fd5x2b61b97376de04be@mail.gmail.com>
On Thu, 17 Aug 2006, Jon Smirl wrote:
Show 15 quoted lines
> On 8/17/06, Nicolas Pitre <nico@cam.org> wrote: > > We're streaving for optimal data storage here so don't be afraid to use > > one of the available types for an "object stream" object. Because when > > you think of it, the deflating of multiple objects into a single zlib > > stream can be applied to all object types not only deltas. If ever > > deflating many blobs into one zlib stream is dimmed worth it then the > > encoding will already be ready for it. Also you can leverage existing > > code to write headers, etc. > > Here are two more case that need to be accounted for in the packs. > > 1) If you zip something and it gets bigger. We need an entry that just > stores the object without it being zipped. Zipping jpegs or mpegs will > likely make them significantly bigger. Or does zlib like already > detect this case and do a null compression?
zlib appears to do the right thing here.
Show 5 quoted lines
> 2) If you delta something and the delta is bigger than the object > being deltaed. The delta code should detect this and store the full > object instead of the delta. Again jpegs and mpegs will trigger this. > You may even want to say that the delta has to be smaller than 80% of > the full object.
I'd invite you to read te try_delta() code in [builtin-]builtin-pack-objects.c. You might find that we're already way more agressive than that with rather simple heuristics that have been tweaked and tuned for over a year now. The commit logs for that file (before the rename to builtin-pack-objects.c) is also quite enlightening.
Nicolas