Re: Revisiting large binary files issue.

From: Carl Baldwin <cnb@fc.hp.com>
Date: Jul 11, 2006, 14:55 UTC
Message-ID: <20060711145527.GA32468@hpsvcnb.fc.hp.com>
In-Reply-To: <Pine.LNX.4.64.0607101623230.5623@g5.osdl.org>

I'd like to get my hands dirty and see for myself where the issue lies. I hope to have some time next week to devote to this. Is it reasonable to hope for a solution that is at least a lot lighter weight than the current status quo?

Carl

On Mon, Jul 10, 2006 at 04:28:24PM -0700, Linus Torvalds wrote:

Show 22 quoted lines

> 
> 
> On Mon, 10 Jul 2006, Carl Baldwin wrote:
> > 
> > When I set the window to 0 I one more issue.  Even though the blobs are
> > already compressed on disk I still seem to pay the penalty of inflating
> > them into memory and then deflating them into the pack.  When the window
> > size is 0 this is just wasted cycles.  With large binary files these
> > wasted cycles slow down the push/fetch operation considerably.  Couldn't
> > the compressed blobs be copied into the pack without first deflating
> > them in this 0 window case?
> 
> The problem is that the individual object disk format isn't actually the 
> same as the pack-file object format for one object. The header is 
> different: a pack-file uses a very dense bit packing, while the individual 
> object format is a bit less dense.
> 
> Sad, really, but it means that right now you can only re-use data that was 
> already packed (when the format matches).
> 
> 		Linus
>

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        RADCAD (R&D CAD)
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -