Re: Revisiting large binary files issue.

From: Linus Torvalds <torvalds@osdl.org>
Date: Jul 10, 2006, 23:28 UTC
Message-ID: <Pine.LNX.4.64.0607101623230.5623@g5.osdl.org>
In-Reply-To: <20060710230132.GA11132@hpsvcnb.fc.hp.com>

On Mon, 10 Jul 2006, Carl Baldwin wrote:

Show 8 quoted lines

> 
> When I set the window to 0 I one more issue.  Even though the blobs are
> already compressed on disk I still seem to pay the penalty of inflating
> them into memory and then deflating them into the pack.  When the window
> size is 0 this is just wasted cycles.  With large binary files these
> wasted cycles slow down the push/fetch operation considerably.  Couldn't
> the compressed blobs be copied into the pack without first deflating
> them in this 0 window case?

The problem is that the individual object disk format isn't actually the same as the pack-file object format for one object. The header is different: a pack-file uses a very dense bit packing, while the individual object format is a bit less dense.

Sad, really, but it means that right now you can only re-use data that was already packed (when the format matches).

		Linus