Re: space compression (again)

From: Derek Fawcus <dfawcus@cisco.com>
Date: Apr 15, 2005, 19:00 UTC
Message-ID: <20050415200054.F6735@mrwint.cisco.com>
In-Reply-To: <Pine.LNX.4.61.0504151437100.27637@cag.csail.mit.edu>

On Fri, Apr 15, 2005 at 02:45:55PM -0400, C. Scott Ananian wrote:

Show 10 quoted lines

> > - we already have wasted space due to the low-level filesystem (as
> >   opposed to "git") usually being block-based, which means that space
> >   utilization for small objects tends to suck. So you really want to
> >   prefer objects that are several kB (compressed), and a small block just
> >   wastes tons of space.
> 
> Not on (say) reiserfs, and not over the network.  I'm proposing (at the 
> moment) easy conversion from chunked to unchunked disk representation,
> so that you can leave things unchunked if (for example) you know you're 
> running ext2 with a large block size.

Or if one does not care about space, and simply want's speed, add another layer of indirection - a flattened container object which has hashses as normal, then as it's content simply has the 'chunk list object' and the 'chunk objects' concatenated.

It's then a per user / database as to if the flattened objects, or the heirarcal objects are storred locally.

DF