chunking (Re: [ANNOUNCEMENT] /Arch/ embraces `git')

From: Linus Torvalds <torvalds@osdl.org>
Date: Apr 20, 2005, 22:22 UTC
Message-ID: <Pine.LNX.4.58.0504201510520.6467@ppc970.osdl.org>
In-Reply-To: <Pine.LNX.4.61.0504201754450.2630@cag.csail.mit.edu>

On Wed, 20 Apr 2005, C. Scott Ananian wrote:

Show 6 quoted lines

> 
> I'm hoping my 'chunking' patches will fix this.  This ought to reduce the 
> size of the object store by (in effect) doing delta compression; rsync
> will then Do The Right Thing and only transfer the needed deltas.
> Running some benchmarks right now to see how well it lives up to this 
> promise...

What's the disk usage results? I'm on ext3, for example, which means that even small files invariably take up 4.125kB on disk (with the inode).

Even uncompressed, most source files tend to be small. Compressed, I'm seeing the median blob size being ~1.6kB in my trivial checks. That's blobs only, btw.

My point being that about 75% of all blobs already take up less than the minimal amount of space that most filesystems can sanely allocate. And I'm _not_ going to say "you have to use reiserfs" with git.

So the disk fragmentation really does matter. It doesn't help to make a file smaller than 4kB, it hurts - while that can be offset by sharing chunks, it might not be.

Also, while network performance is important, so is the handshaking on which objects to get. Lots of small objects potentially need lots of handshaking to figure out _which_ of the objects to do.

		Linus