Re: Unresolved issues #2 (shallow clone again)

From: Jeff King <peff@peff.net>
Date: May 7, 2006, 07:56 UTC
Message-ID: <20060507075631.GA24423@coredump.intra.peff.net>
In-Reply-To: <46a038f90605062308x53995076k7bf45f0aebcae0c6@mail.gmail.com>

On Sun, May 07, 2006 at 06:08:03PM +1200, Martin Langhoff wrote:

Show 5 quoted lines

> >> And in any case commits and trees are lightweight and compress well...
> >Commit maybe, but is this based on a hard fact?
> No hard facts here :( but I think it's reasonable to assume that the
> trees delta/compress reasonably well, as a given commit will change
> just a few entries in each tree.

A few hard facts (using Linus' linux-2.6 tree):
  - original packsize: 120996 kilobytes
  - unpacked: 233338 objects, 1417476 kilobytes
    This is an 11.7:1 compression ratio (of course, much of this is
    wasted space from the 4k block size in the filesystem)
  - There were 87915 total blob objects, of which 19321 were in the
    current tree. I removed all non-current blobs to produce a "shallow"
    tree.
  - The shallow tree unpacked: 164744 objects, 761960 kilobytes
    IOW, about half of the unpacked disk usage was old blobs.
  - Shallow commit/tree/tag objects packed (using 1.3.1
    git-pack-objects):
      Total 164744, written 164744 (delta 92322), reused 0 (delta 0)
      size: 108088
    The compression ratio here is only 7.0:1
  - Total savings by going shallow: 10.7%

So basically, trees and commits DON'T compress as well as historical blobs (potentially because git-pack-objects isn't currently optimized for this -- I haven't checked). As a result, we're saving only 10% by going shallow instead of a potential 50%.

-Peff