Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)

From: Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date: Jul 28, 2010, 07:11 UTC
Message-ID: <AANLkTikcV7BdJ_07yLei9_H_kR2GEOi4707J1gGhS-Mr@mail.gmail.com>
In-Reply-To: <AANLkTi=WjBHYngSDgowD2+0FT3hG+oDYefp2aUostV3U@mail.gmail.com>

On Wed, Jul 28, 2010 at 1:58 PM, Avery Pennarun <apenwarr@gmail.com> wrote:

Show 25 quoted lines

> On Wed, Jul 28, 2010 at 1:06 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>> 2010/7/28 Avery Pennarun <apenwarr@gmail.com>:
>>> 2010/7/27 Elijah Newren <newren@gmail.com>:
>>>> 0) Sparse clones have "all" commit objects, but not all trees/blobs.
>>>>
>>>> Note that "all" only means all that are reachable from the refs being
>>>> downloaded, of course.  I think this is widely agreed upon and has
>>>> been suggested many times on this list.
>>>
>>> I think downloading all commit objects would require very low
>>> bandwidth and storage space, so it should be harmless.
>> >
>> > In fact, I have a pretty strong impression that also downloading
>> > all *tree* objects would be fine too.
>>
>> Here you go. A pack with only commits and trees of git.git#master is
>> 15M. With blobs, it is 28M. Git is not a typical repo with large blobs
>> though.
>
> Hmm, that's very interesting - more than half the repo is just tree
> and commit objects?  Maybe that's not so shocking after all, given the
> tendency in the git project to use long commit messages and relatively
> short patches.
>
> Was your pack carefully ordered for best deltification?

I did not do any optimization. It's pack-objects' defaults. I only filtered blobs out and that's what fetch-pack would receive.

>
> Knowing how much of that is commits vs. trees would also be very interesting.

Commits only takes 7.8 MB. Well.. all those commit messages..

Show 5 quoted lines

> But if so, only saving half the space is kind of disappointing.  If
> you have a script around for generating this, it would be very
> interesting to compare the results with, say, the Linux kernel repo
> (especially since it seems to be the #1 example of "submodules people
> don't want to check out because they're so bloody huge").

I modified pack-objects.c, show_object() to certain objects. Actually I started with git-fetch-pack, but you can do

git rev-parse master | git pack-objects --revs --delta-base-offset pack

Then verify what's in the pack with

git verify-pack -v pack-*.idx

-- 
Duy