Re: Packfile can't be mapped

From: Shawn Pearce <spearce@spearce.org>
Date: Aug 29, 2006, 04:52 UTC
Message-ID: <20060829045239.GB24479@spearce.org>
In-Reply-To: <20060828024720.GD24204@spearce.org>

Shawn Pearce <spearce@spearce.org> wrote:

> I'm going to try to get tree deltas written to the pack sometime this
> week.

I was able to implement and with Jon Smirl's help debug the tree
delta code in fast-import.
 
Earlier this evening Jon sent me the following:

Show 18 quoted lines

> git-fast-import statistics:
> ---------------------------------------------------------------------
> Alloc'd objects:    1980000 (         0 overflow  )
> Total objects:      1967527 (     41856 duplicates                  )
>       blobs  :       633842 (         0 duplicates     576219 deltas)
>       trees  :      1131208 (     41856 duplicates    1019741 deltas)
>       commits:       200921 (         0 duplicates          0 deltas)
>       tags   :         1556 (         0 duplicates          0 deltas)
> Total branches:        1600 (      2228 loads     )
>       marks:        1048576 (    200921 unique    )
>       atoms:          56803
> Memory total:         75213 KiB
>        pools:         13338 KiB
>      objects:         61875 KiB
> Pack remaps:            658
> Pack size:           895983 KiB
> Index size:           46114 KiB
> ---------------------------------------------------------------------

Compared to our last attempt:

> > Pack size:          1713200 KiB
> > Index size:           46114 KiB

This tree delta version came out pretty good. The pack with tree deltas is 874 MiB. Quite a reduction in size. fast-import takes about 20 minutes to convert its 20 GiB input file into this 874 MiB pack. Producing the 20 GiB input file from the 3 GiB CVS ,v files takes about 4 hours with Jon's modified cvs2svn.

Jon has started a `git-repack -a -f` with aggressive depth and window sizes. He estimated it may need another 2.5 hours to process. Hopefully I'll hear more details tomorrow.

-- 
Shawn.