Re: Packfile can't be mapped

From: Shawn Pearce <spearce@spearce.org>
Date: Aug 28, 2006, 02:47 UTC
Message-ID: <20060828024720.GD24204@spearce.org>
In-Reply-To: <9e4733910608271804j762960a8ud83654c78ebe009a@mail.gmail.com>

Jon Smirl <jonsmirl@gmail.com> wrote:

Show 16 quoted lines

> git-repack can't handle my 1.75GB pack file. I am running x86 with 3GB
> address space.
> 
> -rw-rw-r-- 1 jonsmirl jonsmirl    47221712 Aug 27 20:29 testme.idx
> -rw-rw-r-- 1 jonsmirl jonsmirl  1754317619 Aug 27 20:29 testme.pack
> 
> [jonsmirl@jonsmirl t1]$ git-repack -a -f --window=50 --depth=5000
> Generating pack...
> Done counting 1963325 objects.
> fatal: packfile .git/objects/pack/testme.pack cannot be mapped.
> [jonsmirl@jonsmirl t1]$
> 
> It is built from Mozilla CVS but it is an intermediate stage of our
> work. The fast-import tool isn't diffing directory tree which makes
> the pack much bigger than it needs to be. Shawn is working on the
> packing code.

I'm going to try to get tree deltas written to the pack sometime this week. That should compact this intermediate pack down to something that git-pack-objects would be able to successfully mmap into a 32 bit address space. A complete repack with no delta reuse will hopefully generate a pack closer to 400 MB in size. But I know Jon would like to get that pack even smaller. :)

I should point out that the input stream to fast-import was 20 GB (completely decompressed revisions from RCS) plus all commit data. The original CVS ,v files are around 3 GB. An archive .tar.gz'ing the ,v files is around 550 MB. Going to only 1.7 GB without tree or commit deltas is certainly pretty good. :)

Show 17 quoted lines

> ---------------------------------------------------
> Alloc'd objects:    1968000 (   1892000 overflow  )
> Total objects:      1967527 (     41856 duplicates)
>       blobs  :       633842 (         0 duplicates)
>       trees  :      1131208 (     41856 duplicates)
>       commits:       200921 (         0 duplicates)
>       tags   :         1556 (         0 duplicates)
> Total branches:        1600 (      7985 loads     )
>       marks:        1048576 (    200921 unique    )
>       atoms:          56803
> Memory total:         66908 KiB
>        pools:          5408 KiB
>      objects:         61500 KiB
> Pack remaps:           9501
> ---------------------------------------------------
> Pack size:          1713200 KiB
> Index size:           46114 KiB

All of that says that aside from the 1.7 GB output file fast-import ran extremely well. About 1.9 million objects were written into the output pack file, with 41k duplicate trees (duplicate blobs were removed by cvs2svn prior to fast-import so they don't appear). 200k commits were created across 1600 branches. And we did it in only 67 MB of memory.

We also had ~8000 LRU cache misses related to our branch data; this just means that cvs2svn likes to frequently jump around between branches rather than import an entire branch at a time. Boosting the size of the LRU cache (at the expense of needing more memory) should reduce those cache misses as well as 'Pack remaps'.

I'd also like to clean up that pack remapping code and move it into sha1_file.c. Its an implementation of partial pack mapping and it is apparently working quite well for us in fast-import. It may help GIT deal with very large packs (e.g. 1.7 GB) on smaller address space systems (e.g. 32 bit).

We're not confident that this import is completely valid yet. We have a few translation issues we're still working on. But now that we have a complete pack going from start to finish we can start to focus on those issues. Especially since this entire process (,v to .pack) is less than half a day to run.

-- 
Shawn.