Re: git-repack: Outof memory

From: Shawn Pearce <spearce@spearce.org>
Date: Sep 17, 2006, 02:25 UTC
Message-ID: <20060917022534.GB7512@spearce.org>
In-Reply-To: <450CA561.9030602@gmail.com>

Dongsheng Song <dongsheng.song@gmail.com> wrote:

Show 12 quoted lines

> I'm import from subversion. The problem appears to be git-repack phase using too many memory:
> 
> $ git-repack -a -f -d --window=64 --depth=64
> Generating pack...
> Done counting 123497 objects.
> Deltifying 123497 objects.
>   24% (29677/123497) done
> 
> $ top
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  3572 www-data  18   0 2591m 1.9g  528 R   13 94.8  81:48.98 git-pack-object

*ouch* You are probably running out of address space if you are on a 32 bit architecture. A 2.5 GiB virtual address space is pretty close to the maximum allowed on most OSes.

The code that I'm sitting on but haven't yet completed rebasing onto current Git would probably help here.

Do you have any existing .pack files in .git/objects/pack? How big are they and their corresponding .idx files?

git-repack will need to mmap every .pack and .idx in
.git/objects/pack, plus it needs working memory for each object
(123,497 of 'em) but as I recall its pretty frugal on its per-object
allocation.  It can easily work with as many as 2 million objects on
a 32 bit system, assuming the .pack and .idx files aren't too large.

-- 
Shawn.