Re: low memory system to clone larger repo

From: matthew sporleder <msporleder@gmail.com>
Date: Feb 10, 2015, 03:56 UTC
Message-ID: <CAHKF-At1ybThZ54yNJZQDWooMcLPeCi-QKAi4AjzZyiK86_dOA@mail.gmail.com>
In-Reply-To: <CACsJy8A=6m5sWnDhPPMNrWbZ=fOMXPxO_1GVh-WpHycf5gm+rg@mail.gmail.com>

Below is the output from my index-pack/clone over git://

This is with the recent memory patch applied but testing some less crazy tuning. --

The corruption of the signal number shows up in google from other people so I guess it's a lingering bug.

* git-tests $ git clone git://github.com/jsonn/src Cloning into 'src'... remote: Counting objects: 3497569, done. remote: Compressing objects: 100% (640647/640647), done. error: index-pack died of signal 9497569), 990.62 MiB | 8.35 MiB/s fatal: index-pack failed

On Mon, Feb 9, 2015 at 6:32 AM, Duy Nguyen <pclouds@gmail.com> wrote:

Show 17 quoted lines

> On Thu, Jan 8, 2015 at 11:10 PM, matthew sporleder <msporleder@gmail.com> wrote:
>> I am attempting to clone this repo: https://github.com/jsonn/src/
>
> This repo has 3.4M objects. Basic book keeping would cost 200MB (in
> practice it'll be higher because I'm assuming no deltas in my
> calculation). On my 64-bit system, it already uses 400+ MB at the
> beginning of delta resolving phase, and is about 500MB during. 32-bit
> systems cost less but I doubt we could keep it within 256 MB limit. I
> think you just need more powerful machines for a repo this size.
>
> Also, they have some large files (udivmodti4_test.c 16MB, MD5SUMS
> 6MB..) These giant files could make index-pack use more memory
> especially if they are deltified. If you repack the repo with
> core.bigFileThreshold about 1-2MB, then clone, you may get a better
> memory consumption, but at the cost of bigger packs.
> --
> Duy