Re: low memory system to clone larger repo

From: Matt Sporleder <msporleder@gmail.com>
Date: Feb 9, 2015, 11:20 UTC
Message-ID: <EF215DDC-22ED-426B-9C8D-5BA91E6EEACB@gmail.com>
In-Reply-To: <CACsJy8Cx6K3Qdq4hq7T_vxsOR-UJv7+mz9AFSiAeKd3YZxqYHg@mail.gmail.com>

A more "final" version of the tuning exercise I did is here:

http://mail-index.netbsd.org/tech-repository/2015/01/08/msg000520.html

I did try some of these setting on the server and it made the repo much much larger so I guess I am looking for ways to just reduce client memory usage/the best balance of disk, memory, and bandwidth.

If there is a way to turn off some memory hogging speed ups I m interested in trying them.

I will try to get you the output later since I have since started working on the server side.

Let me know if you want to try some server side stuff and I can give you a git:// or http:// off list.

Thanks for looking, Matt

Show 51 quoted lines

> On Feb 9, 2015, at 5:40 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> 
>> On Thu, Jan 8, 2015 at 11:10 PM, matthew sporleder <msporleder@gmail.com> wrote:
>> I am attempting to clone this repo: https://github.com/jsonn/src/
>> 
>> and have been successful on some lower memory systems, but i'm
>> interested in continuing to push down the limit.
>> 
>> I am getting more success running clone via https:// than git:// or
>> ssh (which is confusing to me) and the smallest system that works is a
>> raspberry pi with 256 RAM + 256 swap.
>> 
>> I seem to run out of memory consistently around 16% into Resolving
>> deltas phase but I don't notice an RSS jump so that's another
>> confusing spot.
> 
> Sorry for a really late reply. The command that's running when you run
> out of memory is index-pack. I guess it's verifying the delta chain. I
> think it needs enough memory for two uncompressed objects (or files)
> in a delta chain. I haven't finished cloning this repo yet so I don't
> know what these delta chains look like.
> 
> What does it say when it runs out of memory? I'm thinking maybe we
> could force a core dump, then look at how memory is used.
> 
> What if you "git init", then do "git fetch https://..." manually?
> There's an optimization for git-clone that may make index-pack use a
> bit more memory (and push it over the edge)
> 
>> My config is below and I'd appreciate any more suggestions of getting
>> that down to working on a 128MB box (or smaller).
> 
> I suppose it's ~/.gitconfig or /etc/gitconfig, it's not added after
> the clone is complete, correct? Sounds interesting, let me profile its
> memory usage..
> 
>> [pack]
>>        windowMemory = 1m
>>        packSizeLimit = 1m
>>        deltaCacheSize = 1m
>>        deltaCacheLimit = 10
>>        packSizeLimit = 1m
> 
> I think many of these only affect the server side. If you clone from
> github, then they are useless. You may want to provide your own server
> side with these settings and see if things change. Also play with
> pack.depth (affecting server side)
> 
>>        threads = 1
> -- 
> Duy