git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: low memory system to clone larger repo

From
Duy Nguyen <pclouds@gmail.com>
Date
Feb 9, 2015, 10:40 UTC
Message-ID
<CACsJy8Cx6K3Qdq4hq7T_vxsOR-UJv7+mz9AFSiAeKd3YZxqYHg@mail.gmail.com>
In-Reply-To
<CAHKF-AspyE84_0CVMz2OjFLt3Q62qKDfTkbUk3-+RQ_EZ=0JGg@mail.gmail.com>
On Thu, Jan 8, 2015 at 11:10 PM, matthew sporleder <msporleder@gmail.com> wrote:
Show 12 quoted lines
> I am attempting to clone this repo: https://github.com/jsonn/src/
>
> and have been successful on some lower memory systems, but i'm
> interested in continuing to push down the limit.
>
> I am getting more success running clone via https:// than git:// or
> ssh (which is confusing to me) and the smallest system that works is a
> raspberry pi with 256 RAM + 256 swap.
>
> I seem to run out of memory consistently around 16% into Resolving
> deltas phase but I don't notice an RSS jump so that's another
> confusing spot.

Sorry for a really late reply. The command that's running when you run out of memory is index-pack. I guess it's verifying the delta chain. I think it needs enough memory for two uncompressed objects (or files) in a delta chain. I haven't finished cloning this repo yet so I don't know what these delta chains look like.

What does it say when it runs out of memory? I'm thinking maybe we could force a core dump, then look at how memory is used.

What if you "git init", then do "git fetch https://..." manually? There's an optimization for git-clone that may make index-pack use a bit more memory (and push it over the edge)

> My config is below and I'd appreciate any more suggestions of getting
> that down to working on a 128MB box (or smaller).

I suppose it's ~/.gitconfig or /etc/gitconfig, it's not added after the clone is complete, correct? Sounds interesting, let me profile its memory usage..

Show 6 quoted lines
> [pack]
>         windowMemory = 1m
>         packSizeLimit = 1m
>         deltaCacheSize = 1m
>         deltaCacheLimit = 10
>         packSizeLimit = 1m

I think many of these only affect the server side. If you clone from github, then they are useless. You may want to provide your own server side with these settings and see if things change. Also play with pack.depth (affecting server side)

>         threads = 1
-- 
Duy
Previous: matthew sporlederNext: Matt Sporleder
Message 2 of 12 in “low memory system to clone larger repo”
  1. matthew sporlederJan 8, 2015
  2. Duy NguyenFeb 9, 2015
  3. Matt SporlederFeb 9, 2015
  4. Duy NguyenFeb 9, 2015
  5. index-pack: reduce memory footprint a bitNguyễn Thái Ngọc Duy, Feb 9, 2015
  6. Junio C HamanoFeb 9, 2015
  7. Duy NguyenFeb 10, 2015
  8. matthew sporlederFeb 10, 2015
  9. Junio C HamanoFeb 10, 2015
  10. matthew sporlederFeb 11, 2015
  11. Duy NguyenFeb 11, 2015
  12. matthew sporlederFeb 10, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.