git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: low memory system to clone larger repo

From
Duy Nguyen <pclouds@gmail.com>
Date
Feb 9, 2015, 12:32 UTC
Message-ID
<CACsJy8A=6m5sWnDhPPMNrWbZ=fOMXPxO_1GVh-WpHycf5gm+rg@mail.gmail.com>
In-Reply-To
<CAHKF-AspyE84_0CVMz2OjFLt3Q62qKDfTkbUk3-+RQ_EZ=0JGg@mail.gmail.com>
On Thu, Jan 8, 2015 at 11:10 PM, matthew sporleder <msporleder@gmail.com> wrote:
> I am attempting to clone this repo: https://github.com/jsonn/src/

This repo has 3.4M objects. Basic book keeping would cost 200MB (in practice it'll be higher because I'm assuming no deltas in my calculation). On my 64-bit system, it already uses 400+ MB at the beginning of delta resolving phase, and is about 500MB during. 32-bit systems cost less but I doubt we could keep it within 256 MB limit. I think you just need more powerful machines for a repo this size.

Also, they have some large files (udivmodti4_test.c 16MB, MD5SUMS 6MB..) These giant files could make index-pack use more memory especially if they are deltified. If you repack the repo with core.bigFileThreshold about 1-2MB, then clone, you may get a better memory consumption, but at the cost of bigger packs.

-- 
Duy
Previous: Matt SporlederNext: Nguyễn Thái Ngọc Duy
Message 4 of 12 in “low memory system to clone larger repo”
  1. matthew sporlederJan 8, 2015
  2. Duy NguyenFeb 9, 2015
  3. Matt SporlederFeb 9, 2015
  4. Duy NguyenFeb 9, 2015
  5. index-pack: reduce memory footprint a bitNguyễn Thái Ngọc Duy, Feb 9, 2015
  6. Junio C HamanoFeb 9, 2015
  7. Duy NguyenFeb 10, 2015
  8. matthew sporlederFeb 10, 2015
  9. Junio C HamanoFeb 10, 2015
  10. matthew sporlederFeb 11, 2015
  11. Duy NguyenFeb 11, 2015
  12. matthew sporlederFeb 10, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.