threads / discuss / 12893

git clone out of memory. alternatives?

Subject: git clone out of memory. alternatives?

## tl;dr

3 messages between Mar 28, 2008 and Mar 28, 2008.

replies: 2people: 3as markdown or json

Jeffrey Chang· Mar 28, 2008, 02:29 UTC · lore
Hello,

I am working with a large-ish git repository set up as a shared repository. I am the only user of this repository, and I access it over ssh.

I am running into a problem where I can no longer clone the repository on a new computer. I get a message that seems to indicate that the machine with the repository is running out of memory.

xigua:~/remotecvs jchang$ git clone [...] Initialized empty Git repository in [...] remote: Generating pack... remote: Done counting 9122 objects. remote: Deltifying 9122 objects... error: git-upload-pack: git-pack-objects died with error. fatal: git-upload-pack: aborting due to possible repository corruption on the remote side. remote: fatal: out of memoryremote: remote: aborting due to possible repository corruption on the remote side. fatal: early EOF fatal: index-pack failed fetch-pack from [...] failed. xigua:~/remotecvs jchang$

Moving the repository to another machine is not an option, nor is adding more RAM to that machine.

Is there any way to get around this problem?  For example:
- Can I run git clone in a way that uses less memory, such as cloning
a piece of the repository at a time?
- Can I export the entire repository as a file that can be loaded on
my target machine, like "svnadmin dump" for subversion?
- Can I just rsync the repository from another computer that already has a copy?
- Are there any other work-arounds to set up a copy of the repository
on my local machine?

Thanks, Jeff

Linus Torvalds· Mar 28, 2008, 03:04 UTC · re: Jeffrey Chang · lore

Re: git clone out of memory. alternatives?

On Thu, 27 Mar 2008, Jeffrey Chang wrote:
> 
> Is there any way to get around this problem?  For example:
> - Can I run git clone in a way that uses less memory, such as cloning
> a piece of the repository at a time?

Cloning really is pretty memory-intensive, because it involves going through every single object.

What you *can* do is to limit cloning to the rsync protocol, which is strictly quite horrible (none of the inherent sanity-checks of the native protocol), but it avoids the server-side costs.

However, you'll eventually hit other problems, like the fact that you also won't be able to do a full repack on the server side (because a full repack does the same thing).

> - Can I export the entire repository as a file that can be loaded on
> my target machine, like "svnadmin dump" for subversion?
That's essentially what a repack does. See above about the problem.
> - Can I just rsync the repository from another computer that already has a copy?

Yes. You can. And you can also repack on another host and then rsync the results back to the server. It's not pretty, but it should work.

		Linus
Johannes Schindelin· Mar 28, 2008, 11:55 UTC · re: Jeffrey Chang · lore

Re: git clone out of memory. alternatives?

Hi,
On Thu, 27 Mar 2008, Jeffrey Chang wrote:
Show 7 quoted lines
> I am working with a large-ish git repository set up as a shared 
> repository.  I am the only user of this repository, and I access it over 
> ssh.
> 
> I am running into a problem where I can no longer clone the repository 
> on a new computer.  I get a message that seems to indicate that the 
> machine with the repository is running out of memory.

AFAICT the config variables pack.* (see config.txt) allow to tune the packing, even for cloning.

Hth, Dscho

← back to recent threads