threads / discuss / 17822

Re: git-svn -- out of memory

Subject: Re: git-svn -- out of memory

## tl;dr

2 messages between Feb 16, 2009 and Feb 16, 2009.

replies: 1people: 2as markdown or json

Alfred M. Szmidt· Feb 16, 2009, 14:23 UTC · lore

   >    > when doing a `git svn clone' on a subversion repository that contains
   >    > a large file (>1GiB), `git svn' dies with the following:
   >    > 
   >    > ams@trillian:~$ git svn clone REPO -T trunk -b branches -t tags
   >    > [...]
   >    > fatal: Out of memory? mmap failed: Cannot allocate memory
   >    > hash-object -w --stdin-paths: command returned error: 128

   Ding...

   >    > git version 1.6.1.3
   >
   >    Core git requires objects to be able to fit into (virtual) memory.
   >    Adding more swap space should solve the problem for you, but this is not
   >    only a git-svn problem.
   >
   > In this case, the machine has 32G RAM, including ~20G swap.  So I do
   > not think that this is the case, it could be I suppose...

   The hash-object error message sounds like the issue that was fixed with
   915308b (avoid 31-bit truncation in write_loose_object, 2009-01-29).  Does
   v1.6.2-rc1 help?

ams@trillian:~/# git --version git version 1.6.2.rc1 ams@trillian:~/# git svn clone REPO -T trunk -b branches -t tags [...] fatal: Out of memory? mmap failed: Cannot allocate memory hash-object -w --stdin-paths: command returned error: 128

Nope, behaves the same way.

I'll try adding more swap and see what happens...

Thanks for the help.

Jeff King· Feb 16, 2009, 15:37 UTC · re: Alfred M. Szmidt · lore

On Mon, Feb 16, 2009 at 03:23:40PM +0100, Alfred M. Szmidt wrote:

Show 8 quoted lines

> ams@trillian:~/# git --version
> git version 1.6.2.rc1
> ams@trillian:~/# git svn clone REPO -T trunk -b branches -t tags
> [...]
> fatal: Out of memory? mmap failed: Cannot allocate memory
> hash-object -w --stdin-paths: command returned error: 128
> 
> Nope, behaves the same way.

Hmm. It would help to know what object it is failing on. I assume you repo is too large to be easily shared? If not, can you try instrumenting hash-object with the patch below and seeing which file it is barfing on?

I'd be curious to see the size of the file (in case it is >2G and we have another signed integer problem) and its position in the list (in case we are leaking memory and you are hitting a system limit).

---

diff --git a/hash-object.c b/hash-object.c
index 37e6677..20371d0 100644
--- a/hash-object.c
+++ b/hash-object.c
@@ -44,6 +44,7 @@ static void hash_stdin_paths(const char *type, int write_objects)
 				die("line is badly quoted");
 			strbuf_swap(&buf, &nbuf);
 		}
+		fprintf(stderr, "debug: hashing %s\n", buf.buf);
 		hash_object(buf.buf, type, write_objects, buf.buf);
 	}
 	strbuf_release(&buf);