Hi there,
I'm using Git v1.7.1 on Ubuntu v10.10 and unfortunately Git seems to regularly exhaust the memory on my machine and fails to compress loose objects and/or collect garbage.
My Intel based dual-core machine has 2 GB of RAM and 4 GB of swap space. I need to track a working tree with a handful of really large tarballs that rarely change and loads of really small text files that change frequently. What I'm seeing is that over time whenever "git gc" runs automatically it fails with the message "fatal: Out of memory, malloc failed". So I've been trying to manually run "git repack -ad --window-memory=1g --max-pack-size=1g" in the hope that Git will not exceed the physical memory. But I still get the same error message :(
As I can't make my repository public, I've attached a simple Python script that generates a ~1.3 GB file containing random integers (takes roughly 10 min. on my machine). Then I run the following four commands and get the out-of-memory failure from "git repack". This is effectively emulating the scenario I have with my repository.
$ git init $ git add ./test_data.dat $ git commit ./test_data.dat -m "Test data." $ git repack -ad --window-memory=1g --max-pack-size=1g Counting objects: 3, done. Delta compression using up to 2 threads. Compressing objects: 100% (2/2), done. fatal: Out of memory, malloc failed
I can't find anything on the wiki about out-of-memory failures. Any info/help would be much appreciated.
Regards
Alif
#! /usr/bin/env python
import sys, os, random
if __name__ == '__main__':
fbuff = 2**28 # file buffer size in bytes
fints = 10**8 # number of integers to write
fsize = 12 # number of digits to write per integer
fname = os.path.splitext(os.path.basename(sys.argv[0]))[0]
try:
os.mkdir(fname)
except OSError:
pass # directory already exists
fpath = os.path.join(fname, fname+'.dat')
fhand = open(fpath, 'w', fbuff)
sys.stdout.write('Writing %d MB to file \'%s\'\n' % (fints*(fsize+1)/10**6, fpath))
sys.stdout.write('This will take some time, please wait.\n')
sys.stdout.write('Progress:')
sys.stdout.flush() random.seed(-1)
fform = '%'+str(fsize)+'d\n'
for x in xrange(fints):
fhand.write(fform % (random.randint(0,fints)))
if x % (fints/10) == 0 and x > 0:
sys.stdout.write('%3d/10' % (x/(fints/10)))
sys.stdout.flush() sys.stdout.write(' 10/10\n')
sys.stdout.flush()
fhand.close()
