Re: Errors cloning large repo

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mar 10, 2007, 05:10 UTC
Message-ID: <Pine.LNX.4.64.0703092057570.10832@woody.linux-foundation.org>
In-Reply-To: <645002.46177.qm@web52608.mail.yahoo.com>

On Fri, 9 Mar 2007, Anton Tropashko wrote:

> 
> My problem is git-clone though since for commit it's no big deal
> to git commit [a-c]* , or use xargs as a workaround

Sure, but there were two problems.

The "git commit" problem is trivial, and in no way fundamental. The thing that uses tons of memory is literally just eyecandy, to show you *what* you're committing.

In fact, by the time it starts using tons of memory, the commit has literally already happened. It's just doing statistics afterwards that bloats it up.

> For git clone I got this

The "git clone" problem is different, in that it's due to the 2GB pack-file limit. It's not "fundmentally hard" either, but it's at least not just a small tiny silly detail.

In fact, you can just do

	git add .
	git commit -q

and the "-q" flag (or "--quiet") will mean that the diffstat is never done, and the commit should be almost instantaneous (all the real work is done by the "git add .")

So "git commit" issue really is just a small beauty wart.

Show 8 quoted lines

> Deltifying 144511 objects.
>  100% (144511/144511) done
> 1625.375MB  (1713 kB/s)       
> 1729.057MB  (499 kB/s)       
> /usr/bin/git-clone: line 321: 24360 File size limit exceededgit-fetch-pack --all -k $quiet "$repo"
> 
> again after git repack and don't see how to work around that aside from artifically
> splitting the tree at the top or resorting to a tarball on an ftp site.

So the "git repack" actually worked for you? It really shouldn't have worked.

Is the server side perhaps 64-bit? If so, the limit ends up being 4GB instead of 2GB, and your 8.5GB project may actually fit.

If so, we can trivially fix it with the current index file even for a 32-bit machine. The reason we limit pack-files to 2GB on 32-bit machines is purely that we don't use O_LARGEFILE. If we enable O_LARGEFILE, that moves the limit up from 31 bits to 32 bits, and it might be enough for you. No new data structures for the index necessary at all.

		Linus