Re: git-svnimport failed and now git-repack hates me

From: Linus Torvalds <torvalds@osdl.org>
Date: Jan 5, 2007, 23:17 UTC
Message-ID: <Pine.LNX.4.64.0701051515000.3661@woody.osdl.org>
In-Reply-To: <7v64bldqas.fsf@assigned-by-dhcp.cox.net>

On Fri, 5 Jan 2007, Junio C Hamano wrote:

Show 6 quoted lines

> 
> The problem Linus pointed out was that your SHA1_Update()
> implementations may not be prepared to hash the whole 2.3GB in
> one go.  The one in "master" (and "maint", although I haven't
> done a v1.4.4.4 maintenance release yet) calls SHA1_Update()
> in chunks to work around that potential issue.

Well, I think Chris is worried about having it all mapped at the same time.

It does actually end up forcing the kernel to do more work (it's harder to re-use a mapped page than it is to reuse one that isn't), and in that sense, if you have less than <n> GB of RAM and can't just keep it all in memory at the same time, doing one large mmap is possibly more expensive than chunking things up.

That said, I doubt it's a huge problem. If you can't fit the whole file in memory, your real performance issue is going to be the IO, not the fact that the kernel has to work a bit harder at unmapping pages ;)

		Linus