Problem with git-daemon and mmap.

Hello,

I am trying to use git-daemon on a Debian stable machine. We converted our repositories from darcs to git using tailor (which seemed to work well). Most of the repositories work fine, however our web repository which is quite large (about ~126Mb) is causing problems when trying to clone via git-daemon.

The repository is public, you can find our gitweb here
 http://git.thousandparsec.net/gitweb/gitweb.cgi?p=web.git;a=summary

You can clone using the following URLs,
 http://git.thousandparsec.net/git/web.git
 git://git.thousandparsec.net/git/web.git
 git+ssh://<username>@git.thousandparsec.net/git/web.git

I can clone the repository fine locally (using both cg-clone and git-clone) and with both git+ssh:// and http://

When trying to clone via git-daemon the git-daemon process dies with an the following error (in the git-daemon logs - timestamps trimmed to make the log more readable),

[30957] Connection from 203.122.246.117:20108 [30957] Extended attributes (29 bytes) exist <host=git.thousandparsec.net> [30957] Request upload-pack for '/git/web.git' fatal: Out of memory? mmap failed: Bad file descriptor error: git-upload-pack: git-rev-list died with error. fatal: git-upload-pack: aborting due to possible repository corruption on the remote side. [30957] Disconnected (with error)

On the client side you see, <snip> remote: Generating pack... remote: aborting due to possible repository corruption on the remote side. fatal: early EOF fatal: index-pack died with error code 128 <snip>

After chatting with people on #git (on freenode) I was able to produce a strace for this, you can find a copy of this strace log here, http://www.thousandparsec.net/~tim/strace.log

The machine as only 512mb of Ram. It isn't running much else apart from
a few websites served via apache. Here is some more (hopefully useful)
information about the machine,
verbal:~# cat /proc/meminfo 
MemTotal:       500536 kB
MemFree:         20556 kB
<snip>
SwapTotal:     1542232 kB
SwapFree:      1542200 kB

verbal:~# dpkg --list | grep git ii cogito 0.18.2-1 version control system ii git-core 1.5.1.1-1 fast, scalable, distributed revision control ii git-daemon-run 1.5.1.1-1 fast, scalable, distributed revision control ii gitweb 1.5.1.1-1 fast, scalable, distributed revision control

verbal:~# git --version git version 1.5.1.1

Hopefully this gives you enough information to track down this bug. I'm happy to provide more information if needed. I'll also be on #git, but I'm in Australia so you might have a hard time catching me.

 
Thank you for your time.

Tim Ansell -- http://blog.mithis.net/ - Mithro Rants about Stuff

On Wed, Apr 25, 2007 at 04:21:15PM +0930, Tim Ansell wrote:

In the debian package version 1.4.4.4, git-daemon's memory by default is limited to 64MB. I changed that with 1.5.0, but maybe you chose to not accept the changes to the run script when upgrading to 1.5.5.1-1.

Check /var/service/git-daemon/run, if it has 'exec chpst -m64000000 git-daemon ...'. If so, remove 'chpst -m64000000', and restart the git-daemon service through 'sv restart git-daemon'.

HTH, Gerrit.

<snip>

Doh!

It appears that after upgrading the git-daemon-run from 1.4.4 to 1.5.1, I didn't do a "sv restart git-daemon".

After running the command I can now do a checkout form the web repository, yay!

It appears to use 128m of virtual memory when checking out the repository (which is why it would be hitting the 64Mb limit).

Would adding a "chpst -m256000000" be a good idea?

Thanks for your help.

Tim Ansell

On Wed, Apr 25, 2007 at 05:31:13PM +0930, Tim Ansell wrote:

Probably the package upgrade should have done this for you, my fault.

I personally think so, yes, limiting the memory for long running processes (services) generally is a good idea. But finding a good default for git-daemon isn't easy, as it depends on the repositories it serves, and requires lots of memory per se. So on Debian, it's now unlimited by default.

Regards, Gerrit.

Tim Ansell <mithro@mithis.com> wrote:

Owwww.... I don't know what your bug is, but your bug is showing my own bug. Filtering the strace to the rev-list child shows this gem:

20769 open("./objects/pack/pack-1931b39bd648fd595248e234f43717569ec5c354.pack", O_RDONLY|O_LARGEFILE <unfinished ...> 20769 <... open resumed> ) = 5

OK, so the packfile is fd 5...

20769 mmap2(NULL, 33554432, PROT_READ, MAP_PRIVATE, 5, 0 <unfinished ...> 20769 <... mmap2 resumed> ) = 0xb5e2d000

and we mapped one 32 MiB window from it at position 0...

20769 mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 <unfinished ...> 20769 <... mmap2 resumed> ) = -1 ENOMEM (Cannot allocate memory)

And we asked for another window further into the file.  But got denied.

Now where are we in the code? We're down inside use_pack(), after we have called unuse_one_window() enough times to make sure we stay within our allowed maximum window size. Since we didn't unmap the prior window at 0xb5e2d000 we aren't exceeding the current limit (which probably the defaults).

But we're actually down inside xmmap()...

So we release the window we do have (by calling release_pack_memory), assuming memory pressure...

20769 munmap(0xb5e2d000, 33554432 <unfinished ...> 20769 <... munmap resumed> ) = 0 20769 close(5 <unfinished ...> 20769 <... close resumed> ) = 0

And that was the last window in this packfile. So we closed it. Way to go us. Our xmmap does not expect release_pack_memory to close the fd its about to map...

20769 mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 <unfinished ...> 20769 <... mmap2 resumed> ) = -1 EBADF (Bad file descriptor)

And so the Linux kernel happily tells us f' off.

20769 write(2, "fatal: ", 7 <unfinished ...> 20769 <... write resumed> ) = 7 20769 write(2, "Out of memory? mmap failed: Bad "..., 47 <unfinished ...> 20769 <... write resumed> ) = 47

And we report the bad file descriptor error, and not the ENOMEM.

I'm not really sure why you blew out the memory and couldn't mmap a second window here; you really should have been able to given that there was only one 32 MiB window mapped, and this was fairly earily in the process' life. You can try tuning the packing code down to a smaller window size (see core.packedGitWindowSize and core.packedGitLimit).

But that strace above shows a secondary bug when we get into this sort of situation. Not pretty.

On Wed, 25 Apr 2007, Shawn O. Pearce wrote:

Yeah, that looks like a real bug - it means that if we only have enough memory to have _one_ window open, the whole pack-file windowing doesn't work.

Now, in this case, it looks like Tim's problem was just some way too tight memory limits by his Debian install, and is fixed for him now, but this does seem to be a beauty wart that should be fixed.

Yeah. 

	ulimit -S -v 65536
	git clone linux-2.6 test-linux-2.6

or similar should be able to reproduce it, if you just set the pack-file window to something like 32MB. Maybe.

			Linus

On 4/25/07, Tim Ansell <mithro@mithis.com> wrote:

126Mb sounds so _small_ compared to the repos seen lately (Firefox, KDE, hell, my own work is long in Gb ballpark!). You're lucky :)