Re: Problem with git-daemon and mmap.

From: Shawn O. Pearce <spearce@spearce.org>
Date: Apr 25, 2007, 07:43 UTC
Message-ID: <20070425074354.GH5942@spearce.org>
In-Reply-To: <1177483875.12689.22.camel@localhost>

Tim Ansell <mithro@mithis.com> wrote:

Show 5 quoted lines

> [30957] Request upload-pack for '/git/web.git'
> fatal: Out of memory? mmap failed: Bad file descriptor
> error: git-upload-pack: git-rev-list died with error.
> fatal: git-upload-pack: aborting due to possible repository corruption
> on the remote side.

Owwww.... I don't know what your bug is, but your bug is showing my own bug. Filtering the strace to the rev-list child shows this gem:

20769 open("./objects/pack/pack-1931b39bd648fd595248e234f43717569ec5c354.pack", O_RDONLY|O_LARGEFILE <unfinished ...> 20769 <... open resumed> ) = 5

OK, so the packfile is fd 5...

20769 mmap2(NULL, 33554432, PROT_READ, MAP_PRIVATE, 5, 0 <unfinished ...> 20769 <... mmap2 resumed> ) = 0xb5e2d000

and we mapped one 32 MiB window from it at position 0...

20769 mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 <unfinished ...> 20769 <... mmap2 resumed> ) = -1 ENOMEM (Cannot allocate memory)

And we asked for another window further into the file.  But got denied.

Now where are we in the code? We're down inside use_pack(), after we have called unuse_one_window() enough times to make sure we stay within our allowed maximum window size. Since we didn't unmap the prior window at 0xb5e2d000 we aren't exceeding the current limit (which probably the defaults).

But we're actually down inside xmmap()...

So we release the window we do have (by calling release_pack_memory), assuming memory pressure...

20769 munmap(0xb5e2d000, 33554432 <unfinished ...> 20769 <... munmap resumed> ) = 0 20769 close(5 <unfinished ...> 20769 <... close resumed> ) = 0

And that was the last window in this packfile. So we closed it. Way to go us. Our xmmap does not expect release_pack_memory to close the fd its about to map...

20769 mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 <unfinished ...> 20769 <... mmap2 resumed> ) = -1 EBADF (Bad file descriptor)

And so the Linux kernel happily tells us f' off.

20769 write(2, "fatal: ", 7 <unfinished ...> 20769 <... write resumed> ) = 7 20769 write(2, "Out of memory? mmap failed: Bad "..., 47 <unfinished ...> 20769 <... write resumed> ) = 47

And we report the bad file descriptor error, and not the ENOMEM.

I'm not really sure why you blew out the memory and couldn't mmap a second window here; you really should have been able to given that there was only one 32 MiB window mapped, and this was fairly earily in the process' life. You can try tuning the packing code down to a smaller window size (see core.packedGitWindowSize and core.packedGitLimit).

But that strace above shows a secondary bug when we get into this sort of situation. Not pretty.

-- 
Shawn.