{"thread":{"id":"7826","subject":"Problem with git-daemon and mmap.","startedAt":"2007-04-25T06:51:15Z","lastAt":"2007-04-25T16:13:14Z","messageCount":7,"participants":["Tim Ansell","Gerrit Pape","Shawn O. Pearce","Alex Riesen","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"40379","messageId":"1177483875.12689.22.camel@localhost","threadId":"7826","inReplyTo":null,"subject":"Problem with git-daemon and mmap.","fromName":"Tim Ansell","fromEmail":"mithro@mithis.com","sentAt":"2007-04-25T06:51:15Z","receivedAt":"2007-04-25T06:51:15Z","isPatch":false,"sender":{"key":"mithro@mithis.com","avatar":"https://gravatar.com/avatar/1df49ebfaff79712e08212ff66228e688973c823b6452c23268f9a8c0e4d341b?d=mp&s=160"},"body":"Hello,\n\nI am trying to use git-daemon on a Debian stable machine. We converted\nour repositories from darcs to git using tailor (which seemed to work\nwell). Most of the repositories work fine, however our web repository\nwhich is quite large (about ~126Mb) is causing problems when trying to\nclone via git-daemon.\n\nThe repository is public, you can find our gitweb here\n http://git.thousandparsec.net/gitweb/gitweb.cgi?p=web.git;a=summary\n\nYou can clone using the following URLs,\n http://git.thousandparsec.net/git/web.git\n git://git.thousandparsec.net/git/web.git\n git+ssh://@git.thousandparsec.net/git/web.git\n\nI can clone the repository fine locally (using both cg-clone and\ngit-clone) and with both git+ssh:// and http://\n\nWhen trying to clone via git-daemon the git-daemon process dies with an\nthe following error (in the git-daemon logs - timestamps trimmed to make\nthe log more readable),\n\n[30957] Connection from 203.122.246.117:20108\n[30957] Extended attributes (29 bytes) exist\n\n[30957] Request upload-pack for '/git/web.git'\nfatal: Out of memory? mmap failed: Bad file descriptor\nerror: git-upload-pack: git-rev-list died with error.\nfatal: git-upload-pack: aborting due to possible repository corruption\non the remote side.\n[30957] Disconnected (with error)\n\nOn the client side you see,\n\nremote: Generating pack...\nremote: aborting due to possible repository corruption on the remote\nside.\nfatal: early EOF\nfatal: index-pack died with error code 128\n\n\nAfter chatting with people on #git (on freenode) I was able to produce a\nstrace for this, you can find a copy of this strace log here,\nhttp://www.thousandparsec.net/~tim/strace.log\n\nThe machine as only 512mb of Ram. It isn't running much else apart from\na few websites served via apache. Here is some more (hopefully useful)\ninformation about the machine,\nverbal:~# cat /proc/meminfo \nMemTotal: 500536 kB\nMemFree: 20556 kB\n\nSwapTotal: 1542232 kB\nSwapFree: 1542200 kB\n\nverbal:~# dpkg --list | grep git\nii cogito 0.18.2-1\nversion control system\nii git-core 1.5.1.1-1\nfast, scalable, distributed revision control\nii git-daemon-run 1.5.1.1-1\nfast, scalable, distributed revision control\nii gitweb 1.5.1.1-1\nfast, scalable, distributed revision control\n\nverbal:~# git --version\ngit version 1.5.1.1\n\nHopefully this gives you enough information to track down this bug. I'm\nhappy to provide more information if needed. I'll also be on #git, but\nI'm in Australia so you might have a hard time catching me.\n\n \nThank you for your time.\n\nTim Ansell\n--\nhttp://blog.mithis.net/ - Mithro Rants about Stuff\n"},{"id":"40380","messageId":"20070425074233.7494.qmail@076ec8c0ddad74.315fe32.mid.smarden.org","threadId":"7826","inReplyTo":"1177483875.12689.22.camel@localhost","subject":"Re: Problem with git-daemon and mmap.","fromName":"Gerrit Pape","fromEmail":"pape@smarden.org","sentAt":"2007-04-25T07:42:33Z","receivedAt":"2007-04-25T07:42:33Z","isPatch":false,"sender":{"key":"pape@smarden.org","avatar":"https://avatars.githubusercontent.com/u/143170252?v=4"},"body":"On Wed, Apr 25, 2007 at 04:21:15PM +0930, Tim Ansell wrote:\n> I am trying to use git-daemon on a Debian stable machine. We converted\n> our repositories from darcs to git using tailor (which seemed to work\n> well). Most of the repositories work fine, however our web repository\n> which is quite large (about ~126Mb) is causing problems when trying to\n> clone via git-daemon.\n\n> The machine as only 512mb of Ram. It isn't running much else apart from\n> a few websites served via apache. Here is some more (hopefully useful)\n\n> ii git-daemon-run 1.5.1.1-1\n> fast, scalable, distributed revision control\n\n> Hopefully this gives you enough information to track down this bug. I'm\n> happy to provide more information if needed. I'll also be on #git, but\n> I'm in Australia so you might have a hard time catching me.\n\nIn the debian package version 1.4.4.4, git-daemon's memory by default is\nlimited to 64MB. I changed that with 1.5.0, but maybe you chose to not\naccept the changes to the run script when upgrading to 1.5.5.1-1.\n\nCheck /var/service/git-daemon/run, if it has 'exec chpst -m64000000\ngit-daemon ...'. If so, remove 'chpst -m64000000', and restart the\ngit-daemon service through 'sv restart git-daemon'.\n\nHTH, Gerrit.\n"},{"id":"40381","messageId":"20070425074354.GH5942@spearce.org","threadId":"7826","inReplyTo":"1177483875.12689.22.camel@localhost","subject":"Re: Problem with git-daemon and mmap.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-04-25T07:43:54Z","receivedAt":"2007-04-25T07:43:54Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Tim Ansell wrote:\n> [30957] Request upload-pack for '/git/web.git'\n> fatal: Out of memory? mmap failed: Bad file descriptor\n> error: git-upload-pack: git-rev-list died with error.\n> fatal: git-upload-pack: aborting due to possible repository corruption\n> on the remote side.\n\nOwwww.... I don't know what your bug is, but your bug is showing my\nown bug. Filtering the strace to the rev-list child shows this gem:\n\n20769 open(\"./objects/pack/pack-1931b39bd648fd595248e234f43717569ec5c354.pack\", O_RDONLY|O_LARGEFILE \n20769 <... open resumed> ) = 5\n\nOK, so the packfile is fd 5...\n\n20769 mmap2(NULL, 33554432, PROT_READ, MAP_PRIVATE, 5, 0 \n20769 <... mmap2 resumed> ) = 0xb5e2d000\n\nand we mapped one 32 MiB window from it at position 0...\n\n20769 mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 \n20769 <... mmap2 resumed> ) = -1 ENOMEM (Cannot allocate memory)\n\nAnd we asked for another window further into the file. But got denied.\n\nNow where are we in the code? We're down inside use_pack(), after\nwe have called unuse_one_window() enough times to make sure we stay\nwithin our allowed maximum window size. Since we didn't unmap the\nprior window at 0xb5e2d000 we aren't exceeding the current limit\n(which probably the defaults).\n\nBut we're actually down inside xmmap()...\n\nSo we release the window we do have (by calling release_pack_memory),\nassuming memory pressure...\n\n20769 munmap(0xb5e2d000, 33554432 \n20769 <... munmap resumed> ) = 0\n20769 close(5 \n20769 <... close resumed> ) = 0\n\nAnd that was the last window in this packfile. So we closed it.\nWay to go us. Our xmmap does not expect release_pack_memory\nto close the fd its about to map...\n\n20769 mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 \n20769 <... mmap2 resumed> ) = -1 EBADF (Bad file descriptor)\n\nAnd so the Linux kernel happily tells us f' off.\n\n20769 write(2, \"fatal: \", 7 \n20769 <... write resumed> ) = 7\n20769 write(2, \"Out of memory? mmap failed: Bad \"..., 47 \n20769 <... write resumed> ) = 47\n\nAnd we report the bad file descriptor error, and not the ENOMEM.\n\n\nI'm not really sure why you blew out the memory and couldn't mmap\na second window here; you really should have been able to given\nthat there was only one 32 MiB window mapped, and this was fairly\nearily in the process' life. You can try tuning the packing code\ndown to a smaller window size (see core.packedGitWindowSize and\ncore.packedGitLimit).\n\nBut that strace above shows a secondary bug when we get into this\nsort of situation. Not pretty.\n\n-- \nShawn.\n"},{"id":"40382","messageId":"1177488073.12689.30.camel@localhost","threadId":"7826","inReplyTo":"20070425074233.7494.qmail@076ec8c0ddad74.315fe32.mid.smarden.org","subject":"Re: Problem with git-daemon and mmap.","fromName":"Tim Ansell","fromEmail":"mithro@mithis.com","sentAt":"2007-04-25T08:01:13Z","receivedAt":"2007-04-25T08:01:13Z","isPatch":false,"sender":{"key":"mithro@mithis.com","avatar":"https://gravatar.com/avatar/1df49ebfaff79712e08212ff66228e688973c823b6452c23268f9a8c0e4d341b?d=mp&s=160"},"body":"\n\n> In the debian package version 1.4.4.4, git-daemon's memory by default is\n> limited to 64MB. I changed that with 1.5.0, but maybe you chose to not\n> accept the changes to the run script when upgrading to 1.5.5.1-1.\n> \n> Check /var/service/git-daemon/run, if it has 'exec chpst -m64000000\n> git-daemon ...'. If so, remove 'chpst -m64000000', and restart the\n> git-daemon service through 'sv restart git-daemon'.\n\nDoh!\n\nIt appears that after upgrading the git-daemon-run from 1.4.4 to 1.5.1,\nI didn't do a \"sv restart git-daemon\".\n\nAfter running the command I can now do a checkout form the web\nrepository, yay!\n\nIt appears to use 128m of virtual memory when checking out the\nrepository (which is why it would be hitting the 64Mb limit).\n\nWould adding a \"chpst -m256000000\" be a good idea?\n\nThanks for your help.\n\nTim Ansell\n"},{"id":"40386","messageId":"20070425081136.9611.qmail@f75c6e4144fd72.315fe32.mid.smarden.org","threadId":"7826","inReplyTo":"1177488073.12689.30.camel@localhost","subject":"Re: Problem with git-daemon and mmap.","fromName":"Gerrit Pape","fromEmail":"pape@smarden.org","sentAt":"2007-04-25T08:11:36Z","receivedAt":"2007-04-25T08:11:36Z","isPatch":false,"sender":{"key":"pape@smarden.org","avatar":"https://avatars.githubusercontent.com/u/143170252?v=4"},"body":"On Wed, Apr 25, 2007 at 05:31:13PM +0930, Tim Ansell wrote:\n> > In the debian package version 1.4.4.4, git-daemon's memory by default is\n> > limited to 64MB. I changed that with 1.5.0, but maybe you chose to not\n> > accept the changes to the run script when upgrading to 1.5.5.1-1.\n> > \n> > Check /var/service/git-daemon/run, if it has 'exec chpst -m64000000\n> > git-daemon ...'. If so, remove 'chpst -m64000000', and restart the\n> > git-daemon service through 'sv restart git-daemon'.\n\n> It appears that after upgrading the git-daemon-run from 1.4.4 to 1.5.1,\n> I didn't do a \"sv restart git-daemon\".\n\nProbably the package upgrade should have done this for you, my fault.\n\n> After running the command I can now do a checkout form the web\n> repository, yay!\n> \n> It appears to use 128m of virtual memory when checking out the\n> repository (which is why it would be hitting the 64Mb limit).\n> \n> Would adding a \"chpst -m256000000\" be a good idea?\n\nI personally think so, yes, limiting the memory for long running\nprocesses (services) generally is a good idea. But finding a good\ndefault for git-daemon isn't easy, as it depends on the repositories it\nserves, and requires lots of memory per se. So on Debian, it's now\nunlimited by default.\n\nRegards, Gerrit.\n"},{"id":"40387","messageId":"81b0412b0704250117y5dbe5594ybb672a2b0e350124@mail.gmail.com","threadId":"7826","inReplyTo":"1177483875.12689.22.camel@localhost","subject":"Re: Problem with git-daemon and mmap.","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2007-04-25T08:17:13Z","receivedAt":"2007-04-25T08:17:13Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On 4/25/07, Tim Ansell wrote:\n> which is quite large (about ~126Mb) is causing problems when trying to\n> clone via git-daemon.\n\n126Mb sounds so _small_ compared to the repos seen lately (Firefox,\nKDE, hell, my own work is long in Gb ballpark!). You're lucky :)\n"},{"id":"40413","messageId":"alpine.LFD.0.98.0704250858530.9964@woody.linux-foundation.org","threadId":"7826","inReplyTo":"20070425074354.GH5942@spearce.org","subject":"Re: Problem with git-daemon and mmap.","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-04-25T16:13:14Z","receivedAt":"2007-04-25T16:13:14Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 25 Apr 2007, Shawn O. Pearce wrote:\n> \n> And that was the last window in this packfile. So we closed it.\n> Way to go us. Our xmmap does not expect release_pack_memory\n> to close the fd its about to map...\n\nYeah, that looks like a real bug - it means that if we only have enough \nmemory to have _one_ window open, the whole pack-file windowing doesn't \nwork.\n\nNow, in this case, it looks like Tim's problem was just some way too tight \nmemory limits by his Debian install, and is fixed for him now, but this \ndoes seem to be a beauty wart that should be fixed.\n\n> But that strace above shows a secondary bug when we get into this\n> sort of situation. Not pretty.\n\nYeah. \n\n\tulimit -S -v 65536\n\tgit clone linux-2.6 test-linux-2.6\n\nor similar should be able to reproduce it, if you just set the pack-file \nwindow to something like 32MB. Maybe.\n\n\t\t\tLinus\n"}]}