{"thread":{"id":"34401","subject":"limit memory usage on large repositories","startedAt":"2013-07-10T22:27:57Z","lastAt":"2013-07-27T03:48:43Z","messageCount":2,"participants":["Matt Schoen","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"223040","messageId":"CAJj9RsTjp7j7Ew2pSttKRAZfZ6fLt9jL+Q_vmHQCi16FBBbK=w@mail.gmail.com","threadId":"34401","inReplyTo":null,"subject":"limit memory usage on large repositories","fromName":"Matt Schoen","fromEmail":"mtschoen@gmail.com","sentAt":"2013-07-10T22:27:57Z","receivedAt":"2013-07-10T22:27:57Z","isPatch":false,"sender":{"key":"mtschoen@gmail.com","avatar":null},"body":"Hi there,\n\nI've been using git for some time now, and host my remote bare\nrepositories on my shared hosting account at Dreamhost.com.  As a\nprotective feature on their shared host setup, they enact a policy\nthat kills processes that consume too much memory.  This happens to\ngit sometimes.\n\nBy \"sometimes\" I mean on large repos (>~500MB), when performing\noperations like git gc and git fsck and, most annoyingly, when doing a\nclone.  It seems to happen in the pack phase, but I can't be sure\nexactly.\n\nI've messed around with the config options like pack.threads and\npack.sizeLimit, and basically anything on the git config manpage that\nmentions memory.  I limit all of these things to 1 or 0 or 1m when\napplicable, just to be sure. To be honest, I really don't know what\nI'm doing ;)\n\nOddly enough, I'm having trouble reproducing my issue with anything\nbut git fsck.  Clones were failing in the past, but after a successful\ngit gc, everything seems to be ok(?)\n\nAnyway, I'd like some advice on what settings limit memory usage, and\nexactly how to determine what the memory usage will be with certain\nvalues.\n\nThanks!\n"},{"id":"224167","messageId":"20130727034843.GA20846@sigill.intra.peff.net","threadId":"34401","inReplyTo":"CAJj9RsTjp7j7Ew2pSttKRAZfZ6fLt9jL+Q_vmHQCi16FBBbK=w@mail.gmail.com","subject":"Re: limit memory usage on large repositories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-07-27T03:48:43Z","receivedAt":"2013-07-27T03:48:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 10, 2013 at 05:27:57PM -0500, Matt Schoen wrote:\n\n> I've been using git for some time now, and host my remote bare\n> repositories on my shared hosting account at Dreamhost.com.  As a\n> protective feature on their shared host setup, they enact a policy\n> that kills processes that consume too much memory.  This happens to\n> git sometimes.\n> \n> By \"sometimes\" I mean on large repos (>~500MB), when performing\n> operations like git gc and git fsck and, most annoyingly, when doing a\n> clone.  It seems to happen in the pack phase, but I can't be sure\n> exactly.\n\nDo you know how they measure the memory? One of the problems we've had\nat GitHub in measuring git's memory usage is that git will mmap the\nfairly large packfiles. This can bloat the RSS of the git process. At\nthe same time, not counting the map is not quite right, either; it is\nmemory the process is using, but it could stand to give up some of it if\nother processes needed it (and that giving up is managed by the kernel,\nnot by git). So you end up in a situation where you may have a large RSS\nprecisely _because_ there is no memory pressure on the system, which\nleaves the kernel free to leave the mmap'd pages in RAM.\n\nYou can reduce the amount of memory you map at once with\ncore.packedGitWindowSize.\n\n> I've messed around with the config options like pack.threads and\n> pack.sizeLimit, and basically anything on the git config manpage that\n> mentions memory.  I limit all of these things to 1 or 0 or 1m when\n> applicable, just to be sure. To be honest, I really don't know what\n> I'm doing ;)\n\nI assume you did pack.deltaCacheSize, which can take a fair bit of\nmemory during packing (or cloning).\n\nPacking itself takes up a lot, as I think we keep the whole window's\nworth of objects in memory at one time (so 10 by default). If you have\nlarge objects, that can spike your memory usage for a moment as we keep\nseveral versions of the large object in memory at once.\n\nIf you have such large objects that don't delta well, you can use the\n\"nodelta\" gitattribute so that git doesn't even try them.\n\n> Oddly enough, I'm having trouble reproducing my issue with anything\n> but git fsck.  Clones were failing in the past, but after a successful\n> git gc, everything seems to be ok(?)\n\nMemory usage for clone should improve after a gc, as we will mostly be\nreusing deltas from disk instead of trying to find new ones between\npacks.\n\n-Peff\n"}]}