{"thread":{"id":"32194","subject":"Multiple threads of compression","startedAt":"2012-11-25T16:27:32Z","lastAt":"2012-11-26T09:59:55Z","messageCount":4,"participants":["Thorsten Glaser","Brandon Casey","Sebastian Leske"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"203815","messageId":"loom.20121125T171702-64@post.gmane.org","threadId":"32194","inReplyTo":null,"subject":"Multiple threads of compression","fromName":"Thorsten Glaser","fromEmail":"tg@debian.org","sentAt":"2012-11-25T16:27:32Z","receivedAt":"2012-11-25T16:27:32Z","isPatch":false,"sender":{"key":"tg@debian.org","avatar":"https://avatars.githubusercontent.com/u/861078?v=4"},"body":"Hi,\n\nI’m asking here informally first, because my information relates\nto a quite old version (the one from lenny-backports). A tl;dr\nis at the end.\n\nOn a multi-core machine, the garbage collection of git, as well\nas pack compression on the server side when someone clones a\nrepository remotely, the compression is normally done automatically\nusing multiple threads of execution.\n\nThat may be fine for your typical setups, but in my cases, I have\ntwo scenarios where it isn’t:\n\nⓐ The machine where I want it to use only, say, 2 of my 4 or 8 cores\n  as I’m also running some VMs on the box which eat up a lot of CPU\n  and which I don’t want to slow down.\n\nⓑ The server VM which has been given 2 or 3 VCPUs to cope with all\n  the load done by clients, but which is RAM-constrained to only\n  512 or, when lucky, 768 MiB. It previously served only http/https\n  and *yuk* Subversion, but now, git comes into the play, and I’ve\n  seen the one server box I think about go down *HARD* because git\n  ate up all RAM *and* swap when someone wanted to update their clone\n  of a repository after someone else committed… well, an ~100 MiB large\n  binary file they shouldn’t. (It required manual intervention on the\n  server to kill that revision and then the objects coupled with it,\n  but even *that* didn’t work, read on for more.)\n\nIn both cases, I had to apply a quick hack. One I can reproduce\nby now is, that, on the first box, I added a --threads=2 to the\nline calling git pack-objects in /usr/lib/git-core/git-repack,\nlike this:\n\n   83 args=\"$args $local ${GIT_QUIET:+-q} $no_reuse$extra\"\n   84 names=$(git pack-objects --threads=2 --keep-true-parents --honor-pack-\nkeep --non-empty --all --reflog $arg\n   85         exit 1\n\n(By the way, wrapping source code at 80c is still way to go IMHO.)\n\nOn the second box, IIRC I added --threads=1, but that box got\nsubsequently upgraded from lenny to wheezy so any local modification\nis lost (luckily, the problem didn’t occur again recently, or at\nleast I didn’t notice it, save for the VM load going up to 6-8\nseveral times a day).\n\ntl;dr: I would like to have a *global* option for git to restrict\nthe number of threads of execution it uses. Several subcommands,\nlike pack-objects, are already equipped with an optioin for this,\nbut unfortunately, these are seldom invoked by hand¹, so this can’t\nwork in my situations.\n\n① automatic garbage collection, “git gc --aggressive --prune=now”,\n  and cloning are the use cases I have at hand right now.\n\nÀ propos, while here: is gc --aggressive safe to run on a live,\nonline-shared repository, or does it break other users accessing\nthe repository concurrently? (If it’s safe I’d very much like to do\nthat in a, say weekly, cronjob on FusionForge, our hosting system.)\n\nThanks in advance!\n//mirabilos\n"},{"id":"203843","messageId":"CA+sFfMerMWnJwiBz0=MxO0gn8B2M8g12mt5VzZaRj591oMPVow@mail.gmail.com","threadId":"32194","inReplyTo":"loom.20121125T171702-64@post.gmane.org","subject":"Re: Multiple threads of compression","fromName":"Brandon Casey","fromEmail":"drafnel@gmail.com","sentAt":"2012-11-26T02:37:02Z","receivedAt":"2012-11-26T02:37:02Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"On Sun, Nov 25, 2012 at 8:27 AM, Thorsten Glaser <tg@debian.org> wrote:\n\n> tl;dr: I would like to have a *global* option for git to restrict\n> the number of threads of execution it uses. Several subcommands,\n> like pack-objects, are already equipped with an optioin for this,\n> but unfortunately, these are seldom invoked by hand¹, so this can’t\n> work in my situations.\n\nSee the git-config man page.\n\nThe number of threads that pack uses can be configured in the global\nor system gitconfig file by setting pack.threads.\n\ne.g.\n\n   $ git config --system pack.threads 1\n\nAlso, modern git accepts a '-c' option which allows you to set\nconfiguration options on the command line, e.g. 'git -c pack.threads=1\ngc'.\n\nThe other setting you should probably look at is pack.windowMemory\nwhich should help you control the amount of memory git uses while\npacking.  Also look at core.packedGitWindowSize and\ncore.packedGitLimit if your repository is really large.\n\n>\n> ① automatic garbage collection, “git gc --aggressive --prune=now”,\n>   and cloning are the use cases I have at hand right now.\n>\n> À propos, while here: is gc --aggressive safe to run on a live,\n> online-shared repository, or does it break other users accessing\n> the repository concurrently? (If it’s safe I’d very much like to do\n> that in a, say weekly, cronjob on FusionForge, our hosting system.)\n\nRunning 'git gc' with --aggressive should be as safe as running it\nwithout --aggressive.\n\nBut, you should think about whether you really need to run it more\nthan once, or at all.  When you use --aggressive, git will perform the\nentire delta search again for _every_ object in the repository.  The\ngeneral usecase for --aggressive is when you suspect that the original\ndelta search produced sub-optimal deltas or if you modify the size of\nthe delta search window or depth and want to regenerate your packed\nobjects to improve compression or access speed.  Even then, you will\nnot likely get much benefit from running with --aggressive a second\ntime.\n\nhth,\n-Brandon\n"},{"id":"203944","messageId":"20121126072347.GA25752@localhost","threadId":"32194","inReplyTo":"loom.20121125T171702-64@post.gmane.org","subject":"Re: Multiple threads of compression","fromName":"Sebastian Leske","fromEmail":"sebastian.leske@sleske.name","sentAt":"2012-11-26T07:23:47Z","receivedAt":"2012-11-26T07:23:47Z","isPatch":false,"sender":{"key":"sebastian.leske@sleske.name","avatar":"https://avatars.githubusercontent.com/u/1101993?v=4"},"body":"Hi,\n\n[Thorsten Glaser <tg@debian.org>, 2012-11-25 17:27]:\n> On a multi-core machine, the garbage collection of git, as well\n> as pack compression on the server side when someone clones a\n> repository remotely, the compression is normally done automatically\n> using multiple threads of execution.\n> \n> That may be fine for your typical setups, but in my cases, I have\n> two scenarios where it isn’t:\n> \n> ⓐ The machine where I want it to use only, say, 2 of my 4 or 8 cores\n>   as I’m also running some VMs on the box which eat up a lot of CPU\n>   and which I don’t want to slow down.\n \n> ⓑ The server VM which has been given 2 or 3 VCPUs to cope with all\n>   the load done by clients, but which is RAM-constrained to only\n>   512 or, when lucky, 768 MiB. It previously served only http/https\n>   and *yuk* Subversion, but now, git comes into the play, and I’ve\n>   seen the one server box I think about go down *HARD* because git\n>   ate up all RAM *and* swap when someone wanted to update their clone\n>   of a repository after someone else committed… well, an ~100 MiB large\n>   binary file they shouldn’t.\n\nunfortunately I can't really speak to the git side of things, but both\nof these cases just sound like standard resource starvation. So why\ndon't you address them using the usual OS mechanisms? \n\nIf you want to prevent git from sucking up CPU, nice(1) it, and if it\neats too much RAM, use the parent shell's ulimit mechanism.\n\nGranted, this might also require some changes to git, but wouldn't that\nbe a simpler and more general approach to solving starvation problems?\n"},{"id":"203863","messageId":"Pine.BSM.4.64L.1211260956460.27253@herc.mirbsd.org","threadId":"32194","inReplyTo":"CA+sFfMerMWnJwiBz0=MxO0gn8B2M8g12mt5VzZaRj591oMPVow@mail.gmail.com","subject":"Re: Multiple threads of compression","fromName":"Thorsten Glaser","fromEmail":"tg@debian.org","sentAt":"2012-11-26T09:59:55Z","receivedAt":"2012-11-26T09:59:55Z","isPatch":false,"sender":{"key":"tg@debian.org","avatar":"https://avatars.githubusercontent.com/u/861078?v=4"},"body":"Brandon Casey dixit:\n\n>The number of threads that pack uses can be configured in the global\n>or system gitconfig file by setting pack.threads.\n[…]\n>The other setting you should probably look at is pack.windowMemory\n>which should help you control the amount of memory git uses while\n>packing.  Also look at core.packedGitWindowSize and\n>core.packedGitLimit if your repository is really large.\n\nOK, thanks a lot!\n\nI can’t really say much about the repositories beforehand\nbecause it’s a generic code hosting platform, several instances\nof which we run at my employer’s place (I also run one privately\nnow), and which is also run by e.g. Debian. But I’ll try to figure\nout some somewhat sensible defaults.\n\n>Running 'git gc' with --aggressive should be as safe as running it\n>without --aggressive.\n\nOK, thanks.\n\n>But, you should think about whether you really need to run it more\n>than once, or at all.  When you use --aggressive, git will perform the\n[…]\n\nGreat explanation!\n\nI think that I’d want to run it once, after the repository has\nbeen begun to be used (probably not correct English but you know\nwhat I want to say), but have to figure out a way to do so… but\nI’ll just leave out the --aggressive from the cronjob then.\n\nMuch appreciated,\n//mirabilos\n-- \nSometimes they [people] care too much: pretty printers [and syntax highligh-\nting, d.A.] mechanically produce pretty output that accentuates irrelevant\ndetail in the program, which is as sensible as putting all the prepositions\nin English text in bold font.\t-- Rob Pike in \"Notes on Programming in C\"\n"}]}