{"thread":{"id":"55272","subject":"Performance of \"git gc...\" is extremely bad in some cases","startedAt":"2021-03-08T21:31:59Z","lastAt":"2021-03-09T00:14:43Z","messageCount":5,"participants":["Anthony Muller","Bryan Turner","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"418537","messageId":"17813b232e9.e48d03c3862272.7793967418558853913@monospace.sh","threadId":"55272","inReplyTo":null,"subject":"Performance of \"git gc...\" is extremely bad in some cases","fromName":"Anthony Muller","fromEmail":"anthony@monospace.sh","sentAt":"2021-03-08T21:15:48Z","receivedAt":"2021-03-08T21:31:59Z","isPatch":false,"sender":{"key":"anthony@monospace.sh","avatar":null},"body":"What did you do before the bug happened? (Steps to reproduce your issue)\n\ngit clone https://github.com/notracking/hosts-blocklists\ncd hosts-blocklists\ngit reflog expire --all --expire=now && git gc --prune=now --aggressive\n\n\nWhat did you expect to happen? (Expected behavior)\n\nRunning gc on a ~300 MB repo should not take 1 hour 55 minutes when\nrunning gc on a 2.6 GB repo (LLVM) only takes 24 minutes.\n\n\nWhat happened instead? (Actual behavior)\n\nCommand took 1h 55m to complete on a ~300MB repo and used enough\nresources that the machine is almost unusable.\n\n\nWhat's different between what you expected and what actually happened?\n\nCompression stage uses the majority of the resources and time. Compression\nitself, when compared to something like zlib or lzma, should not take very long.\nWhile more may be happening as objects are compressed, the amount of time\ngc takes to compress the objects and the resources it consumed are both\nunreasonable.\n\nMemory: RSS = 3451152 KB (3.29 GB), VSZ = 29286272 KB (27.92 GB)\nTime: 12902.83s user 8995.41s system 315% cpu 1:55:36.73 total\n\nI've seen this issue with a number of repos and size of the repo does not\ndetermine if this happens. LLVM @ 2.6 GB worked flawlessly, a 900 MB\nrepo never finished, this 300 MB repo takes forever, and if you test something\nlike chromium git will just crash.\n\n\n[System Info]\nhardware: 2.9Ghz Quad Core i7\ngit version:\ngit version 2.30.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64\ncompiler info: clang: 12.0.0 (clang-1200.0.32.28)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /usr/local/bin/zsh\n\n"},{"id":"418544","messageId":"CAGyf7-F6jbs-HQeCSMjf_y8Y=5ZfME=CjBagAfKUbnP_0vDXqA@mail.gmail.com","threadId":"55272","inReplyTo":"17813b232e9.e48d03c3862272.7793967418558853913@monospace.sh","subject":"Re: Performance of \"git gc...\" is extremely bad in some cases","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2021-03-08T22:29:16Z","receivedAt":"2021-03-08T22:30:31Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller wrote:\n>\n> What did you do before the bug happened? (Steps to reproduce your issue)\n>\n> git clone https://github.com/notracking/hosts-blocklists\n> cd hosts-blocklists\n> git reflog expire --all --expire=now && git gc --prune=now --aggressive\n\n--aggressive tells git gc to discard all of its existing delta chains\nand go find new ones, and to be fairly aggressive in how it looks for\ncandidates. This is going to be the primary source of the resource\nusage you see, as well as the time.\n\nAggressive GCs are something you do once in a (very great) while. If\nyou try this without the --aggressive, how does it look?\n\n>\n>\n> What did you expect to happen? (Expected behavior)\n>\n> Running gc on a ~300 MB repo should not take 1 hour 55 minutes when\n> running gc on a 2.6 GB repo (LLVM) only takes 24 minutes.\n>\n>\n> What happened instead? (Actual behavior)\n>\n> Command took 1h 55m to complete on a ~300MB repo and used enough\n> resources that the machine is almost unusable.\n>\n>\n> What's different between what you expected and what actually happened?\n>\n> Compression stage uses the majority of the resources and time. Compression\n> itself, when compared to something like zlib or lzma, should not take very long.\n> While more may be happening as objects are compressed, the amount of time\n> gc takes to compress the objects and the resources it consumed are both\n> unreasonable.\n\nThe compression happening here is delta compression, not simple\ncompression like zip. Git searches across the repository for similar\nobjects and stores them as chains with a base object and (essentially)\ninstructions for converting that base object into another object.\nThat's significantly more resource-intensive work than zipping some\ndata.\n\n>\n> Memory: RSS = 3451152 KB (3.29 GB), VSZ = 29286272 KB (27.92 GB)\n> Time: 12902.83s user 8995.41s system 315% cpu 1:55:36.73 total\n\nGit offers several knobs that can be used to influence (though not\nnecessarily control) its resource usage. On 64-bit Linux the defaults\nare 1 thread per logical CPU (so hyperthreaded CPUs use double) and\n_unlimited_ memory usage per thread. You might want to investigate\nsome options like pack.threads and pack.windowmemory to apply some\nconstraints.\n\n>\n> I've seen this issue with a number of repos and size of the repo does not\n> determine if this happens. LLVM @ 2.6 GB worked flawlessly, a 900 MB\n> repo never finished, this 300 MB repo takes forever, and if you test something\n> like chromium git will just crash.\n>\n>\n> [System Info]\n> hardware: 2.9Ghz Quad Core i7\n> git version:\n> git version 2.30.0\n> cpu: x86_64\n> no commit associated with this build\n> sizeof-long: 8\n> sizeof-size_t: 8\n> shell-path: /bin/sh\n> uname: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64\n> compiler info: clang: 12.0.0 (clang-1200.0.32.28)\n> libc info: no libc information available\n> $SHELL (typically, interactive shell): /usr/local/bin/zsh\n>\n\nHope this helps!\n-b\n"},{"id":"418550","messageId":"CAGyf7-EO3EGgjO5H_8ZXodYraxweA0ez2nm2Zs1BJTGgK-ScKg@mail.gmail.com","threadId":"55272","inReplyTo":"178140c3b3b.c7a29306868075.2037370475662478386@monospace.sh","subject":"Re: Performance of \"git gc...\" is extremely bad in some cases","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2021-03-08T23:55:24Z","receivedAt":"2021-03-08T23:56:45Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"Re-adding the list.\n\nOn Mon, Mar 8, 2021 at 2:54 PM Anthony Muller wrote:\n>\n> ---- On Mon, 08 Mar 2021 22:29:16 +0000 Bryan Turner wrote ----\n> > On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller wrote:\n> > >\n> > > What did you do before the bug happened? (Steps to reproduce your issue)\n> > >\n> > > git clone https://github.com/notracking/hosts-blocklists\n> > > cd hosts-blocklists\n> > > git reflog expire --all --expire=now && git gc --prune=now --aggressive\n> >\n> > --aggressive tells git gc to discard all of its existing delta chains\n> > and go find new ones, and to be fairly aggressive in how it looks for\n> > candidates. This is going to be the primary source of the resource\n> > usage you see, as well as the time.\n> >\n> > Aggressive GCs are something you do once in a (very great) while. If\n> > you try this without the --aggressive, how does it look?\n>\n> Hi Bryan,\n>\n> Without --aggressive it's fine and I do expect it to take longer using aggressive.\n>\n> I find it very odd that a repo ~8x in size and with probably 400x as many objects took 1/4 the time though. I would think size and object count would play a role in time and resources.\n\nLooking at that blocklists repository, it looks like it's not many\nfiles or commits, but the files are pretty large (10-25MB). For delta\ncompression, large files can cause a lot of pain.\n\nIf you set core.bigFileThreshold=5m (a reduction from 512m by default)\nand pack.windowmemory=1g, for me locally, at least, \"fixes\" the\n\"problem\" (which is to say it changes the behavior). The GC runs in\nunder 10 minutes:\n$ /usr/bin/time -l git gc --prune=now --aggressive\nEnumerating objects: 10777, done.\nCounting objects: 100% (10777/10777), done.\nDelta compression using up to 20 threads\nCompressing objects: 100% (8672/8672), done.\nWriting objects: 100% (10777/10777), done.\nReusing bitmaps: 101, done.\nSelecting bitmap commits: 2146, done.\nBuilding bitmaps: 100% (126/126), done.\nTotal 10777 (delta 3986), reused 6784 (delta 0)\n 298.00 real 996.76 user 18.84 sys\n 9284980736 maximum resident set size\n 0 average shared memory size\n 0 average unshared data size\n 0 average unshared stack size\n 2861811 page reclaims\n 1 page faults\n 0 swaps\n 0 block input operations\n 0 block output operations\n 0 messages sent\n 0 messages received\n 296 signals received\n 172 voluntary context switches\n 171245 involuntary context switches\n 20586171 instructions retired\n 28100595 cycles elapsed\n 880640 peak memory footprint\n\nOf course, that also takes the size of the repository from 367MB to\n2.3GB--not exactly your desired outcome if you're trying to save\nspace.\n\nFrom there I tried just reducing the threads from 20 to 8 and using\nthe 1g window memory limit, but leaving the bigFileThreshold at\ndefault. That allows for delta compressing everything, and for me\ncompletes in just under 12 minutes:\n$ /usr/bin/time -l git gc --prune=now --aggressive\nEnumerating objects: 10777, done.\nCounting objects: 100% (10777/10777), done.\nDelta compression using up to 8 threads\nCompressing objects: 100% (10077/10077), done.\nWriting objects: 100% (10777/10777), done.\nReusing bitmaps: 101, done.\nSelecting bitmap commits: 2146, done.\nBuilding bitmaps: 100% (126/126), done.\nTotal 10777 (delta 5387), reused 5383 (delta 0)\n 713.98 real 3053.41 user 31.91 sys\n 13408837632 maximum resident set size\n 0 average shared memory size\n 0 average unshared data size\n 0 average unshared stack size\n 3804319 page reclaims\n 1 page faults\n 0 swaps\n 0 block input operations\n 0 block output operations\n 0 messages sent\n 0 messages received\n 712 signals received\n 57 voluntary context switches\n 1011681 involuntary context switches\n 20568579 instructions retired\n 31734809 cycles elapsed\n 872448 peak memory footprint\n\nThat also reduced the repository from 367MB to 320MB. (Technically\nfrom 2.3GB to 320MB, since I this after the earlier attempt.)\n\nOf course, there's a machine difference to consider here as well. I'm\nguessing you're on a MacBook Pro, based on the specs part of the bug\nreport. My testing here is on a 10 core iMac Pro with 64GB of RAM, so\nsome of the difference may just be that I'm on a less constrained\nsystem.\n\n>\n> What factors would make that happen? Is it a combination of more commits with fewer objects?\n\nBig files are the biggest issue, in my experience. The total number of\nobjects (it's not really about object type too much, as far as I can\ntell) certainly has an impact, but having big files (where \"big\" here\nis anything larger than a normal source code file, which is typically\nwell under 1MB) is likely to balloon both time and resource\nconsumption.\n\n>\n> I've been using aggressive after cloning repos I use primarily for reference/offline/etc to recover a lot of wasted space.\n\nTo some extent I'm not sure there's an easy answer, for this. It may\ncome down to looking at the repositories before you do a local GC to\nsee what \"shape\" they have (starting size on disk, in-repository file\nsizes, etc.) and deciding from there whether the savings is likely to\nbe worth the time investment.\n\n>\n> >\n> > >\n> > >\n> > > What did you expect to happen? (Expected behavior)\n> > >\n> > > Running gc on a ~300 MB repo should not take 1 hour 55 minutes when\n> > > running gc on a 2.6 GB repo (LLVM) only takes 24 minutes.\n> > >\n> > >\n> > > What happened instead? (Actual behavior)\n> > >\n> > > Command took 1h 55m to complete on a ~300MB repo and used enough\n> > > resources that the machine is almost unusable.\n> > >\n> > >\n> > > What's different between what you expected and what actually happened?\n> > >\n> > > Compression stage uses the majority of the resources and time. Compression\n> > > itself, when compared to something like zlib or lzma, should not take very long.\n> > > While more may be happening as objects are compressed, the amount of time\n> > > gc takes to compress the objects and the resources it consumed are both\n> > > unreasonable.\n> >\n> > The compression happening here is delta compression, not simple\n> > compression like zip. Git searches across the repository for similar\n> > objects and stores them as chains with a base object and (essentially)\n> > instructions for converting that base object into another object.\n> > That's significantly more resource-intensive work than zipping some\n> > data.\n> >\n> > >\n> > > Memory: RSS = 3451152 KB (3.29 GB), VSZ = 29286272 KB (27.92 GB)\n> > > Time: 12902.83s user 8995.41s system 315% cpu 1:55:36.73 total\n> >\n> > Git offers several knobs that can be used to influence (though not\n> > necessarily control) its resource usage. On 64-bit Linux the defaults\n> > are 1 thread per logical CPU (so hyperthreaded CPUs use double) and\n> > _unlimited_ memory usage per thread. You might want to investigate\n> > some options like pack.threads and pack.windowmemory to apply some\n> > constraints.\n> >\n> > >\n> > > I've seen this issue with a number of repos and size of the repo does not\n> > > determine if this happens. LLVM @ 2.6 GB worked flawlessly, a 900 MB\n> > > repo never finished, this 300 MB repo takes forever, and if you test something\n> > > like chromium git will just crash.\n\nI should add that for something like Chromium, and potentially\nwhatever 900MB repository you tested with, you're very likely to need\nto do some explicit configuration for things like threads/window\nmemory unless you're on a _very_ beefy machine. The default unlimited\nbehavior is very likely to run afoul of the OOM killer (or something\nsimilar).\n\n> > >\n> > >\n> > > [System Info]\n> > > hardware: 2.9Ghz Quad Core i7\n> > > git version:\n> > > git version 2.30.0\n> > > cpu: x86_64\n> > > no commit associated with this build\n> > > sizeof-long: 8\n> > > sizeof-size_t: 8\n> > > shell-path: /bin/sh\n> > > uname: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64\n> > > compiler info: clang: 12.0.0 (clang-1200.0.32.28)\n> > > libc info: no libc information available\n> > > $SHELL (typically, interactive shell): /usr/local/bin/zsh\n> > >\n> >\n\nHope this helps!\n-b\n"},{"id":"418551","messageId":"YEa5xe0gNDh2wZLB@camp.crustytoothpaste.net","threadId":"55272","inReplyTo":"CAGyf7-F6jbs-HQeCSMjf_y8Y=5ZfME=CjBagAfKUbnP_0vDXqA@mail.gmail.com","subject":"Re: Performance of \"git gc...\" is extremely bad in some cases","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-03-08T23:56:53Z","receivedAt":"2021-03-08T23:57:48Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-03-08 at 22:29:16, Bryan Turner wrote:\n> On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller wrote:\n> >\n> > What did you do before the bug happened? (Steps to reproduce your issue)\n> >\n> > git clone https://github.com/notracking/hosts-blocklists\n> > cd hosts-blocklists\n> > git reflog expire --all --expire=now && git gc --prune=now --aggressive\n> \n> --aggressive tells git gc to discard all of its existing delta chains\n> and go find new ones, and to be fairly aggressive in how it looks for\n> candidates. This is going to be the primary source of the resource\n> usage you see, as well as the time.\n> \n> Aggressive GCs are something you do once in a (very great) while. If\n> you try this without the --aggressive, how does it look?\n\nI should point out that this repository is also rather pathologically\nstructured. Almost every commit is an automatic commit updating the\nsame five files which are text files ranging from 5 MB to 11 MB.\n\nWhen you use --aggressive, as Bryan pointed out, you're asking to throw\naway all the deltas and try really hard to compute all of them fresh.\nThat's going to use a lot of memory because you're loading many large\ntext files into memory. It's also going to use a lot of CPU because\nthese files do indeed delta extremely well, and since computing deltas\non larger files is more expensive, especially when there are many of\nthem.\n\nAnd that's just the blobs. The trees and commits are also going to be\nnearly identically structured and will also delta well with virtually\nevery other similar object of their type. Normally Git sorts by size\nwhich helps pick better candidates, but since these are all going to be\nidentically sized, the performance is going to suffer.\n\nNow, I have the advantage in this case of being a person who's sometimes\non call for the maintenance of Git repositories and in that capacity,\nthat this is pathologically structured is obvious to me. But, yeah, I\nwould definitely not run --aggressive on this repo unless I needed to\nand I would not expect it to perform well.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"418562","messageId":"17814555aec.b8e46da8884253.2263161421793744939@monospace.sh","threadId":"55272","inReplyTo":"YEa5xe0gNDh2wZLB@camp.crustytoothpaste.net","subject":"Re: Performance of \"git gc...\" is extremely bad in some cases","fromName":"Anthony Muller","fromEmail":"anthony@monospace.sh","sentAt":"2021-03-09T00:14:01Z","receivedAt":"2021-03-09T00:14:43Z","isPatch":false,"sender":{"key":"anthony@monospace.sh","avatar":null},"body":"Thank you Brian and Bryan. You both clarified what was happening and now I know what to look for.\n\nI can use a shallow clone for most repos, but there are some I want to keep history for. I don't need a full copy of this repo, but it was a good repo to show the issue I was facing.\n\nThanks again!\n\n\n ---- On Mon, 08 Mar 2021 23:56:53 +0000 brian m. carlson wrote ----\n > On 2021-03-08 at 22:29:16, Bryan Turner wrote:\n > > On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller wrote:\n > > >\n > > > What did you do before the bug happened? (Steps to reproduce your issue)\n > > >\n > > > git clone https://github.com/notracking/hosts-blocklists\n > > > cd hosts-blocklists\n > > > git reflog expire --all --expire=now && git gc --prune=now --aggressive\n > > \n > > --aggressive tells git gc to discard all of its existing delta chains\n > > and go find new ones, and to be fairly aggressive in how it looks for\n > > candidates. This is going to be the primary source of the resource\n > > usage you see, as well as the time.\n > > \n > > Aggressive GCs are something you do once in a (very great) while. If\n > > you try this without the --aggressive, how does it look?\n > \n > I should point out that this repository is also rather pathologically\n > structured. Almost every commit is an automatic commit updating the\n > same five files which are text files ranging from 5 MB to 11 MB.\n > \n > When you use --aggressive, as Bryan pointed out, you're asking to throw\n > away all the deltas and try really hard to compute all of them fresh.\n > That's going to use a lot of memory because you're loading many large\n > text files into memory. It's also going to use a lot of CPU because\n > these files do indeed delta extremely well, and since computing deltas\n > on larger files is more expensive, especially when there are many of\n > them.\n > \n > And that's just the blobs. The trees and commits are also going to be\n > nearly identically structured and will also delta well with virtually\n > every other similar object of their type. Normally Git sorts by size\n > which helps pick better candidates, but since these are all going to be\n > identically sized, the performance is going to suffer.\n > \n > Now, I have the advantage in this case of being a person who's sometimes\n > on call for the maintenance of Git repositories and in that capacity,\n > that this is pathologically structured is obvious to me. But, yeah, I\n > would definitely not run --aggressive on this repo unless I needed to\n > and I would not expect it to perform well.\n > -- \n > brian m. carlson (he/him or they/them)\n > Houston, Texas, US\n > \n"}]}