{"thread":{"id":"43949","subject":"Reducing CPU load on git server","startedAt":"2016-08-28T19:43:31Z","lastAt":"2016-08-31T07:28:47Z","messageCount":16,"participants":["W. David Jarvis","Jakub Narębski","Jeff King","Ævar Arnfjörð Bjarmason","Dennis Kaarsemaker"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"300373","messageId":"CAFMAO9y3LsrAb_jp8XVq2mexaA4bBqmWFwJu55r4S6Dxd2-zxw@mail.gmail.com","threadId":"43949","inReplyTo":null,"subject":"Reducing CPU load on git server","fromName":"W. David Jarvis","fromEmail":"william.d.jarvis@gmail.com","sentAt":"2016-08-28T19:42:52Z","receivedAt":"2016-08-28T19:43:31Z","isPatch":false,"sender":{"key":"william.d.jarvis@gmail.com","avatar":"https://gravatar.com/avatar/4ed36a1dd09f36a54236d394eb73566ebb6ba4aba71182b45cbf585559310e8a?d=mp&s=160"},"body":"Hi all -\n\nI've run into a problem that I'm looking for some help with. Let me\ndescribe the situation, and then some thoughts.\n\nThe company I work for uses git. We use GitHub Enterprise as a\nfrontend for our primary git server. We're using Chef solo to manage a\nfleet of upwards of 10,000 hosts, which regularly pull from our Chef\ngit repository so as to converge.\n\nA few years ago, in order to reduce the load on the primary server,\nthe firm set up a fleet of replication servers. This allowed the\nmajority of our infrastructure to target the replication servers for\nall pull-based activity rather than the primary git server.\n\nThe actual replication process works as follows:\n\n1. The primary git server receives a push and sends a webhook with the\ndetails of the push (repo, ref, sha, some metadata) to a \"publisher\"\nbox\n\n2. The publisher enqueues the details of the webhook into a queue\n\n3. A fleet of \"subscriber\" (replica) boxes each reads the payload of\nthe enqueued message. Each of these then tries to either clone the\nrepository if they don't already have it, or they run `git fetch`.\n\nDuring the course of either of these operations we use a\nrepository-level lockfile. If another request comes in while the repo\nis locked, we re-enqueue it. When that request comes in later, if the\npush event time is earlier than the most recent successful fetch, we\ndon't do any further work.\n\n---\n\nThe problem we've been running into has been that as we've continued\nto add developers, the CPU load on our primary instance has gone\nthrough the roof. We've upgraded the machine to some of the punchiest\nhardware we can get and it's regularly exceeding 70% CPU load. We know\nthat the overwhelming drive of this CPU load is near-constant git\noperations on some of our larger and more active repositories.\n\nMigrating off of Chef solo is essentially a non-starter at this point\nin time, and we've also added a considerable number of non-Chef git\ndependencies such that getting rid of the replication service would be\na massive undertaking. Even if we were to kill the replication fleet,\nwe'd have to figure out where to point that traffic in such a way that\nwe didn't overwhelm the primary git server anyways.\n\nSo I'm looking for some help in figuring out what to do next. At the\nmoment, I have two thoughts:\n\n1. We currently run a blanket `git fetch` rather than specifically\nfetching the ref that was pushed. My understanding from poking around\nthe git source code is that this causes the replication server to send\na list of all of its ref tips to the primary server, and the primary\nserver then has to verify and compare each of these tips to the ref\ntips residing on the server.\n\nThat might not be a ton of work to do on an individual fetch\noperation, but some of our repositories have over 5,000 branches and\nare pushed to 1,000 times a day. Algorithmically, this would suggest\nthat the cost of a fetch will go up in terms of both N -- branches and\nM -- pushes, so we're talking about a cost of N*M, both of which will\nincrease when we hire developers. This implies exponential growth in\nload as we add engineers, which is...not good.\n\nAlso worth noting - if we assume that the replication server should be\nreasonably up-to-date (see lockfile logic description, above), we're\ntalking about typically packing objects for one ref, and at most a\nsmall number (<5).\n\nMy hypothesis is that moving to fetching the specific branch rather\nthan doing a blanket fetch would have a significant and material\nimpact on server load.\n\nIf we do go down this route, we'll potentially need to do some\nrefactoring around how we handle \"failed fetches\", which relates both\nto our locking logic and to the actual potential failure of a git\nfetch. Discussion of the locking mechanism follows:\n\n2. Our current locking mechanism seems to me to be un-necessary. I'm\naware that git uses a few different locking mechanisms internally, and\nthe use of a repo-level lockfile would seem to only guarantee that\nwe're using coarser-grained locking than git actually supports. But I\ndon't know if git supports specific fetch-level ref locking that would\npermit concurrent ref fetch operations. If it does, our current\narchitecture would seem to prevent git from being able to take\nadvantage of those mechanisms.\n\nIn other words, let's imagine a world in which we ditch our current\nrepo-level locking mechanism entirely. Let's also presume we move to\nfetching specific refs rather than using blanket fetches. Does that\nmean that if a fetch for ref A and a fetch for ref B are issued at\nroughly the exact same time, the two will be able to be executed at\nonce without running into some git-internal locking mechanism on a\ngranularity coarser than the ref? i.e. are fetch A and fetch B going\nto be blocked on the other's completion in any way? (let's presume\nthat ref A and ref B are not parents of each other).\n\n---\n\nI'm neither a contributor nor an expert in git, so my inferences thus\nfar as based purely off of what I would describe as \"stumbling around\nthe source code like a drunken baby\".\n\nThe ultimate goal for us is just figuring out how we can best reduce\nthe CPU load on the primary instance so that we don't find ourselves\nin a situation where we're not able to run basic git operations\nanymore.\n\nIf I'm barking up the wrong tree, or if there are other optimizations\nwe should be considering, I'd be eager to learn about those as well.\nOf course, if what I'm describing sounds about right, I'd like\nconfirmation of that from some people who actually know what they're\ntalking about (i.e., not me :) ).\n\nThanks.\n\n - Venantius\n\n-- \n============\nvenanti.us\n203.918.2328\n============\n"},{"id":"300376","messageId":"51205109-5293-3058-f029-0703c33e8cb9@gmail.com","threadId":"43949","inReplyTo":"CAFMAO9y3LsrAb_jp8XVq2mexaA4bBqmWFwJu55r4S6Dxd2-zxw@mail.gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-08-28T21:20:58Z","receivedAt":"2016-08-28T21:21:14Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 28.08.2016 o 21:42, W. David Jarvis pisze:\n\n> The ultimate goal for us is just figuring out how we can best reduce\n> the CPU load on the primary instance so that we don't find ourselves\n> in a situation where we're not able to run basic git operations\n> anymore.\n\nI assume that you have turned on pack bitmaps?  See for example\n\"Counting Objects\" blog post on GitHub Engineering blog\nhttp://githubengineering.com/counting-objects/\n\nThere are a few other articles there worth reading in your\nsituation.\n-- \nJakub Narębski\n"},{"id":"300379","messageId":"CAFMAO9x5hi=drGSZTrJAxntjN+BATLCa_mSsqYZs+o6BO=x2wA@mail.gmail.com","threadId":"43949","inReplyTo":"51205109-5293-3058-f029-0703c33e8cb9@gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"W. David Jarvis","fromEmail":"william.d.jarvis@gmail.com","sentAt":"2016-08-28T23:18:03Z","receivedAt":"2016-08-28T23:18:40Z","isPatch":false,"sender":{"key":"william.d.jarvis@gmail.com","avatar":"https://gravatar.com/avatar/4ed36a1dd09f36a54236d394eb73566ebb6ba4aba71182b45cbf585559310e8a?d=mp&s=160"},"body":"My assumption is that pack bitmaps are enabled since the primary\nserver is a GitHub Enterprise instance, but I'll have to confirm.\n\nOn Sun, Aug 28, 2016 at 2:20 PM, Jakub Narębski <jnareb@gmail.com> wrote:\n> W dniu 28.08.2016 o 21:42, W. David Jarvis pisze:\n>\n>> The ultimate goal for us is just figuring out how we can best reduce\n>> the CPU load on the primary instance so that we don't find ourselves\n>> in a situation where we're not able to run basic git operations\n>> anymore.\n>\n> I assume that you have turned on pack bitmaps?  See for example\n> \"Counting Objects\" blog post on GitHub Engineering blog\n> http://githubengineering.com/counting-objects/\n>\n> There are a few other articles there worth reading in your\n> situation.\n> --\n> Jakub Narębski\n\n\n\n-- \n============\nvenanti.us\n203.918.2328\n============\n"},{"id":"300406","messageId":"20160829054725.r6pqf3xlusxi7ibq@sigill.intra.peff.net","threadId":"43949","inReplyTo":"CAFMAO9y3LsrAb_jp8XVq2mexaA4bBqmWFwJu55r4S6Dxd2-zxw@mail.gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-29T05:47:26Z","receivedAt":"2016-08-29T05:47:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Aug 28, 2016 at 12:42:52PM -0700, W. David Jarvis wrote:\n\n> The actual replication process works as follows:\n> \n> 1. The primary git server receives a push and sends a webhook with the\n> details of the push (repo, ref, sha, some metadata) to a \"publisher\"\n> box\n> \n> 2. The publisher enqueues the details of the webhook into a queue\n> \n> 3. A fleet of \"subscriber\" (replica) boxes each reads the payload of\n> the enqueued message. Each of these then tries to either clone the\n> repository if they don't already have it, or they run `git fetch`.\n\nSo your load is probably really spiky, as you get thundering herds of\nfetchers after every push (the spikes may have a long flatline at the\ntop, as it takes time to process the whole herd).\n\n> 1. We currently run a blanket `git fetch` rather than specifically\n> fetching the ref that was pushed. My understanding from poking around\n> the git source code is that this causes the replication server to send\n> a list of all of its ref tips to the primary server, and the primary\n> server then has to verify and compare each of these tips to the ref\n> tips residing on the server.\n\nYes, though I'd be surprised if this negotiation is that expensive in\npractice. In my experience it's not generally, and even if we ended up\ntraversing every commit in the repository, that's on the order of a few\nseconds even for large, active repositories.\n\nIn my experience, the problem in a mass-fetch like this ends up being\npack-objects preparing the packfile. It has to do a similar traversal,\nbut _also_ look at all of the trees and blobs reachable from there, and\nthen search for likely delta-compression candidates.\n\nDo you know which processes are generating the load? git-upload-pack\ndoes the negotiation, and then pack-objects does the actual packing.\n\n> My hypothesis is that moving to fetching the specific branch rather\n> than doing a blanket fetch would have a significant and material\n> impact on server load.\n\nMaybe. If pack-objects is where your load is coming from, then\ncounter-intuitively things sometimes get _worse_ as you fetch less. The\nproblem is that git will generally re-use deltas it has on disk when\nsending to the clients. But if the clients are missing some of the\nobjects (because they don't fetch all of the branches), then we cannot\nuse those deltas and may need to recompute new ones. So you might see\nsome parts of the fetch get cheaper (negotiation, pack-object's\n\"Counting objects\" phase), but \"Compressing objects\" gets more\nexpensive.\n\nThis is particularly noticeable with shallow fetches, which in my\nexperience are much more expensive to serve.\n\nJakub mentioned bitmaps, and if you are using GitHub Enterprise, they\nare enabled. But they won't really help here. They are essentially\ncached information that git generates at repack time. But if we _just_\ngot a push, then the new objects to fetch won't be part of the cache,\nand we'll fall back to traversing them as normal.  On the other hand,\nthis should be a relatively small bit of history to traverse, so I'd\ndoubt that \"Counting objects\" is that expensive in your case (but you\nshould be able to get a rough sense by watching the progress meter\nduring a fetch).\n\nI'd suspect more that delta compression is expensive (we know we just\ngot some new objects, but we don't know if we can make good deltas\nagainst the objects the client already has). That's a gut feeling,\nthough.\n\nIf the fetch is small, that _also_ shouldn't be too expensive. But\nthings add up when you have a large number of machines all making the\nsame request at once. So it's entirely possible that the machine just\ngets hit with a lot of 5s CPU tasks all at once. If you only have a\ncouple cores, that takes many multiples of 5s to clear out.\n\nThere's nothing in upstream git to help smooth these loads, but since\nyou mentioned GitHub Enterprise, I happen to know that it does have a\nsystem for coalescing multiple fetches into a single pack-objects. I\n_think_ it's in GHE 2.5, so you might check which version you're\nrunning (and possibly also talk to GitHub Support, who might have more\nadvice; there are also tools for finding out which git processes are\ngenerating the most load, etc).\n\n> In other words, let's imagine a world in which we ditch our current\n> repo-level locking mechanism entirely. Let's also presume we move to\n> fetching specific refs rather than using blanket fetches. Does that\n> mean that if a fetch for ref A and a fetch for ref B are issued at\n> roughly the exact same time, the two will be able to be executed at\n> once without running into some git-internal locking mechanism on a\n> granularity coarser than the ref? i.e. are fetch A and fetch B going\n> to be blocked on the other's completion in any way? (let's presume\n> that ref A and ref B are not parents of each other).\n\nGenerally no, they should not conflict. Writes into the object database\ncan happen simultaneously. Ref updates take a per-ref lock, so you\nshould generally be able to write two unrelated refs at once. The big\nexception is that ref deletion required taking a repo-wide lock, but\nthat presumably wouldn't be a problem for your case.\n\nI'm still not convinced that the single-ref fetching will really help,\nthough.\n\n> The ultimate goal for us is just figuring out how we can best reduce\n> the CPU load on the primary instance so that we don't find ourselves\n> in a situation where we're not able to run basic git operations\n> anymore.\n\nI suspect there's room for improvement and tuning of the primary. But\nbarring that, one option would be to have a hierarchy of replicas. Have\n\"k\" first-tier replicas fetch from the primary, then \"k\" second-tier\nreplicas fetch from them, and so on. Trade propagation delay for\ndistributing the load. :)\n\n-Peff\n"},{"id":"300444","messageId":"c1778c9b-2dd1-9cef-0330-204fe9d08d39@gmail.com","threadId":"43949","inReplyTo":"20160829054725.r6pqf3xlusxi7ibq@sigill.intra.peff.net","subject":"Re: Reducing CPU load on git server","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-08-29T10:46:27Z","receivedAt":"2016-08-29T10:47:41Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 29.08.2016 o 07:47, Jeff King pisze:\n> On Sun, Aug 28, 2016 at 12:42:52PM -0700, W. David Jarvis wrote:\n> \n>> The actual replication process works as follows:\n>>\n>> 1. The primary git server receives a push and sends a webhook with the\n>> details of the push (repo, ref, sha, some metadata) to a \"publisher\"\n>> box\n>>\n>> 2. The publisher enqueues the details of the webhook into a queue\n>>\n>> 3. A fleet of \"subscriber\" (replica) boxes each reads the payload of\n>> the enqueued message. Each of these then tries to either clone the\n>> repository if they don't already have it, or they run `git fetch`.\n> \n> So your load is probably really spiky, as you get thundering herds of\n> fetchers after every push (the spikes may have a long flatline at the\n> top, as it takes time to process the whole herd).\n\nOne solution I have heard about, in the context of web cache, to reduce\nthe thundering herd problem (there caused by cache expiring at the same\ntime in many clients) was to add some random or quasi-random distribution\nto expiration time.  In your situation adding a random delay with some\nspecified deviation could help.\n\nNote however that it is, I think, incompatible (to some extent) with\n\"caching\" solution, where the 'thundering herd' get served the same\npackfile.  Or at least one solution can reduce the positive effect\nof the other.\n\n>> 1. We currently run a blanket `git fetch` rather than specifically\n>> fetching the ref that was pushed. My understanding from poking around\n>> the git source code is that this causes the replication server to send\n>> a list of all of its ref tips to the primary server, and the primary\n>> server then has to verify and compare each of these tips to the ref\n>> tips residing on the server.\n\n[...]\n> There's nothing in upstream git to help smooth these loads, but since\n> you mentioned GitHub Enterprise, I happen to know that it does have a\n> system for coalescing multiple fetches into a single pack-objects. I\n> _think_ it's in GHE 2.5, so you might check which version you're\n> running (and possibly also talk to GitHub Support, who might have more\n> advice; there are also tools for finding out which git processes are\n> generating the most load, etc).\n\nI wonder if this system for coalescing multiple fetches is something\ngeneric, or is it something specific to GitHub / GitHub Enterprise\narchitecture?  If it is the former, would it be considered for\nupstreaming, and if so, when it would be in Git itself?\n\n\nOne thing to note: if you have repositories which are to have the\nsame contents, you can distribute the pack-file to them and update\nreferences without going through Git.  It can be done on push\n(push to master, distribute to mirrors), or as part of fetch\n(master fetches from central repository, distributes to mirrors).\nI think; I have never managed large set of replicated Git repositories.\n\nIf mirrors can get out of sync, you would need to ensure that the\nrepository doing the actual fetch / receiving the actual push is\na least common denominator, that it it looks like lagging behind\nall other mirrors in set.  There is no problem if repository gets\npackfile with more objects than it needs.\n\n>> In other words, let's imagine a world in which we ditch our current\n>> repo-level locking mechanism entirely. Let's also presume we move to\n>> fetching specific refs rather than using blanket fetches. Does that\n>> mean that if a fetch for ref A and a fetch for ref B are issued at\n>> roughly the exact same time, the two will be able to be executed at\n>> once without running into some git-internal locking mechanism on a\n>> granularity coarser than the ref? i.e. are fetch A and fetch B going\n>> to be blocked on the other's completion in any way? (let's presume\n>> that ref A and ref B are not parents of each other).\n> \n> Generally no, they should not conflict. Writes into the object database\n> can happen simultaneously. Ref updates take a per-ref lock, so you\n> should generally be able to write two unrelated refs at once. The big\n> exception is that ref deletion required taking a repo-wide lock, but\n> that presumably wouldn't be a problem for your case.\n\nDoesn't Git avoid taking locks, and use lockless synchronization\nmechanisms (though possibly equivalent to locks)?  I think it takes\nlockfile to update reflog together with reference, but if reflogs\nare turned off (and I think they are off for bare repositories by\ndefault), ref update uses \"atomic file write\" (write + rename)\nand compare-and-swap primitive.  Updating repository is lock-free:\nfirst update repository object database, then reference.\n\nThat said, it might be that per-repository global lock that you\nuse is beneficial, limiting the amount of concurrent access; but\nit could be detrimental, that global-lock contention is the cause\nof stalls and latency.\n\n>> The ultimate goal for us is just figuring out how we can best reduce\n>> the CPU load on the primary instance so that we don't find ourselves\n>> in a situation where we're not able to run basic git operations\n>> anymore.\n> \n> I suspect there's room for improvement and tuning of the primary. But\n> barring that, one option would be to have a hierarchy of replicas. Have\n> \"k\" first-tier replicas fetch from the primary, then \"k\" second-tier\n> replicas fetch from them, and so on. Trade propagation delay for\n> distributing the load. :)\n\nI guess that trying to replicate DGit approach that GitHub uses, see\n\"Introducing DGit\" (http://githubengineering.com/introducing-dgit)\nis currently out of question?\n\nI wonder if you can propagate pushes, instead of fetches...\n\nBest regards,\n-- \nJakub Narębski\n \n\n\n"},{"id":"300467","messageId":"20160829171849.psz5j3xy5kesql5e@sigill.intra.peff.net","threadId":"43949","inReplyTo":"c1778c9b-2dd1-9cef-0330-204fe9d08d39@gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-29T17:18:49Z","receivedAt":"2016-08-29T17:18:58Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 29, 2016 at 12:46:27PM +0200, Jakub Narębski wrote:\n\n> > So your load is probably really spiky, as you get thundering herds of\n> > fetchers after every push (the spikes may have a long flatline at the\n> > top, as it takes time to process the whole herd).\n> \n> One solution I have heard about, in the context of web cache, to reduce\n> the thundering herd problem (there caused by cache expiring at the same\n> time in many clients) was to add some random or quasi-random distribution\n> to expiration time.  In your situation adding a random delay with some\n> specified deviation could help.\n\nThat smooths the spikes, but you still have to serve all of the requests\neventually. So if your problem is that the load spikes and the system\nslows to a crawl as a result (or runs out of RAM, etc), then\ndistributing the load helps. But if you have enough load that your\nsystem is constantly busy, queueing the load in a different order just\nshifts it around.\n\nGHE will also introduce delays into starting git when load spikes, but\nthat's a separate system that coalescing identical requests.\n\n> I wonder if this system for coalescing multiple fetches is something\n> generic, or is it something specific to GitHub / GitHub Enterprise\n> architecture?  If it is the former, would it be considered for\n> upstreaming, and if so, when it would be in Git itself?\n\nI've already sent upstream the patch for a \"hook\" that sits between\nupload-pack and pack-objects (and it will be in v2.10). So that can call\nan arbitrary script which can then make scheduling policy for\npack-objects, coalesce similar requests, etc.\n\nGHE has a generic tool for coalescing program invocations that is not\nGit-specific at all (it compares its stdin and command line arguments to\ndecide when two requests are identical, runs the command on its\narguments, and then passes the output to all members of the herd). That\n_might_ be open-sourced in the future, but I don't have a specific\ntimeline.\n\n> One thing to note: if you have repositories which are to have the\n> same contents, you can distribute the pack-file to them and update\n> references without going through Git.  It can be done on push\n> (push to master, distribute to mirrors), or as part of fetch\n> (master fetches from central repository, distributes to mirrors).\n> I think; I have never managed large set of replicated Git repositories.\n\nDoing it naively has some gotchas, because you want to make sure you\nhave all of the necessary objects. But if you are going this route,\nprobably distributed a git-bundle is the simplest way.\n\n> > Generally no, they should not conflict. Writes into the object database\n> > can happen simultaneously. Ref updates take a per-ref lock, so you\n> > should generally be able to write two unrelated refs at once. The big\n> > exception is that ref deletion required taking a repo-wide lock, but\n> > that presumably wouldn't be a problem for your case.\n> \n> Doesn't Git avoid taking locks, and use lockless synchronization\n> mechanisms (though possibly equivalent to locks)?  I think it takes\n> lockfile to update reflog together with reference, but if reflogs\n> are turned off (and I think they are off for bare repositories by\n> default), ref update uses \"atomic file write\" (write + rename)\n> and compare-and-swap primitive.  Updating repository is lock-free:\n> first update repository object database, then reference.\n\nThere is a lockfile to make the compare-and-swap atomic, but yes, it's\nfundamentally based around the compare-and-swap. I don't think that\nmatters to the end user though. Fundamentally they will see \"I hoped to\nmove from X to Y, but somebody else wrote Z, aborting\", which is the\nsame as \"I did not win the lock race, aborting\".\n\nThe point is that updating two different refs is generally independent,\nand updating the same ref is not.\n\n> I guess that trying to replicate DGit approach that GitHub uses, see\n> \"Introducing DGit\" (http://githubengineering.com/introducing-dgit)\n> is currently out of question?\n\nMinor nitpick (that you don't even have any way of knowing about, so\nmaybe more of a public service announcement). GitHub will stop using the\n\"DGit\" name because it's too confusingly similar to \"Git\" (and \"Git\" is\ntrademarked by the project). There's a new blog post coming that\nmentions the name change, and that historic one will have a note added\nretroactively. The new name is \"GitHub Spokes\" (get it, Hub, Spokes?).\n\nBut in response to your question, I'll caution that replicating it is a\nlot of work. :)\n\nSince the original problem report mentions GHE, I'll note that newer\nversions of GHE do support clustering and can share the git load across\nmultiple Spokes servers. So in theory that could make the replica layer\ngo away entirely, because it all happens behind the scenes.\n\n-Peff\n\nPS Sorry, I generally try to avoid hawking GitHub wares on the list, but\n   since the OP mentioned GHE specifically, and because there aren't\n   really generic solutions to most of these things, I do think it's a\n   viable path for a solution for him.\n"},{"id":"300484","messageId":"CAFMAO9yUMY5dqw-oWpKG1H-xska1AtDyt31_WaeJDyTieQLChw@mail.gmail.com","threadId":"43949","inReplyTo":"20160829054725.r6pqf3xlusxi7ibq@sigill.intra.peff.net","subject":"Re: Reducing CPU load on git server","fromName":"W. David Jarvis","fromEmail":"william.d.jarvis@gmail.com","sentAt":"2016-08-29T19:16:20Z","receivedAt":"2016-08-29T19:16:58Z","isPatch":false,"sender":{"key":"william.d.jarvis@gmail.com","avatar":"https://gravatar.com/avatar/4ed36a1dd09f36a54236d394eb73566ebb6ba4aba71182b45cbf585559310e8a?d=mp&s=160"},"body":"> So your load is probably really spiky, as you get thundering herds of\n> fetchers after every push (the spikes may have a long flatline at the\n> top, as it takes time to process the whole herd).\n\nIt is quite spiky, yes. At the moment, however, the replication fleet\nis relatively small (at the moment it's just 4 machines). We had 6\nmachines earlier this month and we had hoped that terminating two of\nthem would lead to a material drop in CPU usage, but we didn't see a\nreally significant reduction.\n\n> Yes, though I'd be surprised if this negotiation is that expensive in\n> practice. In my experience it's not generally, and even if we ended up\n> traversing every commit in the repository, that's on the order of a few\n> seconds even for large, active repositories.\n>\n> In my experience, the problem in a mass-fetch like this ends up being\n> pack-objects preparing the packfile. It has to do a similar traversal,\n> but _also_ look at all of the trees and blobs reachable from there, and\n> then search for likely delta-compression candidates.\n>\n> Do you know which processes are generating the load? git-upload-pack\n> does the negotiation, and then pack-objects does the actual packing.\n\nWhen I look at expensive operations (ones that I can see consuming\n90%+ of a CPU for more than a second), there are often pack-objects\nprocesses running that will consume an entire core for multiple\nseconds (I also saw one pack-object counting process run for several\nminutes while using up a full core). rev-list shows up as a pretty\nactive CPU consumer, as do prune and blame-tree.\n\nI'd say overall that in terms of high-CPU consumption activities,\n`prune` and `rev-list` show up the most frequently.\n\nOn the subject of prune - I forgot to mention that the `git fetch`\ncalls from the subscribers are running `git fetch --prune`. I'm not\nsure if that changes the projected load profile.\n\n> Maybe. If pack-objects is where your load is coming from, then\n> counter-intuitively things sometimes get _worse_ as you fetch less. The\n> problem is that git will generally re-use deltas it has on disk when\n> sending to the clients. But if the clients are missing some of the\n> objects (because they don't fetch all of the branches), then we cannot\n> use those deltas and may need to recompute new ones. So you might see\n> some parts of the fetch get cheaper (negotiation, pack-object's\n> \"Counting objects\" phase), but \"Compressing objects\" gets more\n> expensive.\n\nI might be misunderstanding this, but if the subscriber is already \"up\nto date\" modulo a single updated ref tip, then this problem shouldn't\noccur, right? Concretely: if ref A is built off of ref B, and the\nsubscriber already has B when it requests A, that shouldn't cause this\nbehavior, but it would cause this behavior if the subscriber didn't\nhave B when it requested A.\n\n> This is particularly noticeable with shallow fetches, which in my\n> experience are much more expensive to serve.\n\nI don't think we're doing shallow fetches anywhere in this system.\n\n> Jakub mentioned bitmaps, and if you are using GitHub Enterprise, they\n> are enabled. But they won't really help here. They are essentially\n> cached information that git generates at repack time. But if we _just_\n> got a push, then the new objects to fetch won't be part of the cache,\n> and we'll fall back to traversing them as normal.  On the other hand,\n> this should be a relatively small bit of history to traverse, so I'd\n> doubt that \"Counting objects\" is that expensive in your case (but you\n> should be able to get a rough sense by watching the progress meter\n> during a fetch).\n\nSee comment above about a long-running counting objects process. I\ncouldn't tell which of our repositories it was counting, but it went\nfor about 3 minutes with full core utilization. I didn't see us\ncounting pack-objects frequently but it's an expensive operation.\n\n> I'd suspect more that delta compression is expensive (we know we just\n> got some new objects, but we don't know if we can make good deltas\n> against the objects the client already has). That's a gut feeling,\n> though.\n>\n> If the fetch is small, that _also_ shouldn't be too expensive. But\n> things add up when you have a large number of machines all making the\n> same request at once. So it's entirely possible that the machine just\n> gets hit with a lot of 5s CPU tasks all at once. If you only have a\n> couple cores, that takes many multiples of 5s to clear out.\n\nI think this would show up if I was sitting around running `top` on\nthe machine, but that doesn't end up being what I see. That might just\nbe a function of there being a relatively small number of replication\nmachines, I'm not sure. But I'm not noticing 4 of the same tasks get\nspawned simultaneously, which says to me that we're either utilizing a\ncache or there's some locking behavior involved.\n\n> There's nothing in upstream git to help smooth these loads, but since\n> you mentioned GitHub Enterprise, I happen to know that it does have a\n> system for coalescing multiple fetches into a single pack-objects. I\n> _think_ it's in GHE 2.5, so you might check which version you're\n> running (and possibly also talk to GitHub Support, who might have more\n> advice; there are also tools for finding out which git processes are\n> generating the most load, etc).\n\nWe're on 2.6.4 at the moment.\n\n> I suspect there's room for improvement and tuning of the primary. But\n> barring that, one option would be to have a hierarchy of replicas. Have\n> \"k\" first-tier replicas fetch from the primary, then \"k\" second-tier\n> replicas fetch from them, and so on. Trade propagation delay for\n> distributing the load. :)\n\nYep, this was the first thought we had. I started fleshing out the\narchitecture for it but wasn't sure if the majority of the CPU load\nwas being triggered by the first request to make it in, in which case\nmoving to a multi-tiered architecture wouldn't help us. When we didn't\nnotice a material reduction in CPU load after reducing the replication\nfleet by 1/3, I started to worry about this and stopped moving forward\non the multi-tiered architecture until I understood the behavior\nbetter.\n\n - V\n"},{"id":"300494","messageId":"CACBZZX63DAmFt_ZiUHj-bs9dtwRd4MOxoLfM8r1uRi3q4Mwnkw@mail.gmail.com","threadId":"43949","inReplyTo":"CAFMAO9y3LsrAb_jp8XVq2mexaA4bBqmWFwJu55r4S6Dxd2-zxw@mail.gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2016-08-29T20:14:44Z","receivedAt":"2016-08-29T20:15:10Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sun, Aug 28, 2016 at 9:42 PM, W. David Jarvis\n<william.d.jarvis@gmail.com> wrote:\n> I've run into a problem that I'm looking for some help with. Let me\n> describe the situation, and then some thoughts.\n\nJust a few points that you may not have considered, and I didn't see\nmentioned in this thread:\n\n * Consider having that queue of yours just send the pushed payload\ninstead of \"pull this\", see git-bundle. This can turn this sync entire\nthing into a static file distribution problem.\n\n * It's not clear from your post why you have to worry about all these\nbranches, surely your Chef instances just need the \"master\" branch,\njust push that around.\n\n * If you do need branches consider archiving stale tags/branches\nafter some time. I implemented this where I work, we just have a\n$REPO-archive.git with every tag/branch ever created for a given\n$REPO.git, and delete refs after a certain time.\n\n * If your problem is that you're CPU bound on the master have you\nconsidered maybe solving this with something like NFS, i.e. replace\nyour ad-hoc replication with just a bunch of \"slave\" boxes that mount\nthe remote filesystem.\n\n * Or, if you're willing to deal with occasional transitory repo\ncorruption (just retry?): rsync.\n\n * Theres's no reason for why your replication chain needs to be\nsingle-level if master CPU is really the issue. You could have master\n-> N slaves -> N*X slaves, or some combination thereof.\n\n * Does it really even matter that your \"slave\" machines are all\nup-to-date? We have something similar at work but it's just a minutely\ncronjob that does \"git fetch\" on some repos, since the downstream\nthing (e.g. the chef run) doesn't run more than once every 30m or\nwhatever anyway.\n"},{"id":"300503","messageId":"CAFMAO9wQD5GtGRGv-sMy=NA1q8kbu6n3FFbWuJ+W5-qnRDKW-w@mail.gmail.com","threadId":"43949","inReplyTo":"CACBZZX63DAmFt_ZiUHj-bs9dtwRd4MOxoLfM8r1uRi3q4Mwnkw@mail.gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"W. David Jarvis","fromEmail":"william.d.jarvis@gmail.com","sentAt":"2016-08-29T20:57:56Z","receivedAt":"2016-08-29T20:58:33Z","isPatch":false,"sender":{"key":"william.d.jarvis@gmail.com","avatar":"https://gravatar.com/avatar/4ed36a1dd09f36a54236d394eb73566ebb6ba4aba71182b45cbf585559310e8a?d=mp&s=160"},"body":">  * Consider having that queue of yours just send the pushed payload\n> instead of \"pull this\", see git-bundle. This can turn this sync entire\n> thing into a static file distribution problem.\n\nAs far as I know, GHE doesn't support this out of the box. We've asked\nthem for essentially this, though. Due to the nature of our license we\nmay not be able to configure something like this on the server\ninstance ourselves.\n\n>  * It's not clear from your post why you have to worry about all these\n> branches, surely your Chef instances just need the \"master\" branch,\n> just push that around.\n\nWe allow deployments from non-master branches, so we do need multiple\nbranches. We also use the replication fleet as the target for our\nbuild system, which needs to be able to build essentially any branch\non any repository.\n\n>  * If you do need branches consider archiving stale tags/branches\n> after some time. I implemented this where I work, we just have a\n> $REPO-archive.git with every tag/branch ever created for a given\n> $REPO.git, and delete refs after a certain time.\n\nThis is something else that we're actively considering. Why did your\ncompany implement this -- was it to reduce load, or just to clean up\nyour repositories? Did you notice any change in server load?\n\n>  * If your problem is that you're CPU bound on the master have you\n> considered maybe solving this with something like NFS, i.e. replace\n> your ad-hoc replication with just a bunch of \"slave\" boxes that mount\n> the remote filesystem.\n\nThis is definitely an interesting idea. It'd be a significant\narchitectural change, though, and not one I'm sure we'd be able to get\nsupport for.\n\n>  * Or, if you're willing to deal with occasional transitory repo\n> corruption (just retry?): rsync.\n\nI think this is a cost we're not necessarily okay with having to deal with.\n\n>  * Theres's no reason for why your replication chain needs to be\n> single-level if master CPU is really the issue. You could have master\n> -> N slaves -> N*X slaves, or some combination thereof.\n\nThis was discussed above - if the primary driver of load is the first\nfetch, then moving to a multi-tiered architecture will not solve our\nproblems.\n\n>  * Does it really even matter that your \"slave\" machines are all\n> up-to-date? We have something similar at work but it's just a minutely\n> cronjob that does \"git fetch\" on some repos, since the downstream\n> thing (e.g. the chef run) doesn't run more than once every 30m or\n> whatever anyway.\n\nIt does, because we use the replication fleet for our build server.\n\n - V\n\n-- \n============\nvenanti.us\n203.918.2328\n============\n"},{"id":"300511","messageId":"20160829213101.3ulrw5hrh5pytjii@sigill.intra.peff.net","threadId":"43949","inReplyTo":"CAFMAO9yUMY5dqw-oWpKG1H-xska1AtDyt31_WaeJDyTieQLChw@mail.gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-29T21:31:01Z","receivedAt":"2016-08-29T21:31:11Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 29, 2016 at 12:16:20PM -0700, W. David Jarvis wrote:\n\n> > Do you know which processes are generating the load? git-upload-pack\n> > does the negotiation, and then pack-objects does the actual packing.\n> \n> When I look at expensive operations (ones that I can see consuming\n> 90%+ of a CPU for more than a second), there are often pack-objects\n> processes running that will consume an entire core for multiple\n> seconds (I also saw one pack-object counting process run for several\n> minutes while using up a full core).\n\nPegging CPU for a few seconds doesn't sound out-of-place for\npack-objects serving a fetch or clone on a large repository. And I can\ncertainly believe \"minutes\", especially if it was not serving a fetch,\nbut doing repository maintenance on a large repository.\n\nTalk to GitHub Enterprise support folks about what kind of process\nmonitoring and accounting is available. Recent versions of GHE can\neasily tell things like which repositories and processes are using the\nmost CPU, RAM, I/O, and network, which ones are seeing a lot of\nparallelism, etc.\n\n> rev-list shows up as a pretty active CPU consumer, as do prune and\n> blame-tree.\n> \n> I'd say overall that in terms of high-CPU consumption activities,\n> `prune` and `rev-list` show up the most frequently.\n\nNone of those operations is triggered by client fetches. You'll see\n\"rev-list\" for a variety of operations, so that's hard to pinpoint. But\nI'm surprised that \"prune\" is a common one for you. It is run as part of\nthe post-push, but I happen to know that the version that ships on GHE\nis optimized to use bitmaps, and to avoid doing any work when there are\nno loose objects that need pruning in the first place.\n\nBlame-tree is a GitHub-specific command (it feeds the main repository\nview page), and is a known CPU hog. There's more clever caching for that\ncoming down the pipe, but it's not shipped yet.\n\n> On the subject of prune - I forgot to mention that the `git fetch`\n> calls from the subscribers are running `git fetch --prune`. I'm not\n> sure if that changes the projected load profile.\n\nThat shouldn't change anything; the pruning is purely a client side\nthing.\n\n> > Maybe. If pack-objects is where your load is coming from, then\n> > counter-intuitively things sometimes get _worse_ as you fetch less. The\n> > problem is that git will generally re-use deltas it has on disk when\n> > sending to the clients. But if the clients are missing some of the\n> > objects (because they don't fetch all of the branches), then we cannot\n> > use those deltas and may need to recompute new ones. So you might see\n> > some parts of the fetch get cheaper (negotiation, pack-object's\n> > \"Counting objects\" phase), but \"Compressing objects\" gets more\n> > expensive.\n> \n> I might be misunderstanding this, but if the subscriber is already \"up\n> to date\" modulo a single updated ref tip, then this problem shouldn't\n> occur, right? Concretely: if ref A is built off of ref B, and the\n> subscriber already has B when it requests A, that shouldn't cause this\n> behavior, but it would cause this behavior if the subscriber didn't\n> have B when it requested A.\n\nCorrect. So this shouldn't be a thing you are running into now, but it's\nsomething that might be made worse if you switch to fetching only single\nrefs.\n\n> See comment above about a long-running counting objects process. I\n> couldn't tell which of our repositories it was counting, but it went\n> for about 3 minutes with full core utilization. I didn't see us\n> counting pack-objects frequently but it's an expensive operation.\n\nThat really sounds like repository maintenance. Repacks of\ntorvalds/linux (including all of its forks) on github.com take ~15\nminutes of CPU. There may be some optimization opportunities there (I\nhave a few things I'd like to explore in the next few months), but most\nof it is pretty fundamental. It literally takes a few minutes just to\nwalk the entire object graph for that repo (that's one of the more\nextreme cases, of course, but presumably you are hosting some large\nrepositories).\n\nMaintenance like that should be a very occasional operation, but it's\npossible that you have a very busy repo.\n\n> > There's nothing in upstream git to help smooth these loads, but since\n> > you mentioned GitHub Enterprise, I happen to know that it does have a\n> > system for coalescing multiple fetches into a single pack-objects. I\n> > _think_ it's in GHE 2.5, so you might check which version you're\n> > running (and possibly also talk to GitHub Support, who might have more\n> > advice; there are also tools for finding out which git processes are\n> > generating the most load, etc).\n> \n> We're on 2.6.4 at the moment.\n\nOK, I double-checked, and your version should be coalescing identical\nfetches.\n\nGiven that, and that a lot of the load you mentioned above is coming\nfrom non-fetch sources, it sounds like switching anything with your\nreplica fetch strategy isn't likely to help much. And a multi-tiered\narchitecture won't help if the load is being generated by requests that\nare serving the web-views directly on the box.\n\nI'd really encourage you to talk with GitHub Support about performance\nand clustering. It sounds like there may be some GitHub-specific things\nto tweak. And it may be that the load is just too much for a single\nmachine, and would benefit from spreading the load across multiple git\nservers.\n\n-Peff\n"},{"id":"300512","messageId":"1472506287.4265.36.camel@kaarsemaker.net","threadId":"43949","inReplyTo":"CAFMAO9wQD5GtGRGv-sMy=NA1q8kbu6n3FFbWuJ+W5-qnRDKW-w@mail.gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"Dennis Kaarsemaker","fromEmail":"dennis@kaarsemaker.net","sentAt":"2016-08-29T21:31:27Z","receivedAt":"2016-08-29T21:31:34Z","isPatch":false,"sender":{"key":"dennis@kaarsemaker.net","avatar":"https://avatars.githubusercontent.com/u/200649?v=4"},"body":"On ma, 2016-08-29 at 13:57 -0700, W. David Jarvis wrote:\n\n> >  * If you do need branches consider archiving stale tags/branches\n> > after some time. I implemented this where I work, we just have a\n> > $REPO-archive.git with every tag/branch ever created for a given\n> > $REPO.git, and delete refs after a certain time.\n> \n> This is something else that we're actively considering. Why did your\n> company implement this -- was it to reduce load, or just to clean up\n> your repositories? Did you notice any change in server load?\n\nAt 50k refs, ref negotiation gets expensive, especially over http\n\nD. \n"},{"id":"300520","messageId":"CAFMAO9zfMtF6Vc+Uyt+UdgOVf4hzOqMN3o8QLf8pRHx3_U4DPQ@mail.gmail.com","threadId":"43949","inReplyTo":"20160829213101.3ulrw5hrh5pytjii@sigill.intra.peff.net","subject":"Re: Reducing CPU load on git server","fromName":"W. David Jarvis","fromEmail":"william.d.jarvis@gmail.com","sentAt":"2016-08-29T22:41:59Z","receivedAt":"2016-08-29T22:42:43Z","isPatch":false,"sender":{"key":"william.d.jarvis@gmail.com","avatar":"https://gravatar.com/avatar/4ed36a1dd09f36a54236d394eb73566ebb6ba4aba71182b45cbf585559310e8a?d=mp&s=160"},"body":"> Pegging CPU for a few seconds doesn't sound out-of-place for\n> pack-objects serving a fetch or clone on a large repository. And I can\n> certainly believe \"minutes\", especially if it was not serving a fetch,\n> but doing repository maintenance on a large repository.\n>\n> Talk to GitHub Enterprise support folks about what kind of process\n> monitoring and accounting is available. Recent versions of GHE can\n> easily tell things like which repositories and processes are using the\n> most CPU, RAM, I/O, and network, which ones are seeing a lot of\n> parallelism, etc.\n\nWe have an open support thread with the GHE support folks, but the\nonly feedback we've gotten so far on this subject is that they believe\nour CPU load is being driven by the quantity of fetch operations (an\naverage of 16 fetch requests per second during normal business hours,\nso 4 requests per second per subscriber box). About 4,000 fetch\nrequests on our main repository per day.\n\n> None of those operations is triggered by client fetches. You'll see\n> \"rev-list\" for a variety of operations, so that's hard to pinpoint. But\n> I'm surprised that \"prune\" is a common one for you. It is run as part of\n> the post-push, but I happen to know that the version that ships on GHE\n> is optimized to use bitmaps, and to avoid doing any work when there are\n> no loose objects that need pruning in the first place.\n\nWould regular force-pushing trigger prune operations? Our engineering\nbody loves to force-push.\n\n>> I might be misunderstanding this, but if the subscriber is already \"up\n>> to date\" modulo a single updated ref tip, then this problem shouldn't\n>> occur, right? Concretely: if ref A is built off of ref B, and the\n>> subscriber already has B when it requests A, that shouldn't cause this\n>> behavior, but it would cause this behavior if the subscriber didn't\n>> have B when it requested A.\n>\n> Correct. So this shouldn't be a thing you are running into now, but it's\n> something that might be made worse if you switch to fetching only single\n> refs.\n\nBut in theory if we were always up-to-date (since we'd always fetch\nany updated ref) we wouldn't run into this problem? We could have a\ncron job to ensure that we run a full git fetch every once in a while\nbut I'd hope that if this was written properly we'd almost always have\nthe most recent commit for any dependency ref.\n\n> That really sounds like repository maintenance. Repacks of\n> torvalds/linux (including all of its forks) on github.com take ~15\n> minutes of CPU. There may be some optimization opportunities there (I\n> have a few things I'd like to explore in the next few months), but most\n> of it is pretty fundamental. It literally takes a few minutes just to\n> walk the entire object graph for that repo (that's one of the more\n> extreme cases, of course, but presumably you are hosting some large\n> repositories).\n>\n> Maintenance like that should be a very occasional operation, but it's\n> possible that you have a very busy repo.\n\nOur primary repository is fairly busy. It has about 1/3 the commits of\nLinux and about 1/3 the refs, but seems otherwise to be on the same\nscale. And, of course, it both hasn't been around for as long as Linux\nhas and has been experiencing exponential growth, which means its\ncurrent activity is higher than it has ever been before -- might put\nit on a similar scale to Linux's current activity.\n\n> OK, I double-checked, and your version should be coalescing identical\n> fetches.\n>\n> Given that, and that a lot of the load you mentioned above is coming\n> from non-fetch sources, it sounds like switching anything with your\n> replica fetch strategy isn't likely to help much. And a multi-tiered\n> architecture won't help if the load is being generated by requests that\n> are serving the web-views directly on the box.\n>\n> I'd really encourage you to talk with GitHub Support about performance\n> and clustering. It sounds like there may be some GitHub-specific things\n> to tweak. And it may be that the load is just too much for a single\n> machine, and would benefit from spreading the load across multiple git\n> servers.\n\nWhat surprised us is that we had been running this on an r3.4xlarge\n(16vCPU) on AWS for two years without too much issue. Then in a span\nof months we started experiencing massive CPU load, which forced us to\nupgrade the box to one with 32 vCPU (and better CPUs). We just don't\nunderstand what the precise driver of load is here.\n\nAs noted above, we are talking with GitHub about performance -- we've\nalso been pushing them to start working on a clustering plan, but my\nimpression has been that they're reluctant to go down that path. I\nsuspect that we use GHE much more aggressively than the typical GHE\nclient, but I could be wrong about that.\n\n - V\n\n-- \n============\nvenanti.us\n203.918.2328\n============\n"},{"id":"300550","messageId":"9fe5aa9b-5ba8-2b9a-7feb-58e115be3902@gmail.com","threadId":"43949","inReplyTo":"20160829213101.3ulrw5hrh5pytjii@sigill.intra.peff.net","subject":"git blame <directory> [was: Reducing CPU load on git server]","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-08-30T10:46:20Z","receivedAt":"2016-08-30T10:46:29Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 29.08.2016 o 23:31, Jeff King pisze:\n\n> Blame-tree is a GitHub-specific command (it feeds the main repository\n> view page), and is a known CPU hog. There's more clever caching for that\n> coming down the pipe, but it's not shipped yet.\n\nI wonder if having support for 'git blame <directory>' in Git core would\nbe something interesting to Git users.  I once tried to implement it,\nbut it went nowhere.  Would it be hard to implement?\n\n\nI guess that GitHub offers this view as a part of the home page view\nfor a repository is something of a historical artifact; namely that\nweb interfaces for other version control systems used this.  But I\nwonder how useful it is.  Neither cgit nor gitweb offer such view,\nboth GitLab and Bitbucket start with summary age with no blame-tree.\n\n-- \nJakub Narębski\n\n"},{"id":"300626","messageId":"20160831054201.ldlwptlmcndjmfwu@sigill.intra.peff.net","threadId":"43949","inReplyTo":"9fe5aa9b-5ba8-2b9a-7feb-58e115be3902@gmail.com","subject":"Re: git blame <directory> [was: Reducing CPU load on git server]","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-31T05:42:01Z","receivedAt":"2016-08-31T05:42:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 30, 2016 at 12:46:20PM +0200, Jakub Narębski wrote:\n\n> W dniu 29.08.2016 o 23:31, Jeff King pisze:\n> \n> > Blame-tree is a GitHub-specific command (it feeds the main repository\n> > view page), and is a known CPU hog. There's more clever caching for that\n> > coming down the pipe, but it's not shipped yet.\n> \n> I wonder if having support for 'git blame <directory>' in Git core would\n> be something interesting to Git users.  I once tried to implement it,\n> but it went nowhere.  Would it be hard to implement?\n\nI think there's some interest; I have received a few off-list emails\nover the years about it. There was some preliminary discussion long ago:\n\n  http://public-inbox.org/git/20110302164031.GA18233@sigill.intra.peff.net/\n\nThe code that runs on GitHub is available in my fork of git. I haven't\nsubmitted it upstream because there are some lingering issues. I\nmentioned them on-list in the first few items of:\n\n  http://public-inbox.org/git/20130318121243.GC14789@sigill.intra.peff.net/\n\nThat code is in the jk/blame-tree branch of https://github.com/peff/git\nif you are interested in addressing them (note that I haven't touched\nthat code in a few years except for rebasing it forward, so it may have\nbitrotted a little).\n\nHere's a snippet from an off-list conversation I had with Dennis (cc'd)\nin 2014 (I think he uses that blame-tree code as part of a custom git\nweb interface):\n\n> The things I think it needs are:\n> \n>   1. The max-depth patches need to be reconciled with Duy's pathspec\n>      work upstream. The current implementation works only for tree\n>      diffs, and is not really part of the pathspec at all.\n> \n>   2. Docs/output formats for blame-tree need to be friendlier, as you\n>      noticed.\n> \n>   3. Blame-tree does not use revision pathspecs at all. This makes it\n>      take way longer than it could, because it does not prune away side\n>      branches deep in history that affect only paths whose blame we have\n>      already found. But the current pathspec code is so slow that using\n>      it outweighs the pruning benefit.\n> \n>      I have a series, which I just pushed up to jk/faster-blame-tree,\n>      which tries to improve this.  But it's got a lot of new, untested\n>      code itself (we are not even running it at GitHub yet). It's also\n>      based on v1.9.4; I think there are going to be a lot of conflicts\n>      with the combine-tree work done in v2.0.\n> \n> [...]\n> \n> I also think it would probably make sense for blame-tree to support the\n> same output formats as git-blame (e.g., to have an identical --porcelain\n> mode, to have a reasonable human-readable format by default, etc).\n\nThat's all I could dig out of my archives. I'd be happy if somebody\nwanted to pick it up and run with it. Polishing for upstream has been on\nmy list for several years now, but there's usually something more\nimportant (or interesting) to work on at any given moment.\n\nYou might also look at how GitLab does it. I've never talked to them\nabout it, and as far as I know they do not use blame-tree.\n\n-Peff\n"},{"id":"300627","messageId":"20160831060240.xg6axpqnlclys7ic@sigill.intra.peff.net","threadId":"43949","inReplyTo":"CAFMAO9zfMtF6Vc+Uyt+UdgOVf4hzOqMN3o8QLf8pRHx3_U4DPQ@mail.gmail.com","subject":"Re: Reducing CPU load on git server","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-31T06:02:41Z","receivedAt":"2016-08-31T06:02:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 29, 2016 at 03:41:59PM -0700, W. David Jarvis wrote:\n\n> We have an open support thread with the GHE support folks, but the\n> only feedback we've gotten so far on this subject is that they believe\n> our CPU load is being driven by the quantity of fetch operations (an\n> average of 16 fetch requests per second during normal business hours,\n> so 4 requests per second per subscriber box). About 4,000 fetch\n> requests on our main repository per day.\n\nHmm. That might well be, but I'd have to see the numbers to say more. At\nthis point I think we're exhausting what is useful to talk about on the\nGit list.  I'll point GHE Support at this thread, which might help your\nconversation with them (and they might pull me in behind the scenes).\n\nI'll try to answer any Git-specific questions here, though.\n\n> > None of those operations is triggered by client fetches. You'll see\n> > \"rev-list\" for a variety of operations, so that's hard to pinpoint. But\n> > I'm surprised that \"prune\" is a common one for you. It is run as part of\n> > the post-push, but I happen to know that the version that ships on GHE\n> > is optimized to use bitmaps, and to avoid doing any work when there are\n> > no loose objects that need pruning in the first place.\n> \n> Would regular force-pushing trigger prune operations? Our engineering\n> body loves to force-push.\n\nNo, it shouldn't make a difference. For stock git, \"prune\" will only be\nrun occasionally as part of \"git gc\". On GitHub Enterprise, every push\nkicks off a \"sync\" job that moves objects from a specific fork into\nstorage shared by all of the related forks. So GHE will run prune more\noften than stock git would, but force-pushing wouldn't have any effect\non that.\n\nThere are also some custom patches to optimize prune on GHE, so it\nshouldn't generally be very expensive. Unless perhaps for some reason\nthe reachability bitmaps on your repository aren't performing very well.\n\nYou could try something like comparing:\n\n  time git rev-list --objects --all >/dev/null\n\nand\n\n  time git rev-list --objects --all --use-bitmap-index >/dev/null\n\non your server. The second should be a lot faster. If it's not, that may\nbe an indication that Git could be doing a better job of selecting\nbitmap commits (that code is not GitHub-specific at all).\n\n> >> I might be misunderstanding this, but if the subscriber is already \"up\n> >> to date\" modulo a single updated ref tip, then this problem shouldn't\n> >> occur, right? Concretely: if ref A is built off of ref B, and the\n> >> subscriber already has B when it requests A, that shouldn't cause this\n> >> behavior, but it would cause this behavior if the subscriber didn't\n> >> have B when it requested A.\n> >\n> > Correct. So this shouldn't be a thing you are running into now, but it's\n> > something that might be made worse if you switch to fetching only single\n> > refs.\n> \n> But in theory if we were always up-to-date (since we'd always fetch\n> any updated ref) we wouldn't run into this problem? We could have a\n> cron job to ensure that we run a full git fetch every once in a while\n> but I'd hope that if this was written properly we'd almost always have\n> the most recent commit for any dependency ref.\n\nIt's a little more complicated than that. What you're really going for\nis letting git reuse on-disk deltas when serving fetches. But depending\non when the last repack was run, we might be cobbling together the fetch\nfrom multiple packs on disk, in which case there will still be some\ndelta search. In my experience that's not _generally_ a big deal,\nthough. Small fetches don't have that many deltas to find.\n\n> Our primary repository is fairly busy. It has about 1/3 the commits of\n> Linux and about 1/3 the refs, but seems otherwise to be on the same\n> scale. And, of course, it both hasn't been around for as long as Linux\n> has and has been experiencing exponential growth, which means its\n> current activity is higher than it has ever been before -- might put\n> it on a similar scale to Linux's current activity.\n\nMost of the work for repacking scales with the number of total objects\n(not quite linearly, though).  For torvalds/linux (and its forks),\nthat's around 16 million objects. You might try \"git count-objects -v\"\non your server for comparison (but do it in the \"network.git\" directory,\nas that's the shared object storage).\n\n-Peff\n"},{"id":"300630","messageId":"1472628509.4265.43.camel@kaarsemaker.net","threadId":"43949","inReplyTo":"20160831054201.ldlwptlmcndjmfwu@sigill.intra.peff.net","subject":"Re: git blame <directory> [was: Reducing CPU load on git server]","fromName":"Dennis Kaarsemaker","fromEmail":"dennis@kaarsemaker.net","sentAt":"2016-08-31T07:28:29Z","receivedAt":"2016-08-31T07:28:47Z","isPatch":false,"sender":{"key":"dennis@kaarsemaker.net","avatar":"https://avatars.githubusercontent.com/u/200649?v=4"},"body":"On wo, 2016-08-31 at 01:42 -0400, Jeff King wrote:\n> On Tue, Aug 30, 2016 at 12:46:20PM +0200, Jakub Narębski wrote:\n> \n> > I wonder if having support for 'git blame ' in Git core would\n> > be something interesting to Git users.  I once tried to implement it,\n> > but it went nowhere.  Would it be hard to implement?\n> I think there's some interest; I have received a few off-list emails\n> over the years about it. There was some preliminary discussion long ago:\n> \n>8 <snip>\n>\n> Here's a snippet from an off-list conversation I had with Dennis (cc'd)\n> in 2014 (I think he uses that blame-tree code as part of a custom git\n> web interface):\n\nI still do, but haven't worked on that webinterface in a while. I also\nstill build git with blame-tree merged in (which does occasionally\nrequire a rebase with some tinkering).\n\n> That's all I could dig out of my archives. I'd be happy if somebody> wanted to pick it up and run with it. Polishing for upstream has been on\n> my list for several years now, but there's usually something more\n> important (or interesting) to work on at any given moment.\n\nSame here. I've been meaning to pick this up, but it touches some parts\nof git I'm still not too familiar with and there are always other\nthings to do :)\n\nD.\n"}]}