{"thread":{"id":"38604","subject":"Git Scaling: What factors most affect Git performance for a large repo?","startedAt":"2015-02-19T21:26:58Z","lastAt":"2015-03-02T20:15:39Z","messageCount":28,"participants":["Stephen Morton","Stefan Beller","Ævar Arnfjörð Bjarmason","Duy Nguyen","brian m. carlson","David Turner","Matthieu Moy","Junio C Hamano","Sebastian Schuberth","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"256365","messageId":"CAH8BJxEr_v+aitpZduXPC4oiRhMuySpc7Wb1aGcYe_p3mWjn+w@mail.gmail.com","threadId":"38604","inReplyTo":null,"subject":"Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Stephen Morton","fromEmail":"stephen.c.morton@gmail.com","sentAt":"2015-02-19T21:26:58Z","receivedAt":"2015-02-19T21:26:58Z","isPatch":false,"sender":{"key":"stephen.c.morton@gmail.com","avatar":"https://gravatar.com/avatar/3b826f06cabc070cbd5d53071880caad8badc3b29e0789b510353faca59ffb7b?d=mp&s=160"},"body":"I posted this to comp.version-control.git.user and didn't get any response. I\nthink the question is plumbing-related enough that I can ask it here.\n\nI'm evaluating the feasibility of moving my team from SVN to git. We have a very\nlarge repo. [1] We will have a central repo using GitLab (or similar) that\neverybody works with. Forks, code sharing, pull requests etc. will be done\nthrough this central server.\n\nBy 'performance', I guess I mean speed of day to day operations for devs.\n\n * (Obviously, trivially, a (non-local) clone will be slow with a large repo.)\n * Will a few simultaneous clones from the central server also slow down\n other concurrent operations for other users?\n * Will 'git pull' be slow?\n * 'git push'?\n * 'git commit'? (It is listed as slow in reference [3].)\n * 'git stautus'? (Slow again in reference 3 though I don't see it.)\n * Some operations might not seem to be day-to-day but if they are called\n frequently by the web front-end to GitLab/Stash/GitHub etc then\n they can become bottlenecks. (e.g. 'git branch --contains' seems terribly\n adversely affected by large numbers of branches.)\n * Others?\n\n\nAssuming I can put lots of resources into a central server with lots of CPU,\nRAM, fast SSD, fast networking, what aspects of the repo are most likely to\naffect devs' experience?\n * Number of commits\n * Sheer disk space occupied by the repo\n * Number of tags.\n * Number of branches.\n * Binary objects in the repo that cause it to bloat in size [1]\n * Other factors?\n\nOf the various HW items listed above --CPU speed, number of cores, RAM, SSD,\nnetworking-- which is most critical here?\n\n(Stash recommends 1.5 x repo_size x number of concurrent clones of\navailable RAM.\nI assume that is good advice in general.)\n\nAssume ridiculous numbers. Let me exaggerate: say 1 million commits, 15 GB repo,\n50k tags, 1,000 branches. (Due to historical code fixups, another 5,000 \"fix-up\nbranches\" which are just one little dangling commit required to change the code\na little bit between a commit a tag that was not quite made from it.)\n\nWhile there's lots of information online, much of it is old [3] and with git\nconstantly evolving I don't know how valid it still is. Then there's anecdotal\nevidence that is of questionable value.[2]\n Are many/all of the issues Facebook identified [3] resolved? (Yes, I\nunderstand Facebook went with Mercurial. But I imagine the git team nevertheless\ntook their analysis to heart.)\n\n\nThanks,\nSteve\n\n\n[1] (Yes, I'm investigating ways to make our repo not so large etc. That's\n beyond the scope of the discussion I'd like to have with this\n question. Thanks.)\n[2] The large amounts of anecdotal evidence relate to the \"why don't you try it\n yourself?\" response to my question. I will I I have to but setting up a\n properly methodical study is time consuming and difficult --I don't want to\n produce poor anecdotal numbers that don't really hold up-- and if somebody's\n already done the work, then I should leverage it.\n[3] http://thread.gmane.org/gmane.comp.version-control.git/189776\n"},{"id":"256367","messageId":"CAGZ79kaPhuCXgYvZ49NwVRvrKLVJLTWd0oWHv3jX31sOVCWKVQ@mail.gmail.com","threadId":"38604","inReplyTo":"CAH8BJxEr_v+aitpZduXPC4oiRhMuySpc7Wb1aGcYe_p3mWjn+w@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-02-19T22:21:10Z","receivedAt":"2015-02-19T22:21:10Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Feb 19, 2015 at 1:26 PM, Stephen Morton\n wrote:\n> I posted this to comp.version-control.git.user and didn't get any response. I\n> think the question is plumbing-related enough that I can ask it here.\n>\n> I'm evaluating the feasibility of moving my team from SVN to git. We have a very\n> large repo. [1]\n>\n> [1] (Yes, I'm investigating ways to make our repo not so large etc. That's\n> beyond the scope of the discussion I'd like to have with this\n> question. Thanks.)\n\nWhat do you mean by large?\n* lots of files\n* large files\n* or even large binary files (bad to diff/merge)\n* long history (i.e. lots of small changes)\n* impactful history (changes which rewrite nearly everything from scratch)\n\nFor reference, the linux\n* has 48414 files, in 3128 directories\n* the largest file is 1.1M, the whole repo is 600M\n* no really large binary files\n* more than 500051 changes/commits including merges\n* started in 2004 (when git was invented essentially)\n* the .git folder is 1.4G compared to the 600M files,\n indicating it may have been rewritting 3 times (well this\n metric is bogus, there is lots of compression\n going on in .git)\n\nand linux seems to be doing ok with git.\n\nSo as long as you cannot pinpoint your question on what you are exactly\nconcerned about, there will be no helpful answer I guess.\n\nlinux is by no means a really large project, there are other projects way\nlarger than that (I am thinking about the KDE project for example)\nand they do fine as well.\n\nThanks,\nStefan\n"},{"id":"256368","messageId":"CAH8BJxEK1xwGMsR3JKe72AVS7+5-aaQA4_Q8fpuZndSh1-mong@mail.gmail.com","threadId":"38604","inReplyTo":"CAGZ79kaPhuCXgYvZ49NwVRvrKLVJLTWd0oWHv3jX31sOVCWKVQ@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Stephen Morton","fromEmail":"stephen.c.morton@gmail.com","sentAt":"2015-02-19T23:06:29Z","receivedAt":"2015-02-19T23:06:29Z","isPatch":false,"sender":{"key":"stephen.c.morton@gmail.com","avatar":"https://gravatar.com/avatar/3b826f06cabc070cbd5d53071880caad8badc3b29e0789b510353faca59ffb7b?d=mp&s=160"},"body":"On Thu, Feb 19, 2015 at 5:21 PM, Stefan Beller wrote:\n> On Thu, Feb 19, 2015 at 1:26 PM, Stephen Morton\n> wrote:\n>> I posted this to comp.version-control.git.user and didn't get any response. I\n>> think the question is plumbing-related enough that I can ask it here.\n>>\n>> I'm evaluating the feasibility of moving my team from SVN to git. We have a very\n>> large repo. [1]\n>>\n>> [1] (Yes, I'm investigating ways to make our repo not so large etc. That's\n>> beyond the scope of the discussion I'd like to have with this\n>> question. Thanks.)\n>\n> What do you mean by large?\n> * lots of files\n> * large files\n> * or even large binary files (bad to diff/merge)\n> * long history (i.e. lots of small changes)\n> * impactful history (changes which rewrite nearly everything from scratch)\n>\n> For reference, the linux\n> * has 48414 files, in 3128 directories\n> * the largest file is 1.1M, the whole repo is 600M\n> * no really large binary files\n> * more than 500051 changes/commits including merges\n> * started in 2004 (when git was invented essentially)\n> * the .git folder is 1.4G compared to the 600M files,\n> indicating it may have been rewritting 3 times (well this\n> metric is bogus, there is lots of compression\n> going on in .git)\n>\n> and linux seems to be doing ok with git.\n>\n> So as long as you cannot pinpoint your question on what you are exactly\n> concerned about, there will be no helpful answer I guess.\n>\n> linux is by no means a really large project, there are other projects way\n> larger than that (I am thinking about the KDE project for example)\n> and they do fine as well.\n>\n> Thanks,\n> Stefan\n\nHi Stefan,\n\nI think I addressed most of this in my original post with the paragraph\n\n \"Assume ridiculous numbers. Let me exaggerate: say 1 million commits,\n15 GB repo,\n 50k tags, 1,000 branches. (Due to historical code fixups, another\n5,000 \"fix-up\n branches\" which are just one little dangling commit required to\nchange the code\n a little bit between a commit and a tag that was not quite made from it.)\"\n\nTo that I'd add 25k files,\nno major rewrites,\nno huge binary files, but lots of a few MB binary files with many revisions.\n\nBut even without details of my specific concerns, I thought that\nperhaps the git developers know what limits git's performance even if\nlarge projects like the kernel are not hitting these limits.\n\nSteve\n"},{"id":"256369","messageId":"CAGZ79kaBYENOUAuG4j1itsEq54DoipL7pZSP+b+wTc0wrQZoOw@mail.gmail.com","threadId":"38604","inReplyTo":"CAH8BJxEK1xwGMsR3JKe72AVS7+5-aaQA4_Q8fpuZndSh1-mong@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-02-19T23:15:26Z","receivedAt":"2015-02-19T23:15:26Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Feb 19, 2015 at 3:06 PM, Stephen Morton\n wrote:\n>\n> I think I addressed most of this in my original post with the paragraph\n>\n> \"Assume ridiculous numbers. Let me exaggerate: say 1 million commits,\n> 15 GB repo,\n> 50k tags, 1,000 branches. (Due to historical code fixups, another\n> 5,000 \"fix-up\n> branches\" which are just one little dangling commit required to\n> change the code\n> a little bit between a commit and a tag that was not quite made from it.)\"\n>\n> To that I'd add 25k files,\n> no major rewrites,\n> no huge binary files, but lots of a few MB binary files with many revisions.\n>\n> But even without details of my specific concerns, I thought that\n> perhaps the git developers know what limits git's performance even if\n> large projects like the kernel are not hitting these limits.\n>\n> Steve\n\nI did not realize you gave numbers below, as I started answering after\nreading the\nfirst paragraphs. Sorry about that.\n\nI think lots of files organized in a hierarchical fashion ranging in\nthe small MB range is\nnot a huge deal. Also history is a non issue\n\nThe problem arises with having lots of branches.\n\"640 git branches ought to be enough for everybody -- Linus\" (just kidding)\nGit doesn't really scale efficiently with lots of branches (second\nhand information\nexcept for fetch/pull where I did some patches on another topic recently).\n\nThanks,\nStefan\n"},{"id":"256370","messageId":"CACBZZX6A+35wGBYAYj7E9d4XwLby21TLbTh-zRX+fkSt_e2zeg@mail.gmail.com","threadId":"38604","inReplyTo":"CAH8BJxEr_v+aitpZduXPC4oiRhMuySpc7Wb1aGcYe_p3mWjn+w@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2015-02-19T23:29:30Z","receivedAt":"2015-02-19T23:29:30Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Thu, Feb 19, 2015 at 10:26 PM, Stephen Morton\n wrote:\n> I posted this to comp.version-control.git.user and didn't get any response. I\n> think the question is plumbing-related enough that I can ask it here.\n>\n> I'm evaluating the feasibility of moving my team from SVN to git. We have a very\n> large repo. [1] We will have a central repo using GitLab (or similar) that\n> everybody works with. Forks, code sharing, pull requests etc. will be done\n> through this central server.\n>\n> By 'performance', I guess I mean speed of day to day operations for devs.\n>\n> * (Obviously, trivially, a (non-local) clone will be slow with a large repo.)\n> * Will a few simultaneous clones from the central server also slow down\n> other concurrent operations for other users?\n> * Will 'git pull' be slow?\n> * 'git push'?\n> * 'git commit'? (It is listed as slow in reference [3].)\n> * 'git stautus'? (Slow again in reference 3 though I don't see it.)\n> * Some operations might not seem to be day-to-day but if they are called\n> frequently by the web front-end to GitLab/Stash/GitHub etc then\n> they can become bottlenecks. (e.g. 'git branch --contains' seems terribly\n> adversely affected by large numbers of branches.)\n> * Others?\n>\n>\n> Assuming I can put lots of resources into a central server with lots of CPU,\n> RAM, fast SSD, fast networking, what aspects of the repo are most likely to\n> affect devs' experience?\n> * Number of commits\n> * Sheer disk space occupied by the repo\n> * Number of tags.\n> * Number of branches.\n> * Binary objects in the repo that cause it to bloat in size [1]\n> * Other factors?\n>\n> Of the various HW items listed above --CPU speed, number of cores, RAM, SSD,\n> networking-- which is most critical here?\n>\n> (Stash recommends 1.5 x repo_size x number of concurrent clones of\n> available RAM.\n> I assume that is good advice in general.)\n>\n> Assume ridiculous numbers. Let me exaggerate: say 1 million commits, 15 GB repo,\n> 50k tags, 1,000 branches. (Due to historical code fixups, another 5,000 \"fix-up\n> branches\" which are just one little dangling commit required to change the code\n> a little bit between a commit a tag that was not quite made from it.)\n>\n> While there's lots of information online, much of it is old [3] and with git\n> constantly evolving I don't know how valid it still is. Then there's anecdotal\n> evidence that is of questionable value.[2]\n> Are many/all of the issues Facebook identified [3] resolved? (Yes, I\n> understand Facebook went with Mercurial. But I imagine the git team nevertheless\n> took their analysis to heart.)\n\nAnecdotally I work on a repo at work (where I'm mostly \"the Git guy\") that's:\n\n * Around 500k commits\n * Around 100k tags\n * Around 5k branches\n * Around 500 commits/day, almost entirely to the same branch\n * 1.5 GB .git checkout.\n * Mostly text source, but some binaries (we're trying to cut down[1] on those)\n\nThe main scaling issues we have with Git are:\n\n * \"git pull\" takes around 10 seconds or so\n * Operations like \"git status\" are much slower because they scale\nwith the size of the work tree\n * Similarly \"git rebase\" takes a much longer time for each applied\ncommit, I think because it does the equivalent of \"git status\" for\nevery applied commit. Each commit applied takes around 1-2 seconds.\n * We have a lot of contention on pushes because we're mostly pushing\nto one branch.\n * History spelunking (e.g. git log --reverse -p -G) is taking\nlonger by the day\n\nThe obvious reason for why \"git pull\" is slow is because\ngit-upload-pack spews the complete set of refs at you each time. The\noutput from that command is around 10MB in size for us now. It takes\naround 300 ms to run that locally from hot cache, a bit more to send\nit over the network.\n\nBut actually most of \"git fetch\" is spent in the reachability check\nsubsequently done by \"git-rev-list\" which takes several seconds. I\nhaven't looked into it but there's got to be room for optimization\nthere, surely it only has to do reachability checks for new refs, or\ncould run in some \"I trust this remote not to send me corrupt data\"\ncompletely mode (which would make sense within a company where you can\ntrust your main Git box).\n\nThe \"git status\" operations could be made faster by having something\nlike watchman, there's been some effort on getting that done in Git,\nbut I haven't tried it. This seems to have been the main focus of\nFacebook's Mercurial optimization effort.\n\nSome of this you can \"solve\" mostly by doing e.g. \"git status -uno\",\nhaving support for such unsafe operations (e.g. teaching rebase and\npals to use it) would be nice at the cost of some safety, but having\nsomething that feeds of inotify would be even better.\n\nIt takes around 3 minutes to reclone our repo, we really don't care\n(we rarely re-clone). But I thought I'd mention it because for some\nreason this is important to Facebook and along with inotify were the\ntwo major things they focused on.\n\nAs far as I know every day Git operations don't scale all that badly\nwith a huge history. They will a bit since everything will live in the\nsame pack file, and this'll become especially noticable when your\npackfiles are being evicted out of the page cache.\n\nHowever operations like \"git repack\" seem to be quite bad at handling\nthese sort of repos. It already takes us many GB of RAM to repack\nours. I'd hate to do the same if it was 10x as big.\n\nOverall I'd say Git would work for you for a repo like that, I'd\ncertainly still take it over SVN any day. The main thing you might\nwant to try out is partitioning out any binary assets you may have.\nUsually that's much easier than splitting up the source tree itself.\n\nI haven't yet done this, but I was planning on writing something to\nstart archiving our tags (mostly created by [2]) along with\naggressively deleting branches in the repo. I haven't benchmarked that\nbut I think that'll make the \"pull\" operations much faster, which in\nturn will make the push contention (lots of lemmings pushing to the\nsame ref) better since the pull && push window is reduced.\n\nI do ask myself what we're going to do if we just keep growing and all\nthe numbers I cited will be multiplied by 10-50x. With the current Git\nlimitations on the git implementation I think we'd need to split the\nrepo. The main reason we don't do so is because we like being able to\natomically change a library and its users.\n\nHowever there's nothing in the basic Git file repository format that\ninherently limits Git from being smarter about large repos, it just\nseems to be hard to implement with the way the current client is\nstructured. In particular nothing would stop a Git client from:\n\n * Partially cloning a history but still being able to push upstream.\nYou could just get a partial commit/tree/blob graph and fetch the rest\non-demand as needed.\n * Scaling up to multi TB or PB repos. We'd just have to treat blobs\nas something fetched on-demand, sort of like what git-annex does, but\nbuilt-in. We'd also have to be less stupid about how we pack big blobs\n(or not at all)\n * Being able to partially clone a Git working tree. You could ask the\nserver for the last N commit objects and what you need for some\nsubdirectory in the repo. Then when you commit you ask the server what\nthe other top-level tree objects you need to make a commit are.\n * Nothing in the Git format itself actually requires filesystem\naccess. Not having to deal with external things modifying the tree\nwould be another approach to what the inotify effort is trying to\nsolve.\n\nOf course changes like that will require a major rehaul of the current\ncodebase, or another implementation. Some of those require much more\nactive client/server interaction than what we have now, but they are\npossible, which gives me some hope for the future.\n\nFinally, I'd like to mention that if someone here on-list is\ninterested in doing work on these scalability topics in Git we'd be\nopen to funding that effort on some contract basis. Obviously the\ndetails would have to be worked out blah blah blah, but that came up\nthe last time we had discussions about this internally. Myself and a\nbunch of other people at work /could/ work on this ourselves, but\nwe're busy with other stuff and would much prefer just to pay someone\nto fix them.\n\n1. https://github.com/avar/pre-receive-reject-binaries\n2. https://github.com/git-deploy/git-deploy\n"},{"id":"256371","messageId":"CACsJy8Dortn4fHwF8xSgJ=KoJ9o1qzmc_UyaVq003D2sxFZEuQ@mail.gmail.com","threadId":"38604","inReplyTo":"CAH8BJxEr_v+aitpZduXPC4oiRhMuySpc7Wb1aGcYe_p3mWjn+w@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-19T23:38:43Z","receivedAt":"2015-02-19T23:38:43Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Feb 20, 2015 at 4:26 AM, Stephen Morton\n wrote:\n> By 'performance', I guess I mean speed of day to day operations for devs.\n>\n> * (Obviously, trivially, a (non-local) clone will be slow with a large repo.)\n> * Will a few simultaneous clones from the central server also slow down\n> other concurrent operations for other users?\n\nThere are no locks in server when cloning, so in theory cloning does\nnot affect other operations. Cloning can use lots of memory though\n(and a lot of cpu unless you turn on reachability bitmap feature,\nwhich you should).\n\n> * Will 'git pull' be slow?\n\nIf we exclude the server side, the size of your tree is the main\nfactor, but your 25k files should be fine (linux has 48k files).\n\n> * 'git push'?\n\nThis one is not affected by how deep your repo's history is, or how\nwide your tree is, so should be quick..\n\nAh the number of refs may affect both git-push and git-pull. I think\nStefan knows better than I in this area.\n\n> * 'git commit'? (It is listed as slow in reference [3].)\n> * 'git stautus'? (Slow again in reference 3 though I don't see it.)\n(also git-add)\n\nAgain, the size of your tree. I'm trying to address problems in [3],\nbut at your repo's size, I don't think you need to worry about it.\n\n> * Some operations might not seem to be day-to-day but if they are called\n> frequently by the web front-end to GitLab/Stash/GitHub etc then\n> they can become bottlenecks. (e.g. 'git branch --contains' seems terribly\n> adversely affected by large numbers of branches.)\n> * Others?\n\ngit-blame could be slow when a file is modified a lot.\n-- \nDuy\n"},{"id":"256374","messageId":"20150220000320.GD5021@vauxhall.crustytoothpaste.net","threadId":"38604","inReplyTo":"CAH8BJxEr_v+aitpZduXPC4oiRhMuySpc7Wb1aGcYe_p3mWjn+w@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2015-02-20T00:03:20Z","receivedAt":"2015-02-20T00:03:20Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Thu, Feb 19, 2015 at 04:26:58PM -0500, Stephen Morton wrote:\n> I posted this to comp.version-control.git.user and didn't get any response. I\n> think the question is plumbing-related enough that I can ask it here.\n> \n> I'm evaluating the feasibility of moving my team from SVN to git. We have a very\n> large repo. [1] We will have a central repo using GitLab (or similar) that\n> everybody works with. Forks, code sharing, pull requests etc. will be done\n> through this central server.\n> \n> By 'performance', I guess I mean speed of day to day operations for devs.\n> \n> * (Obviously, trivially, a (non-local) clone will be slow with a large repo.)\n> * Will a few simultaneous clones from the central server also slow down\n> other concurrent operations for other users?\n\nThis hasn't been a problem for us at $DAYJOB. Git doesn't lock anything \non fetches, so each process is independent. We probably have about \nsixty developers (and maybe twenty other occasional users) that manage \nto interact with our Git server all day long. We also have probably \ntwenty smoker (CI) systems pulling at two hour intervals, or, when \nthere's nothing to do, every two minutes, plus probably fifteen to \ntwenty build systems pulling hourly.\n\nI assume you will provide adequate resources for your server.\n\n> * Will 'git pull' be slow?\n> * 'git push'?\n\nThe most pathological case I've seen for git push is a branch with a \nsingle commit merged into the main development branch. As of Git 2.3.0, \nthe performance regression here is fixed.\n\nObviously, the speed of your network connection will affect this. Even \nat 30 MB/s, cloning several gigabytes of data takes time. Git tries \nhard to eliminate sending a lot of data, so if your developers keep \nreasonably up-to-date, the cost of establishing the connection will tend \nto dominate.\n\nI see pull and push times that are less than 2 seconds in most cases.\n\n> * 'git commit'? (It is listed as slow in reference [3].)\n> * 'git stautus'? (Slow again in reference 3 though I don't see it.)\n\nThese can be slow with slow disks or over remote file systems. I \nrecommend not doing that. I've heard rumbles that disk performance is \nbetter on Unix, but I don't use Windows so I can't say.\n\nYou should keep your .gitignore files up-to-date to avoid enumerating \nuntracked files. There's some work towards making this less of an \nissue.\n\ngit blame can be somewhat slow, but it's not something I use more than \nabout once a day, so it doesn't bother me that much.\n\n> Assuming I can put lots of resources into a central server with lots of CPU,\n> RAM, fast SSD, fast networking, what aspects of the repo are most likely to\n> affect devs' experience?\n> * Number of commits\n> * Sheer disk space occupied by the repo\n\nThe number of files can impact performance due to the number of stat()s \nrequired.\n\n> * Number of tags.\n> * Number of branches.\n\nThe number of tags and branches individually is really less relevant \nthan the total number of refs (tags, branches, remote branches, etc). \nVery large numbers of refs can impact performance on pushes and pulls \ndue to the need to enumerate them all.\n\n> * Binary objects in the repo that cause it to bloat in size [1]\n> * Other factors?\n\nIf you want good performance, I'd recommend the latest version of Git \nboth client- and server-side. Newer versions of Git provide pack \nbitmaps, which can dramatically speed up clones and fetches, and Git \n2.3.0 fixes a performance regression with large numbers of refs in \nnon-shallow repositories.\n\nIt is totally worth it to roll your own packages of git if your vendor \nprovides old versions.\n\n> Of the various HW items listed above --CPU speed, number of cores, RAM, SSD,\n> networking-- which is most critical here?\n\nI generally find that having a good disk cache is important with large \nrepositories. It may be advantageous to make sure the developer \nmachines have adequate memory. Performance is notably better on \ndevelopment machines (VMs) with 2 GB or 4 GB of memory instead of 1 GB.\n\nI can't speak to the server side, as I'm not directly involved with its \ndeployment.\n\n> Assume ridiculous numbers. Let me exaggerate: say 1 million commits, 15 GB repo,\n> 50k tags, 1,000 branches. (Due to historical code fixups, another 5,000 \"fix-up\n> branches\" which are just one little dangling commit required to change the code\n> a little bit between a commit a tag that was not quite made from it.)\n\nI routinely work on a repo that's 1.9 GB packed, with 25k (and rapidly \ngrowing) refs. Other developers work on a repo that's 9 GB packed, with \nsomewhat fewer refs. We don't tend to have problems with this.\n\nObviously, performance is better on some of our smaller repos, but it's \nnot unacceptable on the larger ones. I generally find that the 940 KB \nrepo with huge numbers of files performs worse than the 1.9 GB repo with \nsomewhat fewer. If you can split your repository into multiple logical \nrepositories, that will certainly improve performance.\n\nIf you end up having pain points, we're certainly interested in \nworking through those. I've brought up performance problems and people \nare generally responsive.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"256375","messageId":"CACsJy8DkS65axQNY70FrfqR5s-49oOn8j7SAE9BTiRVNrm+ohQ@mail.gmail.com","threadId":"38604","inReplyTo":"CACBZZX6A+35wGBYAYj7E9d4XwLby21TLbTh-zRX+fkSt_e2zeg@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-20T00:04:36Z","receivedAt":"2015-02-20T00:04:36Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Feb 20, 2015 at 6:29 AM, Ævar Arnfjörð Bjarmason\n wrote:\n> Anecdotally I work on a repo at work (where I'm mostly \"the Git guy\") that's:\n>\n> * Around 500k commits\n> * Around 100k tags\n> * Around 5k branches\n> * Around 500 commits/day, almost entirely to the same branch\n> * 1.5 GB .git checkout.\n> * Mostly text source, but some binaries (we're trying to cut down[1] on those)\n\nWould be nice if you could make an anonymized version of this repo\npublic. Working on a \"real\" large repo is better than an artificial\none.\n\n> But actually most of \"git fetch\" is spent in the reachability check\n> subsequently done by \"git-rev-list\" which takes several seconds. I\n\nI wonder if reachability bitmap could help here..\n\n> haven't looked into it but there's got to be room for optimization\n> there, surely it only has to do reachability checks for new refs, or\n> could run in some \"I trust this remote not to send me corrupt data\"\n> completely mode (which would make sense within a company where you can\n> trust your main Git box).\n\nNo, it's not just about trusting the server side, it's about catching\ndata corruption on the wire as well. We have a trick to avoid\nreachability check in clone case, which is much more expensive than a\nfetch. Maybe we could do something further to help the fetch case _if_\nreachability bitmaps don't help.\n-- \nDuy\n"},{"id":"256376","messageId":"1424392969.30029.15.camel@leckie","threadId":"38604","inReplyTo":"CACsJy8Dortn4fHwF8xSgJ=KoJ9o1qzmc_UyaVq003D2sxFZEuQ@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-02-20T00:42:49Z","receivedAt":"2015-02-20T00:42:49Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2015-02-20 at 06:38 +0700, Duy Nguyen wrote:\n> > * 'git push'?\n> \n> This one is not affected by how deep your repo's history is, or how\n> wide your tree is, so should be quick..\n> \n> Ah the number of refs may affect both git-push and git-pull. I think\n> Stefan knows better than I in this area.\n\nI can tell you that this is a bit of a problem for us at Twitter. We\nhave over 100k refs, which adds ~20MiB of downstream traffic to every\npush.\n\nI added a hack to improve this locally inside Twitter: The client sends\na bloom filter of shas that it believes that the server knows about; the\nserver sends only the sha of master and any refs that are not in the\nbloom filter. The client uses its local version of the servers' refs\nas if they had just been sent. This means that some packs will be\nsuboptimal, due to false positives in the bloom filter leading some new\nrefs to not be sent. Also, if there were a repack between the pull and\nthe push, some refs might have been deleted on the server; we repack\nrarely enough and pull frequently enough that this is hopefully not an\nissue.\n\nWe're still testing to see if this works. But due to the number of\nassumptions it makes, it's probably not that great an idea for general\nuse.\n\nThere are probably more complex schemes to compute minimal (or\nsmall-enough) packs; in particular, if the patch is just a few megs off\nof master, it's better to just send the whole pack. That doesn't work\nfor us because we've got a log-based replication scheme that the pack\nappends to, and we don't want the log to get too big; we want\nmore-minimal packs than that. But it might work for others.\n"},{"id":"256413","messageId":"CACBZZX4T38j9YU3eiHTfkDoZKsgyJFrnJQNm5WBmb9RDenDOBg@mail.gmail.com","threadId":"38604","inReplyTo":"CACsJy8DkS65axQNY70FrfqR5s-49oOn8j7SAE9BTiRVNrm+ohQ@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2015-02-20T12:09:06Z","receivedAt":"2015-02-20T12:09:06Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Feb 20, 2015 at 1:04 AM, Duy Nguyen wrote:\n> On Fri, Feb 20, 2015 at 6:29 AM, Ævar Arnfjörð Bjarmason\n> wrote:\n>> Anecdotally I work on a repo at work (where I'm mostly \"the Git guy\") that's:\n>>\n>> * Around 500k commits\n>> * Around 100k tags\n>> * Around 5k branches\n>> * Around 500 commits/day, almost entirely to the same branch\n>> * 1.5 GB .git checkout.\n>> * Mostly text source, but some binaries (we're trying to cut down[1] on those)\n>\n> Would be nice if you could make an anonymized version of this repo\n> public. Working on a \"real\" large repo is better than an artificial\n> one.\n\nYeah, I'll try to do that.\n\n>> But actually most of \"git fetch\" is spent in the reachability check\n>> subsequently done by \"git-rev-list\" which takes several seconds. I\n>\n> I wonder if reachability bitmap could help here..\n\nI could have sworn I had that enabled already but evidently not. I did\ntest it and it cut down on clone times a bit. Now our daily repacking\nis:\n\n git --git-dir={} gc &&\n git --git-dir={} pack-refs --all --prune &&\n git --git-dir={} repack -Ad --window=250 --depth=100\n--write-bitmap-index --pack-kept-objects &&\n\nIt's not clear to me from the documentation whether this should just\nbe enabled on the server, or the clients too. In any case I've enabled\nit on both.\n\nEven then with it enabled on both a \"git pull\" that pulls down just\none commit on one branch is 13s. Trace attached at the end of the\nmail.\n\n>> haven't looked into it but there's got to be room for optimization\n>> there, surely it only has to do reachability checks for new refs, or\n>> could run in some \"I trust this remote not to send me corrupt data\"\n>> completely mode (which would make sense within a company where you can\n>> trust your main Git box).\n>\n> No, it's not just about trusting the server side, it's about catching\n> data corruption on the wire as well. We have a trick to avoid\n> reachability check in clone case, which is much more expensive than a\n> fetch. Maybe we could do something further to help the fetch case _if_\n> reachability bitmaps don't help.\n\nStill, if that's indeed a big bottleneck what's the worst-case\nscenario here? That the local repository gets hosed? The server will\nstill recursively validate the objects it gets sent, right?\n\nI wonder if a better trade-off in that case would be to skip this in\nsome situations and instead put something like \"git fsck\" in a\ncronjob.\n\nHere's a \"git pull\" trace mentioned above:\n\n$ time GIT_TRACE=1 git pull\n13:06:13.603781 git.c:555 trace: exec: 'git-pull'\n13:06:13.603936 run-command.c:351 trace: run_command: 'git-pull'\n13:06:13.620615 git.c:349 trace: built-in: git\n'rev-parse' '--git-dir'\n13:06:13.631602 git.c:349 trace: built-in: git\n'rev-parse' '--is-bare-repository'\n13:06:13.636103 git.c:349 trace: built-in: git\n'rev-parse' '--show-toplevel'\n13:06:13.641491 git.c:349 trace: built-in: git 'ls-files' '-u'\n13:06:13.719923 git.c:349 trace: built-in: git\n'symbolic-ref' '-q' 'HEAD'\n13:06:13.728085 git.c:349 trace: built-in: git 'config'\n'branch.trunk.rebase'\n13:06:13.738160 git.c:349 trace: built-in: git 'config' 'pull.ff'\n13:06:13.743286 git.c:349 trace: built-in: git\n'rev-parse' '-q' '--verify' 'HEAD'\n13:06:13.972091 git.c:349 trace: built-in: git\n'rev-parse' '--verify' 'HEAD'\n13:06:14.149420 git.c:349 trace: built-in: git\n'update-index' '-q' '--ignore-submodules' '--refresh'\n13:06:14.294098 git.c:349 trace: built-in: git\n'diff-files' '--quiet' '--ignore-submodules'\n13:06:14.467711 git.c:349 trace: built-in: git\n'diff-index' '--cached' '--quiet' '--ignore-submodules' 'HEAD' '--'\n13:06:14.683419 git.c:349 trace: built-in: git\n'rev-parse' '-q' '--git-dir'\n13:06:15.189707 git.c:349 trace: built-in: git\n'rev-parse' '-q' '--verify' 'HEAD'\n13:06:15.335948 git.c:349 trace: built-in: git 'fetch'\n'--update-head-ok'\n13:06:15.691303 run-command.c:351 trace: run_command: 'ssh'\n'git.example.com' 'git-upload-pack '\\''/gitrepos/core.git'\\'''\n13:06:17.095662 run-command.c:351 trace: run_command: 'rev-list'\n'--objects' '--stdin' '--not' '--all' '--quiet'\nremote: Counting objects: 6, done.\nremote: Compressing objects: 100% (6/6), done.\n3:06:20.426346 run-command.c:351 trace: run_command:\n'unpack-objects' '--pack_header=2,6'\n13:06:20.431806 exec_cmd.c:130 trace: exec: 'git'\n'unpack-objects' '--pack_header=2,6'\n13:06:20.437343 git.c:349 trace: built-in: git\n'unpack-objects' '--pack_header=2,6'\nremote: Total 6 (delta 0), reused 0 (delta 0)\nUnpacking objects: 100% (6/6), done.\n13:06:20.444196 run-command.c:351 trace: run_command: 'rev-list'\n'--objects' '--stdin' '--not' '--all'\n13:06:20.447135 exec_cmd.c:130 trace: exec: 'git' 'rev-list'\n'--objects' '--stdin' '--not' '--all'\n13:06:20.451283 git.c:349 trace: built-in: git\n'rev-list' '--objects' '--stdin' '--not' '--all'\nFrom ssh://git.example.com/gitrepos/core\n 02d33d2..41e72c4 core -> origin/core\n13:06:22.559609 run-command.c:351 trace: run_command: 'gc' '--auto'\n13:06:22.562176 exec_cmd.c:130 trace: exec: 'git' 'gc' '--auto'\n13:06:22.565661 git.c:349 trace: built-in: git 'gc' '--auto'\n13:06:22.594980 git.c:349 trace: built-in: git\n'rev-parse' '-q' '--verify' 'HEAD'\n13:06:22.845728 git.c:349 trace: built-in: git\n'show-branch' '--merge-base' 'refs/heads/core'\n'41e72c42addc5075e8009a3eebe914fa0ce98b27'\n'02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n13:06:23.087586 git.c:349 trace: built-in: git 'fmt-merge-msg'\n13:06:23.341451 git.c:349 trace: built-in: git\n'rev-parse' '--parseopt' '--stuck-long' '--' '--onto'\n'41e72c42addc5075e8009a3eebe914fa0ce98b27'\n'41e72c42addc5075e8009a3eebe914fa0ce98b27'\n13:06:23.350513 git.c:349 trace: built-in: git\n'rev-parse' '--git-dir'\n13:06:23.362011 git.c:349 trace: built-in: git\n'rev-parse' '--is-bare-repository'\n13:06:23.365282 git.c:349 trace: built-in: git\n'rev-parse' '--show-toplevel'\n13:06:23.372589 git.c:349 trace: built-in: git 'config'\n'--bool' 'rebase.stat'\n13:06:23.377056 git.c:349 trace: built-in: git 'config'\n'--bool' 'rebase.autostash'\n13:06:23.382102 git.c:349 trace: built-in: git 'config'\n'--bool' 'rebase.autosquash'\n13:06:23.389458 git.c:349 trace: built-in: git\n'rev-parse' '--verify' '41e72c42addc5075e8009a3eebe914fa0ce98b27^0'\n13:06:23.608894 git.c:349 trace: built-in: git\n'rev-parse' '--verify' '41e72c42addc5075e8009a3eebe914fa0ce98b27^0'\n13:06:23.894026 git.c:349 trace: built-in: git\n'symbolic-ref' '-q' 'HEAD'\n13:06:23.898918 git.c:349 trace: built-in: git\n'rev-parse' '--verify' 'HEAD'\n13:06:24.102269 git.c:349 trace: built-in: git\n'rev-parse' '--verify' 'HEAD'\n13:06:24.338636 git.c:349 trace: built-in: git\n'update-index' '-q' '--ignore-submodules' '--refresh'\n13:06:24.539912 git.c:349 trace: built-in: git\n'diff-files' '--quiet' '--ignore-submodules'\n13:06:24.729362 git.c:349 trace: built-in: git\n'diff-index' '--cached' '--quiet' '--ignore-submodules' 'HEAD' '--'\n13:06:24.938533 git.c:349 trace: built-in: git\n'merge-base' '41e72c42addc5075e8009a3eebe914fa0ce98b27'\n'02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n13:06:25.197791 git.c:349 trace: built-in: git 'diff'\n'--stat' '--summary' '02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n'41e72c42addc5075e8009a3eebe914fa0ce98b27'\n[details on updated files]\n13:06:25.488275 git.c:349 trace: built-in: git\n'checkout' '-q' '41e72c42addc5075e8009a3eebe914fa0ce98b27^0'\n13:06:26.467413 git.c:349 trace: built-in: git\n'update-ref' 'ORIG_HEAD' '02d33d2be7f8601c3502fdd89b0946447d7cdf15'\nFast-forwarded trunk to 41e72c42addc5075e8009a3eebe914fa0ce98b27.\n13:06:26.716256 git.c:349 trace: built-in: git 'rev-parse' 'HEAD'\n13:06:26.958595 git.c:349 trace: built-in: git\n'update-ref' '-m' 'rebase finished: refs/heads/core onto\n41e72c42addc5075e8009a3eebe914fa0ce98b27' 'refs/heads/core'\n'41e72c42addc5075e8009a3eebe914fa0ce98b27'\n'02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n13:06:27.205320 git.c:349 trace: built-in: git\n'symbolic-ref' '-m' 'rebase finished: returning to refs/heads/core'\n'HEAD' 'refs/heads/core'\n13:06:27.208748 git.c:349 trace: built-in: git 'gc' '--auto'\n"},{"id":"256414","messageId":"CACBZZX402LnNohUre2wayWO-xEhLYL3wi_KRgsOtjQzZ57ucDQ@mail.gmail.com","threadId":"38604","inReplyTo":"CACBZZX4T38j9YU3eiHTfkDoZKsgyJFrnJQNm5WBmb9RDenDOBg@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2015-02-20T12:11:49Z","receivedAt":"2015-02-20T12:11:49Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Feb 20, 2015 at 1:09 PM, Ævar Arnfjörð Bjarmason\n wrote:\n> On Fri, Feb 20, 2015 at 1:04 AM, Duy Nguyen wrote:\n>> On Fri, Feb 20, 2015 at 6:29 AM, Ævar Arnfjörð Bjarmason\n>> wrote:\n>>> Anecdotally I work on a repo at work (where I'm mostly \"the Git guy\") that's:\n>>>\n>>> * Around 500k commits\n>>> * Around 100k tags\n>>> * Around 5k branches\n>>> * Around 500 commits/day, almost entirely to the same branch\n>>> * 1.5 GB .git checkout.\n>>> * Mostly text source, but some binaries (we're trying to cut down[1] on those)\n>>\n>> Would be nice if you could make an anonymized version of this repo\n>> public. Working on a \"real\" large repo is better than an artificial\n>> one.\n>\n> Yeah, I'll try to do that.\n>\n>>> But actually most of \"git fetch\" is spent in the reachability check\n>>> subsequently done by \"git-rev-list\" which takes several seconds. I\n>>\n>> I wonder if reachability bitmap could help here..\n>\n> I could have sworn I had that enabled already but evidently not. I did\n> test it and it cut down on clone times a bit. Now our daily repacking\n> is:\n>\n> git --git-dir={} gc &&\n> git --git-dir={} pack-refs --all --prune &&\n> git --git-dir={} repack -Ad --window=250 --depth=100\n> --write-bitmap-index --pack-kept-objects &&\n>\n> It's not clear to me from the documentation whether this should just\n> be enabled on the server, or the clients too. In any case I've enabled\n> it on both.\n>\n> Even then with it enabled on both a \"git pull\" that pulls down just\n> one commit on one branch is 13s. Trace attached at the end of the\n> mail.\n>\n>>> haven't looked into it but there's got to be room for optimization\n>>> there, surely it only has to do reachability checks for new refs, or\n>>> could run in some \"I trust this remote not to send me corrupt data\"\n>>> completely mode (which would make sense within a company where you can\n>>> trust your main Git box).\n>>\n>> No, it's not just about trusting the server side, it's about catching\n>> data corruption on the wire as well. We have a trick to avoid\n>> reachability check in clone case, which is much more expensive than a\n>> fetch. Maybe we could do something further to help the fetch case _if_\n>> reachability bitmaps don't help.\n>\n> Still, if that's indeed a big bottleneck what's the worst-case\n> scenario here? That the local repository gets hosed? The server will\n> still recursively validate the objects it gets sent, right?\n>\n> I wonder if a better trade-off in that case would be to skip this in\n> some situations and instead put something like \"git fsck\" in a\n> cronjob.\n>\n> Here's a \"git pull\" trace mentioned above:\n>\n> $ time GIT_TRACE=1 git pull\n> 13:06:13.603781 git.c:555 trace: exec: 'git-pull'\n> 13:06:13.603936 run-command.c:351 trace: run_command: 'git-pull'\n> 13:06:13.620615 git.c:349 trace: built-in: git\n> 'rev-parse' '--git-dir'\n> 13:06:13.631602 git.c:349 trace: built-in: git\n> 'rev-parse' '--is-bare-repository'\n> 13:06:13.636103 git.c:349 trace: built-in: git\n> 'rev-parse' '--show-toplevel'\n> 13:06:13.641491 git.c:349 trace: built-in: git 'ls-files' '-u'\n> 13:06:13.719923 git.c:349 trace: built-in: git\n> 'symbolic-ref' '-q' 'HEAD'\n> 13:06:13.728085 git.c:349 trace: built-in: git 'config'\n> 'branch.trunk.rebase'\n> 13:06:13.738160 git.c:349 trace: built-in: git 'config' 'pull.ff'\n> 13:06:13.743286 git.c:349 trace: built-in: git\n> 'rev-parse' '-q' '--verify' 'HEAD'\n> 13:06:13.972091 git.c:349 trace: built-in: git\n> 'rev-parse' '--verify' 'HEAD'\n> 13:06:14.149420 git.c:349 trace: built-in: git\n> 'update-index' '-q' '--ignore-submodules' '--refresh'\n> 13:06:14.294098 git.c:349 trace: built-in: git\n> 'diff-files' '--quiet' '--ignore-submodules'\n> 13:06:14.467711 git.c:349 trace: built-in: git\n> 'diff-index' '--cached' '--quiet' '--ignore-submodules' 'HEAD' '--'\n> 13:06:14.683419 git.c:349 trace: built-in: git\n> 'rev-parse' '-q' '--git-dir'\n> 13:06:15.189707 git.c:349 trace: built-in: git\n> 'rev-parse' '-q' '--verify' 'HEAD'\n> 13:06:15.335948 git.c:349 trace: built-in: git 'fetch'\n> '--update-head-ok'\n> 13:06:15.691303 run-command.c:351 trace: run_command: 'ssh'\n> 'git.example.com' 'git-upload-pack '\\''/gitrepos/core.git'\\'''\n> 13:06:17.095662 run-command.c:351 trace: run_command: 'rev-list'\n> '--objects' '--stdin' '--not' '--all' '--quiet'\n> remote: Counting objects: 6, done.\n> remote: Compressing objects: 100% (6/6), done.\n> 3:06:20.426346 run-command.c:351 trace: run_command:\n> 'unpack-objects' '--pack_header=2,6'\n> 13:06:20.431806 exec_cmd.c:130 trace: exec: 'git'\n> 'unpack-objects' '--pack_header=2,6'\n> 13:06:20.437343 git.c:349 trace: built-in: git\n> 'unpack-objects' '--pack_header=2,6'\n> remote: Total 6 (delta 0), reused 0 (delta 0)\n> Unpacking objects: 100% (6/6), done.\n> 13:06:20.444196 run-command.c:351 trace: run_command: 'rev-list'\n> '--objects' '--stdin' '--not' '--all'\n> 13:06:20.447135 exec_cmd.c:130 trace: exec: 'git' 'rev-list'\n> '--objects' '--stdin' '--not' '--all'\n> 13:06:20.451283 git.c:349 trace: built-in: git\n> 'rev-list' '--objects' '--stdin' '--not' '--all'\n> From ssh://git.example.com/gitrepos/core\n> 02d33d2..41e72c4 core -> origin/core\n> 13:06:22.559609 run-command.c:351 trace: run_command: 'gc' '--auto'\n> 13:06:22.562176 exec_cmd.c:130 trace: exec: 'git' 'gc' '--auto'\n> 13:06:22.565661 git.c:349 trace: built-in: git 'gc' '--auto'\n> 13:06:22.594980 git.c:349 trace: built-in: git\n> 'rev-parse' '-q' '--verify' 'HEAD'\n> 13:06:22.845728 git.c:349 trace: built-in: git\n> 'show-branch' '--merge-base' 'refs/heads/core'\n> '41e72c42addc5075e8009a3eebe914fa0ce98b27'\n> '02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n> 13:06:23.087586 git.c:349 trace: built-in: git 'fmt-merge-msg'\n> 13:06:23.341451 git.c:349 trace: built-in: git\n> 'rev-parse' '--parseopt' '--stuck-long' '--' '--onto'\n> '41e72c42addc5075e8009a3eebe914fa0ce98b27'\n> '41e72c42addc5075e8009a3eebe914fa0ce98b27'\n> 13:06:23.350513 git.c:349 trace: built-in: git\n> 'rev-parse' '--git-dir'\n> 13:06:23.362011 git.c:349 trace: built-in: git\n> 'rev-parse' '--is-bare-repository'\n> 13:06:23.365282 git.c:349 trace: built-in: git\n> 'rev-parse' '--show-toplevel'\n> 13:06:23.372589 git.c:349 trace: built-in: git 'config'\n> '--bool' 'rebase.stat'\n> 13:06:23.377056 git.c:349 trace: built-in: git 'config'\n> '--bool' 'rebase.autostash'\n> 13:06:23.382102 git.c:349 trace: built-in: git 'config'\n> '--bool' 'rebase.autosquash'\n> 13:06:23.389458 git.c:349 trace: built-in: git\n> 'rev-parse' '--verify' '41e72c42addc5075e8009a3eebe914fa0ce98b27^0'\n> 13:06:23.608894 git.c:349 trace: built-in: git\n> 'rev-parse' '--verify' '41e72c42addc5075e8009a3eebe914fa0ce98b27^0'\n> 13:06:23.894026 git.c:349 trace: built-in: git\n> 'symbolic-ref' '-q' 'HEAD'\n> 13:06:23.898918 git.c:349 trace: built-in: git\n> 'rev-parse' '--verify' 'HEAD'\n> 13:06:24.102269 git.c:349 trace: built-in: git\n> 'rev-parse' '--verify' 'HEAD'\n> 13:06:24.338636 git.c:349 trace: built-in: git\n> 'update-index' '-q' '--ignore-submodules' '--refresh'\n> 13:06:24.539912 git.c:349 trace: built-in: git\n> 'diff-files' '--quiet' '--ignore-submodules'\n> 13:06:24.729362 git.c:349 trace: built-in: git\n> 'diff-index' '--cached' '--quiet' '--ignore-submodules' 'HEAD' '--'\n> 13:06:24.938533 git.c:349 trace: built-in: git\n> 'merge-base' '41e72c42addc5075e8009a3eebe914fa0ce98b27'\n> '02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n> 13:06:25.197791 git.c:349 trace: built-in: git 'diff'\n> '--stat' '--summary' '02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n> '41e72c42addc5075e8009a3eebe914fa0ce98b27'\n> [details on updated files]\n> 13:06:25.488275 git.c:349 trace: built-in: git\n> 'checkout' '-q' '41e72c42addc5075e8009a3eebe914fa0ce98b27^0'\n> 13:06:26.467413 git.c:349 trace: built-in: git\n> 'update-ref' 'ORIG_HEAD' '02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n> Fast-forwarded trunk to 41e72c42addc5075e8009a3eebe914fa0ce98b27.\n> 13:06:26.716256 git.c:349 trace: built-in: git 'rev-parse' 'HEAD'\n> 13:06:26.958595 git.c:349 trace: built-in: git\n> 'update-ref' '-m' 'rebase finished: refs/heads/core onto\n> 41e72c42addc5075e8009a3eebe914fa0ce98b27' 'refs/heads/core'\n> '41e72c42addc5075e8009a3eebe914fa0ce98b27'\n> '02d33d2be7f8601c3502fdd89b0946447d7cdf15'\n> 13:06:27.205320 git.c:349 trace: built-in: git\n> 'symbolic-ref' '-m' 'rebase finished: returning to refs/heads/core'\n> 'HEAD' 'refs/heads/core'\n> 13:06:27.208748 git.c:349 trace: built-in: git 'gc' '--auto'\n\nI forgot to include that this took:\n\nreal 0m13.630s\nuser 0m10.739s\nsys 0m4.064s\n\non my local laptop with a ssd + hot cache it was:\n\nreal 0m7.513s\nuser 0m3.796s\nsys 0m0.624s\n\nSo some of that we could speed up with faster systems, but we still\nhave quite a bit of Git overhead.\n\nEven with the hot cache on the ssd I get on this repo:\n\n$ time (git log -1 >/dev/null)\n\nreal 0m0.938s\nuser 0m0.916s\nsys 0m0.020s\n\nv.s. the same on linux.git:\n\n$ time (git log -1 >/dev/null)\n\nreal 0m0.016s\nuser 0m0.008s\nsys 0m0.004s\n\nWhich I suspect is a function of the high ref count, but it could be\nsomething else...\n"},{"id":"256418","messageId":"CACBZZX45eCo6YS4EpHvMQjN32+-w5BztfoLiwh_rJTs7FydgoQ@mail.gmail.com","threadId":"38604","inReplyTo":"CACBZZX4T38j9YU3eiHTfkDoZKsgyJFrnJQNm5WBmb9RDenDOBg@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2015-02-20T14:25:59Z","receivedAt":"2015-02-20T14:25:59Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Feb 20, 2015 at 1:09 PM, Ævar Arnfjörð Bjarmason\n wrote:\n> On Fri, Feb 20, 2015 at 1:04 AM, Duy Nguyen wrote:\n>> On Fri, Feb 20, 2015 at 6:29 AM, Ævar Arnfjörð Bjarmason\n>> wrote:\n>>> Anecdotally I work on a repo at work (where I'm mostly \"the Git guy\") that's:\n>>>\n>>> * Around 500k commits\n>>> * Around 100k tags\n>>> * Around 5k branches\n>>> * Around 500 commits/day, almost entirely to the same branch\n>>> * 1.5 GB .git checkout.\n>>> * Mostly text source, but some binaries (we're trying to cut down[1] on those)\n>>\n>> Would be nice if you could make an anonymized version of this repo\n>> public. Working on a \"real\" large repo is better than an artificial\n>> one.\n>\n> Yeah, I'll try to do that.\n\ntl;dr: After some more testing it turns out the performance issues we\nhave are almost entirely due to the number of refs. Some of these I\nknew about and were obvious (e..g. git pull), but some aren't so\nobvious (why does \"git log\" without \"--all\" slow down as a function of\nthe overall number of refs?).\n\nRather than getting an anonymized version of the repo we have, a\nsimpler isolated test case is just doing this on linux.git:\n\n $ git rev-list --all | perl -ne 'my $cnt; while (<>) {\ns<([a-f0-9]+)>gm; next unless int rand 10\n== 1; $cnt++; s/TAG/tagnr-$cnt/; print }' | sh -x\n\nThat'll create a tag for every 10th commit or so, which is around 50k\ntags for linux.git.\n\nI actually ran this a few times while testing it, so this is a before\nand after on a hot cache of linux.git with 406 tags v.s. ~140k. I ran\nthe gc + repack + bitmaps for both repos noted in an earlier reply of\nmine, and took the fastest run out of 3:\n\n $ time (git log master -100 >/dev/null)\n Before: real 0m0.021s\n After: real 0m2.929s\n $ time (git status >/dev/null)\n # Around 150ms, no noticeable difference\n $ time git fetch\n # I'm fetching from git@github.com:torvalds/linux.git here, the\n # cache is hot but upstream has *no* changes\n Before: real 0m1.826s\n After: real 0m8.458s\n\nDetails on why \"git fetch\" is slow in this situation:\n\n $ time GIT_TRACE=1 git fetch\n 15:15:00.435420 git.c:349 trace: built-in: git 'fetch'\n 15:15:00.654428 run-command.c:341 trace: run_command: 'ssh'\n'git@github.com' 'git-upload-pack '\\''torvalds/linux.git'\\'''\n 15:15:02.426121 run-command.c:341 trace: run_command:\n'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'\n 15:15:05.507327 run-command.c:341 trace: run_command:\n'rev-list' '--objects' '--stdin' '--not' '--all'\n 15:15:05.508329 exec_cmd.c:134 trace: exec: 'git'\n'rev-list' '--objects' '--stdin' '--not' '--all'\n 15:15:05.510490 git.c:349 trace: built-in: git\n'rev-list' '--objects' '--stdin' '--not' '--all'\n 15:15:08.874116 run-command.c:341 trace: run_command: 'gc' '--auto'\n 15:15:08.879570 exec_cmd.c:134 trace: exec: 'git' 'gc' '--auto'\n 15:15:08.882495 git.c:349 trace: built-in: git 'gc' '--auto'\n real 0m8.458s\n user 0m6.548s\n sys 0m0.204s\n\nEven things you'd expect to not be impacted are, like a reverse log\nsearch on the master branch:\n\n $ time (git log --reverse -p --grep=arm64 origin/master >/dev/null)\n Before: real 0m4.473s\n After: real 0m6.194s\n\nOr doing 10 commits and rebasing on the upstream:\n\n $ time (git checkout origin/master~ && for i in {1..10}; do echo\n$i > file && git add file && git commit -m\"moo\" $file; done && git\nrebase origin/master)\n Before: real 0m6.798s\n After: real 0m12.340s\n\nThe remaining slowdown comes from the size of the tree, which we can\ndeal with by either reducing it in size (we have some copied JS\nlibraries and whatnot) or trying the inotify-powered git-status.\n\nIn our case there's no good reason for why we have this many refs in\nthe repository everyone uses. We basically just have a bunch of dated\nrollout tags that have been accumulating for years, and a bunch of\nmostly unused branches people just haven't cleaned up.\n\nSo I'm going to:\n\n 1. Write a hook that rejects tags that aren't new (i.e. forbid\nre-pushes of old tags)\n 2. Create an archive repository that contains all the old tags (i.e.\njust run \"git fetch\" on the main one from cron)\n 3. Run a script to regularly delete tags from the main repo\n 4. Run the same script on the clients that clone the repo\n\nThe branches are slightly harder, deleting those that are fully merged\ninto the same branch is easy, deleting those whose contents 100%\nmatches patch-id's already in the main branch is another thing we can\ndo, and just clean up branches unconditionally after they've reached a\ncertain age (they'll still be archived).\n"},{"id":"256421","messageId":"CAH8BJxEWDb0SDHPS_ZnPzz0QEbryw2GCv2RtJm2u_6rPH566hg@mail.gmail.com","threadId":"38604","inReplyTo":"20150220000320.GD5021@vauxhall.crustytoothpaste.net","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Stephen Morton","fromEmail":"stephen.c.morton@gmail.com","sentAt":"2015-02-20T16:06:44Z","receivedAt":"2015-02-20T16:06:44Z","isPatch":false,"sender":{"key":"stephen.c.morton@gmail.com","avatar":"https://gravatar.com/avatar/3b826f06cabc070cbd5d53071880caad8badc3b29e0789b510353faca59ffb7b?d=mp&s=160"},"body":"This is fantastic. I really appreciate all the answers. And it's great\nthat I think I've sparked some general discussion that could lead\nsomewhere too.\n\nNotes:\n\nI'm currently using 2.1.3. I'll move to 2.3.x\n\nI'm experimenting with git-annex to reduce repo size on disk. We'll see.\n\nI could remove all tags older than /n/ years old in the active repo\nand just maintain them in the historical repo. (We have quite a lot of\nCI-generated tags.) It sounds like that might improve performance.\n\nQuestions:\n\n1. Ævar : I'm a bit concerned by your statement that git rebases take\nabout 1-2 s per commit. Does that mean that a \"git pull --rebase\", if\nit is picking up say 120 commits (not at all unrealistic), could\npotentially take 4 minutes to complete? Or have I misinterpreted your\ncomment.\n\n2. I'd not heard about bitmap indexes before this thread but it sounds\nlike they should help me. In limited searching I can't find much\nuseful documentation about them. It is also not clear to me if I have\nto explicitly run \"git repack --write-bitmap-indexes\" or if git will\nautomatically detect when they're needed; first experiments seem to\nindicate that I need to explicitly generate them. I assume that once\nthe index is there, git will just use it automatically.\n\n\nSteve\n\n\nOn Thu, Feb 19, 2015 at 7:03 PM, brian m. carlson\n wrote:\n> On Thu, Feb 19, 2015 at 04:26:58PM -0500, Stephen Morton wrote:\n>> I posted this to comp.version-control.git.user and didn't get any response. I\n>> think the question is plumbing-related enough that I can ask it here.\n>>\n>> I'm evaluating the feasibility of moving my team from SVN to git. We have a very\n>> large repo. [1] We will have a central repo using GitLab (or similar) that\n>> everybody works with. Forks, code sharing, pull requests etc. will be done\n>> through this central server.\n>>\n>> By 'performance', I guess I mean speed of day to day operations for devs.\n>>\n>> * (Obviously, trivially, a (non-local) clone will be slow with a large repo.)\n>> * Will a few simultaneous clones from the central server also slow down\n>> other concurrent operations for other users?\n>\n> This hasn't been a problem for us at $DAYJOB. Git doesn't lock anything\n> on fetches, so each process is independent. We probably have about\n> sixty developers (and maybe twenty other occasional users) that manage\n> to interact with our Git server all day long. We also have probably\n> twenty smoker (CI) systems pulling at two hour intervals, or, when\n> there's nothing to do, every two minutes, plus probably fifteen to\n> twenty build systems pulling hourly.\n>\n> I assume you will provide adequate resources for your server.\n>\n>> * Will 'git pull' be slow?\n>> * 'git push'?\n>\n> The most pathological case I've seen for git push is a branch with a\n> single commit merged into the main development branch. As of Git 2.3.0,\n> the performance regression here is fixed.\n>\n> Obviously, the speed of your network connection will affect this. Even\n> at 30 MB/s, cloning several gigabytes of data takes time. Git tries\n> hard to eliminate sending a lot of data, so if your developers keep\n> reasonably up-to-date, the cost of establishing the connection will tend\n> to dominate.\n>\n> I see pull and push times that are less than 2 seconds in most cases.\n>\n>> * 'git commit'? (It is listed as slow in reference [3].)\n>> * 'git stautus'? (Slow again in reference 3 though I don't see it.)\n>\n> These can be slow with slow disks or over remote file systems. I\n> recommend not doing that. I've heard rumbles that disk performance is\n> better on Unix, but I don't use Windows so I can't say.\n>\n> You should keep your .gitignore files up-to-date to avoid enumerating\n> untracked files. There's some work towards making this less of an\n> issue.\n>\n> git blame can be somewhat slow, but it's not something I use more than\n> about once a day, so it doesn't bother me that much.\n>\n>> Assuming I can put lots of resources into a central server with lots of CPU,\n>> RAM, fast SSD, fast networking, what aspects of the repo are most likely to\n>> affect devs' experience?\n>> * Number of commits\n>> * Sheer disk space occupied by the repo\n>\n> The number of files can impact performance due to the number of stat()s\n> required.\n>\n>> * Number of tags.\n>> * Number of branches.\n>\n> The number of tags and branches individually is really less relevant\n> than the total number of refs (tags, branches, remote branches, etc).\n> Very large numbers of refs can impact performance on pushes and pulls\n> due to the need to enumerate them all.\n>\n>> * Binary objects in the repo that cause it to bloat in size [1]\n>> * Other factors?\n>\n> If you want good performance, I'd recommend the latest version of Git\n> both client- and server-side. Newer versions of Git provide pack\n> bitmaps, which can dramatically speed up clones and fetches, and Git\n> 2.3.0 fixes a performance regression with large numbers of refs in\n> non-shallow repositories.\n>\n> It is totally worth it to roll your own packages of git if your vendor\n> provides old versions.\n>\n>> Of the various HW items listed above --CPU speed, number of cores, RAM, SSD,\n>> networking-- which is most critical here?\n>\n> I generally find that having a good disk cache is important with large\n> repositories. It may be advantageous to make sure the developer\n> machines have adequate memory. Performance is notably better on\n> development machines (VMs) with 2 GB or 4 GB of memory instead of 1 GB.\n>\n> I can't speak to the server side, as I'm not directly involved with its\n> deployment.\n>\n>> Assume ridiculous numbers. Let me exaggerate: say 1 million commits, 15 GB repo,\n>> 50k tags, 1,000 branches. (Due to historical code fixups, another 5,000 \"fix-up\n>> branches\" which are just one little dangling commit required to change the code\n>> a little bit between a commit a tag that was not quite made from it.)\n>\n> I routinely work on a repo that's 1.9 GB packed, with 25k (and rapidly\n> growing) refs. Other developers work on a repo that's 9 GB packed, with\n> somewhat fewer refs. We don't tend to have problems with this.\n>\n> Obviously, performance is better on some of our smaller repos, but it's\n> not unacceptable on the larger ones. I generally find that the 940 KB\n> repo with huge numbers of files performs worse than the 1.9 GB repo with\n> somewhat fewer. If you can split your repository into multiple logical\n> repositories, that will certainly improve performance.\n>\n> If you end up having pain points, we're certainly interested in\n> working through those. I've brought up performance problems and people\n> are generally responsive.\n> --\n> brian m. carlson / brian with sandals: Houston, Texas, US\n> +1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\n> OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"256423","messageId":"vpq38604jxe.fsf@anie.imag.fr","threadId":"38604","inReplyTo":"CAH8BJxEWDb0SDHPS_ZnPzz0QEbryw2GCv2RtJm2u_6rPH566hg@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2015-02-20T16:38:53Z","receivedAt":"2015-02-20T16:38:53Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Stephen Morton writes:\n\n> 1. Ævar : I'm a bit concerned by your statement that git rebases take\n> about 1-2 s per commit. Does that mean that a \"git pull --rebase\", if\n> it is picking up say 120 commits (not at all unrealistic), could\n> potentially take 4 minutes to complete? Or have I misinterpreted your\n> comment.\n\nÆvar talked about \"applied commits\" during rebase. When you \"git pull\n--rebase\", you fast-forward the history you just fetched, which is\nalmost instantaneous, and then you reapply your local history on top of\nit.\n\nSo, the performance depends on how long your local history is, not on\nhow many commits you're fetching.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"256425","messageId":"20150220171614.GA5730@vauxhall.crustytoothpaste.net","threadId":"38604","inReplyTo":"CAH8BJxEWDb0SDHPS_ZnPzz0QEbryw2GCv2RtJm2u_6rPH566hg@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2015-02-20T17:16:14Z","receivedAt":"2015-02-20T17:16:14Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Feb 20, 2015 at 11:06:44AM -0500, Stephen Morton wrote:\n>2. I'd not heard about bitmap indexes before this thread but it sounds\n>like they should help me. In limited searching I can't find much\n>useful documentation about them. It is also not clear to me if I have\n>to explicitly run \"git repack --write-bitmap-indexes\" or if git will\n>automatically detect when they're needed; first experiments seem to\n>indicate that I need to explicitly generate them. I assume that once\n>the index is there, git will just use it automatically.\n\nPack bitmaps are a way of speeding up clones and fetches by precomputing \nreachability information. Practically, this means that the initial \n\"Counting objects\" phase is instantaneous for clones and much faster for \nfetches.\n\nThe way I've done it in the past is to set repack.writeBitmaps = true in \n/etc/gitconfig on the server. (I highly recommend enabling reflogs in \nthe same place.) Then you'll need to ensure that git gc runs \nperiodically so that bitmaps are generated.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"256436","messageId":"xmqqegpkz4cf.fsf@gitster.dls.corp.google.com","threadId":"38604","inReplyTo":"1424392969.30029.15.camel@leckie","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-20T20:59:44Z","receivedAt":"2015-02-20T20:59:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> On Fri, 2015-02-20 at 06:38 +0700, Duy Nguyen wrote:\n>> > * 'git push'?\n>> \n>> This one is not affected by how deep your repo's history is, or how\n>> wide your tree is, so should be quick..\n>> \n>> Ah the number of refs may affect both git-push and git-pull. I think\n>> Stefan knows better than I in this area.\n>\n> I can tell you that this is a bit of a problem for us at Twitter. We\n> have over 100k refs, which adds ~20MiB of downstream traffic to every\n> push.\n>\n> I added a hack to improve this locally inside Twitter: The client sends\n> a bloom filter of shas that it believes that the server knows about; the\n> server sends only the sha of master and any refs that are not in the\n> bloom filter. The client uses its local version of the servers' refs\n> as if they had just been sent....\n\nInteresting.\n\nCare to extend the discussion to improve the protocol exchange,\nwhich starts at $gmane/263932 [*1*], where I list the known issues\naround the current protocol (and a possible way to correct them in\nfootnotes)?\n\n\n[Footnote]\n\n*1* http://thread.gmane.org/gmane.comp.version-control.git/263898/focus=263932\n"},{"id":"256437","messageId":"xmqqa908z44h.fsf@gitster.dls.corp.google.com","threadId":"38604","inReplyTo":"CACBZZX45eCo6YS4EpHvMQjN32+-w5BztfoLiwh_rJTs7FydgoQ@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-20T21:04:30Z","receivedAt":"2015-02-20T21:04:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> I actually ran this a few times while testing it, so this is a before\n> and after on a hot cache of linux.git with 406 tags v.s. ~140k. I ran\n> the gc + repack + bitmaps for both repos noted in an earlier reply of\n> mine, and took the fastest run out of 3:\n>\n> $ time (git log master -100 >/dev/null)\n> Before: real 0m0.021s\n> After: real 0m2.929s\n\nDo you force --decorate with some config? Or do you see similar\nperformance difference with \"git rev-parse master\", too?\n\n> $ time (git status >/dev/null)\n> # Around 150ms, no noticeable difference\n\nThis is understandable, as it will not look at any ref other than\nHEAD.\n"},{"id":"256439","messageId":"54E7AF11.6000600@gmail.com","threadId":"38604","inReplyTo":"CACBZZX45eCo6YS4EpHvMQjN32+-w5BztfoLiwh_rJTs7FydgoQ@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2015-02-20T22:02:57Z","receivedAt":"2015-02-20T22:02:57Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On 20.02.2015 15:25, Ævar Arnfjörð Bjarmason wrote:\n\n> tl;dr: After some more testing it turns out the performance issues we\n> have are almost entirely due to the number of refs. Some of these I\n\nInteresting. We currently have similar performance issues when pushing \nto a Git repo hosted on Gerrit. The only difference of our repo compared \nto others on the same server that do not have any performance issues is \nthe large number of refs (about 40k) in refs/changes.\n\nI still wonder why that would slow down a push as we have not configured \nour clients to fetch / push refs/changes.\n\n-- \nSebastian Schuberth\n"},{"id":"256440","messageId":"54E7B077.3050501@gmail.com","threadId":"38604","inReplyTo":"20150220000320.GD5021@vauxhall.crustytoothpaste.net","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2015-02-20T22:08:55Z","receivedAt":"2015-02-20T22:08:55Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On 20.02.2015 01:03, brian m. carlson wrote:\n\n> If you want good performance, I'd recommend the latest version of Git\n> both client- and server-side. Newer versions of Git provide pack\n> bitmaps, which can dramatically speed up clones and fetches, and Git\n\nDo you happen now which version, if at all, of JGit and Gerrit support \npack bitmaps?\n\n> 2.3.0 fixes a performance regression with large numbers of refs in\n> non-shallow repositories.\n\nDo you also know in what Git version the regression was introduced?\n\n-- \nSebastian Schuberth\n"},{"id":"256444","messageId":"20150220225827.GA23852@vauxhall.crustytoothpaste.net","threadId":"38604","inReplyTo":"54E7B077.3050501@gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2015-02-20T22:58:27Z","receivedAt":"2015-02-20T22:58:27Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Feb 20, 2015 at 11:08:55PM +0100, Sebastian Schuberth wrote:\n>On 20.02.2015 01:03, brian m. carlson wrote:\n>\n>>If you want good performance, I'd recommend the latest version of Git\n>>both client- and server-side. Newer versions of Git provide pack\n>>bitmaps, which can dramatically speed up clones and fetches, and Git\n>\n>Do you happen now which version, if at all, of JGit and Gerrit support \n>pack bitmaps?\n\nThey were originally implemented in JGit, but I don't know what version, \nsorry. Some googling tells me that it's probably version 3.0.\n\n>>2.3.0 fixes a performance regression with large numbers of refs in\n>>non-shallow repositories.\n>\n>Do you also know in what Git version the regression was introduced?\n\nv1.8.4-rc3-8-gfbd4a70. It was fixed in v2.2.1-65-g2dacf26.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"256453","messageId":"CACsJy8DxTC8xG+emwGUM6qJT-ocv1vWVSLe8S=WdTOe_Kb7+bg@mail.gmail.com","threadId":"38604","inReplyTo":"CACBZZX4T38j9YU3eiHTfkDoZKsgyJFrnJQNm5WBmb9RDenDOBg@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-21T03:51:31Z","receivedAt":"2015-02-21T03:51:31Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Feb 20, 2015 at 7:09 PM, Ævar Arnfjörð Bjarmason\n wrote:\n>>> But actually most of \"git fetch\" is spent in the reachability check\n>>> subsequently done by \"git-rev-list\" which takes several seconds. I\n>>\n>> I wonder if reachability bitmap could help here..\n>\n> I could have sworn I had that enabled already but evidently not. I did\n> test it and it cut down on clone times a bit. Now our daily repacking\n> is:\n>\n> git --git-dir={} gc &&\n> git --git-dir={} pack-refs --all --prune &&\n> git --git-dir={} repack -Ad --window=250 --depth=100\n> --write-bitmap-index --pack-kept-objects &&\n>\n> It's not clear to me from the documentation whether this should just\n> be enabled on the server, or the clients too. In any case I've enabled\n> it on both.\n\nPack bitmaps matter most on the server side. What I was not sure was\nwhether it helped the client side as well because you do rev-list on\nthe client side for reachability test. But thinking again, I don't\nthink enabling pack bitmaps on the client side helps much. The \"--not\n--all\" part in rev-list basically just traverses commits, not trees\nand objects (where pack bitmaps shine). The big problem here is\n\"--all\" which will go examine all refs. So big ref number problem\nagain..\n\n> Even then with it enabled on both a \"git pull\" that pulls down just\n> one commit on one branch is 13s. Trace attached at the end of the\n> mail.\n>\n>>> haven't looked into it but there's got to be room for optimization\n>>> there, surely it only has to do reachability checks for new refs, or\n>>> could run in some \"I trust this remote not to send me corrupt data\"\n>>> completely mode (which would make sense within a company where you can\n>>> trust your main Git box).\n>>\n>> No, it's not just about trusting the server side, it's about catching\n>> data corruption on the wire as well. We have a trick to avoid\n>> reachability check in clone case, which is much more expensive than a\n>> fetch. Maybe we could do something further to help the fetch case _if_\n>> reachability bitmaps don't help.\n>\n> Still, if that's indeed a big bottleneck what's the worst-case\n> scenario here? That the local repository gets hosed? The server will\n> still recursively validate the objects it gets sent, right?\n\nThe server is under pressure to pack and send data fast so it does not\nvalidate as heavily as the client. When deltas are reused, only crc32\nis verified. When deltas are generated, the server must unpack some\nobjects for deltification, but I don't think it rehashes the content\nto see if it produces the same SHA-1. Single bit flips could go\nunnoticed..\n\n> I wonder if a better trade-off in that case would be to skip this in\n> some situations and instead put something like \"git fsck\" in a\n> cronjob.\n\nEither that or be optimistic, accept the pack (i.e. git-fetch returns\nquickly) and validate it in the background. If the pack is indeed\ngood, you don't have to wait until validation is done. If the pack is\nbad, you would know after a minute or two, hopefully you can still\nrecover from that point.\n-- \nDuy\n"},{"id":"256454","messageId":"CACsJy8B6pLnx84Xxb0_p2+YuWTgg_8h0yQt7K+AXiaEaC4jSeQ@mail.gmail.com","threadId":"38604","inReplyTo":"1424392969.30029.15.camel@leckie","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-21T04:01:12Z","receivedAt":"2015-02-21T04:01:12Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Feb 20, 2015 at 7:42 AM, David Turner wrote:\n> On Fri, 2015-02-20 at 06:38 +0700, Duy Nguyen wrote:\n>> > * 'git push'?\n>>\n>> This one is not affected by how deep your repo's history is, or how\n>> wide your tree is, so should be quick..\n>>\n>> Ah the number of refs may affect both git-push and git-pull. I think\n>> Stefan knows better than I in this area.\n>\n> I can tell you that this is a bit of a problem for us at Twitter. We\n> have over 100k refs, which adds ~20MiB of downstream traffic to every\n> push.\n>\n> I added a hack to improve this locally inside Twitter: The client sends\n> a bloom filter of shas that it believes that the server knows about; the\n> server sends only the sha of master and any refs that are not in the\n> bloom filter. The client uses its local version of the servers' refs\n> as if they had just been sent. This means that some packs will be\n> suboptimal, due to false positives in the bloom filter leading some new\n> refs to not be sent. Also, if there were a repack between the pull and\n> the push, some refs might have been deleted on the server; we repack\n> rarely enough and pull frequently enough that this is hopefully not an\n> issue.\n\nI wonder how efficient rsync is for transferring these refs: the\nclient generates a \"file\" containing all refs, the server does the\nsame with their refs, then the client rsync their file to the server..\nThe changes between the server and the client files are usually small,\nI'm hoping rsync can take advantage of that.\n-- \nDuy\n"},{"id":"256542","messageId":"1424722996.27803.29.camel@leckie","threadId":"38604","inReplyTo":"xmqqegpkz4cf.fsf@gitster.dls.corp.google.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-02-23T20:23:16Z","receivedAt":"2015-02-23T20:23:16Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"\nOn Fri, 2015-02-20 at 12:59 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > On Fri, 2015-02-20 at 06:38 +0700, Duy Nguyen wrote:\n> >> > * 'git push'?\n> >> \n> >> This one is not affected by how deep your repo's history is, or how\n> >> wide your tree is, so should be quick..\n> >> \n> >> Ah the number of refs may affect both git-push and git-pull. I think\n> >> Stefan knows better than I in this area.\n> >\n> > I can tell you that this is a bit of a problem for us at Twitter. We\n> > have over 100k refs, which adds ~20MiB of downstream traffic to every\n> > push.\n> >\n> > I added a hack to improve this locally inside Twitter: The client sends\n> > a bloom filter of shas that it believes that the server knows about; the\n> > server sends only the sha of master and any refs that are not in the\n> > bloom filter. The client uses its local version of the servers' refs\n> > as if they had just been sent....\n> \n> Interesting.\n> \n> Care to extend the discussion to improve the protocol exchange,\n> which starts at $gmane/263932 [*1*], where I list the known issues\n> around the current protocol (and a possible way to correct them in\n> footnotes)?\n\nAt Twitter, we changed to an entirely different clone strategy for our\nlargest repo: instead of using git clone, we use bittorrent (on a\ntarball of the repo). For git pull, we maintain a journal of all pushes\never made to the server (data and ref updates); each client keeps track\nof their location in that journal. So now pull does not require any\ncomputation on the server; the client just requests the segment of the\njournal that they don't have. Then the client replays the journal.\nThis scheme isn't perfect: clients end up with data about even\ntransitory and long-dead branches, and there is presently no way to\nredact data (although that would be possible to add). And of course\nshallow and sparse clones are impossible. But it works quite well for\nTwitter's needs. As I understand it, the hope is to implement redaction\nand then submit patches upstream.\n\nI say \"we\", but I personally did not do any of the above work. Because\nI haven't looked into most of these issues personally, I'm reluctant to\nsay too much on protocol improvements. I would want to better\nunderstand the constraints. I do think there is value in having a\ndiversity of possible protocols to handle different use cases. As\nrepositories grow, traditional full-repo clones become less viable.\nNetwork transfer and client-side performance both suffer. In a repo the\nsize of (say) WebKit, the traditional model works. In a repo the size\nof Facebook's monorepo, it starts to break down. So Facebook does\nentirely shallow clones (using hg, but the problems are similar in git).\nCommands like log and blame instead call out to a server to gather\nhistory data. At Google, whose repo is I think two or three orders of\nmagnitude larger than WebKit, all local copies are both shallow and\nsparse; there is also support for \"sparse commits\" -- so that a commit\nthat affects (say) ten thousand files across the entire tree can be kept\nto a reasonable size. \n\n\n\nTwitter's journal scheme explains why I implemented bloom filter pushes\n-- the number of refs does not significantly affect pull performance,\nbut pushes still go through the normal git machinery, so we wanted an\noptimization to reduce latency there.\n"},{"id":"256573","messageId":"54EC7241.7000500@alum.mit.edu","threadId":"38604","inReplyTo":"CACBZZX45eCo6YS4EpHvMQjN32+-w5BztfoLiwh_rJTs7FydgoQ@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-02-24T12:44:49Z","receivedAt":"2015-02-24T12:44:49Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 02/20/2015 03:25 PM, Ævar Arnfjörð Bjarmason wrote:\n> On Fri, Feb 20, 2015 at 1:09 PM, Ævar Arnfjörð Bjarmason\n> wrote:\n>> On Fri, Feb 20, 2015 at 1:04 AM, Duy Nguyen wrote:\n>>> On Fri, Feb 20, 2015 at 6:29 AM, Ævar Arnfjörð Bjarmason\n>>> wrote:\n>>>> Anecdotally I work on a repo at work (where I'm mostly \"the Git guy\") that's:\n>>>>\n>>>> * Around 500k commits\n>>>> * Around 100k tags\n>>>> * Around 5k branches\n>>>> * Around 500 commits/day, almost entirely to the same branch\n>>>> * 1.5 GB .git checkout.\n>>>> * Mostly text source, but some binaries (we're trying to cut down[1] on those)\n>>>\n>>> Would be nice if you could make an anonymized version of this repo\n>>> public. Working on a \"real\" large repo is better than an artificial\n>>> one.\n>>\n>> Yeah, I'll try to do that.\n> \n> tl;dr: After some more testing it turns out the performance issues we\n> have are almost entirely due to the number of refs. Some of these I\n> knew about and were obvious (e..g. git pull), but some aren't so\n> obvious (why does \"git log\" without \"--all\" slow down as a function of\n> the overall number of refs?).\n\nI'm assuming that you pack your references periodically. (If not, you\nshould, because reading lots of loose references is very expensive for\nthe commands that need to iterate over all references!)\n\nOn the other hand, packed refs also have a downside, namely that\nwhenever even a single packed reference has to be read, the whole\npacked-refs file has to be read and parsed. One way that this can bite\nyou, even with innocuous-seeming commands, is if you haven't disabled\nthe use of replace references (i.e., using \"git --no-replace-objects\n\" or GIT_NO_REPLACE_OBJECTS). In that case, almost any Git command\nhas to read the \"refs/replace/*\" namespace, which, in turn, forces the\nwhole packed-refs file to be read and parsed. This can take a\nsignificant amount of time if you have a very large number of references.\n\nSo try your experiments with replace references disabled. If that helps,\nconsider disabling them on your server if you don't need them.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"256633","messageId":"CACsJy8CxdDxP5urp5dY=UjiNFb9vMOB=+BSZCiV4KovjynQ5hg@mail.gmail.com","threadId":"38604","inReplyTo":"CACsJy8B6pLnx84Xxb0_p2+YuWTgg_8h0yQt7K+AXiaEaC4jSeQ@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-02-25T12:02:40Z","receivedAt":"2015-02-25T12:02:40Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Feb 21, 2015 at 11:01 AM, Duy Nguyen wrote:\n> I wonder how efficient rsync is for transferring these refs: the\n> client generates a \"file\" containing all refs, the server does the\n> same with their refs, then the client rsync their file to the server..\n> The changes between the server and the client files are usually small,\n> I'm hoping rsync can take advantage of that.\n\nSome numbers without any actual coding. After the initial clone, we\nstore the server's refs in a file called base-file at client. At the\nnext push or pull, the server saves its refs in 'new-file'. Using\nrsync to avoid initial ref advertisement would involve these steps\n(rdiff command is from librsync)\n\nclient> rdiff signature base-file signature\n(client sends \"signature\" file to server)\nserver> rdiff delta signature new-file delta\n(server sends \"delta\" file back to client)\nclient> rdiff patch base-file delta new-file\n\nThe exchanged files over network are \"signature\" and \"delta\". I used\nmy git.git's packed-refs as the base-file (1416 refs, 78789 bytes) and\nmodifies three lines to create new-file. That produced a signature\nfile of 480 bytes and delta file of 6163 bytes. That's 7% the size of\nthe new file. Good.\n\nWhen I modified more lines in new-file (15 lines), the delta file grew\nto 26644 bytes (\"sig\" file remains the same because it only depends on\nbase-file). Total transferred bytes were 60% the size of new-file.\nLess impressive. Maybe there's some tuning options for better\nresults...\n\nThe same process could be used to transfer the whole client refs to\nserver instead of sending lots of \"have\" lines. I suspect there will\nbe more changes between client's \"have\" file and server ref list. If\nthe changes spread out and cause a lot of blocks to be sent, the\nsaving would not be as high as I wanted. I guess that's it for rsync\nidea.\n-- \nDuy\n"},{"id":"256853","messageId":"CACBZZX7O77zfn75vJXsSyR58UjC_bfTu-DWwJC8n2fhemVBn2w@mail.gmail.com","threadId":"38604","inReplyTo":"xmqqa908z44h.fsf@gitster.dls.corp.google.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2015-03-02T19:36:43Z","receivedAt":"2015-03-02T19:36:43Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Feb 20, 2015 at 10:04 PM, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> I actually ran this a few times while testing it, so this is a before\n>> and after on a hot cache of linux.git with 406 tags v.s. ~140k. I ran\n>> the gc + repack + bitmaps for both repos noted in an earlier reply of\n>> mine, and took the fastest run out of 3:\n>>\n>> $ time (git log master -100 >/dev/null)\n>> Before: real 0m0.021s\n>> After: real 0m2.929s\n>\n> Do you force --decorate with some config? Or do you see similar\n> performance difference with \"git rev-parse master\", too?\n\nYes, I had log.decorate=short set in my config. With --no-decorate:\n\n $ time (git log --no-decorate -100 >/dev/null)\n # Before: real 0m0.010s\n # After: real 0m0.065s\n\n>> $ time (git status >/dev/null)\n>> # Around 150ms, no noticeable difference\n>\n> This is understandable, as it will not look at any ref other than\n> HEAD.\n"},{"id":"256855","messageId":"CACBZZX4rjYoQZRkzrfBYp8jOFwGm64Eh2-6JpJJ=KMFNxCOkbg@mail.gmail.com","threadId":"38604","inReplyTo":"54EC7241.7000500@alum.mit.edu","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2015-03-02T19:42:30Z","receivedAt":"2015-03-02T19:42:30Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Tue, Feb 24, 2015 at 1:44 PM, Michael Haggerty wrote:\n> On 02/20/2015 03:25 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Fri, Feb 20, 2015 at 1:09 PM, Ævar Arnfjörð Bjarmason\n>> wrote:\n>>> On Fri, Feb 20, 2015 at 1:04 AM, Duy Nguyen wrote:\n>>>> On Fri, Feb 20, 2015 at 6:29 AM, Ævar Arnfjörð Bjarmason\n>>>> wrote:\n>>>>> Anecdotally I work on a repo at work (where I'm mostly \"the Git guy\") that's:\n>>>>>\n>>>>> * Around 500k commits\n>>>>> * Around 100k tags\n>>>>> * Around 5k branches\n>>>>> * Around 500 commits/day, almost entirely to the same branch\n>>>>> * 1.5 GB .git checkout.\n>>>>> * Mostly text source, but some binaries (we're trying to cut down[1] on those)\n>>>>\n>>>> Would be nice if you could make an anonymized version of this repo\n>>>> public. Working on a \"real\" large repo is better than an artificial\n>>>> one.\n>>>\n>>> Yeah, I'll try to do that.\n>>\n>> tl;dr: After some more testing it turns out the performance issues we\n>> have are almost entirely due to the number of refs. Some of these I\n>> knew about and were obvious (e..g. git pull), but some aren't so\n>> obvious (why does \"git log\" without \"--all\" slow down as a function of\n>> the overall number of refs?).\n>\n> I'm assuming that you pack your references periodically. (If not, you\n> should, because reading lots of loose references is very expensive for\n> the commands that need to iterate over all references!)\n\nYes, as mentioned in another reply of mine, like this:\n\n git --git-dir={} gc &&\n git --git-dir={} pack-refs --all --prune &&\n git --git-dir={} repack -Ad --window=250 --depth=100\n--write-bitmap-index --pack-kept-objects &&\n\n> On the other hand, packed refs also have a downside, namely that\n> whenever even a single packed reference has to be read, the whole\n> packed-refs file has to be read and parsed. One way that this can bite\n> you, even with innocuous-seeming commands, is if you haven't disabled\n> the use of replace references (i.e., using \"git --no-replace-objects\n> \" or GIT_NO_REPLACE_OBJECTS). In that case, almost any Git command\n> has to read the \"refs/replace/*\" namespace, which, in turn, forces the\n> whole packed-refs file to be read and parsed. This can take a\n> significant amount of time if you have a very large number of references.\n\nInteresting. I tried the rough benchmarks I posted above with\nGIT_NO_REPLACE_OBJECTS=1 and couldn't see any differences, although as\nmentioned in another reply --no-decorate had a big effect on git-log.\n\n> So try your experiments with replace references disabled. If that helps,\n> consider disabling them on your server if you don't need them.\n>\n> Michael\n>\n> --\n> Michael Haggerty\n> mhagger@alum.mit.edu\n>\n"},{"id":"256862","messageId":"xmqqsidn9mvo.fsf@gitster.dls.corp.google.com","threadId":"38604","inReplyTo":"CACBZZX7O77zfn75vJXsSyR58UjC_bfTu-DWwJC8n2fhemVBn2w@mail.gmail.com","subject":"Re: Git Scaling: What factors most affect Git performance for a large repo?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-03-02T20:15:39Z","receivedAt":"2015-03-02T20:15:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Fri, Feb 20, 2015 at 10:04 PM, Junio C Hamano wrote:\n>> Ævar Arnfjörð Bjarmason writes:\n>>\n>>> I actually ran this a few times while testing it, so this is a before\n>>> and after on a hot cache of linux.git with 406 tags v.s. ~140k. I ran\n>>> the gc + repack + bitmaps for both repos noted in an earlier reply of\n>>> mine, and took the fastest run out of 3:\n>>>\n>>> $ time (git log master -100 >/dev/null)\n>>> Before: real 0m0.021s\n>>> After: real 0m2.929s\n>>\n>> Do you force --decorate with some config? Or do you see similar\n>> performance difference with \"git rev-parse master\", too?\n>\n> Yes, I had log.decorate=short set in my config. With --no-decorate:\n>\n> $ time (git log --no-decorate -100 >/dev/null)\n> # Before: real 0m0.010s\n> # After: real 0m0.065s\n\nThere you have the answer to your earlier question, then, which was:\n\n>> tl;dr: After some more testing it turns out the performance issues we\n>> have are almost entirely due to the number of refs. Some of these I\n>> knew about and were obvious (e..g. git pull), but some aren't so\n>> obvious (why does \"git log\" without \"--all\" slow down as a function of\n>> the overall number of refs?).\n"}]}