Re: Git performance results on a large repository
- From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
- Date
- Feb 3, 2012, 14:56 UTC
- Message-ID
- <CACBZZX4BsFZxB6A-Hg-k37FBavgTV8SDiQTK_sVh9Mb9iskiEw@mail.gmail.com>
- In-Reply-To
- <CB5074CF.3AD7A%joshua.redstone@fb.com>
On Fri, Feb 3, 2012 at 15:20, Joshua Redstone <joshua.redstone@fb.com> wrote:
Show 5 quoted lines
> We (Facebook) have been investigating source control systems to meet our > growing needs. We already use git fairly widely, but have noticed it > getting slower as we grow, and we want to make sure we have a good story > going forward. We're debating how to proceed and would like to solicit > people's thoughts.
Where I work we also have a relatively large Git repository. Around 30k files, a couple of hundred thousand commits, clone size around half a GB.
You haven't supplied background info on this but it really seems to me like your testcase is converting something like a humongous Perforce repository directly to Git.
While you /can/ do this it's not a good idea, you should split up repositories at the boundaries code or data doesn't directly cross over, e.g. there's no reason why you need HipHop PHP in the same repository as Cassandra or the Facebook chat system, is there?
While Git could better with large repositories (in particular applying commits in interactive rebase seems to be to slow down on bigger repositories) there's only so much you can do about stat-ing 1.3 million files.
A structure that would make more sense would be to split up that giant repository into a lot of other repositories, most of them probably have no direct dependencies on other components, but even those that do can sometimes just use some other repository as a submodule.
Even if you have the requirement that you'd like to roll out *everything* at a certain point in time you can still solve that with a super-repository that has all the other ones as submodules, and creates a tag for every rollout or something like that.