Re: Git performance results on a large repository

From: David Mohs <dgma@mohsinc.com>
Date: Feb 6, 2012, 07:10 UTC
Message-ID: <loom.20120206T054853-321@post.gmane.org>
In-Reply-To: <243C23AF01622E49BEA3F28617DBF0AD5912CA85@SC-MBX02-5.TheFacebook.com>

Joshua Redstone <joshua.redstone <at> fb.com> writes:

Show 9 quoted lines

> To get a bit abstract for a moment, in an ideal world, it doesn't seem like
> performance constraints of a source-control-system should dictate how we
> choose to structure our code. Ideally, seems like we should be able to choose
> to structure our code in whatever way we feel maximizes developer
> productivity. If development and code/release management seem easier in a
> single repo, than why not make an SCM that can handle it? This is one reason
> I've been leaning towards figuring out an SCM approach that can work well with
> our current practices rather than changing them as a prerequisite for good SCM
> performance.

I certainly agree with this perspective---that our tools should support our use cases and not the other way around. However, I'd like you to consider that the size of this hypothetical repository might be giving you some useful information on the health of the code it contains. You might consider creating separate repositories simply to promote good modularization. It would involve some up-front effort and certainly some pain, but this work itself might be beneficial to your codebase without even considering the improved performance of the version control system.

My concern here is that it may be extremely difficult to make a single piece of software scale for a project that can grow arbitrarily large. You may add some great performance improvements to git to then find that your bottleneck is the filesystem. That would enlarge the scope of your work and would likely make the project more difficult to manage.

If you are able to prove me wrong, the entire software community will benefit from this work. However, before you embark upon a technical solution to your problem, I would urge you to consider the possible benefits of a non-technical solution, specifically restructuring your code and/or teams into more independent modules. You might find benefits from this approach that extend beyond source code control, which could make it the solution with the least amount of overall risk.

Thanks for starting this valuable discussion.

-David