Re: Big project, slow access!

From: R. Tyler Ballance <tyler@slide.com>
Date: Sep 18, 2009, 21:32 UTC
Message-ID: <20090918213216.GJ18785@starfruit.corp.slide.com>
In-Reply-To: <ffb2c0280909181138r7fde8722n80be4bdf95864c37@mail.gmail.com>

On Fri, 18 Sep 2009, Toan Pham wrote:

Show 15 quoted lines

> Hi,
> 
> I use git to maintain a project that is at least 8 gigs in size.
> The project is a Linux from Scratch repository that includes source
> codes to approximately 2000 open source projects,
> gcc tool-chain, 1000+ configurations for different software packages,
> source code for different kernel versions,
> and many linux distributions/flavors resulted from this LFS build environment.
> 
> The git's object repository is now 4.6 gigs and consists of approx.
> 610,000 files and folders.
> The speed of git is now terribly slow.  Each time I use basic commands
> like 'git status' or 'git diff',
> it would take at least 5 minutes for git to give me back a result.
> Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.

Howdy Toan, we have a similarly large repository ~405k files, the .git folder fully packed is ~6GB.

The advise to fully-pack your repository is likely going to have the greatest impact on your performance in the short term, in the long term however you might want to consider using git-filter-branch(1) or other tools available to separate our the components of your current Git reposotory into a series of repos.

The performance hit you're seeing likely has nothing to do with your processor speed either, but rather your disk search speed (i'm waiting for a new fancy SSD to help alleviate my issues ;))

> would  someone please recommend on how i can optimize git's performance?
> Git is so slow, are there better ways to manage a project like this?

Rethink how your project is laid out, and whether certain binaries files need to sit in the tree, or can be build on a need-by-need basis.

Cheers -R. Tyler Ballance Slide, Inc.