Re: Big project, slow access!

From: R. Tyler Ballance <tyler@slide.com>
Date: Sep 22, 2009, 15:22 UTC
Message-ID: <20090922152209.GA31806@starfruit.corp.slide.com>
In-Reply-To: <ffb2c0280909220751r79946d76x3b23b146077781f5@mail.gmail.com>

On Tue, 22 Sep 2009, Toan Pham wrote:

Show 14 quoted lines

> Dear Tyler
> 
> 
> Thank you for your valuable feedback.
> 
> I'll research into git-filter-branch and also dividing a big project
> into several sub-repositories.
> This seems to increase the performance very much; however, there is a
> draw-back that I am a little bit
> concern with.  When we use several sub-repos option, we would probably
> do manual book-keeping as to
> which repo commits are compatible/built-able with other repo. commits.
>  How did you manage to track
> dependencies and their versions between different depos?

This wholly depends on how your project laid out, for example, if your sub-repositories are on different project timelines (shared components come to mind) where you would want to update that submodule's HEAD in the super-project. It's worth making sure you're aware of what you're getting yourself into with git-submodule(1) before you dive in and start splitting your project into submodules:

	http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules
	http://github.com/guides/developing-with-submodules

You could also look at using submodules and Android's repo command:
	http://source.android.com/download/using-repo

> >>i'm waiting for a new fancy SSD to help alleviate my issues.
> 
> Please report the performance increase after you tested on your SS Drive.

This was tongue-in-cheek, I can't afford an SSD in my laptop just yet ;)

Cheers

Show 45 quoted lines

> On Fri, Sep 18, 2009 at 5:32 PM, R. Tyler Ballance <tyler@slide.com> wrote:
> >
> > On Fri, 18 Sep 2009, Toan Pham wrote:
> >
> >> Hi,
> >>
> >> I use git to maintain a project that is at least 8 gigs in size.
> >> The project is a Linux from Scratch repository that includes source
> >> codes to approximately 2000 open source projects,
> >> gcc tool-chain, 1000+ configurations for different software packages,
> >> source code for different kernel versions,
> >> and many linux distributions/flavors resulted from this LFS build environment.
> >>
> >> The git's object repository is now 4.6 gigs and consists of approx.
> >> 610,000 files and folders.
> >> The speed of git is now terribly slow.  Each time I use basic commands
> >> like 'git status' or 'git diff',
> >> it would take at least 5 minutes for git to give me back a result.
> >> Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.
> >
> > Howdy Toan, we have a similarly large repository ~405k files, the .git
> > folder fully packed is ~6GB.
> >
> > The advise to fully-pack your repository is likely going to have the
> > greatest impact on your performance in the short term, in the long term
> > however you might want to consider using git-filter-branch(1) or other
> > tools available to separate our the components of your current Git
> > reposotory into a series of repos.
> >
> > The performance hit you're seeing likely has nothing to do with your
> > processor speed either, but rather your disk search speed (i'm waiting
> > for a new fancy SSD to help alleviate my issues ;))
> >
> >> would  someone please recommend on how i can optimize git's performance?
> >> Git is so slow, are there better ways to manage a project like this?
> >
> > Rethink how your project is laid out, and whether certain binaries files
> > need to sit in the tree, or can be build on a need-by-need basis.
> >
> >
> >
> > Cheers
> > -R. Tyler Ballance
> > Slide, Inc.
> >

-R. Tyler Ballance Slide, Inc.