Re: Performance issue: initial git clone causes massive repack
- From
Robin H. Johnson <robbat2@gentoo.org>
- Date
- Apr 5, 2009, 20:43 UTC
- Message-ID
- <20090405204325.GA31344@curie-int>
- In-Reply-To
- <20090405190213.GA12929@vidovic>
On Sun, Apr 05, 2009 at 09:02:13PM +0200, Nicolas Sebrecht wrote:
Show 17 quoted lines
> > Before I answer the rest of your post, I'd like to note that the matter > > of which choice between single-repo, repo-per-package, repo-per-category > > has been flogged to death within Gentoo. > > > > I did not come to the Git mailing list to rehash those choices. I came > > here to find a solution to the performance problem. > I understand. I know two ways to resolve this: > - by resolving the performance problem itself, > - by changing the workflow to something more accurate and more suitable > against the facts. > > My point is that going from a centralized to a decentralized SCM > involves breacking strongly how developers and maintainers work. What > you're currently suggesting is a way to work with Git in a centralized > way. This sucks. To get the things right with Git I would avoid shared > and global repositories. Gnome is doing it this way: > http://gitorious.org/projects/gnome-svn-hooks/repos/mainline/trees/master
The entire matter of splitting the repository comes down to what should be considered an atomic unit. For GNOME, KDE and all of the other large Git consumers that I'm aware of, there atomic units are individual packages - specifically because they make sense to be consumed without having all the rest of the packages. For the gentoo tree, it is an atomic unit in itself. Changes to the profiles/ directory (for package masks, USE keys are frequently related and need to be always committed and received atomically with changes to one or more packages.
Show 6 quoted lines
> > The GSoC 2009 ideas contain a potential project for caching the > > generated packs, which, while having value in itself, could be partially > > avoided by sending suitable pre-built packs (if they exist) without any > > repacking. > Right. It could be an option to wait and see if the GSoC gives > something.
How hard is it to just look at the git-upload-pack code and make it realize that it doesn't need to repack at all for this case.
Show 8 quoted lines
> > A quick bit of stats run show that while some developers only touch a > > few packages, there are at least 200 developers that have done a major > > change to 100 or more packages. > That's a point that has to be reconsidered. Not the fact that at least > 200 developers work on over 100 packages (this is really not an issue)¹ > but the fact that they do that directly on the main repo/server. The > good way to achieve this is to send his work to the maintainer². The main > issue is a better code reviewing.
This has been shot down by our developer base. One of the grounds is that there is no developer with sufficient time to take a merge-master role on a regular basis like that.
Show 6 quoted lines
> 1. Some or all repo-per-category can be tracked with a simple script. > 2. Maintainers could be - or not be - the same developers as today. > Adding a layer of maintainers in charge of EAPI review (for example) up > to the packages-maintainers could help in fixing a lot of portage issues > and would avoid "simple developers" to do crap on the main repo(s) that > users download.
You imply that there is a problem in that field already, which I disagree with.
> > > One repo per category could be a good compromise assuming one seperate > > > branch per package, then. > > Other downsides to repo-per-category and repo-per-package: > Let's forget a repo-per-package.
One downside unique to repo-per-category is that when a package moves cross-category, you end up with it consuming space in packs on both sides.
> > - Raises difficulty in adding a new package/category. > > You cannot just do 'mkdir && vi ... && git add && git commit' anymore. > Right, but categories are not evolving that much.
There's demand to evolve them, but bulk package moves are painful with CVS, so it's been waiting for Git.
Show 13 quoted lines
> A repo-per-category local workflow would be: > [...] > $ git checkout package_one > $ tree -a > |-- .git > | |-- [...] > | [...] > `-- package_one > |-- ChangeLog > |-- Manifest > |-- metadata.xml > |-- package_one-0.4.ebuild > `-- package_one-0.5.ebuild
Umm, why does package_two not exist in the other branch? If package_one depends on package_two, and you're in for a world of fail the moment it you changes branches here.
> > - Does NOT present a good base for anybody wanting to branch the entire > > tree themselves. > Scriptable.
You dropped my cvsserver list item.
-- Robin Hugh Johnson Gentoo Linux Developer & Infra Guy E-Mail : robbat2@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85