Re: kernel.org and GIT tree rebuilding

From: Jeff Garzik <jgarzik@pobox.com>
Date: Jun 25, 2005, 04:40 UTC
Message-ID: <42BCE026.8050405@pobox.com>
In-Reply-To: <20050624.212009.92584730.davem@davemloft.net>

David S. Miller wrote:

Show 12 quoted lines

> To get a clean history to push to Linus, I typically blow
> away my trees and make fresh ones to stick patches into
> which I want to merge.
> 
> That mostly works fine here on my local systems, but I know this
> brings the master.org mirroring system to it's knees.  So what is the
> generally condoned way to do stuff like this in a more friendly way?
> 
> Should I:
> 
> 1) Do a git pull from Linus's tree once he takes my changes, then
>    ask GIT to prune the tree?  How do I do that and how does it work?

> It takes sometimes an hour for my tree updates on master.kernel.org
> to propagate to rsync.kernel.org so I can ask Linus to pull.
> That's crazy.

Unfortunately you cannot fix this by changing your actions. This is the cumulative effect of all the git kernel trees on kernel.org. It now takes over an hour for my non-git changes to propagate from master to the mirrors, as well.

This is all due to the rsync sweeps, which have to scan metric tons of inodes and dentries. Orders of magnitude over the pre-git days.

ftpadmin@kernel.org folks are supposedly working on an inotify-based system, and an improved rsync application. No ETA or details.

As an aside, cold-cache, git really punishes my disks. Ted T'so noted that it really drains laptop batteries, too.

Show 5 quoted lines

> 2) Should I use .git/object/ database symlinking?
> 
>    Are there any scripts out there which do this automatically?
>    Something as simple to run as "git-pull-script" and it takes
>    care of using links when possible on a local filesystem.

On both kernel.org and locally, I use 'cp -al' to duplicate the initial .git/objects directory, and then rsync (->kernel.org) or git-pull-script (<-kernel.org) to update it after that.

That definitely helps.

Maybe somebody needs to script a relink cron job for kernel.org?

	Jeff