Re: repo.or.cz renovated

From: Petr Baudis <pasky@suse.cz>
Date: Mar 17, 2008, 19:54 UTC
Message-ID: <20080317195422.GF10335@machine.or.cz>
In-Reply-To: <20080317193423.GI8368@mit.edu>

On Mon, Mar 17, 2008 at 03:34:23PM -0400, Theodore Tso wrote:

Show 15 quoted lines

> On Mon, Mar 17, 2008 at 07:10:15PM +0100, Petr Baudis wrote:
> > Actually, it was overwhelmed to so much by its success but by lack of
> > good maintenance. ;-) I gave it some love again for the past week and
> > the improvement was, well, overwhelming. :-)
> > 
> > I finally fixed tons of failures and broken repositories, and most
> > importantly repacked some of the big repositories with object databases
> > in pretty horrid shape. The effect has been immense, having everything
> > in database of 1/3 the size and single big pack drastically reduced the
> > I/O load.
> 
> Are you making sure that repositories which are forks off of some
> parent repository are using objects/info/alternates to share objects?
> (If so you have to be careful when you prune not to drop objects, but
> it can make a huge difference in disk utilization and I/O bandwidth).

Yes, I reuse objects from parent projects, that has always been so.

Show 12 quoted lines

> At least for master.kernel.org, and for those git repositories which I
> own, I make a point of periodically logging in and running git gc,
> copying over the object packs so I can do a prune operation safely,
> etc.  --- and I suspect most of the master.kernel.org git users do
> something similar.  On repo.or.cz we don't have shell access, so the
> project administrators can't do that for you.
> 
> > So for anyone running a hosting site, make sure your repositories are
> > nicely packed. It makes huge difference to the I/O load!
> 
> It seems that a Really Good Idea would be do the the packing and
> pruning via cron scripts that run during the off hours...

Yes, this was done before too, however repo.or.cz has been around for long time and historically the scripts weren't working very well, especially since I had to be careful about the forks problem.

Since I am repacking on live system, I think the current repacking strategy is still not completely error prone, however I believe that I have encountered no breakage because of pruned objects the last at least half a year or so it has been running with the current setup (all of the breakages I have encountered seem to be caused by child process of git-repack dying). Besides, if some fork breaks, it should be possible to fix that very easily (I do not backup the object stores at all anyway - if the server burns down, you will have to re-push ;-).

Show 10 quoted lines

> > My current plan is to have a [Search project] box at the front page,
> > together with direct link to 'show all'. Other than that, what makes
> > sense to display on the front page? I think recently added projects (age
> > < 1 week) for sure. I'm not so sure about recently changed projects -
> > maybe it is better to keep the front page cruft-free.
> 
> There are plenty of ways which sites like freshmeat and sourceforge
> have come up to make it easy to browse a large number of software
> projects.  One way that might make sense is Sourceforge's Software Map
> (i.e., http://sourceforge.net/softwaremap/).

This all feels like a real overkill, besides my main doubt is whether repo.or.cz needs something like this *at all* - but I think I will try the tagging system and see how do people like it.

-- 
				Petr "Pasky" Baudis
Whatever you can do, or dream you can, begin it.
Boldness has genius, power, and magic in it.	-- J. W. von Goethe