Re: How git affects kernel.org performance

From: Theodore Tso <tytso@mit.edu>
Date: Jan 8, 2007, 12:58 UTC
Message-ID: <20070108125819.GA32756@thunk.org>
In-Reply-To: <20070108030555.GA7289@in.ibm.com>

On Mon, Jan 08, 2007 at 08:35:55AM +0530, Suparna Bhattacharya wrote:

Show 13 quoted lines

> > Yeah, slowly-growing directories will get splattered all over the disk.
> > 
> > Possible short-term fixes would be to just allocate up to (say) eight
> > blocks when we grow a directory by one block.  Or teach the
> > directory-growth code to use ext3 reservations.
> > 
> > Longer-term people are talking about things like on-disk rerservations.
> > But I expect directories are being forgotten about in all of that.
> 
> By on-disk reservations, do you mean persistent file preallocation ? (that
> is explicit preallocation of blocks to a given file) If so, you are
> right, we haven't really given any thought to the possibility of directories
> needing that feature.

The fastest and probably most important thing to add is some readahead smarts to directories --- both to the htree and non-htree cases. If you're using some kind of b-tree structure, such as XFS does for directories, preallocation doesn't help you much. Delayed allocation can save you if your delayed allocator knows how to structure disk blocks so that a btree-traversal is efficient, but I'm guessing the biggest reason why we are losing is because we don't have sufficient readahead. This also has the advantage that it will help without needing to doing a backup/restore to improve layout.

Allocating some number of empty blocks when we grow the directory would be a quick hack that I'd probably do as a 2nd priority. It won't help pre-existing directories, but combined with readahead logic, should help us out greatly in the non-btree case.

						- Ted