Re: On the many files problem

From: Martin Langhoff <martin.langhoff@gmail.com>
Date: Dec 31, 2007, 23:31 UTC
Message-ID: <46a038f90712311531x17b6d94aua1bac2d3d186f186@mail.gmail.com>
In-Reply-To: <873atjtbmu.fsf@enceladus.ygingras.net>

On Dec 31, 2007 11:13 PM, Yannick Gingras <ygingras@ygingras.net> wrote:

Show 12 quoted lines

> >    but if you want to check odder cases, try creating a huge
> >    directory, and then deleting most files, and then adding a few
> >    new ones. Some filesystems will take a huge hit because they'll
> >    still scan the whole directory, even though it's mostly empty!
> >
> >    (Also, a "readdir() + stat()" loop will often get *much* worse access
> >    patterns if you've mixed deletions and creations)
>
> This is something that will be interesting to benchmark later on.  So,
> an application with a lot of turnaround, say a mail server, should
> delete and re-create the directories from time to time?  I assume this
> is specific to some file system types.

This is indeed the case. Directories with a lot of movement get fragmented on most FSs -- ext3 is a very bad case for this -- and there are no "directory defrag" tools other than regenarating them. The "Maildir" storage used for many IMAP servers these days shows the problem.

This (longish) threads has some interesting tidbits on getdents() and directory fragmentation. http://kerneltrap.org/mailarchive/git/2007/1/7/235215

cheers,