Re: svn to git, N-squared?

From: Jon Smirl <jonsmirl@gmail.com>
Date: Jun 12, 2006, 19:00 UTC
Message-ID: <9e4733910606121200m402af4a4qa162c6f79cbf7100@mail.gmail.com>
In-Reply-To: <9e4733910606121106ta925b6er49fe68bf3c1031f5@mail.gmail.com>

On 6/12/06, Jon Smirl <jonsmirl@gmail.com> wrote:

Show 14 quoted lines

> On 6/12/06, Linus Torvalds <torvalds@osdl.org> wrote:
> > Having that many files in a single directory (or two) is a total disaster.
> > That said, it works well enough if you don't create new files very often
> > (and _preferably_ don't look them up either, although that is effectively
> > helped by indexing). I _suspect_ that
>
> Posted to the svn list, they said that 220K files is normal. They told
> me to turn on the ext2 dir_index option. Cheking my system I see that
> none of partitions have it turned on so it must not be the default for
> FC5.
>
> I have to unmount the drive to convert existing directories. I can
> trying doing the file move trick while the process is running since
> new directories will use it.

I converted the ext3 directories to dir_index on the fly using the move trick. Switching the directory index makes it look like it is spending even more time in the kernel.

procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us
sy id wa st
 1  0 636188  22380  19176 157200    0    0     0    52  436   415 13
40 48  0  0
 1  0 636188  22504  19176 157200    0    0     0     0  430   373 13
38 49  0  0
 1  0 636188  22628  19176 157064    0    0     0     0  433   380 12
39 49  0  0
 1  0 636188  22628  19184 157056    0    0     0    20  434   390 12
38 49  0  0
 1  0 636188  22628  19184 156920    0    0     0     0  431   376 11
40 49  0  0
 1  0 636188  22752  19192 156912    0    0     0    48  437   376 12
40 49  0  0
 1  0 636188  22876  19192 156912    0    0     0     0  430   386 11
40 49  0  0
 1  0 636188  22752  19192 156776    0    0     0     0  431   370 10
41 49  0  0
 1  0 636188  23016  19192 156776    0    0     8     0  422   500 22
40 37  2  0

The size of the svn directories went from 3.2MB to 4.4MB after they were converted to ext3 indexed mode.

I'll get oprofile running when I do a reboot.

-- 
Jon Smirl
jonsmirl@gmail.com