{"thread":{"id":"4478","subject":"Re: svn to git, N-squared?","startedAt":"2006-06-12T04:39:49Z","lastAt":"2006-06-12T19:00:19Z","messageCount":14,"participants":["linux@horizon.com","Jon Smirl","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"21631","messageId":"20060612043949.20992.qmail@science.horizon.com","threadId":"4478","inReplyTo":null,"subject":"Re: svn to git, N-squared?","fromName":"","fromEmail":"linux@horizon.com","sentAt":"2006-06-12T04:39:49Z","receivedAt":"2006-06-12T04:39:49Z","isPatch":false,"sender":{"key":"linux@horizon.com","avatar":null},"body":">\tfor i in ??\n>\tdo\n>\t\tcp -r $i $i.new\n>\t\trm -rf $i\n>\t\tmv $i.new $i\n>\tdone\n>\n> in your .git/objects/ directory (CAREFUL! Any script that does \"rm -rf\" \n> should be double- and triple-checked for sanity! ;)\n\nInsanity is copying the data rather than just the file name.  Git is\ngood about not reading unnecessary files, and anything necessary should\nbe cached, so on-disk fragmentation is not a concern.\n\nrmdir --ignore-fail-on-non-empty ??\t# Probably unnecessary.\nfor i in ??\ndo\n\tmkdir $i.new\n\tmv $i/* $i.new\n\trmdir $i\n\tmv $i.new $i\ndone\n"},{"id":"21647","messageId":"9e4733910606120832xaf74e77pad7f70df864541fc@mail.gmail.com","threadId":"4478","inReplyTo":"20060612043949.20992.qmail@science.horizon.com","subject":"Re: svn to git, N-squared?","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-12T15:32:08Z","receivedAt":"2006-06-12T15:32:08Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 12 Jun 2006 00:39:49 -0400, linux@horizon.com <linux@horizon.com> wrote:\n> Insanity is copying the data rather than just the file name.  Git is\n> good about not reading unnecessary files, and anything necessary should\n> be cached, so on-disk fragmentation is not a concern.\n\nI've run a pack and I moved the objects to new directories. Directory\nis 746M with 64K files now.\n\nI've stablized like this. 1GB RAM with 2.8Ghz P4 hyperthread. Is there\nanyway to tell what it is doing in the kernel for so much time?\n\nprocs -----------memory---------- ---swap-- -----io---- --system--\n-----cpu------\n r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us\nsy id wa st\n 1  0 599128  24712  38196 247008    0    0     0     0  451   382 12\n39 48  0  0\n 1  0 599128  24836  38196 246872    0    0     0     4  413   354 15\n38 48  0  0\n 1  0 599128  24960  38212 246856    0    0     0    64  453   390 15\n37 48  0  0\n 1  0 599128  24960  38212 246856    0    0     0     0  414   367 12\n40 49  0  0\n 1  0 599128  23504  38212 248216    0    0     0     0  448   365 13\n39 48  0  0\n 1  0 599128  24156  38212 247604    0    0     0     0  407   355 13\n39 49  0  0\n 1  0 599128  25240  38212 246652    0    0     0     0  446   390 13\n39 48  0  0\n 1  0 599128  25240  38224 246572    0    0     4    48  415   418 12\n40 47  0  0\n 1  0 599128  25116  38232 246496    0    0     0    12  452   432 12\n40 48  0  0\n\nStill doesn't seem to be making much forward progress.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21648","messageId":"Pine.LNX.4.64.0606120843340.5498@g5.osdl.org","threadId":"4478","inReplyTo":"9e4733910606120832xaf74e77pad7f70df864541fc@mail.gmail.com","subject":"Re: svn to git, N-squared?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-06-12T15:45:52Z","receivedAt":"2006-06-12T15:45:52Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 12 Jun 2006, Jon Smirl wrote:\n> \n> I've stablized like this. 1GB RAM with 2.8Ghz P4 hyperthread. Is there\n> anyway to tell what it is doing in the kernel for so much time?\n\noprofile will tell you.\n\nI don't see why it would spend a lot of time in the kernel, unless it's \nthe SVN part that does a ton of reads or something. git should have almost \nno kernel footprint apart from the individual objects creation/reading, so \nonce it's repacked, I generally see very little system time.\n\nWhat does top say? (Ie can you see _which_ process spends time in the \nkernel?)\n\n\t\tLinus\n"},{"id":"21649","messageId":"9e4733910606120855p1cec9acfy62dadb89c11756b4@mail.gmail.com","threadId":"4478","inReplyTo":"Pine.LNX.4.64.0606120843340.5498@g5.osdl.org","subject":"Re: svn to git, N-squared?","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-12T15:55:13Z","receivedAt":"2006-06-12T15:55:13Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/12/06, Linus Torvalds <torvalds@osdl.org> wrote:\n>\n>\n> On Mon, 12 Jun 2006, Jon Smirl wrote:\n> >\n> > I've stablized like this. 1GB RAM with 2.8Ghz P4 hyperthread. Is there\n> > anyway to tell what it is doing in the kernel for so much time?\n>\n> oprofile will tell you.\n>\n> I don't see why it would spend a lot of time in the kernel, unless it's\n> the SVN part that does a ton of reads or something. git should have almost\n> no kernel footprint apart from the individual objects creation/reading, so\n> once it's repacked, I generally see very little system time.\n>\n> What does top say? (Ie can you see _which_ process spends time in the\n> kernel?)\n\ntop - 11:54:32 up 4 days,  1:27,  5 users,  load average: 1.85, 1.74, 1.55\nTasks: 135 total,   2 running, 133 sleeping,   0 stopped,   0 zombie\nCpu(s): 14.7% us, 35.3% sy,  0.0% ni, 49.3% id,  0.0% wa,  0.2% hi,  0.5% si,  0\nMem:   1035740k total,  1020836k used,    14904k free,    18368k buffers\nSwap: 118222276k total,   645124k used, 117577152k free,   183172k cached\n\n  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND\n14525 jonsmirl  16   0  604m 391m 1904 S   24 38.7 916:53.39 git-svnimport\n20947 jonsmirl  17   0     0    0    0 R    1  0.0   0:00.03 git-svnimport\n20864 jonsmirl  16   0  2120 1024  788 R    1  0.1   0:00.08 top\n 2436 root      15   0 71184  28m 6100 S    0  2.8 119:13.55 Xorg\n    1 root      16   0  1992  340  312 S    0  0.0   0:00.79 init\n    2 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0\n    3 root      34  19     0    0    0 S    0  0.0   0:01.42 ksoftirqd/0\n    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21650","messageId":"Pine.LNX.4.64.0606120906210.5498@g5.osdl.org","threadId":"4478","inReplyTo":"9e4733910606120855p1cec9acfy62dadb89c11756b4@mail.gmail.com","subject":"Re: svn to git, N-squared?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-06-12T16:12:16Z","receivedAt":"2006-06-12T16:12:16Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 12 Jun 2006, Jon Smirl wrote:\n> \n>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND\n> 14525 jonsmirl  16   0  604m 391m 1904 S   24 38.7 916:53.39 git-svnimport\n> 20947 jonsmirl  17   0     0    0    0 R    1  0.0   0:00.03 git-svnimport\n\nHard to tell, it's obviously got short-lived processes there too that it's \nnot showing, but equally obviously that svnimport script itself is \nspending an alarming amount of CPU time. I don't think it should do that \nmuch processing, but since it's written in perl, I can't read it.\n\nAre there any other directories that seem to be growing (eg some temp-file \ndirectory where the old files aren't cleaned away?). I can't imagine what \nelse it could be doing in kernel space than simply some silly filesystem \noperation, but dang it all, Linux filesystems are usually very efficient \nindeed, unless we're talking huge directories (and if it's not the git \nobject directory any more, it must be something else).\n\nAt least with the cvs importer I have _some_ clue what it's doing, since I \nwrote an earlier version myself (very different, but at least I know what \nthe operations are). SVN has always just confused me, and I have no idea \nwhat svnimport does, so I think I'll have to defer to somebody who \nactually knows the code.\n\nSmurf, have you looked at any larger repositories?\n\n\t\tLinus\n"},{"id":"21651","messageId":"9e4733910606120916o10b0eaf5t65590f88a94e735@mail.gmail.com","threadId":"4478","inReplyTo":"Pine.LNX.4.64.0606120843340.5498@g5.osdl.org","subject":"Re: svn to git, N-squared?","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-12T16:16:46Z","receivedAt":"2006-06-12T16:16:46Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/12/06, Linus Torvalds <torvalds@osdl.org> wrote:\n>\n>\n> On Mon, 12 Jun 2006, Jon Smirl wrote:\n> >\n> > I've stablized like this. 1GB RAM with 2.8Ghz P4 hyperthread. Is there\n> > anyway to tell what it is doing in the kernel for so much time?\n>\n> oprofile will tell you.\n\nI don't have profiling turn on in the kernel. I've turned it on so\nI'll pick it up next time I reboot.\nI'll kill everything and restart when my new RAM arrives tomorrow.\n\nHopefully the SVN import will finish before then but it doesn't look likely.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21653","messageId":"9e4733910606120922g181a5aaal623fd3f29b839f4c@mail.gmail.com","threadId":"4478","inReplyTo":"Pine.LNX.4.64.0606120906210.5498@g5.osdl.org","subject":"Re: svn to git, N-squared?","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-12T16:22:38Z","receivedAt":"2006-06-12T16:22:38Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/12/06, Linus Torvalds <torvalds@osdl.org> wrote:\n>\n>\n> On Mon, 12 Jun 2006, Jon Smirl wrote:\n> >\n> >  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND\n> > 14525 jonsmirl  16   0  604m 391m 1904 S   24 38.7 916:53.39 git-svnimport\n> > 20947 jonsmirl  17   0     0    0    0 R    1  0.0   0:00.03 git-svnimport\n>\n> Hard to tell, it's obviously got short-lived processes there too that it's\n> not showing, but equally obviously that svnimport script itself is\n> spending an alarming amount of CPU time. I don't think it should do that\n> much processing, but since it's written in perl, I can't read it.\n>\n> Are there any other directories that seem to be growing (eg some temp-file\n> directory where the old files aren't cleaned away?). I can't imagine what\n> else it could be doing in kernel space than simply some silly filesystem\n> operation, but dang it all, Linux filesystems are usually very efficient\n> indeed, unless we're talking huge directories (and if it's not the git\n> object directory any more, it must be something else).\n\n64 files in tmp.\nBut the SVN repository itself has 411,000 files in it. Split between\ntwo directories.\n\nIs there some pack equivalent for svn that I haven't found yet?\n\n> At least with the cvs importer I have _some_ clue what it's doing, since I\n> wrote an earlier version myself (very different, but at least I know what\n> the operations are). SVN has always just confused me, and I have no idea\n> what svnimport does, so I think I'll have to defer to somebody who\n> actually knows the code.\n>\n> Smurf, have you looked at any larger repositories?\n>\n>                 Linus\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21655","messageId":"9e4733910606120932k5b6f7acfra3f3a26168454f47@mail.gmail.com","threadId":"4478","inReplyTo":"9e4733910606120922g181a5aaal623fd3f29b839f4c@mail.gmail.com","subject":"Re: svn to git, N-squared?","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-12T16:32:05Z","receivedAt":"2006-06-12T16:32:05Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/12/06, Jon Smirl <jonsmirl@gmail.com> wrote:\n> On 6/12/06, Linus Torvalds <torvalds@osdl.org> wrote:\n> >\n> >\n> > On Mon, 12 Jun 2006, Jon Smirl wrote:\n> > >\n> > >  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND\n> > > 14525 jonsmirl  16   0  604m 391m 1904 S   24 38.7 916:53.39 git-svnimport\n> > > 20947 jonsmirl  17   0     0    0    0 R    1  0.0   0:00.03 git-svnimport\n> >\n> > Hard to tell, it's obviously got short-lived processes there too that it's\n> > not showing, but equally obviously that svnimport script itself is\n> > spending an alarming amount of CPU time. I don't think it should do that\n> > much processing, but since it's written in perl, I can't read it.\n> >\n> > Are there any other directories that seem to be growing (eg some temp-file\n> > directory where the old files aren't cleaned away?). I can't imagine what\n> > else it could be doing in kernel space than simply some silly filesystem\n> > operation, but dang it all, Linux filesystems are usually very efficient\n> > indeed, unless we're talking huge directories (and if it's not the git\n> > object directory any more, it must be something else).\n>\n> 64 files in tmp.\n> But the SVN repository itself has 411,000 files in it. Split between\n> two directories.\n\nI'm doing all of this on ext3. I have plenty of free disk space so I\ncan make another partition and switch to a new file system after I\ninstall the new RAM. What would be the best one to try? Doing that\nwould provide a data point to determine if this is a problem with file\nsystem performance or the misuse of file systems.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21656","messageId":"Pine.LNX.4.64.0606120938490.5498@g5.osdl.org","threadId":"4478","inReplyTo":"9e4733910606120922g181a5aaal623fd3f29b839f4c@mail.gmail.com","subject":"Re: svn to git, N-squared?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-06-12T16:41:59Z","receivedAt":"2006-06-12T16:41:59Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 12 Jun 2006, Jon Smirl wrote:\n>\n> 64 files in tmp.\n> But the SVN repository itself has 411,000 files in it. Split between\n> two directories.\n\nOuch. That sounds like it. \n\n> Is there some pack equivalent for svn that I haven't found yet?\n\nIs this literally what SVN does normally? That's just insane. I mean, even \ngit tried to at least hash out the files (and yeah, admittedly even that \nworked less well than I was hoping for, but I at least fixed it within \njust a few weeks through the pack mechanism).\n\nOr is that 411,000 files a result of how git-svnimport does things, rather \nthan some basic SVN approach to live: does it perhaps end up checking out \neach file under an individual temporary name?\n\n\t\t\tLinus\n"},{"id":"21657","messageId":"9e4733910606120944p4deb170ejc2863846685917f6@mail.gmail.com","threadId":"4478","inReplyTo":"Pine.LNX.4.64.0606120938490.5498@g5.osdl.org","subject":"Re: svn to git, N-squared?","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-12T16:44:07Z","receivedAt":"2006-06-12T16:44:07Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/12/06, Linus Torvalds <torvalds@osdl.org> wrote:\n> > Is there some pack equivalent for svn that I haven't found yet?\n>\n> Is this literally what SVN does normally? That's just insane. I mean, even\n> git tried to at least hash out the files (and yeah, admittedly even that\n> worked less well than I was hoping for, but I at least fixed it within\n> just a few weeks through the pack mechanism).\n>\n> Or is that 411,000 files a result of how git-svnimport does things, rather\n> than some basic SVN approach to live: does it perhaps end up checking out\n> each file under an individual temporary name?\n\nThe svn repository was built by cvs2svn, none of the git tools were involved.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21658","messageId":"Pine.LNX.4.64.0606120942140.5498@g5.osdl.org","threadId":"4478","inReplyTo":"9e4733910606120932k5b6f7acfra3f3a26168454f47@mail.gmail.com","subject":"Re: svn to git, N-squared?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-06-12T16:57:28Z","receivedAt":"2006-06-12T16:57:28Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 12 Jun 2006, Jon Smirl wrote:\n> > \n> > 64 files in tmp.\n> > But the SVN repository itself has 411,000 files in it. Split between\n> > two directories.\n> \n> I'm doing all of this on ext3. I have plenty of free disk space so I\n> can make another partition and switch to a new file system after I\n> install the new RAM. What would be the best one to try? Doing that\n> would provide a data point to determine if this is a problem with file\n> system performance or the misuse of file systems.\n\nI'm sure there are better filesystems to try for this kind of insane \nschenario, but at the same time, I really cannot imaging that the 411,000 \nfiles is a \"normal\" thing. There _must_ be some way to have SVN not do \nthat in the first place (or git-svnimport).\n\nIs this what happened when the SVN people started using fsfs? \n\n\t\t\tLinus\n"},{"id":"21659","messageId":"Pine.LNX.4.64.0606120958230.5498@g5.osdl.org","threadId":"4478","inReplyTo":"9e4733910606120944p4deb170ejc2863846685917f6@mail.gmail.com","subject":"Re: svn to git, N-squared?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-06-12T17:08:05Z","receivedAt":"2006-06-12T17:08:05Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 12 Jun 2006, Jon Smirl wrote:\n> \n> The svn repository was built by cvs2svn, none of the git tools were involved.\n\nOk, so that part is purely a SVN issue.\n\nHaving that many files in a single directory (or two) is a total disaster. \nThat said, it works well enough if you don't create new files very often \n(and _preferably_ don't look them up either, although that is effectively \nhelped by indexing). I _suspect_ that \n\n - the \"cvs->svn\" import process was probably optimized so that it did one \n   file at a time (your \"eight stages\" description certainly sounds as if \n   it could do it), and in that case it's entirely possible that that can \n   be done efficiently (ie you still do file creates and lookups in an \n   increasingly big directory, but you do it only _once_ per file, rather \n   than look up old files all the time). So your lookup ratio would be 1:1 \n   with the files.\n\n   Doing a git-cvsimport would then do basically random lookups in that \n   _huge_ directory, and instead of reading the files one at a time (and \n   fully) and never again, I assume it opens them, reads one revision, \n   closes it, and then goes on to the next revision, so it will have a \n   much higher lookup ratio (you'd look up every file several times).\n\n - I suspect the SVN people must be hurting for performance themselves. I \n   guess they don't expect to be able to do 5-10 commits per second, the \n   way git was designed to do. So they optimized the cvs import part, but \n   their actual regular live usage is probably hitting this same directory \n   inefficiency.\n\nOf course, the old SVN Berkeley DB usage was probably even worse (not in \nsystem time, but I'd expect the access patterns within the BDB file to be \npretty nasty, and probably a lot of user time spent seeking around it). \nBut in this particular case, it might even have been better.\n\nMaybe we could teach the SVN people about pack-files? ;)\n\n\t\t\tLinus\n"},{"id":"21661","messageId":"9e4733910606121106ta925b6er49fe68bf3c1031f5@mail.gmail.com","threadId":"4478","inReplyTo":"Pine.LNX.4.64.0606120958230.5498@g5.osdl.org","subject":"Re: svn to git, N-squared?","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-12T18:06:00Z","receivedAt":"2006-06-12T18:06:00Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/12/06, Linus Torvalds <torvalds@osdl.org> wrote:\n> Having that many files in a single directory (or two) is a total disaster.\n> That said, it works well enough if you don't create new files very often\n> (and _preferably_ don't look them up either, although that is effectively\n> helped by indexing). I _suspect_ that\n\nPosted to the svn list, they said that 220K files is normal. They told\nme to turn on the ext2 dir_index option. Cheking my system I see that\nnone of partitions have it turned on so it must not be the default for\nFC5.\n\nI have to unmount the drive to convert existing directories. I can\ntrying doing the file move trick while the process is running since\nnew directories will use it.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"21666","messageId":"9e4733910606121200m402af4a4qa162c6f79cbf7100@mail.gmail.com","threadId":"4478","inReplyTo":"9e4733910606121106ta925b6er49fe68bf3c1031f5@mail.gmail.com","subject":"Re: svn to git, N-squared?","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-06-12T19:00:19Z","receivedAt":"2006-06-12T19:00:19Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 6/12/06, Jon Smirl <jonsmirl@gmail.com> wrote:\n> On 6/12/06, Linus Torvalds <torvalds@osdl.org> wrote:\n> > Having that many files in a single directory (or two) is a total disaster.\n> > That said, it works well enough if you don't create new files very often\n> > (and _preferably_ don't look them up either, although that is effectively\n> > helped by indexing). I _suspect_ that\n>\n> Posted to the svn list, they said that 220K files is normal. They told\n> me to turn on the ext2 dir_index option. Cheking my system I see that\n> none of partitions have it turned on so it must not be the default for\n> FC5.\n>\n> I have to unmount the drive to convert existing directories. I can\n> trying doing the file move trick while the process is running since\n> new directories will use it.\n\nI converted the ext3 directories to dir_index on the fly using the\nmove trick. Switching the directory index makes it look like it is\nspending even more time in the kernel.\n\nprocs -----------memory---------- ---swap-- -----io---- --system--\n-----cpu------\n r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us\nsy id wa st\n 1  0 636188  22380  19176 157200    0    0     0    52  436   415 13\n40 48  0  0\n 1  0 636188  22504  19176 157200    0    0     0     0  430   373 13\n38 49  0  0\n 1  0 636188  22628  19176 157064    0    0     0     0  433   380 12\n39 49  0  0\n 1  0 636188  22628  19184 157056    0    0     0    20  434   390 12\n38 49  0  0\n 1  0 636188  22628  19184 156920    0    0     0     0  431   376 11\n40 49  0  0\n 1  0 636188  22752  19192 156912    0    0     0    48  437   376 12\n40 49  0  0\n 1  0 636188  22876  19192 156912    0    0     0     0  430   386 11\n40 49  0  0\n 1  0 636188  22752  19192 156776    0    0     0     0  431   370 10\n41 49  0  0\n 1  0 636188  23016  19192 156776    0    0     8     0  422   500 22\n40 37  2  0\n\nThe size of the svn directories went from 3.2MB to 4.4MB after they\nwere converted to ext3 indexed mode.\n\nI'll get oprofile running when I do a reboot.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"}]}