{"thread":{"id":"10336","subject":"[PATCH] gitweb: speed up project listing by limiting find depth","startedAt":"2007-10-17T01:13:26Z","lastAt":"2007-10-17T02:40:18Z","messageCount":3,"participants":["Luke Lu","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"56169","messageId":"1192583606-14893-1-git-send-email-git@vicaya.com","threadId":"10336","inReplyTo":null,"subject":"[PATCH] gitweb: speed up project listing by limiting find depth","fromName":"Luke Lu","fromEmail":"git@vicaya.com","sentAt":"2007-10-17T01:13:26Z","receivedAt":"2007-10-17T01:13:26Z","isPatch":true,"sender":{"key":"git@vicaya.com","avatar":null},"body":"Resubmit patch due to tab/space issue :)\n\nSigned-off-by: Luke Lu <git@vicaya.com>\n---\n Makefile           |    2 ++\n gitweb/gitweb.perl |   11 +++++++++++\n 2 files changed, 13 insertions(+), 0 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 8db4dbe..3e9938e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -165,6 +165,7 @@ GITWEB_CONFIG = gitweb_config.perl\n GITWEB_HOME_LINK_STR = projects\n GITWEB_SITENAME =\n GITWEB_PROJECTROOT = /pub/git\n+GITWEB_PROJECT_MAXDEPTH = 2007\n GITWEB_EXPORT_OK =\n GITWEB_STRICT_EXPORT =\n GITWEB_BASE_URL =\n@@ -831,6 +832,7 @@ gitweb/gitweb.cgi: gitweb/gitweb.perl\n \t    -e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \\\n \t    -e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \\\n \t    -e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \\\n+\t    -e 's|\"++GITWEB_PROJECT_MAXDEPTH++\"|$(GITWEB_PROJECT_MAXDEPTH)|g' \\\n \t    -e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \\\n \t    -e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \\\n \t    -e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \\\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 3064298..d62357f 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -35,6 +35,10 @@ our $GIT = \"++GIT_BINDIR++/git\";\n #our $projectroot = \"/pub/scm\";\n our $projectroot = \"++GITWEB_PROJECTROOT++\";\n \n+# fs traversing limit for getting project list\n+# the number is relative to the projectroot\n+our $project_maxdepth = \"++GITWEB_PROJECT_MAXDEPTH++\";\n+\n # target of the home link on top of all pages\n our $home_link = $my_uri || \"/\";\n \n@@ -1509,16 +1513,23 @@ sub git_get_projects_list {\n \t\t# remove the trailing \"/\"\n \t\t$dir =~ s!/+$!!;\n \t\tmy $pfxlen = length(\"$dir\");\n+\t\tmy $pfxdepth = ($dir =~ tr!/!!);\n \n \t\tFile::Find::find({\n \t\t\tfollow_fast => 1, # follow symbolic links\n \t\t\tfollow_skip => 2, # ignore duplicates\n+\t\t\tno_chdir => 1, # don't chdir into every directory\n \t\t\tdangling_symlinks => 0, # ignore dangling symlinks, silently\n \t\t\twanted => sub {\n \t\t\t\t# skip project-list toplevel, if we get it.\n \t\t\t\treturn if (m!^[/.]$!);\n \t\t\t\t# only directories can be git repositories\n \t\t\t\treturn unless (-d $_);\n+\t\t\t\t# don't traverse too deep (Find is super slow on os x)\n+\t\t\t\tif (tr!/!! - $pfxdepth > $project_maxdepth) {\n+\t\t\t\t\t$File::Find::prune = 1;\n+\t\t\t\t\treturn;\n+\t\t\t\t}\n \n \t\t\t\tmy $subdir = substr($File::Find::name, $pfxlen + 1);\n \t\t\t\t# we check related file in $projectroot\n-- \n1.5.3.4\n"},{"id":"56172","messageId":"20071017013506.GM13801@spearce.org","threadId":"10336","inReplyTo":"1192583606-14893-1-git-send-email-git@vicaya.com","subject":"Re: [PATCH] gitweb: speed up project listing by limiting find depth","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-10-17T01:35:06Z","receivedAt":"2007-10-17T01:35:06Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Luke Lu <git@vicaya.com> wrote:\n> Resubmit patch due to tab/space issue :)\n\nThanks, I have this locally from your prior version but already\nhad fixed the tab/space problem.\n\n> +GITWEB_PROJECT_MAXDEPTH = 2007\n\nCute.  But does what I was asking for, which was to not change\nbehavior for existing users.  Most folks have a MAX_PATH around\n1024-4096.  There's no sane way they would exceed 2000 nested\ndirectories.\n\n-- \nShawn.\n"},{"id":"56189","messageId":"20071017024018.GR13801@spearce.org","threadId":"10336","inReplyTo":"1192583606-14893-1-git-send-email-git@vicaya.com","subject":"Re: [PATCH] gitweb: speed up project listing by limiting find depth","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-10-17T02:40:18Z","receivedAt":"2007-10-17T02:40:18Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Luke Lu <git@vicaya.com> wrote:\n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 3064298..d62357f 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -1509,16 +1513,23 @@ sub git_get_projects_list {\n...\n> +\t\t\t\t# don't traverse too deep (Find is super slow on os x)\n> +\t\t\t\tif (tr!/!! - $pfxdepth > $project_maxdepth) {\n> +\t\t\t\t\t$File::Find::prune = 1;\n> +\t\t\t\t\treturn;\n> +\t\t\t\t}\n>  \n>  \t\t\t\tmy $subdir = substr($File::Find::name, $pfxlen + 1);\n\nYour patch appears to be causing some errors in the test suite\nin t/t9500-gitweb-standalone-no-errors.sh.  Perl is whining about\n$subdir not getting initialized above due to the substr being off\nthe string.  I've got too many other topics tonight to figure out\nwhy yours is failing, can you please run the test and resubmit when\nyou've resolved the error?\n\n-- \nShawn.\n"}]}