threads / patch / 10336

patchgitweb: speed up project listing by limiting find depth

Subject: [PATCH] gitweb: speed up project listing by limiting find depth

## tl;dr

3 messages between Oct 17, 2007 and Oct 17, 2007. Diffs are folded; open one to read it.

replies: 2people: 2as markdown or json

Luke Lu· Oct 17, 2007, 01:13 UTC · lore
Resubmit patch due to tab/space issue :)
Signed-off-by: Luke Lu <git@vicaya.com>
---
 Makefile           |    2 ++
 gitweb/gitweb.perl |   11 +++++++++++
 2 files changed, 13 insertions(+), 0 deletions(-)
Show changes to 2 files +13 −0

Makefile, gitweb/gitweb.perl

diff --git a/Makefile b/Makefile
index 8db4dbe..3e9938e 100644
--- a/Makefile
+++ b/Makefile
@@ -165,6 +165,7 @@ GITWEB_CONFIG = gitweb_config.perl
 GITWEB_HOME_LINK_STR = projects
 GITWEB_SITENAME =
 GITWEB_PROJECTROOT = /pub/git
+GITWEB_PROJECT_MAXDEPTH = 2007
 GITWEB_EXPORT_OK =
 GITWEB_STRICT_EXPORT =
 GITWEB_BASE_URL =
@@ -831,6 +832,7 @@ gitweb/gitweb.cgi: gitweb/gitweb.perl
 	    -e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \
 	    -e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \
 	    -e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \
+	    -e 's|"++GITWEB_PROJECT_MAXDEPTH++"|$(GITWEB_PROJECT_MAXDEPTH)|g' \
 	    -e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \
 	    -e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \
 	    -e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 3064298..d62357f 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -35,6 +35,10 @@ our $GIT = "++GIT_BINDIR++/git";
 #our $projectroot = "/pub/scm";
 our $projectroot = "++GITWEB_PROJECTROOT++";
 
+# fs traversing limit for getting project list
+# the number is relative to the projectroot
+our $project_maxdepth = "++GITWEB_PROJECT_MAXDEPTH++";
+
 # target of the home link on top of all pages
 our $home_link = $my_uri || "/";
 
@@ -1509,16 +1513,23 @@ sub git_get_projects_list {
 		# remove the trailing "/"
 		$dir =~ s!/+$!!;
 		my $pfxlen = length("$dir");
+		my $pfxdepth = ($dir =~ tr!/!!);
 
 		File::Find::find({
 			follow_fast => 1, # follow symbolic links
 			follow_skip => 2, # ignore duplicates
+			no_chdir => 1, # don't chdir into every directory
 			dangling_symlinks => 0, # ignore dangling symlinks, silently
 			wanted => sub {
 				# skip project-list toplevel, if we get it.
 				return if (m!^[/.]$!);
 				# only directories can be git repositories
 				return unless (-d $_);
+				# don't traverse too deep (Find is super slow on os x)
+				if (tr!/!! - $pfxdepth > $project_maxdepth) {
+					$File::Find::prune = 1;
+					return;
+				}
 
 				my $subdir = substr($File::Find::name, $pfxlen + 1);
 				# we check related file in $projectroot
-- 
1.5.3.4
Shawn O. Pearce· Oct 17, 2007, 01:35 UTC · re: Luke Lu · lore

Re: [PATCH] gitweb: speed up project listing by limiting find depth

Luke Lu <git@vicaya.com> wrote:
> Resubmit patch due to tab/space issue :)

Thanks, I have this locally from your prior version but already had fixed the tab/space problem.

> +GITWEB_PROJECT_MAXDEPTH = 2007

Cute. But does what I was asking for, which was to not change behavior for existing users. Most folks have a MAX_PATH around 1024-4096. There's no sane way they would exceed 2000 nested directories.

-- 
Shawn.
Shawn O. Pearce· Oct 17, 2007, 02:40 UTC · re: Luke Lu · lore

Re: [PATCH] gitweb: speed up project listing by limiting find depth

Luke Lu <git@vicaya.com> wrote:
Show 5 quoted lines
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 3064298..d62357f 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -1509,16 +1513,23 @@ sub git_get_projects_list {
...
Show 7 quoted lines
> +				# don't traverse too deep (Find is super slow on os x)
> +				if (tr!/!! - $pfxdepth > $project_maxdepth) {
> +					$File::Find::prune = 1;
> +					return;
> +				}
>  
>  				my $subdir = substr($File::Find::name, $pfxlen + 1);

Your patch appears to be causing some errors in the test suite in t/t9500-gitweb-standalone-no-errors.sh. Perl is whining about $subdir not getting initialized above due to the substr being off the string. I've got too many other topics tonight to figure out why yours is failing, can you please run the test and resubmit when you've resolved the error?

-- 
Shawn.

← back to recent threads