{"thread":{"id":"13341","subject":"[RFC/PATCH] gitweb: Simplify git_project_list_body","startedAt":"2008-05-01T10:20:57Z","lastAt":"2008-05-14T07:59:54Z","messageCount":20,"participants":["Jakub Narebski","Miklos Vajna","J.H.","Lars Hjemli","Kristian Høgsberg"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"75739","messageId":"200805011220.58871.jnareb@gmail.com","threadId":"13341","inReplyTo":null,"subject":"[RFC/PATCH] gitweb: Simplify git_project_list_body","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-01T10:20:57Z","receivedAt":"2008-05-01T10:20:57Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Extract filling project list info, i.e. adding age, description, owner\nand forks information, to fill_project_list_info() subroutine.  This\nis preparation for smart pagination and smart searching (if possible,\nonly calculate info for those projects which are shown).\n\nExtract generating table header cell one can sort by into\nprint_sort_th_str() and print_sort_th_num() subroutines, and\nprint_sort_th() driver subroutine.  This avoids repetition, and should\nmake further improvements (like JavaScript client-side sorting) easier.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis is an RFC patch, first patch of series meant to improve generating\nlist of projects, trying to reduce both amount of work server has to do\nto generate the page, and the size of the page to reduce amount of data\nsend and amount of rendering web browser (client) has to do.\n\nAn RFC because:\n * I don't quote like naming of print_sort_th*() subroutines, but\n   I could not think of a better name\n * I wonder if there is a better way to pass arguments, and better way\n   to deal with numeric sorting (age) and lexical soering (the rest).\n\nI'd like for someone with better Perl knowledge than me examine this\npatch.\n\n\nWhat is planned:\n * Adding description to project_list format, so everything except age\n   (last changed) can be read by parsing single file, making gitweb\n   performance better\n * Paginate projects list in such way so the information such as age\n   is gathered and filled only for projects which are shown, if possible\n   (it is not possible for example if worting by age)\n * Add searching for projects, by project name (project path),\n   description, and owner.  Make gitweb use search page if number of\n   projects is greater than some configurable number, as a front page\n\nWhat is planned in the future\n * Add project's categories support.\n\n gitweb/gitweb.perl |  108 ++++++++++++++++++++++++++++-----------------------\n 1 files changed, 59 insertions(+), 49 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 2facf2d..6a28dca 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3548,21 +3548,23 @@ sub git_patchset_body {\n \n # . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n \n-sub git_project_list_body {\n-\tmy ($projlist, $order, $from, $to, $extra, $no_header) = @_;\n-\n-\tmy ($check_forks) = gitweb_check_feature('forks');\n-\n+# fills project list info (age, description, owner, forks) for each\n+# project in the list, removing invalid projects from returned list\n+sub fill_project_list_info {\n+\tmy ($projlist, $check_forks) = @_;\n \tmy @projects;\n+\n+ PROJECT:\n \tforeach my $pr (@$projlist) {\n-\t\tmy (@aa) = git_get_last_activity($pr->{'path'});\n-\t\tunless (@aa) {\n-\t\t\tnext;\n+\t\tmy (@activity) = git_get_last_activity($pr->{'path'});\n+\t\tunless (@activity) {\n+\t\t\tnext PROJECT;\n \t\t}\n-\t\t($pr->{'age'}, $pr->{'age_string'}) = @aa;\n+\t\t($pr->{'age'}, $pr->{'age_string'}) = @activity;\n \t\tif (!defined $pr->{'descr'}) {\n \t\t\tmy $descr = git_get_project_description($pr->{'path'}) || \"\";\n-\t\t\t$pr->{'descr_long'} = to_utf8($descr);\n+\t\t\t$descr = to_utf8($descr);\n+\t\t\t$pr->{'descr_long'} = $descr;\n \t\t\t$pr->{'descr'} = chop_str($descr, $projects_list_description_width, 5);\n \t\t}\n \t\tif (!defined $pr->{'owner'}) {\n@@ -3574,14 +3576,50 @@ sub git_project_list_body {\n \t\t\t    ($pname !~ /\\/$/) &&\n \t\t\t    (-d \"$projectroot/$pname\")) {\n \t\t\t\t$pr->{'forks'} = \"-d $projectroot/$pname\";\n-\t\t\t}\n-\t\t\telse {\n+\t\t\t}\telse {\n \t\t\t\t$pr->{'forks'} = 0;\n \t\t\t}\n \t\t}\n \t\tpush @projects, $pr;\n \t}\n \n+\treturn @projects;\n+}\n+\n+sub print_sort_th {\n+\tmy ($str_sort, $name, $order, $key, $header, $projlist) = @_;\n+\t$key    ||= $name;\n+\t$header ||= ucfirst($name);\n+\n+\tif ($order eq $name) {\n+\t\tif ($str_sort) {\n+\t\t\t@$projlist = sort {$a->{$key} cmp $b->{$key}} @$projlist;\n+\t\t} else {\n+\t\t\t@$projlist = sort {$a->{$key} <=> $b->{$key}} @$projlist;\n+\t\t}\n+\t\tprint \"<th>$header</th>\\n\";\n+\t} else {\n+\t\tprint \"<th>\" .\n+\t\t      $cgi->a({-href => href(project=>undef, order=>$name),\n+\t\t               -class => \"header\"}, $header) .\n+\t\t      \"</th>\\n\";\n+\t}\n+}\n+\n+sub print_sort_th_str {\n+\tprint_sort_th(1, @_);\n+}\n+\n+sub print_sort_th_num {\n+\tprint_sort_th(0, @_);\n+}\n+\n+sub git_project_list_body {\n+\tmy ($projlist, $order, $from, $to, $extra, $no_header) = @_;\n+\n+\tmy ($check_forks) = gitweb_check_feature('forks');\n+\tmy @projects = fill_project_list_info($projlist, $check_forks);\n+\n \t$order ||= $default_projects_order;\n \t$from = 0 unless defined $from;\n \t$to = $#projects if (!defined $to || $#projects < $to);\n@@ -3592,43 +3630,15 @@ sub git_project_list_body {\n \t\tif ($check_forks) {\n \t\t\tprint \"<th></th>\\n\";\n \t\t}\n-\t\tif ($order eq \"project\") {\n-\t\t\t@projects = sort {$a->{'path'} cmp $b->{'path'}} @projects;\n-\t\t\tprint \"<th>Project</th>\\n\";\n-\t\t} else {\n-\t\t\tprint \"<th>\" .\n-\t\t\t      $cgi->a({-href => href(project=>undef, order=>'project'),\n-\t\t\t               -class => \"header\"}, \"Project\") .\n-\t\t\t      \"</th>\\n\";\n-\t\t}\n-\t\tif ($order eq \"descr\") {\n-\t\t\t@projects = sort {$a->{'descr'} cmp $b->{'descr'}} @projects;\n-\t\t\tprint \"<th>Description</th>\\n\";\n-\t\t} else {\n-\t\t\tprint \"<th>\" .\n-\t\t\t      $cgi->a({-href => href(project=>undef, order=>'descr'),\n-\t\t\t               -class => \"header\"}, \"Description\") .\n-\t\t\t      \"</th>\\n\";\n-\t\t}\n-\t\tif ($order eq \"owner\") {\n-\t\t\t@projects = sort {$a->{'owner'} cmp $b->{'owner'}} @projects;\n-\t\t\tprint \"<th>Owner</th>\\n\";\n-\t\t} else {\n-\t\t\tprint \"<th>\" .\n-\t\t\t      $cgi->a({-href => href(project=>undef, order=>'owner'),\n-\t\t\t               -class => \"header\"}, \"Owner\") .\n-\t\t\t      \"</th>\\n\";\n-\t\t}\n-\t\tif ($order eq \"age\") {\n-\t\t\t@projects = sort {$a->{'age'} <=> $b->{'age'}} @projects;\n-\t\t\tprint \"<th>Last Change</th>\\n\";\n-\t\t} else {\n-\t\t\tprint \"<th>\" .\n-\t\t\t      $cgi->a({-href => href(project=>undef, order=>'age'),\n-\t\t\t               -class => \"header\"}, \"Last Change\") .\n-\t\t\t      \"</th>\\n\";\n-\t\t}\n-\t\tprint \"<th></th>\\n\" .\n+\t\tprint_sort_th_str('project', $order, 'path',\n+\t\t                  'Project', \\@projects);\n+\t\tprint_sort_th_str('descr', $order, 'descr_long',\n+\t\t                  'Description', \\@projects);\n+\t\tprint_sort_th_str('owner', $order, 'owner',\n+\t\t                  'Owner', \\@projects);\n+\t\tprint_sort_th_num('age', $order, 'age',\n+\t\t                  'Last Change', \\@projects);\n+\t\tprint \"<th></th>\\n\" . # for links\n \t\t      \"</tr>\\n\";\n \t}\n \tmy $alternate = 1;\n-- \n1.5.5\n"},{"id":"75820","messageId":"200805021230.05907.jnareb@gmail.com","threadId":"13341","inReplyTo":"200805011220.58871.jnareb@gmail.com","subject":"[RFC/PATCH] gitweb: Allow project description in project_index file","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-02T10:30:04Z","receivedAt":"2008-05-02T10:30:04Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Change format of $projects_list file from\n  <URI-encoded path> SPC <URI-encoded owner> LF\nto\n  <URI-encoded path> SPC <URI-encoded owner> [SPC <description>] LF\nwith optional project description.  To make it easier to read and edit\n$projects_list file the description _is not_ URI encoded.  Please\nremember that only single line of repository (project) description is\nsupported.  Note that SPC can be replaced by any whitespace\ncharacter.\n\nThis change required modifying git_get_projects_list() subroutine\n(part when $projects_list is a file, not a directory to be scanned),\nand git_get_project_description() subroutine.\n\nThe 'project_index' action creates projects list index file in the new\nformat, with project description.\n\nThi change is backwards compatibile: older gitweb with new projects\nlist format would work as it used to work; new gitweb with old format\nwould get (try to get) project description from other sources.\n\n\nWhile at it some comments describing changes and changed subroutines\nwere added, and information about $projects_list format was updated in\ngitweb/README.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis is second patch in the yet to be written series improving projects\nlist page and related things.  I'm sending an early version of patch to\nask for comments.\n\nThe reason why it is an RFC is the decision to _not_ URI-decode (to not\nforce URI-encoding of e.g. spaces) in the project (repository)\ndescription part of projects list page.  It makes projects index file\neasier to read and to edit, but it closes possibility of extending this\nformat further.  And there is another thing that could be reasonably put\nin this file: future project *categories* support.  So should I try to\nadd categories support first?\n\nNote that without this patch future patches, dividing projects list page\ninto 100-items long pages, and adding projects search page, which are\nmeant to bring preformance improvements, would get less performance\nimprovements without this patch.\n\nComments appreciated...\n\n gitweb/README      |    9 +++++--\n gitweb/gitweb.perl |   61 +++++++++++++++++++++++++++++++++++++++++++++------\n 2 files changed, 59 insertions(+), 11 deletions(-)\n\ndiff --git a/gitweb/README b/gitweb/README\nindex 8f7ea36..e58fe18 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -157,9 +157,12 @@ not include variables usually directly set during build):\n  * $projects_list\n    Source of projects list, either directory to scan, or text file\n    with list of repositories (in the \"<URI-encoded repository path> SPC\n-   <URI-encoded repository owner>\" format).  Set to $GITWEB_LIST\n-   during installation.  If empty, $projectroot is used to scan for\n-   repositories.\n+   <URI-encoded repository owner>\" line format, or optionally with\n+   project description in \"<URI-encoded repository path> SPC\n+   <URI-encoded repository owner> SPC <single line description>\";\n+   actually there can be any whitespace in place of SPC).  Set to\n+   $GITWEB_LIST during installation.  If empty, $projectroot is used\n+   to scan for repositories.\n  * $my_url, $my_uri\n    URL and absolute URL of gitweb script; you might need to set those\n    variables if you are using 'pathinfo' feature: see also below.\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 6a28dca..dc3f99a 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1702,13 +1702,43 @@ sub git_get_path_by_hash {\n ## ......................................................................\n ## git utility functions, directly accessing git repository\n \n+# this is helper function for git_get_project_description()\n+sub git_get_project_description_from_list {\n+\tmy $path = shift;\n+\n+\topen(my $fd, $projects_list)\n+\t\tor return;\n+\twhile (my $line = <$fd>) {\n+\t\tchomp $line;\n+\t\tmy ($pr, undef, $descr) = split(' ', $line, 3);\n+\t\t$pr = unescape($pr);\n+\t\tif ($pr eq $path) {\n+\t\t\tclose $fd;\n+\t\t\treturn $descr;\n+\t\t}\n+\t}\n+\tclose $fd;\n+\treturn;\n+}\n+\n+# sources for project description are\n+#  * project_list, if $project_list is a file, and it uses new format:\n+#    URI_encoded(<path>) SPC URI_encoded(<owner>) SPC <description> LF\n+#  * $GIT_DIR/description file in project repository, if it exists\n+#  * gitweb.description configuration variable for a project\n sub git_get_project_description {\n \tmy $path = shift;\n+\tmy $descr;\n+\n+\tif (-f $projects_list) {\n+\t\t$descr = git_get_project_description_from_list($path);\n+\t\treturn $descr if (defined $descr); # try other sources if needed\n+\t}\n \n \t$git_dir = \"$projectroot/$path\";\n \topen my $fd, \"$git_dir/description\"\n \t\tor return git_get_project_config('description');\n-\tmy $descr = <$fd>;\n+\t$descr = <$fd>;\n \tclose $fd;\n \tif (defined $descr) {\n \t\tchomp $descr;\n@@ -1774,20 +1804,24 @@ sub git_get_projects_list {\n \t\t}, \"$dir\");\n \n \t} elsif (-f $projects_list) {\n-\t\t# read from file(url-encoded):\n+\t\t# read from file (whitespace separated, url-encoded):\n \t\t# 'git%2Fgit.git Linus+Torvalds'\n \t\t# 'libs%2Fklibc%2Fklibc.git H.+Peter+Anvin'\n \t\t# 'linux%2Fhotplug%2Fudev.git Greg+Kroah-Hartman'\n+\t\t# optionally with description (which isn't url-encoded)\n+\t\t# 'git/git.git Junio+C+Hamano The core git plumbing'\n+\t\t# 'libs/klibc/klibc.git H.+Peter+Anvin klibc main development tree'\n+\t\t# 'linux/hotplug/udev.git Kay+Sievers udev development tree'\n \t\tmy %paths;\n \t\topen my ($fd), $projects_list or return;\n \tPROJECT:\n \t\twhile (my $line = <$fd>) {\n \t\t\tchomp $line;\n-\t\t\tmy ($path, $owner) = split ' ', $line;\n+\t\t\tmy ($path, $owner, $descr) = split(' ', $line, 3);\n \t\t\t$path = unescape($path);\n \t\t\t$owner = unescape($owner);\n \t\t\tif (!defined $path) {\n-\t\t\t\tnext;\n+\t\t\t\tnext PROJECT;\n \t\t\t}\n \t\t\tif ($filter ne '') {\n \t\t\t\t# looking for forks;\n@@ -1818,9 +1852,14 @@ sub git_get_projects_list {\n \t\t\t}\n \t\t\tif (check_export_ok(\"$projectroot/$path\")) {\n \t\t\t\tmy $pr = {\n-\t\t\t\t\tpath => $path,\n+\t\t\t\t\tpath =>  to_utf8($path),\n \t\t\t\t\towner => to_utf8($owner),\n \t\t\t\t};\n+\t\t\t\tif (defined $descr) {\n+\t\t\t\t\t$descr = to_utf8($descr);\n+\t\t\t\t\t$pr->{'descr_long'} = $descr;\n+\t\t\t\t\t$pr->{'descr'} = chop_str($descr, $projects_list_description_width, 5);\n+\t\t\t\t}\n \t\t\t\tpush @list, $pr;\n \t\t\t\t(my $forks_path = $path) =~ s/\\.git$//;\n \t\t\t\t$paths{$forks_path}++;\n@@ -4006,21 +4045,27 @@ sub git_project_index {\n \tprint $cgi->header(\n \t\t-type => 'text/plain',\n \t\t-charset => 'utf-8',\n-\t\t-content_disposition => 'inline; filename=\"index.aux\"');\n+\t\t-content_disposition => 'inline; filename=\"projects_index.aux\"');\n \n \tforeach my $pr (@projects) {\n \t\tif (!exists $pr->{'owner'}) {\n \t\t\t$pr->{'owner'} = git_get_project_owner(\"$pr->{'path'}\");\n+\t\t\t$pr->{'owner'} = to_utf8($pr->{'owner'});\n+\t\t}\n+\t\tif (!exists $pr->{'descr_long'}) {\n+\t\t\t$pr->{'descr_long'} = git_get_project_description($pr->{'path'}) || \"\";\n+\t\t\t$pr->{'descr_long'} = to_utf8($pr->{'descr_long'});\n \t\t}\n \n-\t\tmy ($path, $owner) = ($pr->{'path'}, $pr->{'owner'});\n+\t\tmy ($path, $owner, $descr) = \n+\t\t\t($pr->{'path'}, $pr->{'owner'}, $pr->{'descr_long'});\n \t\t# quote as in CGI::Util::encode, but keep the slash, and use '+' for ' '\n \t\t$path  =~ s/([^a-zA-Z0-9_.\\-\\/ ])/sprintf(\"%%%02X\", ord($1))/eg;\n \t\t$owner =~ s/([^a-zA-Z0-9_.\\-\\/ ])/sprintf(\"%%%02X\", ord($1))/eg;\n \t\t$path  =~ s/ /\\+/g;\n \t\t$owner =~ s/ /\\+/g;\n \n-\t\tprint \"$path $owner\\n\";\n+\t\tprint \"$path $owner $descr\\n\";\n \t}\n }\n \n-- \n1.5.5\n"},{"id":"75833","messageId":"20080502130456.GN23672@genesis.frugalware.org","threadId":"13341","inReplyTo":"200805021230.05907.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Allow project description in project_index file","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2008-05-02T13:04:56Z","receivedAt":"2008-05-02T13:04:56Z","isPatch":true,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Fri, May 02, 2008 at 12:30:04PM +0200, Jakub Narebski <jnareb@gmail.com> wrote:\n> The reason why it is an RFC is the decision to _not_ URI-decode (to not\n> force URI-encoding of e.g. spaces) in the project (repository)\n> description part of projects list page.  It makes projects index file\n> easier to read and to edit, but it closes possibility of extending this\n> format further.  And there is another thing that could be reasonably put\n> in this file: future project *categories* support.  So should I try to\n> add categories support first?\n\nJust my two cents, I don't think that encoded strings are so unreadable.\nAlso, having the ability to extend the file later as well would be nice.\n"},{"id":"75906","messageId":"200805031103.14960.jnareb@gmail.com","threadId":"13341","inReplyTo":"20080502130456.GN23672@genesis.frugalware.org","subject":"[RFC/PATCH] gitweb: Allow project description in project_index file","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-03T09:03:13Z","receivedAt":"2008-05-03T09:03:13Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Change format of $projects_list file from\n  <URI-encoded path> SPC <URI-encoded owner> LF\nto\n  <URI-encoded path> SPC\n  <URI-encoded owner> [SPC <URI-encoded description>] LF\nwith optional project description.  Please remember that only single\nline of repository (project) description is supported.  Note that SPC\ncan be replaced by any whitespace character.\n\nThis change required modifying git_get_projects_list() subroutine\n(part when $projects_list is a file, not a directory to be scanned),\nand git_get_project_description() subroutine.\n\nThe 'project_index' action creates projects list index file in the new\nformat, with project description.  Also, it now does only minial level\nof escaping / encoding.\n\n\nWhile at it some comments describing changes and changed subroutines\nwere added, and information about $projects_list format was updated in\ngitweb/README and in gitweb/INSTALL.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nOn Fri, 2 May 2008, Miklos Vajna wrote:\n> On Fri, May 02, 2008 at 12:30:04PM +0200, Jakub Narebski <jnareb@gmail.com> wrote:\n>>\n>> The reason why it is an RFC is the decision to _not_ URI-decode (to not\n>> force URI-encoding of e.g. spaces) in the project (repository)\n>> description part of projects list page.  It makes projects index file\n>> easier to read and to edit, but it closes possibility of extending this\n>> format further.  And there is another thing that could be reasonably put\n>> in this file: future project *categories* support.  So should I try to\n>> add categories support first?\n> \n> Just my two cents, I don't think that encoded strings are so unreadable.\n> Also, having the ability to extend the file later as well would be nice.\n\nThis version uses URI-encoded 'description' field in $projects_list\nfile, thus allowing to extend it (e.g. by adding categories support)\nin a backward compatibile way.\n\nAlso, the format is described in more elaborate way; please comment...\n\n gitweb/INSTALL     |   10 +++++--\n gitweb/README      |    9 ++++--\n gitweb/gitweb.perl |   74 ++++++++++++++++++++++++++++++++++++++++++---------\n 3 files changed, 74 insertions(+), 19 deletions(-)\n\ndiff --git a/gitweb/INSTALL b/gitweb/INSTALL\nindex f7194db..7e25a7f 100644\n--- a/gitweb/INSTALL\n+++ b/gitweb/INSTALL\n@@ -140,9 +140,13 @@ Gitweb repositories\n \n   Each line of the projects list file should consist of the url-encoded path\n   to the project repository database (relative to projectroot), followed\n-  by the url-encoded project owner on the same line (separated by a space).\n-  Spaces in both project path and project owner have to be encoded as either\n-  '%20' or '+'.\n+  by the url-encoded project owner on the same line (separated by a space),\n+  and optionally followed by the url-encoded project description (spearated\n+  by space).  Spaces in project path, project owner and project description\n+  have to be encoded as either '%20' or '+'.  Other whitespace (separator),\n+  plus sign '+' (used as replacement for spaces), and percent sign '%' (used\n+  for encoding / escaping) have to be url-encoded, i.e. replaced by '%'\n+  followed by two-digit character number in octal.\n \n   You can generate the projects list index file using the project_index\n   action (the 'TXT' link on projects list page) directly from gitweb.\ndiff --git a/gitweb/README b/gitweb/README\nindex 8f7ea36..130449c 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -157,9 +157,12 @@ not include variables usually directly set during build):\n  * $projects_list\n    Source of projects list, either directory to scan, or text file\n    with list of repositories (in the \"<URI-encoded repository path> SPC\n-   <URI-encoded repository owner>\" format).  Set to $GITWEB_LIST\n-   during installation.  If empty, $projectroot is used to scan for\n-   repositories.\n+   <URI-encoded repository owner>\" line format, or optionally with\n+   project description in \"<URI-encoded repository path> SPC\n+   <URI-encoded repository owner> SPC <URI-encoded description>\";\n+   actually there can be any sequence of whitespace in place of SPC).\n+   Set to $GITWEB_LIST during installation.  If empty, $projectroot is\n+   used to scan for repositories.\n  * $my_url, $my_uri\n    URL and absolute URL of gitweb script; you might need to set those\n    variables if you are using 'pathinfo' feature: see also below.\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 6a28dca..296bfaa 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1702,13 +1702,44 @@ sub git_get_path_by_hash {\n ## ......................................................................\n ## git utility functions, directly accessing git repository\n \n+# this is helper function for git_get_project_description()\n+sub git_get_project_description_from_list {\n+\tmy $path = shift;\n+\n+\topen(my $fd, $projects_list)\n+\t\tor return;\n+\twhile (my $line = <$fd>) {\n+\t\tchomp $line;\n+\t\tmy ($proj, undef, $descr) = split ' ', $line;\n+\t\t$proj = unescape($pr);\n+\t\t$descr = unescape($descr);\n+\t\tif ($proj eq $path) {\n+\t\t\tclose $fd;\n+\t\t\treturn $descr;\n+\t\t}\n+\t}\n+\tclose $fd;\n+\treturn;\n+}\n+\n+# sources for project description are\n+#  * project_list, if $project_list is a file, and it uses new format:\n+#    <encoded path> SPC <encoded owner> SPC <encoded description> LF\n+#  * $GIT_DIR/description file in project repository, if it exists\n+#  * gitweb.description configuration variable for a project\n sub git_get_project_description {\n \tmy $path = shift;\n+\tmy $descr;\n+\n+\tif (-f $projects_list) {\n+\t\t$descr = git_get_project_description_from_list($path);\n+\t\treturn $descr if (defined $descr); # try other sources if needed\n+\t}\n \n \t$git_dir = \"$projectroot/$path\";\n \topen my $fd, \"$git_dir/description\"\n \t\tor return git_get_project_config('description');\n-\tmy $descr = <$fd>;\n+\t$descr = <$fd>;\n \tclose $fd;\n \tif (defined $descr) {\n \t\tchomp $descr;\n@@ -1774,20 +1805,25 @@ sub git_get_projects_list {\n \t\t}, \"$dir\");\n \n \t} elsif (-f $projects_list) {\n-\t\t# read from file(url-encoded):\n+\t\t# read from file (whitespace separated, url-encoded):\n \t\t# 'git%2Fgit.git Linus+Torvalds'\n \t\t# 'libs%2Fklibc%2Fklibc.git H.+Peter+Anvin'\n \t\t# 'linux%2Fhotplug%2Fudev.git Greg+Kroah-Hartman'\n+\t\t# optionally with description\n+\t\t# 'git/git.git Junio+C+Hamano The+core+git+plumbing'\n+\t\t# 'libs/klibc/klibc.git H.+Peter+Anvin klibc+main+development+tree'\n+\t\t# 'linux/hotplug/udev.git Kay+Sievers udev+development+tree'\n \t\tmy %paths;\n \t\topen my ($fd), $projects_list or return;\n \tPROJECT:\n \t\twhile (my $line = <$fd>) {\n \t\t\tchomp $line;\n-\t\t\tmy ($path, $owner) = split ' ', $line;\n+\t\t\tmy ($path, $owner, $descr) = split ' ', $line;\n \t\t\t$path = unescape($path);\n \t\t\t$owner = unescape($owner);\n+\t\t\t$descr = unescape($descr);\n \t\t\tif (!defined $path) {\n-\t\t\t\tnext;\n+\t\t\t\tnext PROJECT;\n \t\t\t}\n \t\t\tif ($filter ne '') {\n \t\t\t\t# looking for forks;\n@@ -1818,9 +1854,14 @@ sub git_get_projects_list {\n \t\t\t}\n \t\t\tif (check_export_ok(\"$projectroot/$path\")) {\n \t\t\t\tmy $pr = {\n-\t\t\t\t\tpath => $path,\n+\t\t\t\t\tpath =>  to_utf8($path),\n \t\t\t\t\towner => to_utf8($owner),\n \t\t\t\t};\n+\t\t\t\tif (defined $descr) {\n+\t\t\t\t\t$descr = to_utf8($descr);\n+\t\t\t\t\t$pr->{'descr_long'} = $descr;\n+\t\t\t\t\t$pr->{'descr'} = chop_str($descr, $projects_list_description_width, 5);\n+\t\t\t\t}\n \t\t\t\tpush @list, $pr;\n \t\t\t\t(my $forks_path = $path) =~ s/\\.git$//;\n \t\t\t\t$paths{$forks_path}++;\n@@ -4006,21 +4047,28 @@ sub git_project_index {\n \tprint $cgi->header(\n \t\t-type => 'text/plain',\n \t\t-charset => 'utf-8',\n-\t\t-content_disposition => 'inline; filename=\"index.aux\"');\n+\t\t-content_disposition => 'inline; filename=\"projects_index.aux\"');\n \n \tforeach my $pr (@projects) {\n \t\tif (!exists $pr->{'owner'}) {\n \t\t\t$pr->{'owner'} = git_get_project_owner(\"$pr->{'path'}\");\n+\t\t\t$pr->{'owner'} = to_utf8($pr->{'owner'});\n+\t\t}\n+\t\tif (!exists $pr->{'descr_long'}) {\n+\t\t\t$pr->{'descr_long'} = git_get_project_description($pr->{'path'}) || \"\";\n+\t\t\t$pr->{'descr_long'} = to_utf8($pr->{'descr_long'});\n \t\t}\n \n-\t\tmy ($path, $owner) = ($pr->{'path'}, $pr->{'owner'});\n-\t\t# quote as in CGI::Util::encode, but keep the slash, and use '+' for ' '\n-\t\t$path  =~ s/([^a-zA-Z0-9_.\\-\\/ ])/sprintf(\"%%%02X\", ord($1))/eg;\n-\t\t$owner =~ s/([^a-zA-Z0-9_.\\-\\/ ])/sprintf(\"%%%02X\", ord($1))/eg;\n-\t\t$path  =~ s/ /\\+/g;\n-\t\t$owner =~ s/ /\\+/g;\n+\t\tmy @pr_info = \n+\t\t\t($pr->{'path'}, $pr->{'owner'}, $pr->{'descr_long'});\n+\t\tforeach (@pr_info) {\n+\t\t\t# quote only minimal set, only what has to be quoted\n+\t\t\ts/([+%])/sprintf(\"%%%02X\", ord($1))/eg;\n+\t\t\ts/ /\\+/g;\n+\t\t\ts/([[:space:]])/sprintf(\"%%%02X\", ord($1))/eg;\n+\t\t}\n \n-\t\tprint \"$path $owner\\n\";\n+\t\tprint join(' ', @pr_info).\"\\n\";\n \t}\n }\n \n-- \n1.5.5\n"},{"id":"75969","messageId":"20080504020309.GR23672@genesis.frugalware.org","threadId":"13341","inReplyTo":"200805031103.14960.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Allow project description in project_index file","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2008-05-04T02:03:09Z","receivedAt":"2008-05-04T02:03:09Z","isPatch":true,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Sat, May 03, 2008 at 11:03:13AM +0200, Jakub Narebski <jnareb@gmail.com> wrote:\n> Also, the format is described in more elaborate way; please comment...\n\nNot that my opinion matters here too much, but I like it. :-)\n"},{"id":"76456","messageId":"200805091523.24591.jnareb@gmail.com","threadId":"13341","inReplyTo":"200805031103.14960.jnareb@gmail.com","subject":"[RFC/PATCH] gitweb: Project search","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-09T13:23:23Z","receivedAt":"2008-05-09T13:23:23Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"If the number of projects is greater than some number, currently for\ntesting set to 1 (in production probably to 100), then projects\nlisting is replaced by project search form.  One can search by project\nname (project path relative to $projectroot), by project description,\nand by owner.  There is also link to show all projects.\n\nBefore searching by some field the information we search for must be\nfilled in.  For this fill_project_list_info() was enhanced to take\nadditional parameter which part of projects info to fill.  This way we\ncan limit doing expensive calculations (like running git-for-each-ref\nto get 'age' / \"Last changed\" info) only to projects which we will\nshow as search results.\n\nTo make sorting by given column of project list / project search\nresults, the 'sort by' links in print_sort_th() subroutine are\ngenerated as 'replay' links.\n\nNOTE: currently match is _not_ highlighted in search results.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch depends on earlier \n  \"[RFC/PATCH] gitweb: Simplify git_project_list_body\"\npatch, and may (but should not) depend textually on\n  \"[RFC/PATCH] gitweb: Allow project description in project_index file\"\n\nAs this is quick'n'dirty proof-of-concept patch the styling of project\nsearch form is done in HTML, instead of using CSS to do that.\n\nThis patch should reduce both server load and browser load for sites\nwith very large number of projects.\n\nNOTE 1: in production one would increase threshold above which project\nsearch is used instead of listing all projects.\n\nNOTE 2: I think that for effective caching one would need to cache\ndata, not final output (to J.H. and Lea Wiemann).\n\nWhat do you think about this concept?\n\n gitweb/gitweb.perl |   92 ++++++++++++++++++++++++++++++++++++++++++++-------\n 1 files changed, 79 insertions(+), 13 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex b117364..7e1a9b4 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -472,7 +472,7 @@ if (defined $page) {\n \n our $searchtype = $cgi->param('st');\n if (defined $searchtype) {\n-\tif ($searchtype =~ m/[^a-z]/) {\n+\tif ($searchtype =~ m/[^a-z_]/) {\n \t\tdie_error(undef, \"Invalid searchtype parameter\");\n \t}\n }\n@@ -3589,29 +3589,61 @@ sub git_patchset_body {\n \n # . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n \n+sub git_project_search_form {\n+\tmy ($projlist, $searchtype, $searchtext, $search_use_regexp);\n+\n+\t$searchtype ||= 'descr';\n+\tprint \"\\n<!-- START: project_list search form -->\\n\" .\n+\t      \"<div style=\\\"text-align: center; margin-top: 20px;\\\">\\n\";\n+\tprint $cgi->startform(-method => 'get', -action => $my_uri) .\n+\t      $cgi->hidden(-name => 'a', -value => 'project_list') . \"\\n\" .\n+\t      $cgi->popup_menu(-name => 'st', -default => $searchtype,\n+\t                       -values => ['project', 'descr', 'owner'],\n+\t                       -labels => {'project' => 'Project',\n+\t                                   'descr' => 'Description',\n+\t                                   'owner' => 'Owner'}) . \"\\n\" .\n+\t      $cgi->textfield(-name => 's', -value => $searchtext,\n+\t                      -size => 60) . \"\\n\" .\n+\t      \"<span title=\\\"Extended regular expression\\\">\" .\n+\t      $cgi->checkbox(-name => 'sr', -value => 1, -label => 're',\n+\t                     -checked => $search_use_regexp) .\n+\t      \"</span>\\n\" .\n+\t      $cgi->submit(-name => 'btnS', -value => 'Search') .\n+\t      $cgi->end_form() . \"\\n\" .\n+\t      $cgi->a({-href => href(project => undef, searchtype => 'list_all')},\n+\t              'List all projects') . \"\\n\";\n+\tprint \"</div>\\n\" .\n+\t      \"<!-- END: project_list search form -->\\n\\n\";\n+}\n+\n # fills project list info (age, description, owner, forks) for each\n-# project in the list, removing invalid projects from returned list\n+# project in the list, removing invalid projects from returned list,\n+# or fill only specified info (removing only when filling 'age')\n sub fill_project_list_info {\n-\tmy ($projlist, $check_forks) = @_;\n+\tmy ($projlist, $check_forks, $fill_only) = @_;\n \tmy @projects;\n \n  PROJECT:\n \tforeach my $pr (@$projlist) {\n-\t\tmy (@activity) = git_get_last_activity($pr->{'path'});\n-\t\tunless (@activity) {\n-\t\t\tnext PROJECT;\n-\t\t}\n-\t\t($pr->{'age'}, $pr->{'age_string'}) = @activity;\n-\t\tif (!defined $pr->{'descr'}) {\n+\t\tif (!exists $pr->{'age'} &&\n+\t\t    (!defined $fill_only || $fill_only eq 'age')) {\n+\t\t\tmy (@activity) = git_get_last_activity($pr->{'path'});\n+\t\t\tnext PROJECT unless (@activity);\n+\t\t\t($pr->{'age'}, $pr->{'age_string'}) = @activity;\n+\t\t}\n+\t\tif (!defined $pr->{'descr'} &&\n+\t\t    (!defined $fill_only || $fill_only eq 'descr')) {\n \t\t\tmy $descr = git_get_project_description($pr->{'path'}) || \"\";\n \t\t\t$descr = to_utf8($descr);\n \t\t\t$pr->{'descr_long'} = $descr;\n \t\t\t$pr->{'descr'} = chop_str($descr, $projects_list_description_width, 5);\n \t\t}\n-\t\tif (!defined $pr->{'owner'}) {\n+\t\tif (!defined $pr->{'owner'} &&\n+\t\t    (!defined $fill_only || $fill_only eq 'owner')) {\n \t\t\t$pr->{'owner'} = git_get_project_owner(\"$pr->{'path'}\") || \"\";\n \t\t}\n-\t\tif ($check_forks) {\n+\t\tif ($check_forks &&\n+\t\t    (!defined $fill_only || $fill_only eq 'forks')) {\n \t\t\tmy $pname = $pr->{'path'};\n \t\t\tif (($pname =~ s/\\.git$//) &&\n \t\t\t    ($pname !~ /\\/$/) &&\n@@ -3627,6 +3659,25 @@ sub fill_project_list_info {\n \treturn @projects;\n }\n \n+# show only projects which match what we search for\n+sub filter_project_list {\n+\tmy ($projlist, $searchtype, $search_regexp) = @_;\n+\tmy %keyname = (\n+\t\t'project' => 'path',\n+\t\t'descr' => 'descr_long',\n+\t\t'owner' => 'owner',\n+\t);\n+\tmy $key = $keyname{$searchtype};\n+\n+\t# fill in the field we search on\n+\t@$projlist = fill_project_list_info($projlist, 0, $searchtype);\n+\t# filter projects list\n+\t@$projlist = grep\n+\t\t{\t$_->{$key} =~ m/$search_regexp/i; } @$projlist;\n+\n+\treturn $projlist;\n+}\n+\n sub print_sort_th {\n \tmy ($str_sort, $name, $order, $key, $header, $projlist) = @_;\n \t$key    ||= $name;\n@@ -3641,7 +3692,7 @@ sub print_sort_th {\n \t\tprint \"<th>$header</th>\\n\";\n \t} else {\n \t\tprint \"<th>\" .\n-\t\t      $cgi->a({-href => href(project=>undef, order=>$name),\n+\t\t      $cgi->a({-href => href(-replay => 1, order=>$name),\n \t\t               -class => \"header\"}, $header) .\n \t\t      \"</th>\\n\";\n \t}\n@@ -4005,6 +4056,10 @@ sub git_project_list {\n \tif (defined $order && $order !~ m/none|project|descr|owner|age/) {\n \t\tdie_error(undef, \"Unknown order parameter\");\n \t}\n+\tif (defined $searchtype &&\n+\t    $searchtype !~ m/^(project|descr|owner|list_all)$/x) {\n+\t\tdie_error(undef, \"Unknown searchtype parameter\");\n+\t}\n \n \tmy @list = git_get_projects_list();\n \tif (!@list) {\n@@ -4019,7 +4074,18 @@ sub git_project_list {\n \t\tclose $fd;\n \t\tprint \"</div>\\n\";\n \t}\n-\tgit_project_list_body(\\@list, $order);\n+\t# 'defined $searchtype' serves as \"was search performed\" test\n+\tif (@list > 1 || defined $searchtype) {\n+\t\tgit_project_search_form(\\@list, $searchtype,\n+\t\t                        $searchtext, $search_use_regexp);\n+\t\tif (defined $searchtype) {\n+\t\t\tfilter_project_list(\\@list, $searchtype, $search_regexp)\n+\t\t\t\tunless ($searchtype eq 'list_all');\n+\t\t\tgit_project_list_body(\\@list, $order);\n+\t\t}\n+\t} else {\n+\t\tgit_project_list_body(\\@list, $order);\n+\t}\n \tgit_footer_html();\n }\n \n-- \n1.5.5\n"},{"id":"76503","messageId":"200805101128.59313.jnareb@gmail.com","threadId":"13341","inReplyTo":"200805091523.24591.jnareb@gmail.com","subject":"[RFC/PATCH] gitweb: Paginate project list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-10T09:28:56Z","receivedAt":"2008-05-10T09:28:56Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Divide projects list and project search results into pages of\n$page_size (currently 100) elements.  Pagination links look\nlike shown below:\n\n  first prev 1 2 3 *4* 5 6 next last\n\nwhere *4* denotes current page, is in bold, and isn't a link.\nThis would need to be improved for extremely large number of projects\n(repositories), where number of projects divided by page size is very\nlarge.\n\nCurrently pagination interacts with sorting in a bit strange way.\nWhen clicking on \"sort by\" link when on some later page the result is\nnot this page sorted, by given page of sorted output.  Perhaps \"sort\nby\" links should always show first page...\n\ngit_project_list_body() subroutine takes care to fill field (key) we\nsort by for all projects, sort by given ordering, and fill the rest of\nproject info only for projects shown.\n\nNOTE: currently code is not optimal, as it contains a few unnecessary\narray copying, or array fragment copying.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch depends on earlier \"gitweb: Project search\", but conceptually\nthey are independent; probably fill_project_list_info() improvement to\ndo partial filling should be in seaparet commit...\n\n gitweb/gitweb.perl |   57 ++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 files changed, 51 insertions(+), 6 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 7e1a9b4..716f7ad 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3709,10 +3709,10 @@ sub print_sort_th_num {\n sub git_project_list_body {\n \tmy ($projlist, $order, $from, $to, $extra, $no_header) = @_;\n \n+\t$order ||= $default_projects_order;\n \tmy ($check_forks) = gitweb_check_feature('forks');\n-\tmy @projects = fill_project_list_info($projlist, $check_forks);\n+\tmy @projects = fill_project_list_info($projlist, $check_forks, $order);\n \n-\t$order ||= $default_projects_order;\n \t$from = 0 unless defined $from;\n \t$to = $#projects if (!defined $to || $#projects < $to);\n \n@@ -3733,9 +3733,10 @@ sub git_project_list_body {\n \t\tprint \"<th></th>\\n\" . # for links\n \t\t      \"</tr>\\n\";\n \t}\n+\t@projects = @projects[$from..$to];\n+\t@projects = fill_project_list_info(\\@projects, $check_forks);\n \tmy $alternate = 1;\n-\tfor (my $i = $from; $i <= $to; $i++) {\n-\t\tmy $pr = $projects[$i];\n+\tforeach my $pr (@projects) {\n \t\tif ($alternate) {\n \t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n@@ -4066,6 +4067,46 @@ sub git_project_list {\n \t\tdie_error(undef, \"No projects found\");\n \t}\n \n+\tmy $page_size = 100;\n+\t# +1 for incomplete page (not having $page_size items)\n+\tmy $npages = int(@list / $page_size)+((@list % $page_size) > 0);\n+\t# pages are numbered 0..$npages-1\n+\t$page = 0 if (!defined $page || $page < 0);\n+\t$page = $npages-1 if ($page >= $npages);\n+\tmy $paging_nav;\n+\n+\tif ($npages > 1) {\n+\t\tmy @paging = ();\n+\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>0)},\n+\t\t                      \"first\");\n+\t\tif ($page > 0) {\n+\t\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>$page-1),\n+\t\t\t                       -accesskey => \"p\", -title => \"Alt-p\"},\n+\t\t\t                      \"prev\");\n+\t\t} else {\n+\t\t\tpush @paging, \"prev\";\n+\t\t}\n+\t\tfor (my $pg = 0; $pg < $npages; $pg++) {\n+\t\t\t# links to individual pages are numbered 1..$npages\n+\t\t\tif ($page == $pg) {\n+\t\t\t\tpush @paging, '<b>'.($pg+1).'</b>';\n+\t\t\t} else {\n+\t\t\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>$pg)},\n+\t\t\t\t                      $pg+1);\n+\t\t\t}\n+\t\t}\n+\t\tif ($page < $npages-1) {\n+\t\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>$page+1),\n+\t\t\t                       -accesskey => \"n\", -title => \"Alt-n\"},\n+\t\t\t                      \"next\");\n+\t\t} else {\n+\t\t\tpush @paging, \"next\";\n+\t\t}\n+\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>$npages-1)},\n+\t\t                      \"last\");\n+\t\t$paging_nav = join(' &sdot; ', @paging);\n+\t}\n+\n \tgit_header_html();\n \tif (-f $home_text) {\n \t\tprint \"<div class=\\\"index_include\\\">\\n\";\n@@ -4081,10 +4122,14 @@ sub git_project_list {\n \t\tif (defined $searchtype) {\n \t\t\tfilter_project_list(\\@list, $searchtype, $search_regexp)\n \t\t\t\tunless ($searchtype eq 'list_all');\n-\t\t\tgit_project_list_body(\\@list, $order);\n+\t\t\tgit_project_list_body(\\@list, $order,\n+\t\t\t                      $page*$page_size, ($page+1)*$page_size - 1,\n+\t\t\t                      $paging_nav);\n \t\t}\n \t} else {\n-\t\tgit_project_list_body(\\@list, $order);\n+\t\tgit_project_list_body(\\@list, $order,\n+\t\t                      $page*$page_size, ($page+1)*$page_size - 1,\n+\t\t                      $paging_nav);\n \t}\n \tgit_footer_html();\n }\n-- \n1.5.5\n"},{"id":"76533","messageId":"1210444108.11526.139.camel@localhost.localdomain","threadId":"13341","inReplyTo":"200805101128.59313.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"J.H.","fromEmail":"warthog19@eaglescrag.net","sentAt":"2008-05-10T18:28:28Z","receivedAt":"2008-05-10T18:28:28Z","isPatch":true,"sender":{"key":"warthog19@eaglescrag.net","avatar":null},"body":"$page_size should be moved to a configuration directive, with a possible\ndefault of 100, and there should be an option to turn it off completely\n(I.E. kernel.org will likely have it turned off).\n\nIt might also be advantageous to generate the entire list and stash that\nsomewhere, (session on the host side?) and run the pagination out of\nthat.  Would even allow the user to choose how many items they wish to\nsee from a drop down maybe?\n\n- John\n\nOn Sat, 2008-05-10 at 11:28 +0200, Jakub Narebski wrote:\n> Divide projects list and project search results into pages of\n> $page_size (currently 100) elements.  Pagination links look\n> like shown below:\n> \n>   first prev 1 2 3 *4* 5 6 next last\n> \n> where *4* denotes current page, is in bold, and isn't a link.\n> This would need to be improved for extremely large number of projects\n> (repositories), where number of projects divided by page size is very\n> large.\n> \n> Currently pagination interacts with sorting in a bit strange way.\n> When clicking on \"sort by\" link when on some later page the result is\n> not this page sorted, by given page of sorted output.  Perhaps \"sort\n> by\" links should always show first page...\n> \n> git_project_list_body() subroutine takes care to fill field (key) we\n> sort by for all projects, sort by given ordering, and fill the rest of\n> project info only for projects shown.\n> \n> NOTE: currently code is not optimal, as it contains a few unnecessary\n> array copying, or array fragment copying.\n> \n> Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n> ---\n> This patch depends on earlier \"gitweb: Project search\", but conceptually\n> they are independent; probably fill_project_list_info() improvement to\n> do partial filling should be in seaparet commit...\n> \n>  gitweb/gitweb.perl |   57 ++++++++++++++++++++++++++++++++++++++++++++++-----\n>  1 files changed, 51 insertions(+), 6 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 7e1a9b4..716f7ad 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -3709,10 +3709,10 @@ sub print_sort_th_num {\n>  sub git_project_list_body {\n>  \tmy ($projlist, $order, $from, $to, $extra, $no_header) = @_;\n>  \n> +\t$order ||= $default_projects_order;\n>  \tmy ($check_forks) = gitweb_check_feature('forks');\n> -\tmy @projects = fill_project_list_info($projlist, $check_forks);\n> +\tmy @projects = fill_project_list_info($projlist, $check_forks, $order);\n>  \n> -\t$order ||= $default_projects_order;\n>  \t$from = 0 unless defined $from;\n>  \t$to = $#projects if (!defined $to || $#projects < $to);\n>  \n> @@ -3733,9 +3733,10 @@ sub git_project_list_body {\n>  \t\tprint \"<th></th>\\n\" . # for links\n>  \t\t      \"</tr>\\n\";\n>  \t}\n> +\t@projects = @projects[$from..$to];\n> +\t@projects = fill_project_list_info(\\@projects, $check_forks);\n>  \tmy $alternate = 1;\n> -\tfor (my $i = $from; $i <= $to; $i++) {\n> -\t\tmy $pr = $projects[$i];\n> +\tforeach my $pr (@projects) {\n>  \t\tif ($alternate) {\n>  \t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n>  \t\t} else {\n> @@ -4066,6 +4067,46 @@ sub git_project_list {\n>  \t\tdie_error(undef, \"No projects found\");\n>  \t}\n>  \n> +\tmy $page_size = 100;\n> +\t# +1 for incomplete page (not having $page_size items)\n> +\tmy $npages = int(@list / $page_size)+((@list % $page_size) > 0);\n> +\t# pages are numbered 0..$npages-1\n> +\t$page = 0 if (!defined $page || $page < 0);\n> +\t$page = $npages-1 if ($page >= $npages);\n> +\tmy $paging_nav;\n> +\n> +\tif ($npages > 1) {\n> +\t\tmy @paging = ();\n> +\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>0)},\n> +\t\t                      \"first\");\n> +\t\tif ($page > 0) {\n> +\t\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>$page-1),\n> +\t\t\t                       -accesskey => \"p\", -title => \"Alt-p\"},\n> +\t\t\t                      \"prev\");\n> +\t\t} else {\n> +\t\t\tpush @paging, \"prev\";\n> +\t\t}\n> +\t\tfor (my $pg = 0; $pg < $npages; $pg++) {\n> +\t\t\t# links to individual pages are numbered 1..$npages\n> +\t\t\tif ($page == $pg) {\n> +\t\t\t\tpush @paging, '<b>'.($pg+1).'</b>';\n> +\t\t\t} else {\n> +\t\t\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>$pg)},\n> +\t\t\t\t                      $pg+1);\n> +\t\t\t}\n> +\t\t}\n> +\t\tif ($page < $npages-1) {\n> +\t\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>$page+1),\n> +\t\t\t                       -accesskey => \"n\", -title => \"Alt-n\"},\n> +\t\t\t                      \"next\");\n> +\t\t} else {\n> +\t\t\tpush @paging, \"next\";\n> +\t\t}\n> +\t\tpush @paging, $cgi->a({-href => href(-replay=>1, page=>$npages-1)},\n> +\t\t                      \"last\");\n> +\t\t$paging_nav = join(' &sdot; ', @paging);\n> +\t}\n> +\n>  \tgit_header_html();\n>  \tif (-f $home_text) {\n>  \t\tprint \"<div class=\\\"index_include\\\">\\n\";\n> @@ -4081,10 +4122,14 @@ sub git_project_list {\n>  \t\tif (defined $searchtype) {\n>  \t\t\tfilter_project_list(\\@list, $searchtype, $search_regexp)\n>  \t\t\t\tunless ($searchtype eq 'list_all');\n> -\t\t\tgit_project_list_body(\\@list, $order);\n> +\t\t\tgit_project_list_body(\\@list, $order,\n> +\t\t\t                      $page*$page_size, ($page+1)*$page_size - 1,\n> +\t\t\t                      $paging_nav);\n>  \t\t}\n>  \t} else {\n> -\t\tgit_project_list_body(\\@list, $order);\n> +\t\tgit_project_list_body(\\@list, $order,\n> +\t\t                      $page*$page_size, ($page+1)*$page_size - 1,\n> +\t\t                      $paging_nav);\n>  \t}\n>  \tgit_footer_html();\n>  }\n"},{"id":"76552","messageId":"200805110032.39729.jnareb@gmail.com","threadId":"13341","inReplyTo":"1210444108.11526.139.camel@localhost.localdomain","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-10T22:32:37Z","receivedAt":"2008-05-10T22:32:37Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sun, 10 May 2008, J.H. <warthog19@eaglescrag.net> wrote:\n\n> $page_size should be moved to a configuration directive, with a possible\n> default of 100, and there should be an option to turn it off completely\n> (I.E. kernel.org will likely have it turned off).\n\nIf it would be configurable (and I agree that it is probably a good\nidea), it should be per gitweb instance (i.e. global variable), and\nshould perhaps be named $project_list_page_size to distinguish it from\nnumber of items shown (page size) for 'shortlog', 'log' and 'history'\nviews.\n\nI guess that value of 0, or perhaps of undef would mean to not divide\nprojects list, or project search results, into pages.\n\nBTW. why do you think that kernel.org would be better to have project\nlist pagination turned off?\n \n> It might also be advantageous to generate the entire list and stash that\n> somewhere, (session on the host side?) and run the pagination out of\n> that.\n\nFirst, when using $projectslist file with new (second patch in series,\n\"gitweb: Allow project description in project_index file\" most of data\n(well, all except age) would be filled by parsing single file.\n\nSecond, the idea is to cache results of filled in @$projlist e.g. using\nStorable, i.e. cache Perl data and not final HTML output.\n\n> Would even allow the user to choose how many items they wish to \n> see from a drop down maybe?\n\nI think it is also a good idea, at least for search results.\n\n\nNOTE that this was an RFC patch, meant mainly to provide comments on\nwhat to improve...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"76583","messageId":"1210485210.11526.145.camel@localhost.localdomain","threadId":"13341","inReplyTo":"200805110032.39729.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"J.H.","fromEmail":"warthog19@eaglescrag.net","sentAt":"2008-05-11T05:53:30Z","receivedAt":"2008-05-11T05:53:30Z","isPatch":true,"sender":{"key":"warthog19@eaglescrag.net","avatar":null},"body":"\nOn Sun, 2008-05-11 at 00:32 +0200, Jakub Narebski wrote:\n> On Sun, 10 May 2008, J.H. <warthog19@eaglescrag.net> wrote:\n> \n> > $page_size should be moved to a configuration directive, with a possible\n> > default of 100, and there should be an option to turn it off completely\n> > (I.E. kernel.org will likely have it turned off).\n> \n> If it would be configurable (and I agree that it is probably a good\n> idea), it should be per gitweb instance (i.e. global variable), and\n> should perhaps be named $project_list_page_size to distinguish it from\n> number of items shown (page size) for 'shortlog', 'log' and 'history'\n> views.\n> \n> I guess that value of 0, or perhaps of undef would mean to not divide\n> projects list, or project search results, into pages.\n> \n> BTW. why do you think that kernel.org would be better to have project\n> list pagination turned off?\n\nIf we have caching on I'd rather leave it as a single page, much easier\nto search on and with firefox /<project or name> then <ctrl>+g a couple\nof times is really fast to find what I need.  Also means less to cache\n(good for us) and faster access for most.  I'm going to guess leaving it\nas a single page, unless it gets *very* unwieldy, will be how we leave\nit.\n\n>  \n> > It might also be advantageous to generate the entire list and stash that\n> > somewhere, (session on the host side?) and run the pagination out of\n> > that.\n> \n> First, when using $projectslist file with new (second patch in series,\n> \"gitweb: Allow project description in project_index file\" most of data\n> (well, all except age) would be filled by parsing single file.\n> \n> Second, the idea is to cache results of filled in @$projlist e.g. using\n> Storable, i.e. cache Perl data and not final HTML output.\n\nI approve of that plan, caching all the html is kinda expensive *hides\nthe 20G of gitweb cache he has*\n\n> \n> > Would even allow the user to choose how many items they wish to \n> > see from a drop down maybe?\n> \n> I think it is also a good idea, at least for search results.\n> \n> \n> NOTE that this was an RFC patch, meant mainly to provide comments on\n> what to improve...\n\nI know, that's why I'm chiming in with my thoughts, I like the idea and\nbonus points go to the user configurable display on that.  There are\ndefinitely sites that can benefit from this and likely at some point\nk.org is going to need have to paginate - just not sure when that point\nis going to get hit.\n\n- John 'Warthog9' Hawley\n"},{"id":"76667","messageId":"200805120151.20741.jnareb@gmail.com","threadId":"13341","inReplyTo":"1210485210.11526.145.camel@localhost.localdomain","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-11T23:51:19Z","receivedAt":"2008-05-11T23:51:19Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sun, 11 May 2008, J.H. wrote:\n> \n> On Sun, 2008-05-11 at 00:32 +0200, Jakub Narebski wrote:\n>> On Sun, 10 May 2008, J.H. <warthog19@eaglescrag.net> wrote:\n>>\n>> BTW. why do you think that kernel.org would be better to have project\n>> list pagination turned off?\n> \n> If we have caching on I'd rather leave it as a single page, much easier\n> to search on and with firefox /<project or name> then <ctrl>+g a couple\n> of times is really fast to find what I need.  Also means less to cache\n> (good for us) and faster access for most.  I'm going to guess leaving it\n> as a single page, unless it gets *very* unwieldy, will be how we leave\n> it.\n\nFirst, you can cache individual pages of projects list output with\nsmall overhead if number of projects per page is large enough.  Second,\nsee my earlier patch adding project search capability to gitweb\n(it is server-side search, though).  Searching needs caching data, not\noutput, while pagination only can be done with both.\n\n\nAs to single-page projects list being being unwieldy...  You should\ntake into account how large the page is (having page size in megabytes\nisn't too good), how much time it takes to download (and not only\ngenerate), and how much time it takes for web browser to render it.\n\nHow many repositories is \"too many\", that I'm not sure about...\n\n>>> It might also be advantageous to generate the entire list and stash that\n>>> somewhere, (session on the host side?) and run the pagination out of\n>>> that.\n>> \n>> First, when using $projectslist file with new (second patch in series,\n>> \"gitweb: Allow project description in project_index file\" most of data\n>> (well, all except age) would be filled by parsing single file.\n>> \n>> Second, the idea is to cache results of filled in @$projlist e.g. using\n>> Storable, i.e. cache Perl data and not final HTML output.\n> \n> I approve of that plan, caching all the html is kinda expensive *hides\n> the 20G of gitweb cache he has*\n\nCaching output (HTML) has the advantage of being simple, and avoiding\ntime to generate page.  Caching data has the advantage of taking less\ndisk space due to being able to reuse the same data for different pages.\n \n-- \nJakub Narebski\nPoland\n"},{"id":"76681","messageId":"200805120903.25040.jnareb@gmail.com","threadId":"13341","inReplyTo":"8c5c35580805102356p7e5532aah319af921f9b19392@mail.gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-12T07:03:24Z","receivedAt":"2008-05-12T07:03:24Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[The original email by Lars didn't get to git mailing list because of\n lack of quotes around J.H. in \"J.H.\" <warthog19@eaglescrag.net>\n email address in Cc:]\n\nDnia niedziela 11. maja 2008 08:56, Lars Hjemli napisał:\n> On 5/11/08, J.H. <warthog19@eaglescrag.net> wrote:\n>>  On Sun, 2008-05-11 at 00:32 +0200, Jakub Narebski wrote:\n>>>\n>>> First, when using $projectslist file with new (second patch in series,\n>>> \"gitweb: Allow project description in project_index file\" most of data\n>>> (well, all except age) would be filled by parsing single file.\n>>>\n>>> Second, the idea is to cache results of filled in @$projlist e.g. using\n>>> Storable, i.e. cache Perl data and not final HTML output.\n>>\n>> I approve of that plan, caching all the html is kinda expensive *hides\n>>  the 20G of gitweb cache he has*\n> \n> It seems to me that \"projectlist in a single file\" and \"cache results\n> of filled in @$projlist\" are different solutions to the same problem:\n> rapidly filling a perl datastructure.\n\nWell, yes and no.  \"Projectlist in single file\" is about _static_ data\n(which changes only if projects are added, deleted, its description\nchanged; those are usually rare events), and avoiding mainly I/O and\nnot CPU (scanning filesystem for repositories, reading config and\ndescription, etc.).\n\n\"Cache data\" is about caching _variable_ data, such as \"Last changed\"\ninformation for project.  Caching data instead of caching output\n(caching HTML) allows to share cache for different presentation of\nthe very same data (e.g. 'history'/'shortlog' vs 'rss').  And for some\npages, like project search results, caching HTML output doesn't make\nmuch sense, while caching data has it.\n\n> In cgit I've chosen \"projectlist in a single file\" and \"cache html\n> output\". This makes it cheap (in terms of cpu and io) to both generate\n> and serve the cached page (and the cache works for all pages).\n\nAs I said, for some pages, like for search results, caching output\ndoesn't make sense, while caching data has.\n\n> This used to be expensive in terms of cache size (similar to k.orgs\n> 20G), but current cgit solves this by treating the cache as a hash\n> table; cgitrc has an option to set the cache size (number of files),\n> each filename is generated as `hash(url) % cachesize` and each file\n> contains the full url (to detect hash collisions) followed by the\n> cached content for that url (see\n> http://hjemli.net/git/cgit/tree/cache.c for the details).\n\nI guess that is the simplest solution, but I don't think that is\nthe best solution to have size-limited cache.  For example CPAN Perl\nmodule Cache::SizeAwareCache and its derivatives use the following\nalgorithm\n\n  The default cache size limiting algorithm works by removing cache\n  objects in the following order until the desired limit is reached:\n\n    1) objects that have expired\n    2) objects that are least recently accessed\n    3) objects that that expire next\n\n\nBTW. if majority of your clients support transparent compression\n(J.H., could you check it for kernel.org; Pasky, could you check\nit for repo.or.cz?) then you can reduce cache size by storing pages\ncompressed.\n\n> Btw: gitweb and cgit seems to aquire the same features these days:\n> cgit recently got pagination + search on the project list.\n\nI haven't checked what features cgit has lately...\n\nGitweb development seems a bit stalled; I got no response to latest\nturn od gitweb TODO and wishlist list...\n-- \nJakub Narebski\nPoland\n"},{"id":"76718","messageId":"8c5c35580805120843j9b401f8mfa104806880a51c2@mail.gmail.com","threadId":"13341","inReplyTo":"200805120903.25040.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2008-05-12T15:43:06Z","receivedAt":"2008-05-12T15:43:06Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On 5/12/08, Jakub Narebski <jnareb@gmail.com> wrote:\n> [The original email by Lars didn't get to git mailing list because of\n>   lack of quotes around J.H. in \"J.H.\" <warthog19@eaglescrag.net>\n>   email address in Cc:]\n\nGaah, bad gmail...\n\n\n>  Dnia niedziela 11. maja 2008 08:56, Lars Hjemli napisał:\n>\n>  > It seems to me that \"projectlist in a single file\" and \"cache results\n>  > of filled in @$projlist\" are different solutions to the same problem:\n>  > rapidly filling a perl datastructure.\n>\n> Well, yes and no.  \"Projectlist in single file\" is about _static_ data\n>  (which changes only if projects are added, deleted, its description\n>  changed; those are usually rare events), and avoiding mainly I/O and\n>  not CPU (scanning filesystem for repositories, reading config and\n>  description, etc.).\n>\n>  \"Cache data\" is about caching _variable_ data, such as \"Last changed\"\n>  information for project.  Caching data instead of caching output\n>  (caching HTML) allows to share cache for different presentation of\n>  the very same data (e.g. 'history'/'shortlog' vs 'rss').  And for some\n>  pages, like project search results, caching HTML output doesn't make\n>  much sense, while caching data has it.\n\nWhile I agree that caching search result output almost never makes\nsense, I think it's more important that cache hits requires minimal\nprocessing. This is why I've chosen to cache the final result instead\nof an intermediate state, but both solutions obviously got some pros\nand cons.\n\n>  > This used to be expensive in terms of cache size (similar to k.orgs\n>  > 20G), but current cgit solves this by treating the cache as a hash\n>  > table; cgitrc has an option to set the cache size (number of files),\n>  > each filename is generated as `hash(url) % cachesize` and each file\n>  > contains the full url (to detect hash collisions) followed by the\n>  > cached content for that url (see\n>  > http://hjemli.net/git/cgit/tree/cache.c for the details).\n>\n>\n> I guess that is the simplest solution, but I don't think that is\n>  the best solution to have size-limited cache.  For example CPAN Perl\n>  module Cache::SizeAwareCache and its derivatives use the following\n>  algorithm\n>\n>   The default cache size limiting algorithm works by removing cache\n>   objects in the following order until the desired limit is reached:\n>\n>     1) objects that have expired\n>     2) objects that are least recently accessed\n>     3) objects that that expire next\n\nAgain, minimal processing is the goal of cgits cache implementation,\nhence the simple solution.\n\n>  > Btw: gitweb and cgit seems to aquire the same features these days:\n>  > cgit recently got pagination + search on the project list.\n>\n>\n> I haven't checked what features cgit has lately...\n>\n>  Gitweb development seems a bit stalled; I got no response to latest\n>  turn od gitweb TODO and wishlist list...\n\nWell, I for one found the wishlist interesting; I've been pondering on\nimplementing a graphic log in cgit (inspired by git-forest and\ngit-graph), but I refuse to perform a  topo-sort ;-)\n\nHopefully I can exploit the fact that cgit never uses more than one\ncommit as starting point for log traversal, combined with heuristics\non commit date, to enable a fast graphic log that will be correct for\nall but the most pathological cases.\n\n--\nlarsh\n"},{"id":"76838","messageId":"200805130855.08133.jnareb@gmail.com","threadId":"13341","inReplyTo":"8c5c35580805120843j9b401f8mfa104806880a51c2@mail.gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-13T06:55:07Z","receivedAt":"2008-05-13T06:55:07Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dnia poniedziałek 12. maja 2008 17:43, Lars Hjemli napisał:\n> On 5/12/08, Jakub Narebski <jnareb@gmail.com> wrote:\n>>  Dnia niedziela 11. maja 2008 08:56, Lars Hjemli napisał:\n>>\n>>> It seems to me that \"projectlist in a single file\" and \"cache results\n>>> of filled in @$projlist\" are different solutions to the same problem:\n>>> rapidly filling a perl datastructure.\n>>\n>> Well, yes and no.  \"Projectlist in single file\" is about _static_ data\n>> (which changes only if projects are added, deleted, its description\n>> changed; those are usually rare events), and avoiding mainly I/O and\n>> not CPU (scanning filesystem for repositories, reading config and\n>> description, etc.).\n>>\n>> \"Cache data\" is about caching _variable_ data, such as \"Last changed\"\n>> information for project.  Caching data instead of caching output\n>> (caching HTML) allows to share cache for different presentation of\n>> the very same data (e.g. 'history'/'shortlog' vs 'rss').  And for some\n>> pages, like project search results, caching HTML output doesn't make\n>> much sense, while caching data has it.\n> \n> While I agree that caching search result output almost never makes\n> sense, I think it's more important that cache hits requires minimal\n> processing. This is why I've chosen to cache the final result instead\n> of an intermediate state, but both solutions obviously got some pros\n> and cons.\n\nTrue.  In most cases caching final output is enough.  Only in some\ncases caching data is better solution.  I hope that \"Gitweb caching\"\nGit's Google Summer of Code 2008 project would examine this in more\ndetail.\n\nBut please take into account that gitweb performance, and I guess any\ngit web interface performance, is I/O bound and not CPU bound (at least\naccording to what I remember from J.H. emails).  So a little more\nprocessing is I think less important than avoiding hitting the repos.\n\n\nJ.H. (kernel.org) gitweb from what I remember does adaptive caching\nof HTML output, while Pasky (repo.or.cz) gitweb does data caching only\nfor projects list page.\n\n>>> This used to be expensive in terms of cache size (similar to k.orgs\n>>> 20G), but current cgit solves this by treating the cache as a hash\n>>> table; cgitrc has an option to set the cache size (number of files),\n>>> each filename is generated as `hash(url) % cachesize` and each file\n>>> contains the full url (to detect hash collisions) followed by the\n>>> cached content for that url (see\n>>> http://hjemli.net/git/cgit/tree/cache.c for the details).\n>>\n>>\n>> I guess that is the simplest solution, but I don't think that is\n>> the best solution to have size-limited cache.  For example CPAN Perl\n>> module Cache::SizeAwareCache and its derivatives use the following\n>> algorithm\n>>\n>>   The default cache size limiting algorithm works by removing cache\n>>   objects in the following order until the desired limit is reached:\n>>\n>>     1) objects that have expired\n>>     2) objects that are least recently accessed\n>>     3) objects that that expire next\n> \n> Again, minimal processing is the goal of cgits cache implementation,\n> hence the simple solution.\n\nI would really like if some comp-sci could calculate amortized cost\nof this solution, and what I think is more important, cost of worst\ncase and what is the probability of hitting worst case or next to\nworst case.\n\nBy the way, you have to take into account the time it takes to\ncalculate hash when comparing performance.  Note that for LRU cache\nyou can use heap / priority queue, or splice / self organizing binary\ntree.\n\n>>> Btw: gitweb and cgit seems to aquire the same features these days:\n>>> cgit recently got pagination + search on the project list.\n>>\n>> I haven't checked what features cgit has lately...\n>>\n>> Gitweb development seems a bit stalled; I got no response to latest\n>> turn od gitweb TODO and wishlist list...\n\n...so you would have to turn for example to git-php, gitorious and\ngithub for inspiration.\n\n> Well, I for one found the wishlist interesting; I've been pondering on\n> implementing a graphic log in cgit (inspired by git-forest and\n> git-graph), but I refuse to perform a  topo-sort ;-)\n> \n> Hopefully I can exploit the fact that cgit never uses more than one\n> commit as starting point for log traversal, combined with heuristics\n> on commit date, to enable a fast graphic log that will be correct for\n> all but the most pathological cases.\n\nI think if you wait for graphing API to make it into released version,\nyou (well, cgit) would be able to use it for \"fast graphic log\".\n\n-- \nJakub Narebski\nPoland\n"},{"id":"76875","messageId":"8c5c35580805130946m691b9389sad20d1949cbc896c@mail.gmail.com","threadId":"13341","inReplyTo":"8c5c35580805130939m1a1ef8e0yd72402f3c79190ea@mail.gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2008-05-13T16:46:06Z","receivedAt":"2008-05-13T16:46:06Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On Tue, May 13, 2008 at 6:39 PM, Lars Hjemli <hjemli@gmail.com> wrote:\n>  $ time git-rev-list -n 100 master >/dev/null\n>\n>  real    0m0.006s\n>  user    0m0.008s\n>  sys     0m0.000s\n>\n>  $ time git-rev-list -n 100 --graph master >/dev/null\n>\n>  real    0m1.670s\n>  user    0m1.636s\n>  sys     0m0.032s\n\nI forgot to mention: That's on a recent linux-2.6.git, using current 'next'.\n\n-- \nlarsh\n"},{"id":"76876","messageId":"200805131904.38317.jnareb@gmail.com","threadId":"13341","inReplyTo":"8c5c35580805130939m1a1ef8e0yd72402f3c79190ea@mail.gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-13T17:04:37Z","receivedAt":"2008-05-13T17:04:37Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 13 May 2008, Lars Hjemli wrote:\n> On Tue, May 13, 2008 at 8:55 AM, Jakub Narebski <jnareb@gmail.com> wrote:\n>>\n>> But please take into account that gitweb performance, and I guess any\n>> git web interface performance, is I/O bound and not CPU bound (at least\n>> according to what I remember from J.H. emails).  So a little more\n>> processing is I think less important than avoiding hitting the repos.\n> \n> Yeah, that's a pretty convincing argument for \"cache the data, not the\n> output\", at least for gitweb (cgit never touches the repos to generate\n> the project list/search).\n\nFirst, this is less argument for \"cache data\" against \"cache output\",\nonly against disregarding \"cache data\" for assumed performance reasons,\nwhile it \"cache data\" can be useful in some places.\n\n\nSecond, gitweb's projects list page contains \"Last Changed\" column,\nand you _*have*_ to hit repositories for this data, and backwards\ncompatibility prevents from removing that.\n\nYou can configure gitweb in such way that project name, owner, and\nproject description are read from single 'project_index' file with \n  \"gitweb: Allow project description in project_index file\"\npatch.  repo.or.cz can do this.\n\nOn the other hand mechanism to scan directory, getting owner from\nfilesystem owner (plus GECOS) or repository's gitweb.owner configuration,\nand project description from 'description' file in projects repository,\nallow to treat gitweb in similar way to mod_userdir i.e. as service.\nYou don't need to register project with gitweb; you need only to create\nit in appropriate place (or put symbolic link there).  kernel.org\nfunctions this way.\n\nI wonder how Freedesktop.org does it: it offers both gitweb and cgit\ninterfaces, if I remember correctly...\n-- \nJakub Narebski\nPoland\n"},{"id":"76879","messageId":"1210705862.1313.9.camel@gaara.bos.redhat.com","threadId":"13341","inReplyTo":"200805131904.38317.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Kristian Høgsberg","fromEmail":"krh@redhat.com","sentAt":"2008-05-13T19:11:02Z","receivedAt":"2008-05-13T19:11:02Z","isPatch":true,"sender":{"key":"krh@redhat.com","avatar":"https://gravatar.com/avatar/763dee6f9594ac474f725b137a39565792928e583ddf59b32befc2907409027e?d=mp&s=160"},"body":"On Tue, 2008-05-13 at 19:04 +0200, Jakub Narebski wrote:\n\n> I wonder how Freedesktop.org does it: it offers both gitweb and cgit\n> interfaces, if I remember correctly...\n\nI have a cron script in my homedir that scans the places we have repos\nand generates the list of repos.  This generates a list of repo\ndescription in the cgit config file format, which is included in the\ncgit config file main.  It runs every hour or so.\n\nKristian\n"},{"id":"76880","messageId":"8c5c35580805131230p37953e33he97803c0609012fa@mail.gmail.com","threadId":"13341","inReplyTo":"200805131904.38317.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2008-05-13T19:30:45Z","receivedAt":"2008-05-13T19:30:45Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On Tue, May 13, 2008 at 7:04 PM, Jakub Narebski <jnareb@gmail.com> wrote:\n> On Tue, 13 May 2008, Lars Hjemli wrote:\n>  > On Tue, May 13, 2008 at 8:55 AM, Jakub Narebski <jnareb@gmail.com> wrote:\n>  >>\n> >> But please take into account that gitweb performance, and I guess any\n>  >> git web interface performance, is I/O bound and not CPU bound (at least\n>  >> according to what I remember from J.H. emails).  So a little more\n>  >> processing is I think less important than avoiding hitting the repos.\n>  >\n>  > Yeah, that's a pretty convincing argument for \"cache the data, not the\n>  > output\", at least for gitweb (cgit never touches the repos to generate\n>  > the project list/search).\n>\n>  First, this is less argument for \"cache data\" against \"cache output\",\n>  only against disregarding \"cache data\" for assumed performance reasons,\n>  while it \"cache data\" can be useful in some places.\n\nOk, then I misinterpreted your argument.\n\n\n>  Second, gitweb's projects list page contains \"Last Changed\" column,\n>  and you _*have*_ to hit repositories for this data\n\nNo you don't. One alternative is to use the post-update hook in each\nrepo to update a separate file with info about last-changed-time.\nAnother (less accurate) alternative is to stat one or more of\npacked-refs and refs/heads/*; cgit uses both of these alternatives to\navoid hitting the repo (i.e. object-db) when the project list/search\npage is generated.\n\n--\nlarsh\n"},{"id":"76895","messageId":"200805140128.40853.jnareb@gmail.com","threadId":"13341","inReplyTo":"8c5c35580805131230p37953e33he97803c0609012fa@mail.gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-13T23:28:39Z","receivedAt":"2008-05-13T23:28:39Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 13 May 2008, Lars Hjemli wrote:\n> On Tue, May 13, 2008 at 7:04 PM, Jakub Narebski <jnareb@gmail.com>\n> wrote: \n>> On Tue, 13 May 2008, Lars Hjemli wrote:\n>>>\n>>> [...] (cgit never touches the repos to generate\n>>> the project list/search).\n[...]\n>>  Second, gitweb's projects list page contains \"Last Changed\" column,\n>>  and you _*have*_ to hit repositories for this data\n> \n> No you don't. One alternative is to use the post-update hook in each\n> repo to update a separate file with info about last-changed-time.\n\nWell, adding setting / caching last-changed data using post-update hook\nwas one of my early patches to gitweb, but it got shot down. Mainly \nbecause the fact that it doesn't work with gitweb as \"usergit\" service,\nas you won't be able to install hooks into other users repositories, \nlike in the kernel.org case (it should/would work for repo.or.cz etc.).\nAnother issue is if you publish via web interface non-bare repo; then\nupdate is not the only way data arrives at repository (yes, I know that\npost-commit would solve this).\n\n> Another (less accurate) alternative is to stat one or more of\n> packed-refs and refs/heads/*; \n\nAnd now, in the times of detached HEAD, also HEAD.\n\nBTW. how this work with hierarchical branch names?\n\n>                               cgit uses both of these alternatives to \n> avoid hitting the repo (i.e. object-db) when the project list/search\n> page is generated.\n\nFirst is implicit caching of data, made easier because in this case you\ncan invalidate cache when it becomes invalid.  Second doesn't hit the \nobject-db, but it hits I/O (perhaps very slightly); and the second \nsolution is what we can pursue for gitweb in addition to caching data.\n\nAdditional complication is that currently there is no place for \ntimestamp in project_index format (backward compatibility, unnamed \ntuple not good design to storing structured/tabular data).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"76926","messageId":"200805140959.58263.jnareb@gmail.com","threadId":"13341","inReplyTo":"200805140128.40853.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Paginate project list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-14T07:59:54Z","receivedAt":"2008-05-14T07:59:54Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 14 May 2008, Jakub Narebski wrote:\n> On Tue, 13 May 2008, Lars Hjemli wrote:\n\n>> Another (less accurate) alternative is to stat one or more of\n>> packed-refs and refs/heads/*; cgit uses both of these alternatives to \n>> avoid hitting the repo (i.e. object-db) when the project list/search\n>> page is generated.\n> \n>                                              [It] doesn't hit the \n> object-db, but it hits I/O (perhaps very slightly); and the second \n> solution is what we can pursue for gitweb in addition to caching data.\n\nNOTE, NOTE, NOTE! This changes _semantics_ from \"Last Change\"\n(i.e. commit date of latest commit), to \"Last Update\" (when latest\nbatch of objects arrived at given repository).\n\n-- \nJakub Narebski\nPoland\n"}]}