Lubomir Rintel[RESEND] Pagination for gitweb
I tought something like this could be a starter for better handling long gitweb project lists (such as http://pkgs.fedoraproject.org/gitweb/).
Could anyone please take a look? Thanks!
threads / patch / 25068
patch, 2 partsgitweb: Make it possible to paginate projects
Subject: [PATCH 1/2] gitweb: Make it possible to paginate projects
Lubomir RintelLubomir RintelThis adds simple pagination (next and prev links), to project lists, analogous to what is done for commit history lists. --- gitweb/gitweb.perl | 26 ++++++++++++++++++++++++++ 1 files changed, 26 insertions(+), 0 deletions(-)
Show changes to gitweb/gitweb.perl +26 −0
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl index a85e2f6..8dc7f29 100755 --- a/gitweb/gitweb.perl +++ b/gitweb/gitweb.perl @@ -255,6 +255,9 @@ our %highlight_ext = ( map { $_ => 'xml' } qw(xhtml html htm), ); +# Set this to non-zero to enable project list pagination +our $projects_per_page = 0; + # You define site-wide feature defaults here; override them with # $GITWEB_CONFIG as necessary. our %feature = ( @@ -4613,9 +4616,19 @@ sub git_project_list_body { my @projects = fill_project_list_info($projlist, $check_forks); $order ||= $default_projects_order; + $page ||= 0; + if ($projects_per_page) { + $from = $page * $projects_per_page unless defined $from; + $to = $from + $projects_per_page - 1 unless defined $to; + } $from = 0 unless defined $from; $to = $#projects if (!defined $to || $#projects < $to); + my $prev_link = $cgi->a({-href => href(-replay=>1, page=>$page-1), + -accesskey => "p", -title => "Alt-p"}, "prev") if ($page > 0); + my $next_link = $cgi->a({-href => href(-replay=>1, page=>$page+1), + -accesskey => "n", -title => "Alt-n"}, "next") if ($#$projlist > $to); + my %order_info = ( project => { key => 'path', type => 'str' }, descr => { key => 'descr_long', type => 'str' }, @@ -4709,6 +4722,19 @@ sub git_project_list_body { print "<td colspan=\"5\">$extra</td>\n" . "</tr>\n"; } + + if ($prev_link or $next_link) { + print "<tr>\n"; + if ($check_forks) { + print "<td></td>\n"; + } + print "<td colspan=\"5\">"; + print $prev_link if $prev_link; + print " ⋅ " if $prev_link and $next_link; + print $next_link if $next_link; + print "</td>\n</tr>\n"; + } + print "</table>\n"; }
-- 1.7.2.1
Lubomir Rintel[PATCH 2/2] gitweb: Optimize paging when sorted by path
There's no need to get authors, description and last modification time of a project that's not being shown on a current page. We can only tell that in advance if the list is sorted by pathname. --- gitweb/gitweb.perl | 47 ++++++++++++++++++++++++++++++++++------------- 1 files changed, 34 insertions(+), 13 deletions(-)
Show changes to gitweb/gitweb.perl +34 −13
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl index 8dc7f29..a2e9a95 100755 --- a/gitweb/gitweb.perl +++ b/gitweb/gitweb.perl @@ -4608,12 +4608,30 @@ sub format_sort_th { return $sort_th; } +sub git_try_to_order { + my ($projects, $order) = @_; + + my %order_info = ( + project => { key => 'path', type => 'str' }, + descr => { key => 'descr_long', type => 'str' }, + owner => { key => 'owner', type => 'str' }, + age => { key => 'age', type => 'num' } + ); + my $oi = $order_info{$order}; + return undef unless exists $projects->[0]->{$oi->{'key'}}; + if ($oi->{'type'} eq 'str') { + @$projects = sort {$a->{$oi->{'key'}} cmp $b->{$oi->{'key'}}} @$projects; + } else { + @$projects = sort {$a->{$oi->{'key'}} <=> $b->{$oi->{'key'}}} @$projects; + } + return 1; +} + sub git_project_list_body { # actually uses global variable $project my ($projlist, $order, $from, $to, $extra, $no_header) = @_; my $check_forks = gitweb_check_feature('forks'); - my @projects = fill_project_list_info($projlist, $check_forks); $order ||= $default_projects_order; $page ||= 0; @@ -4622,26 +4640,29 @@ sub git_project_list_body { $to = $from + $projects_per_page - 1 unless defined $to; } $from = 0 unless defined $from; - $to = $#projects if (!defined $to || $#projects < $to); + $to = $#$projlist if (!defined $to || $#$projlist < $to); my $prev_link = $cgi->a({-href => href(-replay=>1, page=>$page-1), -accesskey => "p", -title => "Alt-p"}, "prev") if ($page > 0); my $next_link = $cgi->a({-href => href(-replay=>1, page=>$page+1), -accesskey => "n", -title => "Alt-n"}, "next") if ($#$projlist > $to); - my %order_info = ( - project => { key => 'path', type => 'str' }, - descr => { key => 'descr_long', type => 'str' }, - owner => { key => 'owner', type => 'str' }, - age => { key => 'age', type => 'num' } - ); - my $oi = $order_info{$order}; - if ($oi->{'type'} eq 'str') { - @projects = sort {$a->{$oi->{'key'}} cmp $b->{$oi->{'key'}}} @projects; - } else { - @projects = sort {$a->{$oi->{'key'}} <=> $b->{$oi->{'key'}}} @projects; + # If we're paginating and can order the list now (by pathname), we + # don't need to do an unnecessary and expensive query of the details + # of the projects we're not going to display. Attempt the sort and + # remove the other projects from the list if the sort is successful. + # Can't be used with ctags, since it needs a complete project list. + my $ordered = git_try_to_order($projlist, $order) + unless gitweb_check_feature('ctags'); + if ($ordered) { + @$projlist = @$projlist[$from..$to]; + $to -= $from; + $from = 0; } + my @projects = fill_project_list_info($projlist, $check_forks); + git_try_to_order(\@projects, $order) unless $ordered; + my $show_ctags = gitweb_check_feature('ctags'); if ($show_ctags) { my %ctags;
-- 1.7.2.1
Jakub NarebskiRe: [PATCH 2/2] gitweb: Optimize paging when sorted by path
Lubomir Rintel <lkundrak@v3.sk> writes:
> There's no need to get authors, description and last modification time > of a project that's not being shown on a current page. We can only tell > that in advance if the list is sorted by pathname.
More advanced version of this trick can be found in
[RFC/PATCH] gitweb: Paginate project list http://thread.gmane.org/gmane.comp.version-control.git/80896/focus=81613
from 2008-05-10. Perhaps it would be a good idea to merge these two patches?
Show 23 quoted lines
> ---
> gitweb/gitweb.perl | 47 ++++++++++++++++++++++++++++++++++-------------
> 1 files changed, 34 insertions(+), 13 deletions(-)
>
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 8dc7f29..a2e9a95 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -4608,12 +4608,30 @@ sub format_sort_th {
> return $sort_th;
> }
>
> +sub git_try_to_order {
> + my ($projects, $order) = @_;
> +
> + my %order_info = (
> + project => { key => 'path', type => 'str' },
> + descr => { key => 'descr_long', type => 'str' },
> + owner => { key => 'owner', type => 'str' },
> + age => { key => 'age', type => 'num' }
> + );
> + my $oi = $order_info{$order};
> + return undef unless exists $projects->[0]->{$oi->{'key'}};Nitpick: there is no need to use '->' between [0] and {$oi->{'key'}}.
BTW is this sanity check necessary?Show 7 quoted lines
> + if ($oi->{'type'} eq 'str') {
> + @$projects = sort {$a->{$oi->{'key'}} cmp $b->{$oi->{'key'}}} @$projects;
> + } else {
> + @$projects = sort {$a->{$oi->{'key'}} <=> $b->{$oi->{'key'}}} @$projects;
> + }
> + return 1;
> +}Note that separating this part of code into subroutine is a good idea anyway, even if the final patch would look different.
Show 55 quoted lines
> +
> sub git_project_list_body {
> # actually uses global variable $project
> my ($projlist, $order, $from, $to, $extra, $no_header) = @_;
>
> my $check_forks = gitweb_check_feature('forks');
> - my @projects = fill_project_list_info($projlist, $check_forks);
>
> $order ||= $default_projects_order;
> $page ||= 0;
> @@ -4622,26 +4640,29 @@ sub git_project_list_body {
> $to = $from + $projects_per_page - 1 unless defined $to;
> }
> $from = 0 unless defined $from;
> - $to = $#projects if (!defined $to || $#projects < $to);
> + $to = $#$projlist if (!defined $to || $#$projlist < $to);
>
> my $prev_link = $cgi->a({-href => href(-replay=>1, page=>$page-1),
> -accesskey => "p", -title => "Alt-p"}, "prev") if ($page > 0);
> my $next_link = $cgi->a({-href => href(-replay=>1, page=>$page+1),
> -accesskey => "n", -title => "Alt-n"}, "next") if ($#$projlist > $to);
>
> - my %order_info = (
> - project => { key => 'path', type => 'str' },
> - descr => { key => 'descr_long', type => 'str' },
> - owner => { key => 'owner', type => 'str' },
> - age => { key => 'age', type => 'num' }
> - );
> - my $oi = $order_info{$order};
> - if ($oi->{'type'} eq 'str') {
> - @projects = sort {$a->{$oi->{'key'}} cmp $b->{$oi->{'key'}}} @projects;
> - } else {
> - @projects = sort {$a->{$oi->{'key'}} <=> $b->{$oi->{'key'}}} @projects;
> + # If we're paginating and can order the list now (by pathname), we
> + # don't need to do an unnecessary and expensive query of the details
> + # of the projects we're not going to display. Attempt the sort and
> + # remove the other projects from the list if the sort is successful.
> + # Can't be used with ctags, since it needs a complete project list.
> + my $ordered = git_try_to_order($projlist, $order)
> + unless gitweb_check_feature('ctags');
> + if ($ordered) {
> + @$projlist = @$projlist[$from..$to];
> + $to -= $from;
> + $from = 0;
> }
>
> + my @projects = fill_project_list_info($projlist, $check_forks);
> + git_try_to_order(\@projects, $order) unless $ordered;
> +
> my $show_ctags = gitweb_check_feature('ctags');
> if ($show_ctags) {
> my %ctags;
> --
> 1.7.2.1
> -- Jakub Narebski Poland ShadeHawk on #git
Jakub NarebskiRe: [PATCH 1/2] gitweb: Make it possible to paginate projects
Lubomir Rintel <lkundrak@v3.sk> writes:
> This adds simple pagination (next and prev links), to project lists, > analogous to what is done for commit history lists.
Lack signoff (see Documentation/SubmittingPatches).
Show 15 quoted lines
> ---
> gitweb/gitweb.perl | 26 ++++++++++++++++++++++++++
> 1 files changed, 26 insertions(+), 0 deletions(-)
>
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index a85e2f6..8dc7f29 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -255,6 +255,9 @@ our %highlight_ext = (
> map { $_ => 'xml' } qw(xhtml html htm),
> );
>
> +# Set this to non-zero to enable project list pagination
> +our $projects_per_page = 0;
> +Good idea of being able to enable or disable pagination of projects list.
I am not sure though if this is the correct solution. First, pagination in all other places is hardcoded to 100 items per page; IMHO for consistency it would be good to use the same page size everywhere.
Second, perhaps instead of yet another global variable a better solution would be non-verridable %feature, like 'pathinfo' or 'forks' features?
Show 12 quoted lines
> # You define site-wide feature defaults here; override them with
> # $GITWEB_CONFIG as necessary.
> our %feature = (
> @@ -4613,9 +4616,19 @@ sub git_project_list_body {
> my @projects = fill_project_list_info($projlist, $check_forks);
>
> $order ||= $default_projects_order;
> + $page ||= 0;
> + if ($projects_per_page) {
> + $from = $page * $projects_per_page unless defined $from;
> + $to = $from + $projects_per_page - 1 unless defined $to;
> + }Hmmm...
Show 8 quoted lines
> $from = 0 unless defined $from;
> $to = $#projects if (!defined $to || $#projects < $to);
>
> + my $prev_link = $cgi->a({-href => href(-replay=>1, page=>$page-1),
> + -accesskey => "p", -title => "Alt-p"}, "prev") if ($page > 0);
> + my $next_link = $cgi->a({-href => href(-replay=>1, page=>$page+1),
> + -accesskey => "n", -title => "Alt-n"}, "next") if ($#$projlist > $to);
> +In other places we have 'first' ⋅ 'prev' ⋅ 'next'...
Show 19 quoted lines
> my %order_info = (
> project => { key => 'path', type => 'str' },
> descr => { key => 'descr_long', type => 'str' },
> @@ -4709,6 +4722,19 @@ sub git_project_list_body {
> print "<td colspan=\"5\">$extra</td>\n" .
> "</tr>\n";
> }
> +
> + if ($prev_link or $next_link) {
> + print "<tr>\n";
> + if ($check_forks) {
> + print "<td></td>\n";
> + }
> + print "<td colspan=\"5\">";
> + print $prev_link if $prev_link;
> + print " ⋅ " if $prev_link and $next_link;
> + print $next_link if $next_link;
> + print "</td>\n</tr>\n";
> + }... is there a reason to not use format_paging_nav() subroutine?
Show 7 quoted lines
> + > print "</table>\n"; > } > > -- > 1.7.2.1 >
See also comments to next patch in series
-- Jakub Narebski Poland ShadeHawk on #git
Jakub NarebskiRe: [RESEND] Pagination for gitweb
Lubomir Rintel <lkundrak@v3.sk> writes:
> I thought something like this could be a starter for better handling long > gitweb project lists (such as http://pkgs.fedoraproject.org/gitweb/). > > Could anyone please take a look?
What do you mean here by "better handling"?
Is the problem server performance for large number of projects? If this is the problem, perhaps better solution would be to use caching (work in progress).
Is the problem large projects-list page and bandwidth? There was a patch adding transparent compression of pages generated by gitweb would be a better solution; perhaps this together with caching (to avoid performance hit on CPU; note that usually gitweb performance is I/O and not CPU-bound).
Is the problem client rendering performance on large page with large table? If it is, then paginating output, or adding project search like in gitweb fork used on http://repo.or.cz is correct solution.
So which is it?
-- Jakub Narebski Poland ShadeHawk on #git
Re: [RESEND] Pagination for gitweb
On 09/10/2010 11:57 AM, Jakub Narebski wrote:
Show 12 quoted lines
> Lubomir Rintel <lkundrak@v3.sk> writes: > >> I thought something like this could be a starter for better handling long >> gitweb project lists (such as http://pkgs.fedoraproject.org/gitweb/). >> >> Could anyone please take a look? > > What do you mean here by "better handling"? > > Is the problem server performance for large number of projects? If > this is the problem, perhaps better solution would be to use caching > (work in progress).
They already moved to using my caching layer, mainly because I could create an RPM for them and the fact that my caching code is slightly more battle tested.
Show 12 quoted lines
> Is the problem large projects-list page and bandwidth? There was a > patch adding transparent compression of pages generated by gitweb > would be a better solution; perhaps this together with caching (to > avoid performance hit on CPU; note that usually gitweb performance is > I/O and not CPU-bound). > > Is the problem client rendering performance on large page with large > table? If it is, then paginating output, or adding project search > like in gitweb fork used on http://repo.or.cz is correct solution. > > > So which is it?
I think the issue is just having something like 10K projects all suddenly staring you in the face on a single page, it's not so much a technical problem with gitweb itself as a mental problem of dealing with a giant webpage like that.
So far the Fedora guys seem happy with the way things are (well except for the fact that their front page takes something like 2 hours to build, which has it's own set of problems).
Just some additional food for thought.
- John 'Warthog9' Hawley
Jakub NarebskiRe: [RESEND] Pagination for gitweb
On Fri, 10 Sep 2010, J.H. wrote:
Show 17 quoted lines
> On 09/10/2010 11:57 AM, Jakub Narebski wrote: > > Lubomir Rintel <lkundrak@v3.sk> writes: > > > > > I thought something like this could be a starter for better handling long > > > gitweb project lists (such as http://pkgs.fedoraproject.org/gitweb/). > > > > > > Could anyone please take a look? > > > > What do you mean here by "better handling"? > > > > Is the problem server performance for large number of projects? If > > this is the problem, perhaps better solution would be to use caching > > (work in progress). > > They already moved to using my caching layer, mainly because I could > create an RPM for them and the fact that my caching code is slightly > more battle tested.
Note that with project list pagination, and especially with project search feature, a better solution might be to cache *data* rather than HTML output, like in fork used (or used to be used) by http://repo.or.cz, or in Lea Wiemann's GSoC 2008 project [1]
[1] http://repo.or.cz/w/git/gitweb-caching.git
-- Jakub Narebski Poland
Jakub NarebskiRe: [RESEND] Pagination for gitweb
On Fri, 10 Sep 2010, J.H. wrote:
> On 09/10/2010 11:57 AM, Jakub Narebski wrote:
Show 7 quoted lines
> > Is the problem server performance for large number of projects? If > > this is the problem, perhaps better solution would be to use caching > > (work in progress). > > They already moved to using my caching layer, mainly because I could > create an RPM for them and the fact that my caching code is slightly > more battle tested.
Is this RPM site-specific, or generic one? Could you share *.spec file? Or do you use git.spec from git, just use patched gitweb?
-- Jakub Narebski Poland