git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 2/6] gitweb: Change the way "content tags" ('ctags') are handled

From
Jakub Narebski <jnareb@gmail.com>
Date
Apr 29, 2011, 17:51 UTC
Message-ID
<1304099521-27617-3-git-send-email-jnareb@gmail.com>
In-Reply-To
<1304099521-27617-1-git-send-email-jnareb@gmail.com>

The major change is removing the ability to edit content tags (ctags) in a web browser.

The interface was created by gitweb, while actual editing of tags was
to be done by external script; the API was not defined, and neither
was provided example implementation.  Such split is also a bit fragile
- interface and implementation have to be kept in sync.  Gitweb
provided only ability to add tags; you could not edit tags nor delete
them.

Format of ctags is now described in the comment above git_get_project_ctags subroutine. Gitweb now is more robust with respect to original ctags format; it also accepts two new formats: $GIT_DIR/ctags file, with one content tag per line, and multi-value `gitweb.ctag' config variable.

Gathering all ctags of all project is now put in git_gather_all_ctags subroutine, making git_project_list_body more clear.

git_populate_project_tagcloud subroutine now generates data used for tag cloud, including generation of ctag link, also in the case HTML::TagCloud module is unavailable. Links are now generated using href() subroutine - this is more robust, as ctags might contain '?', ';' and '=' special characters that need to be escaped in query param. Shown tags are HTML-escaped.

The generation of tag cloud in git_show_project_tagcloud in the case when HTML::TagCloud is not available is now changed slightly.

The 'content tags' field on project summary page is made more in line with other fields in "projects_list" table. Because one cannot now add new tags from web interface, this field is no longer displayed when there are no content tags for given project.

Ctags-issue-Reported-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Ctags-issue-Reported-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This patch was originally sent to git mailing list in
  [RFC/PATCH] gitweb: Change the way "content tags" ('ctags') are handled
  http://thread.gmane.org/gmane.linux.debian.devel.bugs.general/802865/focus=168599
as part of
  gitweb: cloud tags feature produces malformed XML for errors
  http://thread.gmane.org/gmane.linux.debian.devel.bugs.general/802865/focus=168266
thread, crossposted to git mailing list.

Hardening parsing of ctags files, so that gitweb does not crash on malformed entries, but just ignores them, sort of fixes original issue reported in

  Bug#616005: libhtml-tagcloud-perl breaks gitweb
  http://thread.gmane.org/gmane.linux.debian.devel.bugs.general/802865/
  http://bugs.debian.org/616005

by Uwe Kleine-König and crossposted to git mailing list by Jonathan Nieder.

I say "sort of fixes" because this patch doesn't attempt to solve deeper fundamental problem with error / exception handling in gitweb after some data was already sent to a browser.

Note also that the bug was caused by mishandling (incorrectly generating) underdocumented then 'ctags' feature.

 gitweb/gitweb.perl |  141 +++++++++++++++++++++++++++++++++++----------------
 1 files changed, 97 insertions(+), 44 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 014b33b..60cb772 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -412,20 +412,23 @@ our %feature = (
 		'override' => 0,
 		'default' => []},
 
-	# Allow gitweb scan project content tags described in ctags/
-	# of project repository, and display the popular Web 2.0-ish
-	# "tag cloud" near the project list. Note that this is something
-	# COMPLETELY different from the normal Git tags.
+	# Allow gitweb scan project content tags of project repository,
+	# and display the popular Web 2.0-ish "tag cloud" near the projects
+	# list.  Note that this is something COMPLETELY different from the
+	# normal Git tags.
 
 	# gitweb by itself can show existing tags, but it does not handle
-	# tagging itself; you need an external application for that.
-	# For an example script, check Girocco's cgi/tagproj.cgi.
+	# tagging itself; you need to do it externally, outside gitweb.
+	# The format is described in git_get_project_ctags() subroutine.
 	# You may want to install the HTML::TagCloud Perl module to get
 	# a pretty tag cloud instead of just a list of tags.
 
 	# To enable system wide have in $GITWEB_CONFIG
-	# $feature{'ctags'}{'default'} = ['path_to_tag_script'];
+	# $feature{'ctags'}{'default'} = [1];
 	# Project specific override is not supported.
+
+	# In the future whether ctags editing is enabled might depend
+	# on the value, but using 1 should always mean no editing of ctags.
 	'ctags' => {
 		'override' => 0,
 		'default' => [0]},
@@ -703,6 +706,7 @@ our @cgi_param_mapping = (
 	snapshot_format => "sf",
 	extra_options => "opt",
 	search_use_regexp => "sr",
+	ctag => "by_tag",
 	# this must be last entry (for manipulation from JavaScript)
 	javascript => "js"
 );
@@ -2572,23 +2576,66 @@ sub git_get_project_description {
 	return $descr;
 }
 
+# supported formats:
+# * $GIT_DIR/ctags/<tagname> file (in 'ctags' subdirectory)
+#   - if its contents is a number, use it as tag weight,
+#   - otherwise add a tag with weight 1
+# * $GIT_DIR/ctags file, each line is a tag (with weight 1)
+#   the same value multiple times increases tag weight
+# * `gitweb.ctag' multi-valued repo config variable
 sub git_get_project_ctags {
-	my $path = shift;
+	my $project = shift;
 	my $ctags = {};
 
-	$git_dir = "$projectroot/$path";
-	opendir my $dh, "$git_dir/ctags"
-		or return $ctags;
-	foreach (grep { -f $_ } map { "$git_dir/ctags/$_" } readdir($dh)) {
-		open my $ct, '<', $_ or next;
-		my $val = <$ct>;
-		chomp $val;
-		close $ct;
-		my $ctag = $_; $ctag =~ s#.*/##;
-		$ctags->{$ctag} = $val;
+	$git_dir = "$projectroot/$project";
+	if (opendir my $dh, "$git_dir/ctags") {
+		my @files = grep { -f $_ } map { "$git_dir/ctags/$_" } readdir($dh);
+		foreach my $tagfile (@files) {
+			open my $ct, '<', $tagfile
+				or next;
+			my $val = <$ct>;
+			chomp $val if $val;
+			close $ct;
+
+			(my $ctag = $tagfile) =~ s#.*/##;
+			if ($val =~ /\d+/) {
+				$ctags->{$ctag} = $val;
+			} else {
+				$ctags->{$ctag} = 1;
+			}
+		}
+		closedir $dh;
+
+	} elsif (open my $fh, '<', "$git_dir/ctags") {
+		while (my $line = <$fh>) {
+			chomp $line;
+			$ctags->{$line}++ if $line;
+		}
+		close $fh;
+
+	} else {
+		my $taglist = config_to_multi(git_get_project_config('ctag'));
+		foreach my $tag (@$taglist) {
+			$ctags->{$tag}++;
+		}
 	}
-	closedir $dh;
-	$ctags;
+
+	return $ctags;
+}
+
+# return hash, where keys are content tags ('ctags'),
+# and values are sum of weights of given tag in every project
+sub git_gather_all_ctags {
+	my $projects = shift;
+	my $ctags = {};
+
+	foreach my $p (@$projects) {
+		foreach my $ct (keys %{$p->{'ctags'}}) {
+			$ctags->{$ct} += $p->{'ctags'}->{$ct};
+		}
+	}
+
+	return $ctags;
 }
 
 sub git_populate_project_tagcloud {
@@ -2608,31 +2655,41 @@ sub git_populate_project_tagcloud {
 	my $cloud;
 	if (eval { require HTML::TagCloud; 1; }) {
 		$cloud = HTML::TagCloud->new;
-		foreach (sort keys %ctags_lc) {
+		foreach my $ctag (sort keys %ctags_lc) {
 			# Pad the title with spaces so that the cloud looks
 			# less crammed.
-			my $title = $ctags_lc{$_}->{topname};
+			my $title = esc_html($ctags_lc{$ctag}->{topname});
 			$title =~ s/ /&nbsp;/g;
 			$title =~ s/^/&nbsp;/g;
 			$title =~ s/$/&nbsp;/g;
-			$cloud->add($title, $home_link."?by_tag=".$_, $ctags_lc{$_}->{count});
+			$cloud->add($title, href(project=>undef, ctag=>$ctag),
+			            $ctags_lc{$ctag}->{count});
 		}
 	} else {
-		$cloud = \%ctags_lc;
+		$cloud = {};
+		foreach my $ctag (keys %ctags_lc) {
+			my $title = $ctags_lc{$ctag}->{topname};
+			$cloud->{$ctag}{count} = $ctags_lc{$ctag}->{count};
+			$cloud->{$ctag}{ctag} =
+				$cgi->a({-href=>href(project=>undef, ctag=>$ctag)},
+			          esc_html($title, -nbsp=>1));
+		}
 	}
-	$cloud;
+	return $cloud;
 }
 
 sub git_show_project_tagcloud {
 	my ($cloud, $count) = @_;
-	print STDERR ref($cloud)."..\n";
 	if (ref $cloud eq 'HTML::TagCloud') {
 		return $cloud->html_and_css($count);
 	} else {
-		my @tags = sort { $cloud->{$a}->{count} <=> $cloud->{$b}->{count} } keys %$cloud;
-		return '<p align="center">' . join (', ', map {
-			$cgi->a({-href=>"$home_link?by_tag=$_"}, $cloud->{$_}->{topname})
-		} splice(@tags, 0, $count)) . '</p>';
+		my @tags = sort { $cloud->{$a}->{'count'} <=> $cloud->{$b}->{'count'} } keys %$cloud;
+		return
+			'<div id="htmltagcloud"'.($project ? '' : ' align="center"').'>' .
+			join (', ', map {
+				$cloud->{$_}->{'ctag'}
+			} splice(@tags, 0, $count)) .
+			'</div>';
 	}
 }
 
@@ -4920,13 +4977,8 @@ sub git_project_list_body {
 	@projects = sort_projects_list(\@projects, $order);
 
 	if ($show_ctags) {
-		my %ctags;
-		foreach my $p (@projects) {
-			foreach my $ct (keys %{$p->{'ctags'}}) {
-				$ctags{$ct} += $p->{'ctags'}->{$ct};
-			}
-		}
-		my $cloud = git_populate_project_tagcloud(\%ctags);
+		my $ctags = git_gather_all_ctags(\@projects);
+		my $cloud = git_populate_project_tagcloud($ctags);
 		print git_show_project_tagcloud($cloud, 64);
 	}
 
@@ -5521,13 +5573,14 @@ sub git_summary {
 	my $show_ctags = gitweb_check_feature('ctags');
 	if ($show_ctags) {
 		my $ctags = git_get_project_ctags($project);
-		my $cloud = git_populate_project_tagcloud($ctags);
-		print "<tr id=\"metadata_ctags\"><td>Content tags:<br />";
-		print "</td>\n<td>" unless %$ctags;
-		print "<form action=\"$show_ctags\" method=\"post\"><input type=\"hidden\" name=\"p\" value=\"$project\" />Add: <input type=\"text\" name=\"t\" size=\"8\" /></form>";
-		print "</td>\n<td>" if %$ctags;
-		print git_show_project_tagcloud($cloud, 48);
-		print "</td></tr>";
+		if (%$ctags) {
+			# without ability to add tags, don't show if there are none
+			my $cloud = git_populate_project_tagcloud($ctags);
+			print "<tr id=\"metadata_ctags\">" .
+			      "<td>content tags</td>" .
+			      "<td>".git_show_project_tagcloud($cloud, 48)."</td>" .
+			      "</tr>\n";
+		}
 	}
 
 	print "</table>\n";

-- 
1.7.3
Previous: Jakub NarebskiNext: Jakub Narebski
Message 4 of 13 in “gitweb: Improve ctags, introduce categories”
  1. 0/6 gitweb: Improve ctags, introduce categoriesJakub Narebski, Apr 29, 2011
  2. 1/6 gitweb: Restructure projects list generationJakub Narebski, Apr 29, 2011
  3. Jakub NarebskiMay 7, 2011
  4. 2/6 gitweb: Change the way "content tags" ('ctags') are handledJakub Narebski, Apr 29, 2011
  5. 3/6 gitweb: Mark matched 'ctag' / contents tag (?by_tag=foo)Jakub Narebski, Apr 29, 2011
  6. 4/6 gitweb: Split git_project_list_body in two functionsJakub Narebski, Apr 29, 2011
  7. 5/6 gitweb: Modularized git_get_project_description to be more genericJakub Narebski, Apr 29, 2011
  8. 6/6 gitweb: Optional grouping of projects by categoryJakub Narebski, Apr 29, 2011
  9. Junio C HamanoApr 29, 2011
  10. Jakub NarebskiApr 29, 2011
  11. Øyvind A. HolmApr 30, 2011
  12. Jakub NarebskiMay 3, 2011
  13. git-send-email: Do not encode Subject if not required (was: git-send-email and non 7bit clean message)Jakub Narebski, May 4, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.