{"thread":{"id":"14072","subject":"[RFC/PATCH] gitweb: Extend project_index file format by project description","startedAt":"2008-06-21T13:40:57Z","lastAt":"2008-06-21T16:09:07Z","messageCount":3,"participants":["Jakub Narebski","Lea Wiemann"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"80548","messageId":"200806211540.58929.jnareb@gmail.com","threadId":"14072","inReplyTo":null,"subject":"[RFC/PATCH] gitweb: Extend project_index file format by project description","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-21T13:40:57Z","receivedAt":"2008-06-21T13:40:57Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Change format of $projects_list file from\n  <URI-encoded path> SPC <URI-encoded owner> LF\nto\n  <URI-encoded path> SPC \\\n  <URI-encoded owner> [SPC <URI-encoded description>] LF\nwith optional project description.  Please remember that only single\nline of repository (project) description is supported.  Note that SPC\ncan be replaced by any whitespace character.\n\nThis change required modifying git_get_projects_list() subroutine\n(part when $projects_list is a file, and not a directory to be\nscanned), and git_get_project_description() subroutine.\n\nThe 'project_index' action now creates projects list index file in the\nnew format, with project description.  Also, it now does only minimal\nlevel of escaping / encoding.\n\n\nWhile at it, add some comments describing changes and changed\nsubroutines.  Update and extend information about $projects_list\nformat in gitweb/README and in gitweb/INSTALL.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis is an RFC mainly because I don't know if it wouldn't conflict\nwith Lea Wiemann work on adding caching support to gitweb; if it would,\nI'll send separately the parts that enhance documentation (modified\nof course to not include new feature).\n\nThe goal is to first, improve performance; and second, to be possible\nto have single place for all info (well, amost all info) needed to\ngenerate projects list.\n\nI plan to submit to contrib small Perl script which would generate\nprojects list given (as argument) $projectroot, with minimal\ndependencies, and minimal footprint.  The question is how to name\nit...\n\n gitweb/INSTALL     |   10 +++++--\n gitweb/README      |   50 +++++++++++++++++++++++++++++++----\n gitweb/gitweb.perl |   74 ++++++++++++++++++++++++++++++++++++++++++---------\n 3 files changed, 112 insertions(+), 22 deletions(-)\n\ndiff --git a/gitweb/INSTALL b/gitweb/INSTALL\nindex f7194db..7e25a7f 100644\n--- a/gitweb/INSTALL\n+++ b/gitweb/INSTALL\n@@ -140,9 +140,13 @@ Gitweb repositories\n \n   Each line of the projects list file should consist of the url-encoded path\n   to the project repository database (relative to projectroot), followed\n-  by the url-encoded project owner on the same line (separated by a space).\n-  Spaces in both project path and project owner have to be encoded as either\n-  '%20' or '+'.\n+  by the url-encoded project owner on the same line (separated by a space),\n+  and optionally followed by the url-encoded project description (separated\n+  by space).  Spaces in project path, project owner and project description\n+  have to be encoded as either '%20' or '+'.  Other whitespace (separator),\n+  plus sign '+' (used as replacement for spaces), and percent sign '%' (used\n+  for encoding / escaping) have to be url-encoded, i.e. replaced by '%'\n+  followed by two-digit character number in octal.\n \n   You can generate the projects list index file using the project_index\n   action (the 'TXT' link on projects list page) directly from gitweb.\ndiff --git a/gitweb/README b/gitweb/README\nindex 356ab7b..a32a177 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -157,9 +157,12 @@ not include variables usually directly set during build):\n  * $projects_list\n    Source of projects list, either directory to scan, or text file\n    with list of repositories (in the \"<URI-encoded repository path> SPC\n-   <URI-encoded repository owner>\" format).  Set to $GITWEB_LIST\n-   during installation.  If empty, $projectroot is used to scan for\n-   repositories.\n+   <URI-encoded repository owner>\" line format, or optionally with\n+   project description in \"<URI-encoded repository path> SPC\n+   <URI-encoded repository owner> SPC <URI-encoded description>\";\n+   actually there can be any sequence of whitespace in place of SPC).\n+   Set to $GITWEB_LIST during installation.  If empty, $projectroot is\n+   used to scan for repositories.\n  * $my_url, $my_uri\n    URL and absolute URL of gitweb script; you might need to set those\n    variables if you are using 'pathinfo' feature: see also below.\n@@ -214,6 +217,40 @@ not include variables usually directly set during build):\n    ('-M'); set it to ('-C') or ('-C', '-C') to also detect copies, or\n    set it to () if you don't want to have renames detection.\n \n+\n+Projects list file format\n+~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Instead of having gitweb find repositories by scanning filesystem starting\n+from $projectroot (or $projects_list, if it points to directory), you can\n+provide list of projects by setting $projects_list to a text file with list\n+of projects (and some additional info).  This file uses the following\n+format:\n+\n+One record (for project / repository) per line, whitespace separated fields;\n+does not support (at least for now) lines continuation (newline escaping).\n+Leading and trailing whitespace are ignored, any run of whitespace can be\n+used as field separator (rules for Perl's \"split(' ', $line)\".  Keyed by\n+the first field, which is project name, i.e. path to repository GIT_DIR\n+relative to $projectroot.  Fields use modified URI encoding, defined in\n+RFC 3986, section 2.1 (Percent-Encoding), or rather \"Query string encoding\"\n+(see http://en.wikipedia.org/wiki/Query_string#URL_encoding), the difference\n+being that SPC (' ') can be encoded as '+' (and therefore '+' has to be also\n+percent-encoded).  Reserved characters are: '%' (used for encoding), '+'\n+(can be used to encode SPACE), all whitespace characters as defined in Perl,\n+including SPC, TAB and LF, (used to separate fields in a record).\n+\n+Currently list of fields is\n+ * <repository path>  - path to repository GIT_DIR, relative to $projectroot\n+ * <repository owner> - displayed as repository owner, preferably full name,\n+                        or email, or both\n+ * [<description>]    - one line repository description, OPTIONAL\n+\n+You can additionally use $projects_list file together with to limit which\n+repositories are visible, and together with $strict_export to limit access\n+to repositories (see \"Gitweb repositories\" section in gitweb/INSTALL).\n+\n+\n Per-repository gitweb configuration\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n \n@@ -225,8 +262,8 @@ You can use the following files in repository:\n  * README.html\n    A .html file (HTML fragment) which is included on the gitweb project\n    summary page inside <div> block element. You can use it for longer\n-   description of a project, to provide links for example to projects\n-   homepage, etc.\n+   description of a project, to provide links (for example to project's\n+   homepage), etc.\n  * description (or gitweb.description)\n    Short (shortened by default to 25 characters in the projects list page)\n    single line description of a project (of a repository). Plain text file;\n@@ -243,7 +280,8 @@ You can use the following files in repository:\n  * gitweb.owner\n    You can use the gitweb.owner repository configuration variable to set\n    repository's owner. It is displayed in the project list and summary\n-   page. If it's not set, filesystem directory's owner is used.\n+   page. If it's not set, filesystem directory's owner is used\n+   (via GECOS field / real name field from getpwiud(3)).\n  * various gitweb.* config variables (in config)\n    Read description of %feature hash for detailed list, and some\n    descriptions.\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 9c0dfbd..58144e7 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1710,13 +1710,44 @@ sub git_get_path_by_hash {\n ## ......................................................................\n ## git utility functions, directly accessing git repository\n \n+# this is helper function for git_get_project_description()\n+sub git_get_project_description_from_list {\n+\tmy $path = shift;\n+\n+\topen(my $fd, $projects_list)\n+\t\tor return;\n+\twhile (my $line = <$fd>) {\n+\t\tchomp $line;\n+\t\tmy ($proj, undef, $descr) = split ' ', $line;\n+\t\t$proj = unescape($proj);\n+\t\t$descr = unescape($descr);\n+\t\tif ($proj eq $path) {\n+\t\t\tclose $fd;\n+\t\t\treturn $descr;\n+\t\t}\n+\t}\n+\tclose $fd;\n+\treturn;\n+}\n+\n+# sources for project description are\n+#  * project_list, if $project_list is a file, and it uses new format:\n+#    <encoded path> SPC <encoded owner> SPC <encoded description> LF\n+#  * $GIT_DIR/description file in project repository, if it exists\n+#  * gitweb.description configuration variable for a project\n sub git_get_project_description {\n \tmy $path = shift;\n+\tmy $descr;\n+\n+\tif (-f $projects_list) {\n+\t\t$descr = git_get_project_description_from_list($path);\n+\t\treturn $descr if (defined $descr); # try other sources if needed\n+\t}\n \n \t$git_dir = \"$projectroot/$path\";\n \topen my $fd, \"$git_dir/description\"\n \t\tor return git_get_project_config('description');\n-\tmy $descr = <$fd>;\n+\t$descr = <$fd>;\n \tclose $fd;\n \tif (defined $descr) {\n \t\tchomp $descr;\n@@ -1782,20 +1813,25 @@ sub git_get_projects_list {\n \t\t}, \"$dir\");\n \n \t} elsif (-f $projects_list) {\n-\t\t# read from file(url-encoded):\n+\t\t# read from file (whitespace separated, url-encoded):\n \t\t# 'git%2Fgit.git Linus+Torvalds'\n \t\t# 'libs%2Fklibc%2Fklibc.git H.+Peter+Anvin'\n \t\t# 'linux%2Fhotplug%2Fudev.git Greg+Kroah-Hartman'\n+\t\t# optionally with description\n+\t\t# 'git/git.git Junio+C+Hamano The+core+git+plumbing'\n+\t\t# 'libs/klibc/klibc.git H.+Peter+Anvin klibc+main+development+tree'\n+\t\t# 'linux/hotplug/udev.git Kay+Sievers udev+development+tree'\n \t\tmy %paths;\n \t\topen my ($fd), $projects_list or return;\n \tPROJECT:\n \t\twhile (my $line = <$fd>) {\n \t\t\tchomp $line;\n-\t\t\tmy ($path, $owner) = split ' ', $line;\n+\t\t\tmy ($path, $owner, $descr) = split ' ', $line;\n \t\t\t$path = unescape($path);\n \t\t\t$owner = unescape($owner);\n+\t\t\t$descr = unescape($descr);\n \t\t\tif (!defined $path) {\n-\t\t\t\tnext;\n+\t\t\t\tnext PROJECT;\n \t\t\t}\n \t\t\tif ($filter ne '') {\n \t\t\t\t# looking for forks;\n@@ -1826,9 +1862,14 @@ sub git_get_projects_list {\n \t\t\t}\n \t\t\tif (check_export_ok(\"$projectroot/$path\")) {\n \t\t\t\tmy $pr = {\n-\t\t\t\t\tpath => $path,\n+\t\t\t\t\tpath =>  to_utf8($path),\n \t\t\t\t\towner => to_utf8($owner),\n \t\t\t\t};\n+\t\t\t\tif (defined $descr) {\n+\t\t\t\t\t$descr = to_utf8($descr);\n+\t\t\t\t\t$pr->{'descr_long'} = $descr;\n+\t\t\t\t\t$pr->{'descr'} = chop_str($descr, $projects_list_description_width, 5);\n+\t\t\t\t}\n \t\t\t\tpush @list, $pr;\n \t\t\t\t(my $forks_path = $path) =~ s/\\.git$//;\n \t\t\t\t$paths{$forks_path}++;\n@@ -3981,21 +4022,28 @@ sub git_project_index {\n \tprint $cgi->header(\n \t\t-type => 'text/plain',\n \t\t-charset => 'utf-8',\n-\t\t-content_disposition => 'inline; filename=\"index.aux\"');\n+\t\t-content_disposition => 'inline; filename=\"projects_index.aux\"');\n \n \tforeach my $pr (@projects) {\n \t\tif (!exists $pr->{'owner'}) {\n \t\t\t$pr->{'owner'} = git_get_project_owner(\"$pr->{'path'}\");\n+\t\t\t$pr->{'owner'} = to_utf8($pr->{'owner'});\n+\t\t}\n+\t\tif (!exists $pr->{'descr_long'}) {\n+\t\t\t$pr->{'descr_long'} = git_get_project_description($pr->{'path'}) || \"\";\n+\t\t\t$pr->{'descr_long'} = to_utf8($pr->{'descr_long'});\n \t\t}\n \n-\t\tmy ($path, $owner) = ($pr->{'path'}, $pr->{'owner'});\n-\t\t# quote as in CGI::Util::encode, but keep the slash, and use '+' for ' '\n-\t\t$path  =~ s/([^a-zA-Z0-9_.\\-\\/ ])/sprintf(\"%%%02X\", ord($1))/eg;\n-\t\t$owner =~ s/([^a-zA-Z0-9_.\\-\\/ ])/sprintf(\"%%%02X\", ord($1))/eg;\n-\t\t$path  =~ s/ /\\+/g;\n-\t\t$owner =~ s/ /\\+/g;\n+\t\tmy @pr_info = \n+\t\t\t($pr->{'path'}, $pr->{'owner'}, $pr->{'descr_long'});\n+\t\tforeach (@pr_info) {\n+\t\t\t# quote only minimal set, only what has to be quoted\n+\t\t\ts/([+%])/sprintf(\"%%%02X\", ord($1))/eg;\n+\t\t\ts/ /\\+/g;\n+\t\t\ts/([[:space:]])/sprintf(\"%%%02X\", ord($1))/eg;\n+\t\t}\n \n-\t\tprint \"$path $owner\\n\";\n+\t\tprint join(' ', @pr_info).\"\\n\";\n \t}\n }\n \n-- \n1.5.6\n"},{"id":"80550","messageId":"485D1E76.6090709@gmail.com","threadId":"14072","inReplyTo":"200806211540.58929.jnareb@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Extend project_index file format by project description","fromName":"Lea Wiemann","fromEmail":"lewiemann@gmail.com","sentAt":"2008-06-21T15:29:58Z","receivedAt":"2008-06-21T15:29:58Z","isPatch":true,"sender":{"key":"lewiemann@gmail.com","avatar":null},"body":"Jakub Narebski wrote:\n> This is an RFC mainly because I don't know if it wouldn't conflict\n> with Lea Wiemann work on adding caching support to gitweb;\n\nIf you're talking about merge conflicts, it applies fine on my current \nworking copy.  (There's trailing whitespace on the \"@pr_info =\" line, by \nthe way.)\n\n> The goal is to first, improve performance; and second, to be possible\n> to have single place for all info (well, amost all info) needed to\n> generate projects list.\n\nI'm not sure if I understand your objective correctly.  If this is \nexclusively about performance, may I suggest you wait with this till I \nhave implemented caching?  Regenerating the project list might end up \nbeing fast enough that it won't matter at all; so no reason to \ncomplicate the code.\n\nIf it's about the convenience of maintaining project descriptions in a \ncentral place, sure, that's fine.\n\n-- Lea\n"},{"id":"80552","messageId":"200806211809.07763.jnareb@gmail.com","threadId":"14072","inReplyTo":"485D1E76.6090709@gmail.com","subject":"Re: [RFC/PATCH] gitweb: Extend project_index file format by project description","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-21T16:09:07Z","receivedAt":"2008-06-21T16:09:07Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Lea Wiemann wrote:\n> Jakub Narebski wrote:\n>> \n>> The goal is to first, improve performance; and second, to be possible\n>> to have single place for all info (well, amost all info) needed to\n>> generate projects list.\n> \n> I'm not sure if I understand your objective correctly.  If this is \n> exclusively about performance, may I suggest you wait with this till I \n> have implemented caching?  Regenerating the project list might end up \n> being fast enough that it won't matter at all; so no reason to \n> complicate the code.\n> \n> If it's about the convenience of maintaining project descriptions in a \n> central place, sure, that's fine.\n\nIt is about both, but I think mainly about convenience of maintaining \n(having) all static data about project in one place (well, almost all: \nthere is README.html, but it is not visible in projects list page).\n\nWhat we gain in performance generating projects list (when caching is \ndisabled for some reason, like limited quota or/and not installed \nmemcached), we might lose when generating project pages (with project \ndescription).\n\n-- \nJakub Narebski\nPoland\n"}]}