Re: [spf:guess,iffy] Re: "Integrated Web Client for git" GSoC proposal

On Thu, 15 April 2010, Sam Vilain wrote:

By the way, is the above statement correct? What should be the *goal* for gitweb? Is it running on oldest Perl possible (which IIRC is Perl 5.8.6, because gitweb needs good Unicode support)? Is it minimal required non-core dependencies (core as of Perl 5.8.6)? Is it being easy to install (single monolithic script + some minimal number of static files like CSS, JavaScript, images + perhaps extra files with optional features like output caching) by hand? Is it being easy to configure by hand?

With local::lib it is easy to install Perl modules from CPAN locally (in home directory), so perhaps we could ease on minimal dependencies rule. On the other hand gitweb is web app, and one would have to be able to configure web server to use locally installed Perl modules, which might be not possible; this swicthes the problem from "not being able to install Perl modules as non-root" to "not being able to install Perl modules that web server can use without help from sysadmin".

One of requirements is backwards compatibility of gitweb links (URLs), both query string based, and path_info URLs. Yet another is ability to *scan* for projects (this is IIRC requirement for git.kernel.org), rather than using project index file (perhaps (re)generated (semi)automatically).

There exists very many web frameworks in Perl[1][2][3][4]: Catalyst (one of more popular, used by Gitalist), Jifty, CGI::Application (and its offshot Titanium), Mojolicus, Dancer, Squatting, Web::Simple,...

Nowadays many (all mentioned) of those web frameworks are either built on top of PSGI / Plack (Perl Superglue for Web Frameworks and Web Servers), or have PSGI / Plack adapter (see http://plackperl.org). So another solution would be to use bare-bones Plack instead of CGI with help of CGI.pm, perhaps also using one of URI dispatchers[5][6]. Plack/PSGI looks like the future of Perl web scripting... but is currently quite new, at version 0.9930.

[1]: http://www.perlfoundation.org/perl5/index.cgi?web_frameworks [2]: http://plackperl.org/ ("Frameworks" section) [3]: http://perl.worldken.org/search?q=web+framework [4]: http://search.cpan.org/search?query=web+framework&mode=all

[5]: http://www.perlfoundation.org/perl5/index.cgi?web_application_dispatchers [6]: http://github.com/miyagawa/plack-dispatching-samples

As to splitting gitweb into smaller modules (like e.g. SVN::Web)... Gitweb isn't the largest script in git sources: gitk is. But splitting gitk should be much easier - git-gui is already split. Close after gitweb is git-svn, which probably also need splitting; and again it would be easier to split parts of git-svn e.g into Git::SVN module, as git-svn already depends on Git.pm.

The trouble with splitting is installing split web script. I have no idea how to do this in cross-webserver way, cross-distribution and cross-system way (the filesystem hierarchy used by web server may differ from distribution to distribution, and from operating system to operating system).

On Fri, 2010-04-16 at 11:18 +0200, Jakub Narebski wrote: 

This is correct for XS modules, using DynaLoader to link in arbitrary .so's may be restricted in a web server :). But for pure Perl modules it should be no problem. Many modules have both XS and Pure-Perl versions, for instance Template Toolkit has both and so is suitable for this style of bundling.

Ok so PSGI is the port of Python's WSGI to Perl. Plack is the reference implementation, and also quite heavy at 2.5MB tarball.

Titanium is an extension to CGI::Application and requires DBI for instance. That's probably not right.

Jifty, Mojolicious, will also have prohibitively difficult dependencies for the run-anywhere case.

Squatting has a few XS dependencies, but perhaps they could be excised if required. Dancer however seems to stand out at only 94kB tarball, minimal non-core dependencies and support for PSGI. The HTTP::Server::Simple::PSGI dependency should let it support the 'instaweb' case with pure perl.

It should be possible for the script to figure out what filesystem path it is being run from, perhaps find a local lib/ dir and then add that to @INC. In shell scripts you just use FindBin, I don't know whether that is still expected to work from eg mod_perl but there's bound to be a solution for that. So yeah I'd say just aim to ship lots of .pm files in a nearby dir alongside the script...

Sam

Sam Vilain <sam <at> vilain.net> writes:

I admit that it's quite new - not old anyway, but don't get fooled by its
version number. It's already adapted by almost all perl frameworks and
used by lots of companies on production.
 

No, the tarball just contains 2.4MB of .jpeg file for large-file download and upload testing. Its dependencies are kept minimal - only requires LWP, URI and some pure perl modules.

You can already use CGI::Emulate::PSGI (via Plack::App::CGIBin) to run gitweb.cgi on Plack, or whatever web server that supports PSGI - like HTTP::Server::Simple::PSGI you just mentioned. http://gist.github.com/346068

Or use App::FatPacker.

-- Tatsuhiko Miyagawa

Restored some of Cc list...

Tatsuhiko Miyagawa <miyagawa@bulknews.net> writes:

FindBin, Dir::Self (or just relevant code from it), and then "use lib <dir>".

But try to make this build-time dependency optional. I wonder also what is performance cost of using App::FatPacker trick.

It is a bit pity that PAR is not in core.  We could use 

  $ git archive --format=zip --output=gitweb.par --prefix=lib/ \
    HEAD -- gitweb/lib

to generate packfile, and then

  use PAR 'gitweb.par';

in gitweb.

On Sat, 17 Apr 2010, Sam Vilain wrote:

Errr... how the above addresses "not being able to install Perl modules that web server can use without help from sysadmin", or in other words "not being able to tell web server where to find locally installed / / additional Perl modules"? If you can't set PERL5LIB for 'apache' or 'web' or 'nobody' user, and you can't edit web server configuration files...

Well, perhaps

  use lib (grep { -d } split /:/, "++GITWEBPERLLIB++");

and associated change to gitweb/Makefile would help here.

Tatsuhiko Miyagawa addressed the issue of "2.5MB tarball". PSGI is specification, Plack is utilities (and includes reference implementation).

Using PSGI / Plack / PSGI compatibile web framework would give us ability to run gitweb on many web servers: CGI, FastCGI, mod_perl (all those via adapters from Plack), mod_psgi, HTTP::Server::PSGI (included in Plack),... and give us Test::Plack.

But not less important would be ability to use Plack middleware; for example gitweb output caching could be as simple as

  use Plack::Builder;
  ...

  builder {
     enable "Cache", ...
     $app;
  };

Thanks for the research on those Perl web frameworks.

Or HTTP::Server::PSGI from Plack.

Errr... isn't FindBin considered harmful, and current best practice is using Self::Dir, or just

        # __DIR__ is taken from Dir::Self __DIR__ fragment
        sub __DIR__ () {
                File::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);
        }
        use lib __DIR__ . '/lib';

And there is also App::FatPacker and PAR solution; pity that neither is in core (well, for App::FatPacker it would be build time dependency only).

Dnia środa 14. kwietnia 2010 15:50, Junio C Hamano napisał:

Well, I try to either save interesting gitweb patches (which doesn't help
much), or import them to my repository at
  http://repo.or.cz/w/git/jnareb-git.git

The interesting branches would be gitwe/* branches. Some of them are regularly rebased against 'master' branch of main git repository.

Those branches are:

gitweb/web::

  The branch that I am doing miscelanous gitweb development on.  Managed
  using StGit, not all patches applied (visible).  Regularly rebased 
  against 'origin' branch.

  Currently contains syntax highlighting support from Johannes Schindelin
  (based in turn on Daniel Svensson's and Sham Chukoury's work in
  gitweb-xmms2.git), plus my refactoring of that.

  Needs decision on names of subroutines, and on semantics of 'highlight'
  feature (on/off, or highlighter name).

gitweb/cache-kernel-pu::

  Simple file-based output caching for gitweb, based on work by John 
  'Warthog9' Hawley (J.H.), split into smaller patches for easier review.
  Much more configurable than original patch(es) by J.H., with pluggable
  caching engine (it works for example with Cache::FileCache).

  J.H. ideas were: no serialization and mtime based expiration of simple
  file-based cache, using locking to avoid 'cache miss stampede' problem,
  serving stale data when cache entry is being regenerated, generating
  cache in background, trick for non-AJAXy progress info, having everything
  related to cache in 'cache.pm' file.

  My ideas were: tests, separating simple file-based cache into separate
  package and making it easy to replace this cache engine by any
  Cache::Cache or CHI cache driver, select(FILEHANLE) based capturing,
  separating capturing output into package to be able to replace it by
  other captyring mechanism, daemonizing for background cache generation
  (for mod_perl), being able to 'require' the 'cache.pm' file, gathering
  options in %cache_options and adding configurability, documentation.

  This is work in progress, and a bit of mess currently.  I have posted
  information about current state of this branch to git mailing list.

  Rebased on top of 'origin'.

gitweb/cache-kernel-v3:: gitweb/cache-kernel-v2:: gitweb/cache-kernel::

  Gitweb caching series, as send to git mailing list.  Not modified, here
  for historical reasons.

gitweb/category::

  The "nitpicked to death" series by Sebastien Cevey.

  I have meant to fix the way list of projects is displayed first, to
  separate filtering list of projects and printing list of projects,
  but never did managed to get to it.

  Rebased on top of 'origin'.  Series is stalled.

gitweb/committags::

  Series by Marcel M. Cary.

  I have meant to comment on this series, but forgot about it.  The major
  problem is deciding how to configure this feature... and how to name it.
  It is decision about API (configuration), so it is important to get it
  right the first time.

  Needs review.  Series is stalled.

gitweb/cache::

  Old caching (pre-GSoC2009 gitweb caching project even) of Perl data,
  for projects_list page only, by Petr Baudis.  Some version of it might
  be used on http://repo.or.cz

  Abandoned for now.  Needs checking which method of caching is better.

On Sat, Apr 17, 2010 at 11:57 AM, Tatsuhiko Miyagawa <miyagawa@bulknews.net> wrote:

We need to keep gitweb dependancies as less as possible. So, using frameworks will be a bad idea, as everybody might don't want to use them.

Coming to splitting of gitweb, as Jakub said installing a split web script would be a problem, but we can have a solution the problem by using proper algorithms. First, we parse the URL and get to know which function to be implemented and then we include the necessary file and call the function which will then execute and return the proper HTML output.

This way, we can no longer worry about installing the split webscript. All we need is to install the basic gitweb.pl file which does all this basic parsing and calling other perl files. The other files just needed to be in the path.

-pavan

Pavan Kumar Sunkara <pavan.sss1991@gmail.com> writes:

Well, hopefully in the future PSGI / Plack would be as widely available as CGI.pm is currently...

Not a solution, I think. If you could munge PATH for web server [user], you could munge PERL5LIB and use locally (local::lib) installed packages.