Re: [RFC PATCH v7 11/9] [PoC] gitweb/lib - tee, i.e. print and capture during cache entry generation

From: Jakub Narebski <jnareb@gmail.com>
Date: Jan 4, 2011, 13:20 UTC
Message-ID: <201101041420.11577.jnareb@gmail.com>
In-Reply-To: <201101040128.26826.jnareb@gmail.com>

Jakub Narebski wrote:

Show 12 quoted lines

> On Tue, 4 Jan 2011, J.H. wrote:
> > On 01/03/2011 01:33 PM, Jakub Narebski wrote:
> 
> > > Instead of having gitweb use progress info indicator / throbber to
> > > notify user that data is being generated by current process, gitweb
> > > can now (provided that PerlIO::tee from PerlIO::Util is available)
> > > send page to web browser while simultaneously saving it to cache
> > > (print and capture, i.e. tee), thus having incremental generating of
> > > page serve as a progress indicator.
> > 
> > In general, and particularly for the large sites that caching is
> > targeted at, teeing is a really bad idea.

[...]

Show 9 quoted lines

> > 1) Errors may still be generated in flight as the cache is being
> > generated.  It would be better to let the cache run with a progress
> > indicator and should an error occur, display the error instead of giving
> > any output that may have been generated (and thus likely a broken page).
> 
> On the contrary, with tee-ing (and zero size sanity check) you would be
> able to see pages even if there are errors saving cache entry.  Though
> this wouldn't help very large sites which cannot function without caching,
> it could be useful for smaller sites.

I was not sure how Perl reacts to ENOSPC (No space left on device), which I think it is only error that can be generated in flight as cache is being generated (or as gitweb output is printed i.e. sent to browser and captured/tee-ed i.e. saved to cache entry file), so I have checked this (using loopback to create small filesystem).

The outcomes one worry about are the following:
* Perl dies during printing - this leads to broken page send to
  browser, and no cache entry generated
* Perl prints output without dying at all; the page send to browser
  via tee-in is all right, but cache entry is truncated which results
  in broken page shown to other clients.

But what actually happens is actually different, and quite safe:
* Perl prints output without dying, and dies on closing cache entry
  file with ENOSPC.  This means that client generating data gets correct
  output, and cache entry is not generated.  Other clients with my code
  try their hand at generation and also get correct page, but not save
  it to cache.

This means that no error page about problems with cache is shown, which
is bad.  On the other hand, at least for smaller sites, gitweb keeps 
working as if without cache for newer entries.
  
Note that observed behaviour might depend on operating system / filesystem
parameters, such as buffer sizes.

> But see below.

[...]

> Note also that in my rewrite you can simply (by changing one single 
> configuration knob) configure gitweb to also cache error pages.  This
> might be best and safest solution for very large sites with very large
> disk space, but not so good for smaller sites.

Errr... now after rereading your email I see that caching error pages has one problem: errors that come from the caching engine or capturing engine - those errors you cannot cache. Sorry, my mistake.

Show 7 quoted lines

> > - John 'Warthog9' Hawley
> > 
> > P.S. I'm back to work full-time on Wednesday, which I'll be catching up
> > on gitweb and trying to make forward progress on my gitweb code again.
> 
> I'll try to send much simplified (and easier to use in caching) error
> handling using exceptions (die / eval used as throw / catch) today.

Sent as
  [RFC PATCH v7 2.5/9] gitweb: Make die_error just die, and use send_error
    to create error pages
  Message-ID: <201101040135.08638.jnareb@gmail.com>
  http://permalink.gmane.org/gmane.comp.version-control.git/164466

-- 
Jakub Narebski
Poland