git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH v7 11/9] [PoC] gitweb/lib - tee, i.e. print and capture during cache entry generation

From
J.H. <warthog9@eaglescrag.net>
Date
Jan 3, 2011, 23:31 UTC
Message-ID
<4D225C6E.9000108@eaglescrag.net>
In-Reply-To
<201101032233.16174.jnareb@gmail.com>
On 01/03/2011 01:33 PM, Jakub Narebski wrote:
Show 6 quoted lines
> Instead of having gitweb use progress info indicator / throbber to
> notify user that data is being generated by current process, gitweb
> can now (provided that PerlIO::tee from PerlIO::Util is available)
> send page to web browser while simultaneously saving it to cache
> (print and capture, i.e. tee), thus having incremental generating of
> page serve as a progress indicator.

In general, and particularly for the large sites that caching is targeted at, teeing is a really bad idea. I've mentioned this several times before, and the progress indicator is a *MUCH* better idea. I'm not sure how many times I can say that, even if this was added it would have the potential to exacerbate disk thrashing and overall make things a lot more complex.

1) Errors may still be generated in flight as the cache is being
generated.  It would be better to let the cache run with a progress
indicator and should an error occur, display the error instead of giving
any output that may have been generated (and thus likely a broken page).
2) Having multiple clients all waiting on the same page (in particular
the index page) can lead to invalid output.  In particular if you are
teeing the output a reading client now must come in, read the current
contents of the file (as written), then pick up on the the tee after
that.  It's actually possible for the reading client to miss data as it
may be in flight to be written and the client is switching from reading
the file to reading the tee.  I don't see anything in your code to
handle that kind of switch over.
3) This makes no allowance for the file to be generated completely in
the background while serving stale data in the interim.  Keep in mind
that it can (as Fedora has experienced) take *HOURS* to generate the
index page, teeing that output just means brokenness and isn't useful.

It's much better to have a simple, lightweight waiting message get displayed while things happen. When they are done, output the completed page to all waiting clients.

- John 'Warthog9' Hawley

P.S. I'm back to work full-time on Wednesday, which I'll be catching up on gitweb and trying to make forward progress on my gitweb code again.

Previous: Jakub NarebskiNext: Jakub Narebski
Message 29 of 34 in “gitweb: Output caching, with eval/die based error handling”
  1. 0/9 gitweb: Output caching, with eval/die based error handlingJakub Narebski, Dec 22, 2010
  2. 1/9 gitweb: Go to DONE_REQUEST rather than DONE_GITWEB in die_errorJakub Narebski, Dec 22, 2010
  3. Jonathan NiederDec 23, 2010
  4. Jakub NarebskiDec 25, 2010
  5. diff: funcname and word patterns for perlJonathan Nieder, Dec 26, 2010
  6. Jakub NarebskiDec 26, 2010
  7. Junio C HamanoDec 27, 2010
  8. Jakub NarebskiDec 27, 2010
  9. Jeff KingDec 28, 2010
  10. Jonathan NiederDec 26, 2010
  11. Jakub NarebskiDec 26, 2010
  12. 2/9 gitweb: use eval + die for error (exception) handlingJakub Narebski, Dec 22, 2010
  13. Jonathan NiederDec 23, 2010
  14. Jakub NarebskiDec 25, 2010
  15. 5/9 gitweb: Make die_error just die, and use send_error to create error pagesJakub Narebski, Jan 4, 2011
  16. 3/9 gitweb: Introduce %actions_info, gathering information about actionsJakub Narebski, Dec 22, 2010
  17. 4/9 gitweb: Prepare for splitting gitwebJakub Narebski, Dec 22, 2010
  18. Jonathan NiederDec 24, 2010
  19. Jakub NarebskiDec 26, 2010
  20. 5/9 t/test-lib.sh: Export also GIT_BUILD_DIR in test_externalJakub Narebski, Dec 22, 2010
  21. 6/9 gitweb/lib - Simple output capture by redirecting STDOUT to fileJakub Narebski, Dec 22, 2010
  22. Jonathan NiederDec 24, 2010
  23. Jakub NarebskiDec 26, 2010
  24. 7/9 gitweb/lib - Very simple file based cacheJakub Narebski, Dec 22, 2010
  25. 8/9 gitweb/lib - Cache captured output (using compute_fh)Jakub Narebski, Dec 22, 2010
  26. 9/9 gitweb: Add optional output cachingJakub Narebski, Dec 22, 2010
  27. 10/9 gitweb: Background cache generation and progress indicatorJakub Narebski, Dec 31, 2010
  28. 11/9 [PoC] gitweb/lib - tee, i.e. print and capture during cache entry generationJakub Narebski, Jan 3, 2011
  29. J.H.Jan 3, 2011
  30. Jakub NarebskiJan 4, 2011
  31. Jakub NarebskiJan 4, 2011
  32. 11/9 [PoC] gitweb/lib - HTTP-aware output cachingJakub Narebski, Jan 5, 2011
  33. Jonathan NiederDec 26, 2010
  34. Jonathan NiederDec 26, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.