git/list[1] front-page[2] threads[3] people[4] search[5] about
 

repo.or.cz renovated

From
Petr Baudis <pasky@suse.cz>
Date
Mar 17, 2008, 18:10 UTC
Message-ID
<20080317181015.GC10335@machine.or.cz>
In-Reply-To
<m3ve3nwtl3.fsf@localhost.localdomain>
On Sat, Mar 15, 2008 at 02:44:42PM -0700, Jakub Narebski wrote:
Show 9 quoted lines
> Petr Baudis <pasky@suse.cz> writes:
> 
> > On repo.or.cz (permanently I/O overloaded and hosting 1050 project +
> > forks), 
> 
> It looks like repo.or.cz is overwhelmed by its success. I hope that
> now that there are other software hosting sites with git hosting
> (Savannah, GitHub, Gitorious,...) the number of projects wouldn't grow
> as rapidly.

Actually, it was overwhelmed to so much by its success but by lack of good maintenance. ;-) I gave it some love again for the past week and the improvement was, well, overwhelming. :-)

I finally fixed tons of failures and broken repositories, and most importantly repacked some of the big repositories with object databases in pretty horrid shape. The effect has been immense, having everything in database of 1/3 the size and single big pack drastically reduced the I/O load.

Scenario: Site with about 1100 repositories weighting 13GB, running a
fetch job for about 200 of them hourly. About two git-daemon requests
per minute and 10 gitweb requests per minute (the last two numbers are
taken quite sloppily over a small sample of the last ten minutes ;-).
Site is running on 2x1GHz P3 with 2G RAM, repository is on hw RAID5.
(We are currently preparing to migrate it to a more powerful machine.)

Before, the load on the server would be normally about 6 to 15 _all the time_ and bunch of git-related processes would be permanently eating some CPU and crunch on the disk.

After introducing the index caching and repacking the repositories, the load seems to be around 1 at most and hardly seems to come above 3; all feels very snappy.

So for anyone running a hosting site, make sure your repositories are nicely packed. It makes huge difference to the I/O load!

Show 6 quoted lines
> Another solution would be to divide projects list page into pages,
> perhaps adding search box for searching for a project (by name, by
> description and by owner).
> 
> Nevertheless even with pagination, if we want to have "sort by last
> update" we do need caching.

Yes, I'm pondering about pagination, but because of web clients, not the server load; it takes firefox on my notebook noticeable time to render this list already, and it's rather big too. Ideas are welcome here.

My current plan is to have a [Search project] box at the front page, together with direct link to 'show all'. Other than that, what makes sense to display on the front page? I think recently added projects (age < 1 week) for sure. I'm not so sure about recently changed projects - maybe it is better to keep the front page cruft-free.

-- 
				Petr "Pasky" Baudis
Whatever you can do, or dream you can, begin it.
Boldness has genius, power, and magic in it.	-- J. W. von Goethe
Previous: Jakub NarebskiNext: Junio C Hamano
Message 26 of 30 in “gitweb: Support caching projects list”
  1. gitweb: Support caching projects listPetr Baudis, Mar 13, 2008
  2. Jay SoffianMar 14, 2008
  3. Petr BaudisMar 14, 2008
  4. Jay SoffianMar 14, 2008
  5. J.H.Mar 14, 2008
  6. Jakub NarebskiMar 14, 2008
  7. J.H.Mar 14, 2008
  8. repo.or.cz renovationPetr Baudis, Mar 17, 2008
  9. Petr BaudisMar 17, 2008
  10. J.H.Mar 17, 2008
  11. Jakub NarebskiMar 17, 2008
  12. Jakub NarebskiMar 17, 2008
  13. Jakub NarebskiMar 14, 2008
  14. Jay SoffianMar 14, 2008
  15. Junio C HamanoMar 14, 2008
  16. Frank LichtenheldMar 14, 2008
  17. Jakub NarebskiMar 14, 2008
  18. Petr BaudisMar 17, 2008
  19. Jakub NarebskiMar 15, 2008
  20. Miklos VajnaMar 16, 2008
  21. Frank LichtenheldMar 16, 2008
  22. J.H.Mar 16, 2008
  23. Jakub NarebskiMar 16, 2008
  24. J.H.Mar 16, 2008
  25. Jakub NarebskiMar 16, 2008
  26. repo.or.cz renovatedPetr Baudis, Mar 17, 2008
  27. Junio C HamanoMar 17, 2008
  28. Petr BaudisMar 17, 2008
  29. Theodore TsoMar 17, 2008
  30. Petr BaudisMar 17, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.