cgit vs. gitweb

Hi

I wanted to compare cgit against gitweb to see the effect of using libgit and internal caching. So I ran the attached scripts against http://hjemli.net/git/ and http://hjemli.net/cgi-bin/gitweb.cgi as a very simplistic benchmark.

The scripts forks a number of similar child processes (specified by $1) and then uses curl(1) to request the first log page of three repositories $2 times. This is an attempt to simulate concurrent requests for different pages/repositories (I chose the log-pages since they're pretty similar in cgit and gitweb, both showing info for 100 commits).

The numbers are the average of three runs, executed from another box on my subnet. To avoid a roundtrip to my ISP, the pages are accessed by a local hostname.

So, here's the results, as measured by time(1):

fork count|cached uncached gitweb.cgi
---- -----+------ -------- ----------
   0    10|  0.28     0.46      15.33
   1    10|  0.56     0.77      29.72
  10    10|  2.77     4.22     166.40

fork: number of child processes count: number of loop iterations, each requesting 3 different log-pages cached: results for cgit with caching enabled uncached: results for cgit with caching disabled gitweb.cgi: results for gitweb

If ((fork+1) * count * 3) is taken as a rough estimate of the total number of page requests, then requests/second looks like this:

 req| cached uncached gitweb.cgi
----+------- -------- ----------
  30| 107.14    65.22       1.96
  60| 107.14    77.92       2.01
 330| 119.13    78.20       1.98

[Cc: git@vger.kernel.org]

Lars Hjemli wrote:

First, could you try this also with mod_perl, not only CGI?

Could you include ApacheBench (ab) results?

On 12/25/06, Jakub Narebski <jnareb@gmail.com> wrote:

Well, not easily I'm afraid: I'm running slackware 11 w. apache 1.3.37, w.o. mod_perl, and when trying to install mod_perl tonight I ended up w.o. a working httpd :-(

But I guess someone who has tested gitweb w. and w.o. mod_perl could come up with an estimate on the difference between them?

Sure (I didn't know about this tool, thanks for the pointer)

Here's the result of a few requests:

$ ab -n 1000 -c 1 http://box1/git/cgit/log/?h=master This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3 [...snip...] Document Path: /git/cgit/log/?h=master Document Length: 9373 bytes

Concurrency Level: 1 Time taken for tests: 5.788 seconds Complete requests: 1000 Failed requests: 0 Broken pipe errors: 0 Total transferred: 9604000 bytes HTML transferred: 9373000 bytes Requests per second: 172.77 [#/sec] (mean) Time per request: 5.79 [ms] (mean) Time per request: 5.79 [ms] (mean, across all concurrent requests) Transfer rate: 1659.30 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     0
Processing:     3     5   19.0      3   206
Waiting:        3     5   19.0      3   206
Total:          3     5   19.0      3   206

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      3
  80%      3
  90%      4
  95%      4
  98%      4
  99%      4
 100%    206 (last request)

$ ab -n 1000 -c 10 http://box1/git/cgit/log/?h=master [..snip..] Concurrency Level: 10 Time taken for tests: 3.421 seconds Complete requests: 1000 Failed requests: 0 Broken pipe errors: 0 Total transferred: 9604000 bytes HTML transferred: 9373000 bytes Requests per second: 292.31 [#/sec] (mean) Time per request: 34.21 [ms] (mean) Time per request: 3.42 [ms] (mean, across all concurrent requests) Transfer rate: 2807.37 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     1
Processing:     3    31   98.8     13  1037
Waiting:        3    31   98.8     13  1037
Total:          3    31   98.8     13  1037
ERROR: The median and mean for the initial connection time are more
than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%     13
  66%     16
  75%     17
  80%     20
  90%     27
  95%     61
  98%    335
  99%    665
 100%   1037 (last request)

$ ab -n 1000 -c 100 http://box1/git/cgit/log/?h=master [..snip..]Concurrency Level: 100 Time taken for tests: 3.450 seconds Complete requests: 1000 Failed requests: 0 Broken pipe errors: 0 Total transferred: 9612547 bytes HTML transferred: 9381316 bytes Requests per second: 289.86 [#/sec] (mean) Time per request: 345.00 [ms] (mean) Time per request: 3.45 [ms] (mean, across all concurrent requests) Transfer rate: 2786.25 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.3      0     3
Processing:     9   322  143.8    311  1780
Waiting:        6   322  143.8    311  1780
Total:          9   322  143.6    311  1780

Percentage of the requests served within a certain time (ms)
  50%    311
  66%    316
  75%    321
  80%    323
  90%    334
  95%    383
  98%    737
  99%   1146
 100%   1780 (last request)

$ ab -n 100 -c 1 http://box1/cgi-bin/gitweb.cgi?p=cgit;a=shortlog [...snip...] Document Path: /cgi-bin/gitweb.cgi?p=cgit Document Length: 16636 bytes

Concurrency Level: 1 Time taken for tests: 21.908 seconds Complete requests: 100 Failed requests: 0 Broken pipe errors: 0 Total transferred: 1678100 bytes HTML transferred: 1663600 bytes Requests per second: 4.56 [#/sec] (mean) Time per request: 219.08 [ms] (mean) Time per request: 219.08 [ms] (mean, across all concurrent requests) Transfer rate: 76.60 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     0
Processing:   216   219    1.0    219   222
Waiting:      216   219    0.9    219   221
Total:        216   219    1.0    219   222

Percentage of the requests served within a certain time (ms)
  50%    219
  66%    219
  75%    220
  80%    220
  90%    221
  95%    221
  98%    221
  99%    222
 100%    222 (last request)

$ ab -n 100 -c 10 http://box1/cgi-bin/gitweb.cgi?p=cgit;a=shortlog [...snip...] Concurrency Level: 10 Time taken for tests: 23.480 seconds Complete requests: 100 Failed requests: 0 Broken pipe errors: 0 Total transferred: 1744006 bytes HTML transferred: 1728636 bytes Requests per second: 4.26 [#/sec] (mean) Time per request: 2348.00 [ms] (mean) Time per request: 234.80 [ms] (mean, across all concurrent requests) Transfer rate: 74.28 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     0
Processing:   216  2244  897.8   2044  4905
Waiting:      216  2244  897.8   2043  4905
Total:        216  2244  897.8   2044  4905

Percentage of the requests served within a certain time (ms)
  50%   2044
  66%   2566
  75%   3008
  80%   3119
  90%   3487
  95%   3591
  98%   4094
  99%   4171
 100%   4905 (last request)

$ ab -n 100 -c 50 http://box1/cgi-bin/gitweb.cgi?p=cgit;a=shortlog [...snip...] Concurrency Level: 50 Time taken for tests: 29.552 seconds Complete requests: 100 Failed requests: 0 Broken pipe errors: 0 Total transferred: 1978127 bytes HTML transferred: 1958697 bytes Requests per second: 3.38 [#/sec] (mean) Time per request: 14776.00 [ms] (mean) Time per request: 295.52 [ms] (mean, across all concurrent requests) Transfer rate: 66.94 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     1    0.0      1     1
Processing:   221 11464 5725.4  11769 22556
Waiting:      221 11464 5725.5  11769 22556
Total:        221 11465 5725.6  11770 22557
ERROR: The median and mean for the initial connection time are more
than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%  11770
  66%  14899
  75%  16229
  80%  16535
  90%  19184
  95%  20065
  98%  22227
  99%  22397
 100%  22557 (last request)

This does seem to confirm the results of my homemade scripts.

Lars Hjemli wrote:

[...] 

[snip]

By the way, which version of gitweb did you use (shortened sha1 of commit, or result of git-describe)? I'm interested if it is before or after Robert Fitzsimons patches which replaced 101 forks in log-like views with 1 fork, which I think affects gitweb performance a lot.

The page which is optimized is summary view, if you use projects_list and not projectroot directory...

Here are the numbers for a few tests I ran yesterday, this is with gitweb without mod_perl.

v267, 1842 38.5 +/- ms (with v1.4.4.3 git tools) v1.4.4.3, 445 2.5 +/- ms master, 445 2.5 +/- ms master + bulk patches, 340 3.7 +/- ms (--skip patch)

(/usr/sbin/ab -n 20 -k "http://localhost/git/?p=home/bob/src/git/git/.git;a=summary")

Robert

On 12/26/06, Jakub Narebski <jnareb@gmail.com> wrote:

Results from ab used gitweb/1.4.4.3.gd4ada (pulled tonight), but my homemade scripts used c902c9a6 (from dec 21)

ab is after the patches, my homemade scripts was before

I tested 'a=shortlog', since it seems to be functionally equivalent to cgits log view.