Tag: Benchmark

nginx vs Tornado Web vs Apache Performance Benchmark

Posted on by

Let’s put the nginx, Tornado and Apache and see who is the winner.

What to test:

A simple Hello World webpage.

How:

ab -n 10000 -c 300 http://myserver.ip

Test Environment:

OS: Ubuntu 9.10 (32-bit) / Linux 2.6.31
CPU: Pentium Celeron M 1.5 GHz
Ram: 512 MB
Apache: 2.2.12
PHP: 5.2.10
Python: 2.6
Tornado: 0.2
nginx: 0.6.13

Result:

Apache

Server Software:        Apache/2.2.12
Server Port:            80

Document Path:          /
Document Length:        13 bytes

Concurrency Level:      300
Time taken for tests:   5.295 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      2863146 bytes
HTML transferred:       130143 bytes
Requests per second:    1888.58 [#/sec] (mean)
Time per request:       158.850 [ms] (mean)
Time per request:       0.529 [ms] (mean, across all concurrent requests)
Transfer rate:          528.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    8 151.0      0    3028
Processing:     0  122 602.0     48    5289
Waiting:        0  122 600.8     48    5284
Total:         10  130 622.1     49    5292

Percentage of the requests served within a certain time (ms)
  50%     49
  66%     50
  75%     50
  80%     51
  90%     53
  95%     63
  98%     68
  99%   5271
 100%   5292 (longest request)

nginx

Server Software:        nginx/0.6.13
Server Port:            8000

Document Path:          /
Document Length:        13 bytes

Concurrency Level:      300
Time taken for tests:   3.172 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      2230000 bytes
HTML transferred:       130000 bytes
Requests per second:    3152.12 [#/sec] (mean)
Time per request:       95.174 [ms] (mean)
Time per request:       0.317 [ms] (mean, across all concurrent requests)
Transfer rate:          686.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2  68.3      0    3057
Processing:     0   70 359.1     27    3164
Waiting:        0   70 359.1     26    3164
Total:         10   72 365.6     27    3169

Percentage of the requests served within a certain time (ms)
  50%     27
  66%     27
  75%     27
  80%     27
  90%     28
  95%     29
  98%     32
  99%   3157
 100%   3169 (longest request)

Tornado Web:

Server Software:        TornadoServer/0.1
Server Port:            8888

Document Path:          /
Document Length:        12 bytes

Concurrency Level:      300
Time taken for tests:   5.109 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1680000 bytes
HTML transferred:       120000 bytes
Requests per second:    1957.33 [#/sec] (mean)
Time per request:       153.270 [ms] (mean)
Time per request:       0.511 [ms] (mean, across all concurrent requests)
Transfer rate:          321.12 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   32 309.3      0    3035
Processing:    33  103  28.5    103     375
Waiting:       33  102  28.5    103     375
Total:         36  135 311.3    103    3154

Percentage of the requests served within a certain time (ms)
  50%    103
  66%    105
  75%    108
  80%    112
  90%    116
  95%    126
  98%    313
  99%   3121
 100%   3154 (longest request)

What do these numbers mean?

In a given time, if nginx and Tornado Web can handle 1.66 and 1.036 times more requests than Apache, respectively! In the other words, the server is more efficient and it can use the saved resource to do something else!

–Derrick

Our sponsors:

Tornado Web Server – Very Fast!

Posted on by

I just did a benchmark comparison on Tornado Web Server and Apache + PHP server. The result is pretty amazing.

What to test:

A simple Hello World application.

How:

ab -n 1000 -c 300 http://myserver.ip

Test Environment

OS: FreeBSD 7.1
CPU: Pentium III 933 MHz
Ram: 256 MB
PHP: 5.2.9
Python: 2.5.4
Tornado: 0.2

Result:

Apache + PHP:

Concurrency Level:      300
Time taken for tests:   3.515 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      255000 bytes
HTML transferred:       12000 bytes
Requests per second:    284.50 [#/sec] (mean)
Time per request:       1054.499 [ms] (mean)
Time per request:       3.515 [ms] (mean, across all concurrent requests)
Transfer rate:          70.85 [Kbytes/sec] received

Tornado Web Server:

Concurrency Level:      300
Time taken for tests:   1.907 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      168000 bytes
HTML transferred:       12000 bytes
Requests per second:    524.48 [#/sec] (mean)
Time per request:       571.993 [ms] (mean)
Time per request:       1.907 [ms] (mean, across all concurrent requests)
Transfer rate:          86.05 [Kbytes/sec] received

What do these numbers mean?

Tornado Web Server can handle 1.84 times more requests than Apache + PHP server in a given time!

–Derrick

Our sponsors:

Does the most popular imply the best?

Posted on by

Today, I had a discussion with my co-worker on the best languages for web applications. While I argued that Perl, and Ruby performs better than PHP, my co-worker had an opposite opinion. He believed that PHP is the best language overall because it is widely used. It brings me to think of a question, does the most popular imply the best?

Let’s answer this question by examples:

Category Most Popular The Best Why The Best?
OS (Desktop) Windows OS X User friendly, secure, stability, and performance.
OS (Server) Linux, Windows Server FreeBSD Fast, stable, secure, simple design, and centralize management.
Browsers Internet Explorer, Firefox Safari Fast, secure, stable
Programming languages (Desktop/GUI) C#, Java C, C++ Fast, secure
Programming languages (Web/Server) PHP, ASP Ruby, Python, Perl Fast, fewer lines of code.
Database MySQL, Oracle Berkeley DB, Tokyo Cabinet Fast I/O, simple design, smaller size of the database file.
Software company to work for Microsoft Start-up Rewards, opportunity to grow

I think there are several reasons why the most popular things are not the best.

1. Most popular thing is always backed by the best marketing strategies. Marketing strategies is simply a way to let people know about the product and convince them to use it. It has nothing to do with the quality of the product at all.

Example: The ads of Internet Explorer 8 look very attractive. Is it the most secure browser ever?

2. Most people are followers. They are not good in selecting good products from bad products(i.e., filtering the noise). They believe that the choice form most people (most popular) should be the best one. This is similar to election. The choice of most people is more likely to be better.

Example: Is MySQL really the best database?

3. Different people have different preferences. So a popular product must have the general taste. For example, an extremely hot and spicy food may be better to the body, but it is not as popular as a mild one, because the number of people that can take extremely hot and spicy food is smaller.

Example: Linux is very easy to configure comparing to FreeBSD (GUI vs command line) from the end-user perspective, but it is less powerful and stable overall.

Conclusions:

Think twice before your move.

Our sponsors:

Tokyo Cabinet is 31 times faster than MySQL!!!

Posted on by

Today I was comparing the performance of Tokyo Cabinet, MySQL and Memcached using PHP. The result of Tokyo Cabinet is really impressive!

(Note that these results are generated from my computer. If you perform your own benchmarks, then you might find different results due to different system configurations. In fact, I encourage you to perform your own benchmarks. )

1. Quick Result

Total time used to write 10000 records and retrieve them back by each candidate  (The lower the better):
Tokyo Cabinet: 18.87s
Memcached: 11.309s
MySQL: 562.21s

A completed result will be given below.

2. Testing environment

Hardware

  • Intel Pentium II 450MHz
  • 160 MB RAM

Software

  • FreeBSD 7.2
  • Apache 2.2.11
  • PHP 5.2.9
  • Tokyo Cabinet 1.4.20
  • Tokyo Tyrant 1.1.26
  • MySQL 5.1.34
  • Memcached 1.2.6

3. Test scenario

For each candidate, I measure its performance using the following strategies through PHP:

1.) The time used to create all of the necessary objects.

2.) The time used to connect to the database.

3.) The time used to store 10000 randomly generated data.

4.) The time used to read 10000 randomly generated data.

5.) The time used to close the connection.

4. Test results

Tokyo Cabinet

time index ex time %
Start 1243136772.72818600 – 0.00%
create 1243136772.72838700 0.000201 0.00%
connect 1243136772.73224200 0.003855 0.02%
vanish 1243136772.73526800 0.003026 0.02%
put 1243136781.53287200 8.797604 46.60%
get 1243136791.60574800 10.072876 53.35%
close 1243136791.60755900 0.001811 0.01%
Stop 1243136791.60782300 0.000264 0.00%
total – 18.879637 100.00%


Memcached

time index ex time %
Start 1243131185.86078100 – 0.00%
create 1243131185.86092500 0.000144 0.00%
connect 1243131185.86688300 0.005958 0.05%
put 1243131191.49702900 5.630146 49.78%
get 1243131197.16868100 5.671652 50.15%
close 1243131197.16982400 0.001143 0.01%
Stop 1243131197.17003400 0.000210 0.00%
total – 11.309253 100.00%

MySQL

time index ex time %
Start 1243136138.19049000 – 0.00%
create 1243136138.19064700 0.000157 0.00%
connect 1243136138.22529900 0.034652 0.01%
put 1243136150.70760600 12.482307 2.22%
get 1243136700.40063700 549.693031 97.77%
close 1243136700.40078900 0.000152 0.00%
Stop 1243136700.40104500 0.000256 0.00%
total – 562.210555 100.00%

The overhead time (create objects, making connections, closing connections etc) of all candidates are about the same. The only difference is storing the record and retrieving the record. Both Tokyo Cabinet and MySQL took about 10 seconds to store 10000 records in the database. However, Tokyo Cabinet took about 10 seconds to retrieve the records while MySQL took about 550 seconds! That’s about 31 times longer!

benchmark-tokyocabinet-memcached-mysql

5. Materials

Here is the material I used to make this benchmark test.
Click here to download the package.

Please feel free to let me know if you have any question or comment.

Our sponsors:

Performance Benchmark: FreeBSD vs Ubuntu vs OS X, which one is better?

Posted on by

If you cannot decide whether you should go with FreeBSD or Ubuntu (or Linux), check out this article: FreeBSD or Linux in 6 Simple Questions

Recently I am working on building a web farm which is formed by three ancient time computers (The sum of their ages is older than me). Therefore, I need to pick an operating system to make them running efficient enough. Basically here are my criterias:

  • It must be secure. (Sorry, Microsoft Windows, I tried installing Windows Server and the OS got infected by virus in 20 minutes after connecting to Web.)
  • It must be stable and require no reboot. (I have a very good experience with FreeBSD. My longest record of uptime is 6 months so far.)
  • It must be efficient and effective on performance.

So after trying different operating systems, I ran down to three choices: FreeBSD 8, Ubuntu 10 and OS X.

And I finally go with FreeBSD 8. Before I talked about why I kick the Ubuntu and OS X away, let me discuss what methodology I used.

First Test: Extreme Computation

I write a program to calculate the value of pi using Monte-Carlo simulations. For those who are not familiar with this simulation method, basically it try to repeat a calculation for n times and try to determine the value of pi based on the results of calculations. This calculation will use all available CPU resources.

Here is the result from a computer: Pentium II 400MHz + 160MB

  • FreeBSD 8.1 ~ 5 seconds
  • FreeBSD 4.11~ 10 seconds
  • Ubuntu 10.04 ~ 8 seconds
  • OS X 10.3.9 ~ 11 seconds

The result is very obvious, FreeBSD 8.1 is the best candidate in this test. However, the result of my program only gives a very small portion of the picture. So I decide to run another test:

Second Test: Apache Benchmark

No test is better than the real world test drive. Since my primary purpose is for web server, so I decide to test the performance on how Apache perform on different operating systems. My test is pretty simple, I basically ask the Apache Benchmark client to download a page from the test server. The page does three things:

  1. Display a picture
  2. Insert a record into MySQL database
  3. Retrieve the current count from MySQL database

This way I can test the overall performance on how the system handles the file I/O, database I/O and computations.

So the result is pretty interesting (1000 total requests and 100 concurrent requests)

  • FreeBSD 8.1 ~ 200 requests/sec, no failure
  • FreeBSD 4.11~ 150 requests/sec, approx. 20% failure.
  • Ubuntu 10.04 ~ 180 requests/sec, no failure
  • OS X 10.3.9 ~ 160 requests/sec, approx. 50% failure.

I am not surprised with the result because FreeBSD 8 really does a very good job in kernel optimization. Since kernel is the heart(or brain) of all computations, an OS with good kernel is expected to give good results always.

So, I finally pick FreeBSD 8 as the core OS of my web farm, and I am loving it.

Our sponsors: