hhvm performance similar to PHP 5.5 Ubuntu 14.04

Question

I must have missed something. But I'm getting the same performance with php and hhvm by running
ab -n 100 -c 10 http://127.0.0.1:8080/

php -v returns:

HipHop VM 3.2.0 (rel)
Compiler: tags/HHVM-3.2.0-0-g01228273b8cf709aacbd3df1c51b1e690ecebac8
Repo schema: c52ba40f4a246d35a88f1dfc1daf959851ced8aa`

tail -3 /var/log/nginx/access.log returns

127.0.0.1 - - [13/Sep/2014:02:46:33 +0300] "GET / HTTP/1.0" 200 116 "-" "ApacheBench/2.3"
127.0.0.1 - - [13/Sep/2014:02:46:33 +0300] "GET / HTTP/1.0" 200 116 "-" "ApacheBench/2.3"
127.0.0.1 - - [13/Sep/2014:02:46:33 +0300] "GET / HTTP/1.0" 200 116 "-" "ApacheBench/2.3"`

nginx conf:

location ~ \.(hh|php)$ {
                fastcgi_keep_conn on;
                fastcgi_pass   127.0.0.1:9000;
                fastcgi_index  index.php;
                fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include        fastcgi_params;
 }

this is my /etc/hhvm/php.ini file:

hhvm.log.level = Warning
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.runtime_error_reporting_level = 8191
hhvm.mysql.typed_results = false
hhvm.eval.jit_warmup_requests = 0
hhvm.eval.jit = true

and this is my /etc/hhvm/server.ini file:

pid = /var/run/hhvm/pid

; hhvm specific

hhvm.server.port = 9000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.eval.jit_warmup_requests = 0
hhvm.eval.jit = true

I made sure to restart hhvm nginx and also rebooted my server.

Monkhmer

Answer 1

I work on the HHVM team, and recently I've been looking at benchmarking. A few problems stand out:

1. hhvm.eval.jit_warmup_requests = 0 - why are you setting this?
2. Are you actually doing any warmup requests? Every jit is slow for a start - I would expect php5 to beat HHVM squarely for the first 10-50 requests.
3. -n 100 -c 10 is fairly low

Benchmarking is very hard; I've been trying to automate a 'good' benchmark suite - you can find the work in progress here: https://github.com/facebook/hhvm/tree/master/hphp/test/frameworks/perf/

It currently only supports wordpress, but I'm hoping to change that soon.

The key things it does:

• runs -n 300 -c 10 as a warmup (using siege instead of ab)
• hits multiple addresses, not just /
• makes sure HHVM is a release build, without asserts, and has the jit turned on
• makes sure PHP5 (or PHP-NG) have the opcode cache turned on
• runs -c 60 for 1 minute for the actual benchmark

Also, what code are you trying to run? It's completely possible that we're actually slower at running your code; for example, we're fast at running mediawiki, but we're slow at running 'print' in a loop, fibonacci, or the other classical benchmarks.

Gerrigerrie

I removed hhvm.eval.jit_warmup_requests = 0 and increased the benchmark to -n 1000 -c 100. The code is a bunch of libraries bootstrap via composer and a few classes I've written on top of it – Monkhmer

Did you run separate warmup requests? That's also not really enough information about the code - is there any chance you could share it? – Gerrigerrie

Can you please elaborate more on how to run separate warm-up requests? If you can link me to a document. I am sorry but I cannot share the code. I can only share its dependencies – Monkhmer

The short version is 'run apachebench/siege two or more times, and ignore the first set of results' :) – Gerrigerrie