Guzzle TransferStats HandlerStats documentation/explanation?

Question

In Guzzle documentation you can find a request option callable called on_stats which allows you to get access to transfer statistics of a request and access the lower level transfer details of the handler associated with your client.

This TransferStats object gives you some methods to inspect your request, one of which is getHandlerStats(). According to the docblock this method provides you with an array of all of the handler specific transfer data.

However i cannot find any documentation for the specific keys of this array. Some are easy offcourse like primary_ip or url but for others i have questions below.

$handlerStats = [
    "url" => "https://example.com",
    "content_type" => "application/json; charset=utf-8",
    "http_code" => 200,
    "header_size" => 569,
    "request_size" => 731,  // is request size purely the body? and is this in bytes or kb or..?
    "filetime" => -1,
    "ssl_verify_result" => 0, // what are the options here?
    "redirect_count" => 0, 
    "total_time" => 0.33132, // is this in seconds, i guess so? this is supposed to be the same as the getTransferTime() method.
    "namelookup_time" => 0.067308, // i assume purely for the dns lookup?
    "connect_time" => 0.078286, // what is this?
    "pretransfer_time" => 0.111673, // what is this? and do some of these times overlap.
    "size_upload" => 0.0, // again are sizes in bytes?
    "size_download" => 7717.0,
    "speed_download" => 23314.0, // is this in bytes per second?
    "speed_upload" => 0.0,
    "download_content_length" => -1.0,
    "upload_content_length" => -1.0,
    "starttransfer_time" => 0.320876,
    "redirect_time" => 0.0,
    "redirect_url" => "",
    "primary_ip" => "1.1.1.1",
    "certinfo" => [],
    "primary_port" => 443,
    "local_ip" => "192.168.208.5",
    "local_port" => 39868,
    "http_version" => 2,
    "protocol" => 2,
    "ssl_verifyresult" => 0, // why is this duplicated, deprecated? 
    "scheme" => "HTTPS",
    "appconnect_time_us" => 111513,
    "connect_time_us" => 78286,
    "namelookup_time_us" => 67308,
    "pretransfer_time_us" => 111673,
    "redirect_time_us" => 0,
    "starttransfer_time_us" => 320876,
    "total_time_us" => 331320,
    "appconnect_time" => 0.111513, // what does the appconnectime do?
  ];

Does anyone have a documentation link or can explain these keys? Many thanks.

Snowfall

doesn't seems too obvious from the key names? Which one is unknown to you? – Prejudice

@Prejudice some are obvious off course indeed, questions are behind the keys in the comments. Like for instance what is appconnect time or pretransfer time? That's why i was looking for official documentation. – Snowfall

that would take a lot of space commenting that here, so I move my next comment to the Answer... – Prejudice

Answer 1

Transfer statistics are delivered by TransferStats class, however the Guzzle documentation does not describe the return keys and values of the statistics. Reading TransferStats comments gives a bit of a clue.

TransferStats are created in three classes:

CurlFactory

private static function invokeStats(EasyHandle $easy): void
{
    $curlStats = \curl_getinfo($easy->handle);
    $curlStats['appconnect_time'] = \curl_getinfo($easy->handle, \CURLINFO_APPCONNECT_TIME);
    $stats = new TransferStats(
        $easy->request,
        $easy->response,
        $curlStats['total_time'],
        $easy->errno,
        $curlStats
    );
    ($easy->options['on_stats'])($stats);
}

MockHandler

private function invokeStats(
        RequestInterface $request,
        array $options,
        ResponseInterface $response = null,
        $reason = null
): void {
    if (isset($options['on_stats'])) {
        $transferTime = $options['transfer_time'] ?? 0;
        $stats = new TransferStats($request, $response, $transferTime, $reason);
        ($options['on_stats'])($stats);
    }
}

StreamHandler

private function invokeStats(
    array $options,
    RequestInterface $request,
    ?float $startTime,
    ResponseInterface $response = null,
    \Throwable $error = null
): void {
    if (isset($options['on_stats'])) {
        $stats = new TransferStats($request, $response, Utils::currentTime() - $startTime, $error, []);
        ($options['on_stats'])($stats);
    }
}

Note from the documentation states:

Guzzle no longer requires cURL in order to send HTTP requests. Guzzle will use the PHP stream wrapper to send HTTP requests if cURL is not installed. Alternatively, you can provide your own HTTP handler used to send requests. Keep in mind that cURL is still required for sending concurrent requests.

Therefore the keys and values you receive from stats will most likely depend on using cURL by Guzzle or not. You may also see at the examples of creation TransferStats that it varies for different Classes (CurlFactory/MockHandler/StramHandler). That's probably why the documentation for the statistics is missing.

Note that the keys and values you may receive from stats will also depend on version of cURL you use and even the PHP version you use. The Changelog of the PHP's curl_getinfo states:

Version     Description
7.3.0       Introduced CURLINFO_CONTENT_LENGTH_DOWNLOAD_T, CURLINFO_CONTENT_LENGTH_UPLOAD_T (...)

so versions of PHP below 7.3.0 should not have any stats like CURLINFO_CONTENT_LENGTH_DOWNLOAD_T and many others.

Caution about expecting any particular keys you may receive is a good idea thus I would always recommend using array_key_exists with the status array of the trasfer.

Assuming that Guzzle uses the cURL for requests then the statistics are taken by PHP's native function curl_getinfo that has described many statistics:

CURLINFO_EFFECTIVE_URL - Last effective URL
CURLINFO_HTTP_CODE - The last response code. As of PHP 5.5.0 and cURL 7.10.8, this is a legacy alias of CURLINFO_RESPONSE_CODE 
(...)

and possible array keys:

"url"
"content_type"
"http_code" 
(...)

however you may notice an inconsistency of ordering descriptions provided by the PHP's predefined constants:

CURLINFO_HTTP_CODE - The last response code. As of PHP 5.5.0 and cURL 7.10.8, this is a legacy alias of CURLINFO_RESPONSE_CODE 

and the list of keys:

"content_type"

CURLINFO_HTTP_CODE is second at the list of the constants and does not match with the second key content_type at the list of returned keys.
Moreover some of the predefined constants like: APPCONNECT_TIME_T do not have their respective names at the list of the returned array keys. Assuming that APPCONNECT_TIME_T has the array key appconnect_time_t is wrong because it is in fact appconnect_time_us.

PHP's doc of curl_getinfo is not extensive nor consistent.

For getting PHP's predefined constants and respective array key names it is better to have a look at the PHP 7.4 source of ext/curl/interface.c code where you may see calls to the native cURL function:

if (curl_easy_getinfo(ch->cp, CURLINFO_HTTP_CODE, &l_code) == CURLE_OK) {
    CAAL("http_code", l_code);
}

and you can easly match CURLINFO_HTTP_CODE that is the second arg of the curl_easy_getinfo with the http_code that is the first arg of CAAL.
This way you can build your own matching list like
[CURLINFO_HTTP_CODE => 'http_code', <PHPConstant> => <array-key>]
from the most reliable source because from the source of the PHP itself.

Because PHP uses the cURL's curl_easy_getinfo then looking at the doc of:

curl_easy_getinfo - extract information from a curl

will bring you even more details, especially interesting are these about TIMES:

An overview of the six time values available from curl_easy_getinfo()

curl_easy_perform()
    |
    |--NAMELOOKUP
    |--|--CONNECT
    |--|--|--APPCONNECT
    |--|--|--|--PRETRANSFER
    |--|--|--|--|--STARTTRANSFER
    |--|--|--|--|--|--TOTAL
    |--|--|--|--|--|--REDIRECT

for example:

CURLINFO_APPCONNECT_TIME_T

Time from start until SSL/SSH handshake completed. See CURLINFO_APPCONNECT_TIME_T

and following that link you can get even more detailed explanation:

(...) This time is most often very near to the CURLINFO_PRETRANSFER_TIME_T time, except for cases such as HTTP pipelining where the pretransfer time can be delayed due to waits in line for the pipeline and more.

When a redirect is followed, the time from each request is added together. (...)

I hope that the whole knowledge combined answers your question and you liked the whole process of digging into it that I revealed instead of just saying X is Y without any proof.

Let's have some fun now :)

running this code:

<?php

error_reporting(E_ALL);
header("Content-Type: text/plain");

$urlPhpSource = 'https://raw.githubusercontent.com/php/php-src/5caaf40b43303887a38d738a9c2a2f4cf6dc9b1a/ext/curl/interface.c';
$phpSourceArr = explode(\PHP_EOL, file_get_contents($urlPhpSource));

$option = '';
$optionNameArr = [];
foreach ($phpSourceArr as $line) {

    $line = trim($line);
    if (preg_match('/curl_easy_getinfo.+(CURLINFO_[A-Z_]+)/', $line, $matches)) {
        $option = $matches[1];
    }

    if (!empty($option) && preg_match('/^CAA.+\"(.+)\"/', $line, $matches)) {
        $optionNameArr[$option] = $matches[1];
        $option = '';
    }
}

$urlDescriptions = 'https://curl.se/libcurl/c/easy_getinfo_options.html';
$descriptionsArr = explode(\PHP_EOL, file_get_contents($urlDescriptions));

$descriptionResultArr = [];
foreach ($descriptionsArr as $line) {
    $pattern = '/(CURLINFO_[A-Z_]+).*';
    $pattern .= '<\/td><td>(.*)<\/td><\/tr>/';
    if (preg_match($pattern, $line, $matches)) {
        $descriptionResultArr[trim($matches[1])] = ucfirst(trim($matches[2]));
    }
}

$resultCombinedArr = [];
foreach($optionNameArr as $option => $name) {
    if (!key_exists($option, $descriptionResultArr)) {
        $desciption = '<not found>';
    } else {
        $desciption = $descriptionResultArr[$option];
    }

    $tmpArr = [];
    $tmpArr['name'] = $name;
    $tmpArr['option'] = $option;
    if (defined($option)) {
        $tmpArr['optionValue'] = constant('\\' . $option);
    }
    $tmpArr['desciption'] = $desciption;
    $resultCombinedArr[$name] = $tmpArr;
}

echo str_repeat("-", 80) . \PHP_EOL . \PHP_EOL;
echo "All Options:\n";
print_r($resultCombinedArr);
echo str_repeat("-", 80) . \PHP_EOL . \PHP_EOL;

echo "Available Options:\n";
print_r(array_filter($resultCombinedArr, function($data) {
    return key_exists('optionValue', $data);
}));
echo str_repeat("-", 80) . \PHP_EOL . \PHP_EOL;

echo "Not available Options:\n";
print_r(array_filter($resultCombinedArr, function($data) {
    return !key_exists('optionValue', $data);
}));

Will give the list of all possible

• predefined constants (option)
• option's int value (optionValue)
• option's respective array key (name)
• option's description (description)

this way:

All Options:
Array
(
    [url] => Array
        (
            [name] => url
            [option] => CURLINFO_EFFECTIVE_URL
            [optionValue] => 1048577
            [desciption] => Get the last used URL
        )

    [content_type] => Array
        (
            [name] => content_type
            [option] => CURLINFO_CONTENT_TYPE
            [optionValue] => 1048594
            [desciption] => Get Content-Type
        )

    [http_code] => Array
        (
            [name] => http_code
            [option] => CURLINFO_HTTP_CODE
            [optionValue] => 2097154
            [desciption] => <not found>
        )

        (...)

Supported by your PHP options:

Available Options:
Array
(
    [url] => Array
        (
            [name] => url
            [option] => CURLINFO_EFFECTIVE_URL
            [optionValue] => 1048577
            [desciption] => Get the last used URL
        )

    [content_type] => Array
        (
            [name] => content_type
            [option] => CURLINFO_CONTENT_TYPE
            [optionValue] => 1048594
            [desciption] => Get Content-Type
        )

    [http_code] => Array
        (
            [name] => http_code
            [option] => CURLINFO_HTTP_CODE
            [optionValue] => 2097154
            [desciption] => <not found>
        )

     (...)

and not supported:

Not available Options:
Array
(
    [appconnect_time_us] => Array
        (
            [name] => appconnect_time_us
            [option] => CURLINFO_APPCONNECT_TIME_T
            [desciption] => Get the time until the SSL/SSH handshake is completed
        )

    [connect_time_us] => Array
        (
            [name] => connect_time_us
            [option] => CURLINFO_CONNECT_TIME_T
            [desciption] => Get the time until connect
        )

    [namelookup_time_us] => Array
        (
            [name] => namelookup_time_us
            [option] => CURLINFO_NAMELOOKUP_TIME_T
            [desciption] => Get the name lookup time in microseconds
        )

    (...)

Note that if a predefined constant like CURLINFO_APPCONNECT_TIME_T is not defined in the PHP the option CURLINFO_APPCONNECT_TIME_T is considered not supported but this does not neccessarly mean that you will get transfer stat without the respective key appconnect_time_us or the key appconnect_time_us will have some undefined value.

the other odd thing is that I ran this scipt for the PHP version:

PHP 7.3.27-9+ubuntu18.04.1+deb.sury.org+1 (cli) (built: Feb 23 2021 15:10:08) ( NTS )

and I got these constants not defined:

Array
(
    [0] => CURLINFO_APPCONNECT_TIME_T
    [1] => CURLINFO_CONNECT_TIME_T
    [2] => CURLINFO_NAMELOOKUP_TIME_T
    [3] => CURLINFO_PRETRANSFER_TIME_T
    [4] => CURLINFO_REDIRECT_TIME_T
    [5] => CURLINFO_STARTTRANSFER_TIME_T
    [6] => CURLINFO_TOTAL_TIME_T
)

that is mutually exclusive to the statement in the documentation of the curl_getinfo that states under the Changelog section all of them should be already introduced for PHP version 7.3.0 and above. Either I did something wrong or the documentation of curl_getinfo proved to be inaccurate again.

However on my own image based on the official

PHP 7.3.8 (cli) (built: Aug 2 2019 05:16:32) ( NTS )

Docker image every option was available.

Here's the generated unofficial documentation for the parameters you can just copy and paste to your project:

<?php

array (
  'url' => 
  array (
    'name' => 'url',
    'option' => 'CURLINFO_EFFECTIVE_URL',
    'optionValue' => 1048577,
    'desciption' => 'Get the last used URL',
  ),
  'content_type' => 
  array (
    'name' => 'content_type',
    'option' => 'CURLINFO_CONTENT_TYPE',
    'optionValue' => 1048594,
    'desciption' => 'Get Content-Type',
  ),
  'http_code' => 
  array (
    'name' => 'http_code',
    'option' => 'CURLINFO_HTTP_CODE',
    'optionValue' => 2097154,
    'desciption' => '<not found>',
  ),
  'header_size' => 
  array (
    'name' => 'header_size',
    'option' => 'CURLINFO_HEADER_SIZE',
    'optionValue' => 2097163,
    'desciption' => 'Get size of retrieved headers',
  ),
  'request_size' => 
  array (
    'name' => 'request_size',
    'option' => 'CURLINFO_REQUEST_SIZE',
    'optionValue' => 2097164,
    'desciption' => 'Get size of sent request',
  ),
  'filetime' => 
  array (
    'name' => 'filetime',
    'option' => 'CURLINFO_FILETIME',
    'optionValue' => 2097166,
    'desciption' => 'Get the remote time of the retrieved document',
  ),
  'ssl_verify_result' => 
  array (
    'name' => 'ssl_verify_result',
    'option' => 'CURLINFO_SSL_VERIFYRESULT',
    'optionValue' => 2097165,
    'desciption' => 'Get the result of the certificate verification',
  ),
  'redirect_count' => 
  array (
    'name' => 'redirect_count',
    'option' => 'CURLINFO_REDIRECT_COUNT',
    'optionValue' => 2097172,
    'desciption' => 'Get the number of redirects',
  ),
  'total_time' => 
  array (
    'name' => 'total_time',
    'option' => 'CURLINFO_TOTAL_TIME',
    'optionValue' => 3145731,
    'desciption' => 'Get total time of previous transfer',
  ),
  'namelookup_time' => 
  array (
    'name' => 'namelookup_time',
    'option' => 'CURLINFO_NAMELOOKUP_TIME',
    'optionValue' => 3145732,
    'desciption' => 'Get the name lookup time',
  ),
  'connect_time' => 
  array (
    'name' => 'connect_time',
    'option' => 'CURLINFO_CONNECT_TIME',
    'optionValue' => 3145733,
    'desciption' => 'Get the time until connect',
  ),
  'pretransfer_time' => 
  array (
    'name' => 'pretransfer_time',
    'option' => 'CURLINFO_PRETRANSFER_TIME',
    'optionValue' => 3145734,
    'desciption' => 'Get the time until the file transfer start',
  ),
  'size_upload' => 
  array (
    'name' => 'size_upload',
    'option' => 'CURLINFO_SIZE_UPLOAD',
    'optionValue' => 3145735,
    'desciption' => 'Get the number of uploaded bytes',
  ),
  'size_download' => 
  array (
    'name' => 'size_download',
    'option' => 'CURLINFO_SIZE_DOWNLOAD',
    'optionValue' => 3145736,
    'desciption' => 'Get the number of downloaded bytes',
  ),
  'speed_download' => 
  array (
    'name' => 'speed_download',
    'option' => 'CURLINFO_SPEED_DOWNLOAD',
    'optionValue' => 3145737,
    'desciption' => 'Get download speed',
  ),
  'speed_upload' => 
  array (
    'name' => 'speed_upload',
    'option' => 'CURLINFO_SPEED_UPLOAD',
    'optionValue' => 3145738,
    'desciption' => 'Get upload speed',
  ),
  'download_content_length' => 
  array (
    'name' => 'download_content_length',
    'option' => 'CURLINFO_CONTENT_LENGTH_DOWNLOAD',
    'optionValue' => 3145743,
    'desciption' => 'Get content-length of download',
  ),
  'upload_content_length' => 
  array (
    'name' => 'upload_content_length',
    'option' => 'CURLINFO_CONTENT_LENGTH_UPLOAD',
    'optionValue' => 3145744,
    'desciption' => 'Get the specified size of the upload',
  ),
  'starttransfer_time' => 
  array (
    'name' => 'starttransfer_time',
    'option' => 'CURLINFO_STARTTRANSFER_TIME',
    'optionValue' => 3145745,
    'desciption' => 'Get the time until the first byte is received',
  ),
  'redirect_time' => 
  array (
    'name' => 'redirect_time',
    'option' => 'CURLINFO_REDIRECT_TIME',
    'optionValue' => 3145747,
    'desciption' => 'Get the time for all redirection steps',
  ),
  'redirect_url' => 
  array (
    'name' => 'redirect_url',
    'option' => 'CURLINFO_REDIRECT_URL',
    'optionValue' => 1048607,
    'desciption' => 'Get the URL a redirect would go to',
  ),
  'primary_ip' => 
  array (
    'name' => 'primary_ip',
    'option' => 'CURLINFO_PRIMARY_IP',
    'optionValue' => 1048608,
    'desciption' => 'Get IP address of last connection',
  ),
  'certinfo' => 
  array (
    'name' => 'certinfo',
    'option' => 'CURLINFO_CERTINFO',
    'optionValue' => 4194338,
    'desciption' => 'Get the TLS certificate chain',
  ),
  'primary_port' => 
  array (
    'name' => 'primary_port',
    'option' => 'CURLINFO_PRIMARY_PORT',
    'optionValue' => 2097192,
    'desciption' => 'Get the latest destination port number',
  ),
  'local_ip' => 
  array (
    'name' => 'local_ip',
    'option' => 'CURLINFO_LOCAL_IP',
    'optionValue' => 1048617,
    'desciption' => 'Get local IP address of last connection',
  ),
  'local_port' => 
  array (
    'name' => 'local_port',
    'option' => 'CURLINFO_LOCAL_PORT',
    'optionValue' => 2097194,
    'desciption' => 'Get the latest local port number',
  ),
  'http_version' => 
  array (
    'name' => 'http_version',
    'option' => 'CURLINFO_HTTP_VERSION',
    'optionValue' => 2097198,
    'desciption' => 'Get the http version used in the connection',
  ),
  'protocol' => 
  array (
    'name' => 'protocol',
    'option' => 'CURLINFO_PROTOCOL',
    'optionValue' => 2097200,
    'desciption' => 'Get the protocol used in the connection',
  ),
  'ssl_verifyresult' => 
  array (
    'name' => 'ssl_verifyresult',
    'option' => 'CURLINFO_PROXY_SSL_VERIFYRESULT',
    'optionValue' => 2097199,
    'desciption' => 'Get the result of the proxy certificate verification',
  ),
  'scheme' => 
  array (
    'name' => 'scheme',
    'option' => 'CURLINFO_SCHEME',
    'optionValue' => 1048625,
    'desciption' => 'Get the URL scheme (sometimes called protocol) used in the connection',
  ),
  'appconnect_time_us' => 
  array (
    'name' => 'appconnect_time_us',
    'option' => 'CURLINFO_APPCONNECT_TIME_T',
    'optionValue' => 6291512,
    'desciption' => 'Get the time until the SSL/SSH handshake is completed',
  ),
  'connect_time_us' => 
  array (
    'name' => 'connect_time_us',
    'option' => 'CURLINFO_CONNECT_TIME_T',
    'optionValue' => 6291508,
    'desciption' => 'Get the time until connect',
  ),
  'namelookup_time_us' => 
  array (
    'name' => 'namelookup_time_us',
    'option' => 'CURLINFO_NAMELOOKUP_TIME_T',
    'optionValue' => 6291507,
    'desciption' => 'Get the name lookup time in microseconds',
  ),
  'pretransfer_time_us' => 
  array (
    'name' => 'pretransfer_time_us',
    'option' => 'CURLINFO_PRETRANSFER_TIME_T',
    'optionValue' => 6291509,
    'desciption' => 'Get the time until the file transfer start',
  ),
  'redirect_time_us' => 
  array (
    'name' => 'redirect_time_us',
    'option' => 'CURLINFO_REDIRECT_TIME_T',
    'optionValue' => 6291511,
    'desciption' => 'Get the time for all redirection steps',
  ),
  'starttransfer_time_us' => 
  array (
    'name' => 'starttransfer_time_us',
    'option' => 'CURLINFO_STARTTRANSFER_TIME_T',
    'optionValue' => 6291510,
    'desciption' => 'Get the time until the first byte is received',
  ),
  'total_time_us' => 
  array (
    'name' => 'total_time_us',
    'option' => 'CURLINFO_TOTAL_TIME_T',
    'optionValue' => 6291506,
    'desciption' => 'Get total time of previous transfer in microseconds',
  ),
)

Scanner

This is brilliant and more then what i was looking for. Cool to see that you can actually test per environment which keys are available and what they represent. At least i now know how to interpet the keys which was the goals for this question and bounty. – Snowfall

PS: if you scroll down on the curl_getinfo docs there is also a list of all the available ssl_verify_result options. Nice:-) php.net/manual/en/function.curl-getinfo.php#111678 – Snowfall

Answer 2

appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed

pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.

... well I'm not sure about the others, Could you share which keys you want to describe ?

Prejudice