Can Goutte/Guzzle be forced into UTF-8 mode?

Asked 13/9, 2013 at 9:3 Answered 3/12, 2022 at 18:23

Solved php web-scraping guzzle goutte symfony-components

I'm scraping from a UTF-8 site, using Goutte, which internally uses Guzzle. The site declares a meta tag of UTF-8, thus:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

However, the content type header is thus:

Content-Type: text/html

and not:

Content-Type: text/html; charset=utf-8

Thus, when I scrape, Goutte does not spot that it is UTF-8, and grabs data incorrectly. The remote site is not under my control, so I can't fix the problem there! Here's a set of scripts to replicate the problem. First, the scraper:

<?php

require_once realpath(__DIR__ . '/..') . '/vendor/goutte/goutte.phar';

$url = 'http://crawler-tests.local/utf-8.php';
use Goutte\Client;

$client = new Client();
$crawler = $client->request('get', $url);
$text = $crawler->text();
echo 'Whole page: ' . $text . "\n";

Now a test page to be placed on a web server:

<?php
// Correct
#header('Content-Type: text/html; charset=utf-8');

// Incorrect
header('Content-Type: text/html');
?>  
<!DOCTYPE html>
<html>
    <head>
        <title>UTF-8 test</title>
        <meta charset="utf-8" />
    </head>
    <body>
        <p>When the Content-Header header is incomplete, the pound sign breaks:

        £15,216</p>
    </body>
</html>

Here's the output of the Goutte test:

Whole page: UTF-8 test When the Content-Header header is incomplete, the pound sign breaks: Â£15,216

As you can see from the comments in the last script, properly declaring the character set in the header fixes things. I've hunted around in Goutte to see if there is anything that looks like it would force the character set, but to no avail. Any ideas?

Biaxial answered 13/9, 2013 at 9:3 Comment(1)

What problem are you talking about exactly? How does it manifest? – Righthander 13/9, 2013 at 17:11

The issue is actually with symfony/browser-kit and symfony/domcrawler. The browserkit's Client does not examine the HTML meta tags to determine the charset, content-type header only. When the response body is handed over to the domcrawler, it is treated as the default charset ISO-8859-1. After examining the meta tags that decision should be reverted and the DomDocument rebuilt, but that never happens.

The easy workaround is to wrap $crawler->text() with utf8_decode():

$text = utf8_decode($crawler->text());

This works if the input is UTF-8. I suppose for other encodings something similar can be achieved with iconv() or so. However, you have to remember to do that every time you call text().

A more generic approach is to make the Domcrawler believe that it deals with UTF-8. To that end I've come up with a Guzzle plugin that overwrites (or adds) the charset in the content-type response header. You can find it at https://gist.github.com/pschultz/6554265. Usage is like this:

<?php

use Goutte\Client;


$plugin = new ForceCharsetPlugin();
$plugin->setForcedCharset('utf-8');

$client = new Client();
$client->getClient()->addSubscriber($plugin);
$crawler = $client->request('get', $url);

echo $crawler->text();

Righthander answered 13/9, 2013 at 19:10 Comment(8)

This looks great, Peter - many thanks. I'll give it a whirl tomorrow and will let you know how I get on! I've updated the question with the faulty output of the test I wrote - the £ sign is corrupted. – Biaxial 13/9, 2013 at 23:38

Just tried this in standalone mode, works fine. Again, thanks! I was thinking of using the workaround you suggested, but it seemed somewhat inelegant - your plugin is much nicer. – Biaxial 14/9, 2013 at 12:17

Symfony 2.3.5 contains a commit Crawler guess charset from html. That seems to tackle the issue. – Righthander 30/9, 2013 at 12:31

Great, appreciated. I think it'll take a little longer to reach Goutte, but I'll keep an eye on it. – Biaxial 30/9, 2013 at 12:40

Do subsequent releases of symfony/browser-kit and symfony/domcrawler address the issue? – Apropos 23/3, 2015 at 13:51

As mentioned above this is fixed starting with v2.3.5 and has also been backported to v2.2.7+ – Righthander 23/3, 2015 at 17:27

When I run this ,I am getting error saying PHP Fatal error: Class App\Console\Commands\ForceCharsetPlugin not found and [Symfony\Component\Debug\Exception\FatalErrorException] Class App\Console\Commands\ForceCharsetPlugin not found – Thorndike 25/6, 2016 at 17:34

@MohdAbrarAhmed: you'd need to include that class or add it to your autoloading system. – Biaxial 10/9, 2016 at 9:54

I seem to have been hitting two bugs here, one of which was identified by Peter's answer. The other was the way in which I am separately using the Symfony Crawler class to explore HTML snippets.

I was doing this (to parse the HTML for a table row):

$subCrawler = new Crawler($rowHtml);

Adding HTML via the constructor, however, does not appear to give a way in which the character set can be specified, and I assume ISO-8859-1 is again the default.

Simply using addHtmlContent gets it right; the second parameter specifies the character set, and it defaults to UTF-8 if it is not specified.

$subCrawler = new Crawler();
$subCrawler->addHtmlContent($rowHtml);

Biaxial answered 14/9, 2013 at 13:39 Comment(2)

holy crap ... you saved me from killing my self ... I was figuring this out for whole day – Pontificals 10/9, 2016 at 9:33

No worries @Umair, pleased it was of assistance! – Biaxial 10/9, 2016 at 9:53

Crawler tries detect charset from <meta charset tag but frequently it's missing and then Crawler uses charset by default (ISO-8859-1) - it is source of problem described in this thread.

When we are passing content to Crawler through constructor we miss Content-Type header that usually contains charset.

Here's how we can handle it:

$crawler = new Crawler();
$crawler->addContent(
    $response->getBody()->getContents(), 
    $response->getHeaderLine('Content-Type')
);

With this solution we are using correct charset from server response and don't bind our solution to any single charset and of course after that we don't need decode every single received line from Crawler (using utf8_decode() or somehow else).

Thracian answered 21/11, 2017 at 11:25 Comment(0)

Guzzle is true about what it gets, so the best way is to do the conversion like this:

  // $client = \Drupal::httpClient();
  $client = new \GuzzleHttp\Client();
  $response = $client->get($remoteUrl);
  if ($response->getStatusCode() !== 200) {
    return NULL;
  }
  $originalBody = $response->getBody()->getContents();
  $contentTypeHeader = $response->getHeader('content-type');
  $originalEncoding = \GuzzleHttp\Psr7\Header::parse($contentTypeHeader)[0]['charset'] ?? NULL;
  $body = !$originalEncoding ? $originalBody :
    mb_convert_encoding($originalBody, 'UTF-8', $originalEncoding);

Of course if the response lies about its encoding, you're lost until you work around or fix that.

Merger answered 3/12, 2022 at 18:23 Comment(0)

Recommended topics

Hot tags