How to download any(!) webpage with correct charset in python?

Question

Problem

When screen-scraping a webpage using python one has to know the character encoding of the page. If you get the character encoding wrong than your output will be messed up.

People usually use some rudimentary technique to detect the encoding. They either use the charset from the header or the charset defined in the meta tag or they use an encoding detector (which does not care about meta tags or headers). By using only one these techniques, sometimes you will not get the same result as you would in a browser.

Browsers do it this way:

• Meta tags always takes precedence (or xml definition)
• Encoding defined in the header is used when there is no charset defined in a meta tag
• If the encoding is not defined at all, than it is time for encoding detection.

(Well... at least that is the way I believe most browsers do it. Documentation is really scarce.)

What I'm looking for is a library that can decide the character set of a page the way a browser would. I'm sure I'm not the first who needs a proper solution to this problem.

Solution (I have not tried it yet...)

According to Beautiful Soup's documentation.

Beautiful Soup tries the following encodings, in order of priority, to turn your document into Unicode:

• An encoding you pass in as the fromEncoding argument to the soup constructor.
• An encoding discovered in the document itself: for instance, in an XML declaration or (for HTML documents) an http-equiv META tag. If Beautiful Soup finds this kind of encoding within the document, it parses the document again from the beginning and gives the new encoding a try. The only exception is if you explicitly specified an encoding, and that encoding actually worked: then it will ignore any encoding it finds in the document.
• An encoding sniffed by looking at the first few bytes of the file. If an encoding is detected at this stage, it will be one of the UTF-* encodings, EBCDIC, or ASCII.
• An encoding sniffed by the chardet library, if you have it installed.
• UTF-8
• Windows-1252

Vicegerent

You can't download "any" page with a correct character set. Browsers guess wrong all the time, when the correct charset isn't specified. I use the view->encoding menu in FF to fix incorrect guesses on a daily basis. You want to do as well as you can, but give up on guessing every page correctly. – Darr

Guessing character sets is evil and has got us into this mess in the first place. If the browsers had never attempted to guess, developers would be forced to learn about HTTP headers and always specify the encoding properly. Guessing means sometime you are going to get it wrong – Tonina

gnibbler, guessing is a last resort – Paphian

This may be helpful: https://mcmap.net/q/393558/-what-is-a-nice-reliable-short-way-to-get-the-charset-of-a-webpage – Tibbs

Answer 1

I would use html5lib for this.

Bothnia

This looks really nice. Documentation about how it does its encoding discovery: html5lib.readthedocs.org/en/latest/… – Paphian

Answer 2

When you download a file with urllib or urllib2, you can find out whether a charset header was transmitted:

fp = urllib2.urlopen(request)
charset = fp.headers.getparam('charset')

You can use BeautifulSoup to locate a meta element in the HTML:

soup = BeatifulSoup.BeautifulSoup(data)
meta = soup.findAll('meta', {'http-equiv':lambda v:v.lower()=='content-type'})

If neither is available, browsers typically fall back to user configuration, combined with auto-detection. As rajax proposes, you could use the chardet module. If you have user configuration available telling you that the page should be Chinese (say), you may be able to do better.

Glazer

@kaizer.se: right; it's get_param in 3.x (but then, it's also urllib.request) – Ptolemaic

Unfortunately (at least in Python 2.7) urllib2 doesn't parse out charset from the Content-Type header, so you'll need to do something like the answer in https://mcmap.net/q/25702/-urllib2-read-to-unicode – Etalon

It is close, but still have a few pieces missing - BOM marks are not taken in account, it is not said how to resolve HTTP header and meta tag ambiguity; encoding names defined in HTTP headers and meta tags don't match names supported by Python stdlib. Using a library function which does all of that (like w3lib.encoding.html_to_unicode) instead of trying to get it right manually is usually a better idea. – Moonstone

Answer 3

Use the Universal Encoding Detector:

>>> import chardet
>>> chardet.detect(urlread("http://google.cn/"))
{'encoding': 'GB2312', 'confidence': 0.99}

The other option would be to just use wget:

  import os
  h = os.popen('wget -q -O foo1.txt http://foo.html')
  h.close()
  s = open('foo1.txt').read()

Scifi

This is no good as it fails sometimes. Also see: chardet.feedparser.org/docs/faq.html#faq.yippie (Yippie!) – Paphian

The main problem with this approach that you ignore the page's explicitly specified character encoding. – Paphian

Ok, then there isn't a silver bullet here I'm afraid - so write it yourself. :) – Scifi

For example chardet detects origo.hu as ISO-8859-8 while that page is actually ISO-8859-2 as defined by a meta tag. That would mess things up badly. – Paphian

Well... and how is foo1.txt encoded? :) The same way the webpage was encoded and we still don't what the encoding is. – Paphian

And you've lost data, since the very first place to look is the Content-Type HTTP header, which was thrown away. – Darr

@Kalmi: You link to the chardet faq; less than 10 lines down, he links to feedparser, which does what you want: code.google.com/p/feedparser/source/browse/trunk/feedparser/… (Granted, he only handles xml files, but 90% of the machinery you need is in there...) – Malley

@Vicegerent - There simply doesn't exist a solution that works every time, since many byte sequences can appear in many encodings. – Inconformity

Stobor, Your answer(um... comment...) is the best so far. :) – Paphian

Answer 4

It seems like you need a hybrid of the answers presented:

1. Fetch the page using urllib
2. Find <meta> tags using beautiful soup or other method
3. If no meta tags exist, check the headers returned by urllib
4. If that still doesn't give you an answer, use the universal encoding detector.

I honestly don't believe you're going to find anything better than that.

In fact if you read further into the FAQ you linked to in the comments on the other answer, that's what the author of detector library advocates.

If you believe the FAQ, this is what the browsers do (as requested in your original question) as the detector is a port of the firefox sniffing code.

Schipperke

What I find odd is that there is no existing library/snippet for this. – Paphian

Stobor pointed out the existence of feedparser.py (which is unfortunately only for XML), but contains most of the things I need. – Paphian

The algorithm is not correct, as HTTP headers should take precedance over meta tags. It also misses BOM marks and an encoding normalisation step (encoding names in HTML/HTTP are not the same as names provided by Python). – Moonstone

Answer 5

Scrapy downloads a page and detects a correct encoding for it, unlike requests.get(url).text or urlopen. To do so it tries to follow browser-like rules - this is the best one can do, because website owners have incentive to make their websites work in a browser. Scrapy needs to take HTTP headers, <meta> tags, BOM marks and differences in encoding names in account.

Content-based guessing (chardet, UnicodeDammit) on its own is not a correct solution, as it may fail; it should be only used as a last resort when headers or <meta> or BOM marks are not available or provide no information.

You don't have to use Scrapy to get its encoding detection functions; they are released (among with some other stuff) in a separate library called w3lib: https://github.com/scrapy/w3lib.

To get page encoding and unicode body use w3lib.encoding.html_to_unicode function, with a content-based guessing fallback:

import chardet
from w3lib.encoding import html_to_unicode

def _guess_encoding(data):
    return chardet.detect(data).get('encoding')

detected_encoding, html_content_unicode = html_to_unicode(
    content_type_header,
    html_content_bytes,
    default_encoding='utf8', 
    auto_detect_fun=_guess_encoding,
)

Moonstone

Answer 6

BeautifulSoup dose this with UnicodeDammit : Unicode, Dammit

Stasiastasis

Answer 7

instead of trying to get a page then figuring out the charset the browser would use, why not just use a browser to fetch the page and check what charset it uses..

from win32com.client import DispatchWithEvents
import threading


stopEvent=threading.Event()

class EventHandler(object):
    def OnDownloadBegin(self):
        pass

def waitUntilReady(ie):
    """
    copypasted from
    http://mail.python.org/pipermail/python-win32/2004-June/002040.html
    """
    if ie.ReadyState!=4:
        while 1:
            print "waiting"
            pythoncom.PumpWaitingMessages()
            stopEvent.wait(.2)
            if stopEvent.isSet() or ie.ReadyState==4:
                stopEvent.clear()
                break;

ie = DispatchWithEvents("InternetExplorer.Application", EventHandler)
ie.Visible = 0
ie.Navigate('http://kskky.info')
waitUntilReady(ie)
d = ie.Document
print d.CharSet

Abase

just tested this on origo.hu and it works, albeit incredibly slowly - maybe try with the firefox activex component instead – Abase