How to preserve namespace information when parsing HTML with lxml?

Asked 6/7, 2011 at 13:34 Answered 14/4, 2015 at 6:4

python html lxml facebook-like xml-namespaces

>>> from lxml.etree import HTML, tostring
>>> tostring(HTML('<fb:like>'))
'<html><body><like/></body></html>'

Note how the tag turns from <fb:like> to simply <like>.

This makes processing pages that incorporate XFBML with lxml much harder. (Same thing happens to <g:plusone></g:plusone>)

Any help is appreciated.

Softy answered 6/7, 2011 at 13:34 Comment(2)

HTML doesn't have namespaces. Welcome to the consequences of expando-elements. If it is XHTML + FBML (instead of HTML + FBML) then you could use an XML parser. – Heliochrome 6/7, 2011 at 15:21

I would have, if it was XML. The point of resilient HTML parsers is that they get the job done. While I don't like these magic tag names either, I think it's reasonable to expect from lxml to preserve them. I'll try to talk to the maintainers to see if this can be accommodated. – Softy 6/7, 2011 at 16:9

One way to fix this issue is to patch libxml2.

Referring to the source code of libxml2.9.2 (https: //git.gnome.org/browse/libxml2/tree/?id=v2.9.2), in SAX2.c (https: //git.gnome.org/browse/libxml2/tree/SAX2.c?id=v2.9.2) (the internal SAX parser used to create the DOM tree) at line 1699 attributes with xmlns are not parsed when in HTML mode, and they are parsed like any other attributes at line and 1740. Consequently, it makes sense to adjust line 1622, which splits the name into prefix and local part. Change:

name = xmlSplitQName(ctxt, fullname, &prefix);

into

if (!ctxt->html) {
    name = xmlSplitQName(ctxt, fullname, &prefix);
} else {
    name = xmlStrdup(fullname);
    prefix = NULL;
}

Then libxml2 will consider tags such as <o:p> to be for elements with name o:p, that is, the colon is included in the element name with no special meaning. This is the correct interpretation in HTML. For example, the HTML5 specification says:

In the HTML syntax, namespace prefixes and namespace declarations do not have the same effect as in XML. For instance, the colon has no special meaning in HTML element names.

Hopefully this change will be approved for a future version of libxml2. There is an open bug report (https: //bugzilla.gnome.org/show_bug.cgi?id=654146).

Mozell answered 14/4, 2015 at 6:4 Comment(2)

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – Laundromat 14/4, 2015 at 6:41

@Laundromat OK, have added details here. Sorry about the ugly links. I don't have sufficient reputation to include them as real links. – Mozell 15/4, 2015 at 5:51

Try adding the namespace prefix definitions that are missing. lxml will avoid the namespaces otherwise, supposedly to make it easier for you.

Most likely the sites you try to parse will not contain these namespace definitions, so you should add them.

Something like this: xmlns:adlcp="http://xxx/yy/zzz"

Goshen answered 6/7, 2011 at 15:25 Comment(0)

Recommended topics

Hot tags