How can i grab CData out of BeautifulSoup

Asked 9/1, 2010 at 2:53 Answered 14/3, 2019 at 21:15

python screen-scraping beautifulsoup cdata

I have a website that I'm scraping that has a similar structure the following. I'd like to be able to grab the info out of the CData block.

I'm using BeautifulSoup to pull other info off the page, so if the solution can work with that, it would help keep my learning curve down as I'm a python novice. Specifically, I want to get at the two different types of data hidden in the CData statement. the first which is just text I'm pretty sure I can throw a regex at it and get what I need. For the second type, if i could drop the data that has html elements into it's own beautifulsoup, I can parse that.

I'm just learning python and beautifulsoup, so I'm struggling to find the magical incantation that will give me just the CData by itself.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">  
<head>  
<title>
   Cows and Sheep
  </title>
</head>
<body>
 <div id="main">
  <div id="main-precontents">
   <div id="main-contents" class="main-contents">
    <script type="text/javascript">
       //<![CDATA[var _ = g_cow;_[7654]={cowname_enus:'cows rule!',leather_quality:99,icon:'cow_level_23'};_[37357]={sheepname_enus:'baa breath',wool_quality:75,icon:'sheep_level_23'};_[39654].cowmeat_enus = '<table><tr><td><b class="q4">cows rule!</b><br></br>
       <!--ts-->
       get it now<table width="100%"><tr><td>NOW</td><th>NOW</th></tr></table><span>244 Cows</span><br></br>67 leather<br></br>68 Brains
       <!--yy-->
       <span class="q0">Cow Bonus: +9 Cow Power</span><br></br>Sheep Power 60 / 60<br></br>Sheep 88<br></br>Cow Level 555</td></tr></table>
       <!--?5695:5:40:45-->
       ';
        //]]>
      </script>
     </div>
     </div>
    </div>
 </body>
</html>

Taima answered 9/1, 2010 at 2:53 Comment(2)

Ouch, that's a desperately malformed script block! If that's the real markup, it won't actually work anywhere, neither XHTML nor HTML... – Hyksos 9/1, 2010 at 4:23

it's not real, i wanted to condense a much much larger block. guess i ripped out too much. – Taima 9/1, 2010 at 12:41

One thing you need to be careful of BeautifulSoup grabbing CData is not to use a lxml parser.

By default, the lxml parser will strip CDATA sections from the tree and replace them by their plain text content, Learn more here

#Trying it with html.parser


>>> from bs4 import BeautifulSoup
>>> import bs4
>>> s='''<?xml version="1.0" ?>
<foo>
    <bar><![CDATA[
        aaaaaaaaaaaaa
    ]]></bar>
</foo>'''
>>> soup = BeautifulSoup(s, "html.parser")
>>> soup.find(text=lambda tag: isinstance(tag, bs4.CData)).string.strip()
'aaaaaaaaaaaaa'
>>>

Hamblin answered 28/12, 2016 at 14:58 Comment(1)

thank you! I did something very similar in the end :) – Zohara 26/10, 2022 at 13:35

BeautifulSoup sees CData as a special case (subclass) of "navigable strings". So for example:

import BeautifulSoup

txt = '''<foobar>We have
       <![CDATA[some data here]]>
       and more.
       </foobar>'''

soup = BeautifulSoup.BeautifulSoup(txt)
for cd in soup.findAll(text=True):
  if isinstance(cd, BeautifulSoup.CData):
    print 'CData contents: %r' % cd

In your case of course you could look in the subtree starting at the div with the 'main-contents' ID, rather than all over the document tree.

Kailakaile answered 9/1, 2010 at 3:31 Comment(4)

thanks. this will do nicely, it even cleaned off the starting and end <![CDATA //]]> bits. i had tried BeautifulSoup.CData before, but it didn't work for me. I was getting the following error: "AttributeError: class BeautifulSoup has no attribute 'CData'" guess i needed "import BeautifulSoup" instead of "from BeautifulSoup import BeautifulSoup". – Taima 9/1, 2010 at 12:47

@hary, yes, this kind of thing is part of why I recommend always importing the module (import BeautifulSoup) rather than bits and pieces from within it!-) – Kailakaile 9/1, 2010 at 15:25

It seems this approach only works for CDATA tags that have not been commented out. In the original question's example, the CDATA would not be found. – Lacking 13/10, 2017 at 21:51

node.find_all(text=True, recursive=False) can give a better accuracy on what you find. – Hagy 11/8, 2022 at 14:22

You could try this:

from BeautifulSoup import BeautifulSoup

// source.html contains your html above
f = open('source.html')
soup = BeautifulSoup(''.join(f.readlines()))
s = soup.findAll('script')
cdata = s[0].contents[0]

That should give you the contents of cdata.

Update

This may be a little cleaner:

from BeautifulSoup import BeautifulSoup
import re

// source.html contains your html above
f = open('source.html')
soup = BeautifulSoup(''.join(f.readlines()))
cdata = soup.find(text=re.compile("CDATA"))

Just personal preference, but I like the bottom one a little better.

Dextrosinistral answered 9/1, 2010 at 3:19 Comment(1)

thanks for the response, this website is a vast wealth of knowledge – Taima 9/1, 2010 at 12:50

import re
from bs4 import BeautifulSoup

soup = BeautifulSoup(content)
for x in soup.find_all('item'):
    print re.sub('[\[CDATA\]]', '', x.string)

Aedes answered 20/5, 2014 at 8:16 Comment(0)

For anyone using BeautifulSoup4, Alex Martelli's solution works but do this:

from bs4 import BeautifulSoup, CData

soup = BeautifulSoup(txt)
for cd in soup.findAll(text=True):
  if isinstance(cd, Cdata):
    print 'CData contents: %r' % cd

Troup answered 14/3, 2019 at 21:15 Comment(0)

Recommended topics

Hot tags