from mechanize import Browser
br = Browser()
br.open('http://somewebpage')
html = br.response().readlines()
for line in html:
print line
When printing a line in an HTML file, I'm trying to find a way to only show the contents of each HTML element and not the formatting itself. If it finds '<a href="whatever.example">some text</a>', it will only print 'some text', '<b>hello</b>' prints 'hello', etc. How would one go about doing this?
I always used this function to strip HTML tags, as it requires only the Python stdlib:
For Python 3:
from io import StringIO
from html.parser import HTMLParser
class MLStripper(HTMLParser):
def __init__(self):
super().__init__()
self.reset()
self.strict = False
self.convert_charrefs= True
self.text = StringIO()
def handle_data(self, d):
self.text.write(d)
def get_data(self):
return self.text.getvalue()
def strip_tags(html):
s = MLStripper()
s.feed(html)
return s.get_data()
For Python 2:
from HTMLParser import HTMLParser
from StringIO import StringIO
class MLStripper(HTMLParser):
def __init__(self):
self.reset()
self.text = StringIO()
def handle_data(self, d):
self.text.write(d)
def get_data(self):
return self.text.getvalue()
def strip_tags(html):
s = MLStripper()
s.feed(html)
return s.get_data()
&) as well as tags. –
Bassarisk return ''.join(self.fed) into return self.fed. Now it's perfect. –
Campbellite __init__ function. See here: #11061558. –
Lotuseater entityref = re.compile('&([a-zA-Z][-.a-zA-Z0-9]*)[^a-zA-Z0-9]') in Python 2.7. –
Fafnir parser = HTMLParser() and html = parser.unescape(html) to the beginning of the strip_tags function. –
Novgorod <<sc<script>script>alert(1)<</sc</script>/script>. If you pass that through this code, the output will be <script>alert(1)</script>. To be sure, I wrapped your solution with html.escape() to make sure no tags are left in the output. –
Horsa get data to return ' '.join([x for x in self.fed if not x.isspace()]) –
Rammer <p>...</p><p>...</p> from getting mushed together? –
Gearard <<b>b>Bold!<</b>/b> becomes <b>Bold!</b>. –
Parthinia If you need to strip HTML tags to do text processing, a simple regular expression will do. Do not use this if you are looking to sanitize user-generated HTML to prevent XSS attacks. It is not a secure way to remove all <script> tags or tracking <img>s. The following regular expression will fairly reliably strip most HTML tags:
import re
re.sub('<[^<]+?>', '', text)
For those that don't understand regex, this searches for a string <...>, where the inner content is made of one or more (+) characters that isn't a <. The ? means that it will match the smallest string it can find. For example given <p>Hello</p>, it will match <'p> and </p> separately with the ?. Without it, it will match the entire string <..Hello..>.
If non-tag < appears in html (eg. 2 < 3), it should be written as an escape sequence &... anyway so the ^< may be unnecessary.
&) unchanged in the output. –
Bassarisk < as < to make sure it's actually safe. –
Kweisui cgi.escape(s, True)), even if you "know" that it doesn't contain HTML (e.g. because you stripped HTML content). However, this is not what OP asked about. –
Bassarisk [^>] instead of [^<] make it a little better –
Grinder import re –
Tula cgi.escape(s, True))" -- cgi.escape: "Deprecated since version 3.2.... Use html.escape() instead." Removed in 3.8 -- so for 3.8+, use html.escape(). –
N You can use BeautifulSoup get_text() feature.
from bs4 import BeautifulSoup
html_str = '''
<td><a href="http://www.fakewebsite.example">Please can you strip me?</a>
<br/><a href="http://www.fakewebsite.example">I am waiting....</a>
</td>
'''
soup = BeautifulSoup(html_str)
print(soup.get_text())
#or via attribute of Soup Object: print(soup.text)
It is advisable to explicitly specify the parser, for example as BeautifulSoup(html_str, features="html.parser"), for the output to be reproducible.
import re, html
tag_re = re.compile(r'(<!--.*?-->|<[^>]*>)')
# Remove well-formed tags, fixing mistakes by legitimate users
no_tags = tag_re.sub('', user_input)
# Clean up anything else by escaping
ready_for_web = html.escape(no_tags)
Regex source: MarkupSafe. Their version handles HTML entities too, while this quick one doesn't.
It's one thing to keep people from <i>italicizing</i> things, without leaving is floating around. But it's another to take arbitrary input and make it completely harmless. Most of the techniques on this page will leave things like unclosed comments (<!--) and angle-brackets that aren't part of tags (blah <<<><blah) intact. The HTMLParser version can even leave complete tags in, if they're inside an unclosed comment.
What if your template is {{ firstname }} {{ lastname }}? firstname = '<a' and lastname = 'href="http://evil.example/">' will be let through by every tag stripper on this page (except @Medeiros!), because they're not complete tags on their own. Stripping out normal HTML tags is not enough.
Django's strip_tags, an improved (see next heading) version of the top answer to this question, gives the following warning:
Absolutely NO guarantee is provided about the resulting string being HTML safe. So NEVER mark safe the result of a
strip_tagscall without escaping it first, for example withescape().
Follow their advice!
It's easy to circumvent the top answer to this question.
Look at this string (source and discussion):
<img<!-- --> src=x onerror=alert(1);//><!-- -->
The first time HTMLParser sees it, it can't tell that the <img...> is a tag. It looks broken, so HTMLParser doesn't get rid of it. It only takes out the <!-- comments -->, leaving you with
<img src=x onerror=alert(1);//>
This problem was disclosed to the Django project in March, 2014. Their old strip_tags was essentially the same as the top answer to this question. Their new version basically runs it in a loop until running it again doesn't change the string:
# _strip_once runs HTMLParser once, pulling out just the text of all the nodes.
def strip_tags(value):
"""Returns the given HTML with all tags stripped."""
# Note: in typical case this loop executes _strip_once once. Loop condition
# is redundant, but helps to reduce number of executions of _strip_once.
while '<' in value and '>' in value:
new_value = _strip_once(value)
if len(new_value) >= len(value):
# _strip_once was not able to detect more tags
break
value = new_value
return value
Of course, none of this is an issue if you always escape the result of strip_tags().
Update 19 March, 2015: There was a bug in Django versions before 1.4.20, 1.6.11, 1.7.7, and 1.8c1. These versions could enter an infinite loop in the strip_tags() function. The fixed version is reproduced above. More details here.
My example code doesn't handle HTML entities - the Django and MarkupSafe packaged versions do.
My example code is pulled from the excellent MarkupSafe library for cross-site scripting prevention. It's convenient and fast (with C speedups to its native Python version). It's included in Google App Engine, and used by Jinja2 (2.7 and up), Mako, Pylons, and more. It works easily with Django templates from Django 1.7.
Django's strip_tags and other HTML utilities from a recent version are good, but I find them less convenient than MarkupSafe. They're pretty self-contained, you could copy what you need from this file.
If you need to strip almost all tags, the Bleach library is good. You can have it enforce rules like "my users can italicize things, but they can't make iframes."
Understand the properties of your tag stripper! Run fuzz tests on it! Here is the code I used to do the research for this answer.
sheepish note - The question itself is about printing to the console, but this is the top Google result for "python strip HTML from string", so that's why this answer is 99% about the web.
ready_for_web = cgi.escape(no_tags) -- cgi.escape is "Deprecated since version 3.2: This function is unsafe because quote is false by default, and therefore deprecated. Use html.escape() instead." Removed in 3.8. –
N I needed a way to strip tags and decode HTML entities to plain text. The following solution is based on Eloff's answer (which I couldn't use because it strips entities).
import html.parser
class HTMLTextExtractor(html.parser.HTMLParser):
def __init__(self):
super(HTMLTextExtractor, self).__init__()
self.result = [ ]
def handle_data(self, d):
self.result.append(d)
def get_text(self):
return ''.join(self.result)
def html_to_text(html):
"""Converts HTML to plain text (stripping tags and converting entities).
>>> html_to_text('<a href="#">Demo<!--...--> <em>(¬ \u0394ημώ)</em></a>')
'Demo (\xac \u0394\u03b7\u03bc\u03ce)'
"Plain text" doesn't mean result can safely be used as-is in HTML.
>>> html_to_text('<script>alert("Hello");</script>')
'<script>alert("Hello");</script>'
Always use html.escape to sanitize text before using in an HTML context!
HTMLParser will do its best to make sense of invalid HTML.
>>> html_to_text('x < y < z <!--b')
'x < y < z '
Named entities are handled as per HTML 5.
>>> html_to_text('&nosuchentity; ' ')
"&nosuchentity; ' "
"""
s = HTMLTextExtractor()
s.feed(html)
return s.get_text()
A quick test:
html = '<a href="#">Demo <em>(¬ \u0394ημώ)</em></a>'
print(repr(html_to_text(html)))
Result:
'Demo (¬ Δημώ)'
Security note: Do not confuse HTML stripping (converting HTML into plain text) with HTML sanitizing (converting plain text into HTML). This answer will remove HTML and decode entities into plain text – that does not make the result safe to use in a HTML context.
Example: <script>alert("Hello");</script> will be converted to <script>alert("Hello");</script>, which is 100% correct behavior, but obviously not sufficient if the resulting plain text is inserted as-is into an HTML page.
The rule is not hard: Any time you insert a plain-text string into HTML output, always HTML escape it (using html.escape(s)), even if you "know" that it doesn't contain HTML (e.g. because you stripped HTML content).
However, the OP asked about printing the result to the console, in which case no HTML escaping is needed. Instead you may want to strip ASCII control characters, as they can trigger unwanted behavior (especially on Unix systems):
import re
text = html_to_text(untrusted_html_input)
clean_text = re.sub(r'[\0-\x1f\x7f]+', '', text)
# Alternatively, if you want to allow newlines:
# clean_text = re.sub(r'[\0-\x09\x0b-\x1f\x7f]+', '', text)
print(clean_text)
There's a simple way to this:
def remove_html_markup(s):
tag = False
quote = False
out = ""
for c in s:
if c == '<' and not quote:
tag = True
elif c == '>' and not quote:
tag = False
elif (c == '"' or c == "'") and tag:
quote = not quote
elif not tag:
out = out + c
return out
The idea is explained here: http://youtu.be/2tu9LTDujbw
You can see it working here: http://youtu.be/HPkNPcYed9M?t=35s
PS - If you're interested in the class(about smart debugging with python) I give you a link: http://www.udacity.com/overview/Course/cs259/CourseRev/1. It's free!
You're welcome! :)
<b class="o'>x</b> as input function outputs x. But actually this input is invalid. I think that's why people prefer libs. –
Tressa out = out + c will be the worst of your nightmares. Instead, you could use a list that you’ll "".join at the end. –
Kissel An lxml.html-based solution (lxml is a native library and can be more performant than a pure python solution).
To install the lxml module use pip install lxml
from lxml import html
## from file-like object or URL
tree = html.parse(file_like_object_or_url)
## from string
tree = html.fromstring('safe <script>unsafe</script> safe')
print(tree.text_content().strip())
### OUTPUT: 'safe unsafe safe'
from lxml import html
from lxml.html.clean import clean_html
tree = html.fromstring("""<script>dangerous</script><span class="item-summary">
Detailed answers to any questions you might have
</span>""")
## text only
print(clean_html(tree).text_content().strip())
### OUTPUT: 'Detailed answers to any questions you might have'
Also see http://lxml.de/lxmlhtml.html#cleaning-up-html for what exactly the lxml.cleaner does.
If you need more control over which specific tags should be removed before converting to text then create a custom lxml Cleaner with the desired options, e.g:
from lxml.html.clean import Cleaner
cleaner = Cleaner(page_structure=True,
meta=True,
embedded=True,
links=True,
style=True,
processing_instructions=True,
inline_style=True,
scripts=True,
javascript=True,
comments=True,
frames=True,
forms=True,
annoying_tags=True,
remove_unknown_tags=True,
safe_attrs_only=True,
safe_attrs=frozenset(['src','color', 'href', 'title', 'class', 'name', 'id']),
remove_tags=('span', 'font', 'div')
)
sanitized_html = cleaner.clean_html(unsafe_html)
To customize how plain text is generated you can use lxml.etree.tostring instead of text_content():
from lxml.etree import tostring
print(tostring(tree, method='text', encoding=str))
Here is a simple solution that strips HTML tags and decodes HTML entities based on the amazingly fast lxml library:
from lxml import html
def strip_html(s):
return str(html.fromstring(s).text_content())
strip_html('Ein <a href="">schöner</a> Text.') # Output: Ein schöner Text.
text_content() returns lxml.etree._ElementUnicodeResult so you might have to cast it to string first –
Bezant str for string operations like + and indexing []. Added a cast for good measure anyhow. –
Milepost If you need to preserve HTML entities (i.e. &), I added "handle_entityref" method to Eloff's answer.
from HTMLParser import HTMLParser
class MLStripper(HTMLParser):
def __init__(self):
self.reset()
self.fed = []
def handle_data(self, d):
self.fed.append(d)
def handle_entityref(self, name):
self.fed.append('&%s;' % name)
def get_data(self):
return ''.join(self.fed)
def html_to_text(html):
s = MLStripper()
s.feed(html)
return s.get_data()
If you want to strip all HTML tags the easiest way I found is using BeautifulSoup:
from bs4 import BeautifulSoup # Or from BeautifulSoup import BeautifulSoup
def stripHtmlTags(htmlTxt):
if htmlTxt is None:
return None
else:
return ''.join(BeautifulSoup(htmlTxt).findAll(text=True))
I tried the code of the accepted answer but I was getting "RuntimeError: maximum recursion depth exceeded", which didn't happen with the above block of code.
''.join(BeautifulSoup('<em>he</em>llo<br>world').find_all(text=True)). Here the output is "helloworld", while you probably want it to be "hello world". ' '.join(BeautifulSoup('<em>he</em>llo<br>world').find_all(text=True)) does not help as it becomes "he llo world". –
Extenuatory The Beautiful Soup package does this immediately for you.
from bs4 import BeautifulSoup
soup = BeautifulSoup(html)
text = soup.get_text()
print(text)
Here's a solution similar to the currently accepted answer (https://mcmap.net/q/92394/-strip-html-from-strings-in-python), except that it uses the internal HTMLParser class directly (i.e. no subclassing), thereby making it significantly more terse:
def strip_html(text):
parts = []
parser = HTMLParser()
parser.handle_data = parts.append
parser.feed(text)
return ''.join(parts)
Here's my solution for python 3.
import html
import re
def html_to_txt(html_text):
## unescape html
txt = html.unescape(html_text)
tags = re.findall("<[^>]+>",txt)
print("found tags: ")
print(tags)
for tag in tags:
txt=txt.replace(tag,'')
return txt
Not sure if it is perfect, but solved my use case and seems simple.
You can use either a different HTML parser (like lxml, or Beautiful Soup) -- one that offers functions to extract just text. Or, you can run a regex on your line string that strips out the tags. See Python docs for more.
lxml.html.fromstring(s).text_content() –
Gules &) to text. –
Bassarisk For one project, I needed so strip HTML, but also css and js. Thus, I made a variation of Eloffs answer:
class MLStripper(HTMLParser):
def __init__(self):
self.reset()
self.strict = False
self.convert_charrefs= True
self.fed = []
self.css = False
def handle_starttag(self, tag, attrs):
if tag == "style" or tag=="script":
self.css = True
def handle_endtag(self, tag):
if tag=="style" or tag=="script":
self.css=False
def handle_data(self, d):
if not self.css:
self.fed.append(d)
def get_data(self):
return ''.join(self.fed)
def strip_tags(html):
s = MLStripper()
s.feed(html)
return s.get_data()
2020 Update
Use the Mozilla Bleach library, it really lets you customize which tags to keep and which attributes to keep and also filter out attributes based on values
Here are 2 cases to illustrate
1) Do not allow any HTML tags or attributes
Take sample raw text
raw_text = """
<p><img width="696" height="392" src="https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-768x432.jpg" class="attachment-medium_large size-medium_large wp-post-image" alt="Ethereum Classic 51% Attack: Okex Crypto Exchange Suffers $5.6 Million Loss, Contemplates Delisting ETC" style="float:left; margin:0 15px 15px 0;" srcset="https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-768x432.jpg 768w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-300x169.jpg 300w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-1024x576.jpg 1024w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-696x392.jpg 696w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-1068x601.jpg 1068w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-747x420.jpg 747w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-190x107.jpg 190w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-380x214.jpg 380w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-760x428.jpg 760w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc.jpg 1280w" sizes="(max-width: 696px) 100vw, 696px" />Cryptocurrency exchange Okex reveals it suffered the $5.6 million loss as a result of the double-spend carried out by the attacker(s) in Ethereum Classic 51% attack. Okex says it fully absorbed the loss as per its user-protection policy while insisting that the attack did not cause any loss to the platform’s users. Also as part […]</p>
<p>The post <a rel="nofollow" href="https://news.bitcoin.com/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc/">Ethereum Classic 51% Attack: Okex Crypto Exchange Suffers $5.6 Million Loss, Contemplates Delisting ETC</a> appeared first on <a rel="nofollow" href="https://news.bitcoin.com">Bitcoin News</a>.</p>
"""
2) Remove all HTML tags and attributes from raw text
# DO NOT ALLOW any tags or any attributes
from bleach.sanitizer import Cleaner
cleaner = Cleaner(tags=[], attributes={}, styles=[], protocols=[], strip=True, strip_comments=True, filters=None)
print(cleaner.clean(raw_text))
Output
Cryptocurrency exchange Okex reveals it suffered the $5.6 million loss as a result of the double-spend carried out by the attacker(s) in Ethereum Classic 51% attack. Okex says it fully absorbed the loss as per its user-protection policy while insisting that the attack did not cause any loss to the platform’s users. Also as part […]
The post Ethereum Classic 51% Attack: Okex Crypto Exchange Suffers $5.6 Million Loss, Contemplates Delisting ETC appeared first on Bitcoin News.
3 Allow Only img tag with srcset attribute
from bleach.sanitizer import Cleaner
# ALLOW ONLY img tags with src attribute
cleaner = Cleaner(tags=['img'], attributes={'img': ['srcset']}, styles=[], protocols=[], strip=True, strip_comments=True, filters=None)
print(cleaner.clean(raw_text))
Output
<img srcset="https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-768x432.jpg 768w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-300x169.jpg 300w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-1024x576.jpg 1024w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-696x392.jpg 696w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-1068x601.jpg 1068w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-747x420.jpg 747w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-190x107.jpg 190w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-380x214.jpg 380w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc-760x428.jpg 760w, https://news.bitcoin.com/wp-content/uploads/2020/08/ethereum-classic-51-attack-okex-crypto-exchange-suffers-5-6-million-loss-contemplates-delisting-etc.jpg 1280w">Cryptocurrency exchange Okex reveals it suffered the $5.6 million loss as a result of the double-spend carried out by the attacker(s) in Ethereum Classic 51% attack. Okex says it fully absorbed the loss as per its user-protection policy while insisting that the attack did not cause any loss to the platform’s users. Also as part […]
The post Ethereum Classic 51% Attack: Okex Crypto Exchange Suffers $5.6 Million Loss, Contemplates Delisting ETC appeared first on Bitcoin News.
I have used Eloff's answer successfully for Python 3.1 [many thanks!].
I upgraded to Python 3.2.3, and ran into errors.
The solution, provided here thanks to the responder Thomas K, is to insert super().__init__() into the following code:
def __init__(self):
self.reset()
self.fed = []
... in order to make it look like this:
def __init__(self):
super().__init__()
self.reset()
self.fed = []
... and it will work for Python 3.2.3.
Again, thanks to Thomas K for the fix and for Eloff's original code provided above!
The solutions with HTML-Parser are all breakable, if they run only once:
html_to_text('<<b>script>alert("hacked")<</b>/script>
results in:
<script>alert("hacked")</script>
what you intend to prevent. if you use a HTML-Parser, count the Tags until zero are replaced:
from HTMLParser import HTMLParser
class MLStripper(HTMLParser):
def __init__(self):
self.reset()
self.fed = []
self.containstags = False
def handle_starttag(self, tag, attrs):
self.containstags = True
def handle_data(self, d):
self.fed.append(d)
def has_tags(self):
return self.containstags
def get_data(self):
return ''.join(self.fed)
def strip_tags(html):
must_filtered = True
while ( must_filtered ):
s = MLStripper()
s.feed(html)
html = s.get_data()
must_filtered = s.has_tags()
return html
html_to_text and you embed the text being output from that function inside html without escaping that text, then it is the lack of escaping, which is a security vulnerability, not the html_to_text function. The html_to_text function never promised you the output would be text. And inserting text into html without escaping is a potential security vulnerability regardless of whether you got the text from html_to_text or some other source. –
Popular This is a quick fix and can be even more optimized but it will work fine. This code will replace all non empty tags with "" and strips all html tags form a given input text .You can run it using ./file.py input output
#!/usr/bin/python
import sys
def replace(strng,replaceText):
rpl = 0
while rpl > -1:
rpl = strng.find(replaceText)
if rpl != -1:
strng = strng[0:rpl] + strng[rpl + len(replaceText):]
return strng
lessThanPos = -1
count = 0
listOf = []
try:
#write File
writeto = open(sys.argv[2],'w')
#read file and store it in list
f = open(sys.argv[1],'r')
for readLine in f.readlines():
listOf.append(readLine)
f.close()
#remove all tags
for line in listOf:
count = 0;
lessThanPos = -1
lineTemp = line
for char in lineTemp:
if char == "<":
lessThanPos = count
if char == ">":
if lessThanPos > -1:
if line[lessThanPos:count + 1] != '<>':
lineTemp = replace(lineTemp,line[lessThanPos:count + 1])
lessThanPos = -1
count = count + 1
lineTemp = lineTemp.replace("<","<")
lineTemp = lineTemp.replace(">",">")
writeto.write(lineTemp)
writeto.close()
print "Write To --- >" , sys.argv[2]
except:
print "Help: invalid arguments or exception"
print "Usage : ",sys.argv[0]," inputfile outputfile"
A python 3 adaption of søren-løvborg's answer
from html.parser import HTMLParser
from html.entities import html5
class HTMLTextExtractor(HTMLParser):
""" Adaption of https://mcmap.net/q/92394/-strip-html-from-strings-in-python """
def __init__(self):
super().__init__()
self.result = []
def handle_data(self, d):
self.result.append(d)
def handle_charref(self, number):
codepoint = int(number[1:], 16) if number[0] in (u'x', u'X') else int(number)
self.result.append(unichr(codepoint))
def handle_entityref(self, name):
if name in html5:
self.result.append(unichr(html5[name]))
def get_text(self):
return u''.join(self.result)
def html_to_text(html):
s = HTMLTextExtractor()
s.feed(html)
return s.get_text()
You can write your own function:
def StripTags(text):
finished = 0
while not finished:
finished = 1
start = text.find("<")
if start >= 0:
stop = text[start:].find(">")
if stop >= 0:
text = text[:start] + text[start+stop+1:]
finished = 0
return text
I'm parsing Github readmes and I find that the following really works well:
import re
import lxml.html
def strip_markdown(x):
links_sub = re.sub(r'\[(.+)\]\([^\)]+\)', r'\1', x)
bold_sub = re.sub(r'\*\*([^*]+)\*\*', r'\1', links_sub)
emph_sub = re.sub(r'\*([^*]+)\*', r'\1', bold_sub)
return emph_sub
def strip_html(x):
return lxml.html.fromstring(x).text_content() if x else ''
And then
readme = """<img src="https://raw.githubusercontent.com/kootenpv/sky/master/resources/skylogo.png" />
sky is a web scraping framework, implemented with the latest python versions in mind (3.4+).
It uses the asynchronous `asyncio` framework, as well as many popular modules
and extensions.
Most importantly, it aims for **next generation** web crawling where machine intelligence
is used to speed up the development/maintainance/reliability of crawling.
It mainly does this by considering the user to be interested in content
from *domains*, not just a collection of *single pages*
([templating approach](#templating-approach))."""
strip_markdown(strip_html(readme))
Removes all markdown and html correctly.
Using BeautifulSoup, html2text or the code from @Eloff, most of the time, it remains some html elements, javascript code...
So you can use a combination of these libraries and delete markdown formatting (Python 3):
import re
import html2text
from bs4 import BeautifulSoup
def html2Text(html):
def removeMarkdown(text):
for current in ["^[ #*]{2,30}", "^[ ]{0,30}\d\\\.", "^[ ]{0,30}\d\."]:
markdown = re.compile(current, flags=re.MULTILINE)
text = markdown.sub(" ", text)
return text
def removeAngular(text):
angular = re.compile("[{][|].{2,40}[|][}]|[{][*].{2,40}[*][}]|[{][{].{2,40}[}][}]|\[\[.{2,40}\]\]")
text = angular.sub(" ", text)
return text
h = html2text.HTML2Text()
h.images_to_alt = True
h.ignore_links = True
h.ignore_emphasis = False
h.skip_internal_links = True
text = h.handle(html)
soup = BeautifulSoup(text, "html.parser")
text = soup.text
text = removeAngular(text)
text = removeMarkdown(text)
return text
It works well for me but it can be enhanced, of course...
Simple code!. This will remove all kind of tags and content inside of it.
def rm(s):
start=False
end=False
s=' '+s
for i in range(len(s)-1):
if i<len(s):
if start!=False:
if s[i]=='>':
end=i
s=s[:start]+s[end+1:]
start=end=False
else:
if s[i]=='<':
start=i
if s.count('<')>0:
self.rm(s)
else:
s=s.replace(' ', ' ')
return s
But it won't give full result if text contains <> symbols inside it.
# This is a regex solution.
import re
def removeHtml(html):
if not html: return html
# Remove comments first
innerText = re.compile('<!--[\s\S]*?-->').sub('',html)
while innerText.find('>')>=0: # Loop through nested Tags
text = re.compile('<[^<>]+?>').sub('',innerText)
if text == innerText:
break
innerText = text
return innerText.strip()
This is how I do it, but I have no idea what I am doing. I take data from a HTML table by stripping out the HTML tags.
This takes the string "name" and returns the string "name1" without the HTML tags.
x = 0
anglebrackets = 0
name1 = ""
while x < len(name):
if name[x] == "<":
anglebrackets = anglebrackets + 1
if name[x] == ">":
anglebrackets = anglebrackets - 1
if anglebrackets == 0:
if name[x] != ">":
name1 = name1 + name[x]
x = x + 1
import re
def remove(text):
clean = re.compile('<.*?>')
return re.sub(clean, '', text)
nh3 should also work:
>>> import nh3
>>> print(nh3.clean("<s><b>text</s></b>", tags={"b"}))
<b>text</b>
>>> print(nh3.clean("<s><b>text</s></b>", tags=set()))
text
performance should be good as it's wrapper for rust libtrary
This method works flawlessly for me and requires no additional installations:
import re
import htmlentitydefs
def convertentity(m):
if m.group(1)=='#':
try:
return unichr(int(m.group(2)))
except ValueError:
return '&#%s;' % m.group(2)
try:
return htmlentitydefs.entitydefs[m.group(2)]
except KeyError:
return '&%s;' % m.group(2)
def converthtml(s):
return re.sub(r'&(#?)(.+?);',convertentity,s)
html = converthtml(html)
html.replace(" ", " ") ## Get rid of the remnants of certain formatting(subscript,superscript,etc).
© 2022 - 2024 — McMap. All rights reserved.
&). You can either 1) remove them along with the tags (often undesirable, and unnecessary as they are equivalent to plain text), 2) leave them unchanged (a suitable solution if the stripped text is going right back into an HTML context) or 3) decode them to plain text (if the stripped text is going into a database or some other non-HTML context, or if your web framework automatically performs HTML escaping of text for you). – Bassarisk