How do I parse and scrape the meta tags of a URL with Nokogiri?

Question

I am using Nokogiri to pull the <h1> and <title> tags, but I am having trouble getting these:

<meta name="description" content="I design and develop websites and applications.">
<meta name="keywords" content="web designer,web developer">

I have this code:

url = 'https://en.wikipedia.org/wiki/Emma_Watson' 
page = Nokogiri::HTML(open(url))

puts page.css('title')[0].text puts page.css('h1')[0].text
puts page.css('description')
puts META DESCRIPTION
puts META KEYWORDS

I looked in the docs and didn't find anything. Would I use regex to do this?

Thanks.

Steak

give the full html.. your need is unclear.. – Pneumatometer

Just to clarify: Nokogiri doesn't crawl anything. It only does parsing. Your code, in conjunction with gems like OpenURI and Nokogiri, does the crawling. – Skees

Use github.com/BorisBresciani/rails_parse_head – Melliemelliferous

Answer 1

Here's how I'd go about it:

require 'nokogiri'

doc = Nokogiri::HTML(<<EOT)
<meta name="description" content="I design and develop websites and applications.">
<meta name="keywords" content="web designer,web developer">
EOT

contents = %w[description keywords].map { |name|
  doc.at("meta[name='#{name}']")['content']
}
contents # => ["I design and develop websites and applications.", "web designer,web developer"]

Or:

contents = doc.search("meta[name='description'], meta[name='keywords']").map { |n| 
  n['content'] 
}
contents # => ["I design and develop websites and applications.", "web designer,web developer"]

Skees

FYI: The only works when you have the meta tags in the document. If you run this code with one of them removed it'll error: undefined method '[]' for nil:NilClass. – Heavyduty

Then something changed in one of many revisions since 2013, because the examples are the exact output from Ruby when the answer was created. – Skees

Answer 2

That would be:

page.at('meta[name="keywords"]')['content']

Cordie

Answer 3

Another solution: You can use XPath or CSS.

puts page.xpath('/html/head/meta[@name="description"]/@content').to_s
puts page.xpath('/html/head/meta[@name="keywords"]/@content').to_s

Loehr