January 14th 2010 By Aaron Patterson

Getting Started with Nokogiri

We’re decided to mix up the Engine Yard blog a little and invite some community members to contribute guest posts. This one (our first!) is from Aaron Patterson – a long-time member of the Ruby community, and the creator of Nokogiri. He hacks with the developers of Seattle.rb, and travels the world to speak about Nokogiri and other Ruby topics at industry conferences and events.

Nokogiri is a library for dealing with XML and HTML documents. I wrote Nokogiri along with my (more attractive) partner in crime, Mike Dalessio. We both use and enjoy working with Nokogiri for dealing with HTML and XML on a daily basis, and I’d like to share it with you! In this post, we’ll be covering:

  • Getting Nokogiri installed
  • Basic document parsing
  • Basic data extraction

Hopefully by the end of this article you will also be able to use and enjoy Nokogiri on a day to day basis too!

Installation

Nokogiri is actually a wrapper around Daniel Veillard’s excellent HTML/XML parsing library written, libxml2. Since Nokogiri simply wraps and builds upon this already existing library, installing libxml2 is a prerequisite for installing Nokogiri. Fortunately, libxml2 has been ported to most systems, so the installation is pretty easy.

###OS X

I recommend installing libxml2 on OS X from macports. OS X ships with libxml2 installed, but macports is more up to date, so I’d recommend using it instead.

To install libxml2 from macports:

	$ sudo port install libxml2 libxslt

Then to install nokogiri:

	$ sudo gem install nokogiri

And that should be it!

###Linux

On Linux, we still need to install libxml2. The command for installing libxml2 will change depending on the package manager and linux distribution you’re using, but we’ll cover Fedora and Ubuntu here.

On Fedora:

	$ sudo yum install libxml2-devel libxslt-devel
	$ gem install nokogiri

On Ubuntu:

	$ sudo apt-get install libxml2 libxml2-dev libxslt libxslt-dev
	$ gem install nokogiri

###Windows

Dealing with libxml2 on Windows is so much work, that we built libxml2 for you, and now ship it along with Nokogiri. On Windows, to install, simply do gem install nokogiri.

###Oh Noes! Something Went Wrong!

Nokogiri ships with some basic intelligence for finding your installation of libxml2, but clever developers can easily fool it! If you have problems, first check that the libxml2 and libxslt development packages are installed. If everything seems OK, and Nokogiri still won’t install, send an email to the Nokogiri mailing list. We’re here to help!

Basic Parsing

Now that we have installation out of the way, it’s time to get Nokogiri to do some work for us. Nokogiri lets you parse an HTML or XML document using a few different strategies:

  • DOM
  • SAX
  • Reader
  • Pull

Each of these strategies have different advantages and disadvantages. We won’t go through all the differences in this post; the DOM interface is the most common, and generally regarded as the easiest to use, so that’s what we’ll focus on here.

There are two main entry points to Nokogiri depending on the kind of document you wish to parse: one for HTML documents and one for XML documents. Parsing HTML documents looks like this:

	doc = Nokogiri::HTML(html_document)

Parsing XML documents looks like this:

	doc = Nokogiri::XML(xml_document)

Both of these functions will take an IO object or a String object. Since both forms accept IO objects, we can even feed open-uri straight in to Nokogiri like this:

	doc = Nokogiri::HTML(open("http://www.google.com/search?q=doughnuts"))

Feeding Nokogiri an IO object is slightly more efficient than using a String, but you should choose the one that is most convenient.

###Data Structures

To become data extraction Zen Masters, we first need to understand the data structure returned by Nokogiri. Notably, we need to understand that Nokogiri converts HTML and XML documents into a tree data structure.

For example, an HTML document that looks like this:

	<html>
	  <head>
	    <title>Hello!</title>
	  </head>
	  <body id="uniq">
	    <h1>Hello World!</h1>
	  </body>
	</html>

…will be represented in memory with a tree that looks like this:

HTML Tree

Any data extraction technique used is simply a way for traversing this in-memory tree. If we keep this structure in mind while trying to do data extraction, we can enter data extraction nirvana!

Data Extraction

We’ve seen how to turn an HTML or XML document into an in-memory tree. Now we’re going to try to do something useful with this tree: extract some data. Let’s take a look at a few different strategies for unlocking the data in our tree.

There are three different ways to traverse our in-memory tree. The first two, XPath and CSS, are small languages built specifically for tree traversal. The last one we’ll examine is the Nokogiri API for manual tree traversal.

###Basic XPath

The XPath language was written to easily traverse an XML tree structure, but we can use it with HTML trees as well. Here’s a sample program for extracting search result links from a google search. We’ll use XPath to find the data we want, and then pick apart the XPath syntax:

	require 'open-uri'
	require 'nokogiri'

	doc = Nokogiri::HTML(open("http://www.google.com/search?q=doughnuts"))
	doc.xpath('//h3/a').each do |node|
	  puts node.text
	end

The XPath used in this program is:

	//h3/a

In English, this XPath says:

Find all "a" tags with a parent tag whose name is "h3"

Thus, our program finds all “a” tags with “h3” parents, loops over them, and prints out the text content.

XPath works like a directory structure where the leading “/” indicates the root of the tree. Slashes separate the tag matching information. When there’s nothing between slashes, it’s a sort of wild card—meaning “any tag matches”. The “h3” and “a” are tag name matchers, and only match when the tag name matches.

Finding tag names is great, but if you run the previous program, you might find that it returns more “a” tags than we actually want. We need to narrow down our search based on some attributes of the tags, specifically the “class” values. To match attribute values in XPath, we use brackets. Now let’s look at a couple of examples.

To match “h3” tags that have a class attribute, we write:

	h3[@class]

To match “h3” tags whose class attribute is equal to the string “r”, we write:

	h3[@class = "r"]

Using the attribute matching construct, we can modify our previous query to:

	//h3[@class = "r"]/a[@class = "l"]

which in English terms is:

Find all "a" tags with a class attribute equal to "l" and an immediate parent tag "h3" that has a class attribute equal to "r"

If we substitute that XPath back in to our original program, we’ll get the expected results.

For more information on doing XPath queries, I recommend checking out the tutorial at w3schools as well as the w3 recommendation.

For more information on using XPath within Nokogiri, check out the Nokogiri tutorials as well as the RDoc.

Next, let’s look at CSS syntax.

###Basic CSS

CSS is similar to XPath in that it’s another language for searching a tree data structure. In this section, we’ll perform the same task as the XPath section, but we’ll examine the CSS syntax.

CSS does not separate tag matching patterns by slashes, but rather by whitespace or “greater than” characters (actually, there are more, but we’re just going to talk about those two for now). Let’s rewrite our previous XPath as CSS and examine the syntax.

	//h3/a

…can be written in CSS as:

	h3 > a

The “>” character indicates that the “a” tag must be a direct descendant of the “h3” tag. Most CSS that I see uses space separators like this:

	h3 a

Using a space indicates that there could be any number of tags between the “h3” tag and the “a” tag. The space is similar to “//” in XPath, and this CSS query could be written in XPath like this:

	//h3//a

Similar to XPath, CSS can use brackets for matching attributes. Let’s do a couple more XPath to CSS translations. On the left is XPath, on the right is CSS:

	h3[@class]        =>  h3[class]
	h3[@class = "r"]  =>  h3[class = "r"]

This syntax works, but CSS provides us with a shorthand for matching the “class” attribute. To find all h3 tags whose class attribute contains “r”, we can say:

	h3.r

There’s a subtle difference between the two previous examples. The selector h3[@class = "r"] must be an exact match; the class value must exactly equal the string r. In the second example, the selector h3.r means “the class attribute must contain the value r”. That means h3.r will match the following tag, but h3[@class = "r"] will not:

	<h3 class="r foo">Hi!</h3>

The XPath selector and our translated CSS selector would not match this tag, but the “h3.r” selector would. Most of the time, the CSS class selectors do what we want. Only when I need something very specific do I use the bracket form in my CSS selectors.

With this knowledge in hand, we can rewrite our original program using CSS selectors:

	doc = Nokogiri::HTML(open("http://www.google.com/search?q=doughnuts"))
	doc.css('h3.r > a.l').each do |node|
	  puts node.text
	end

I think the CSS selectors usually result in more concise and clear queries than XPath, so I usually stick to CSS queries in my code. There are some tasks which CSS cannot accomplish that XPath can though, so it’s nice to be able to fall back to XPath queries when I need to.

Next, let’s look at some basic node API’s provided by Nokogiri.

###Basic Node API

Since we’re dealing with a tree data structure, Nokogiri provides methods for navigating that tree. In fact, all of the tree traversal we’ve seen so far using XPath and CSS can be accomplished manually via Ruby. Manual tree traversal is, however, cumbersome and verbose, which is why languages like XPath and CSS exist. Sometimes a combination of XPath or CSS plus manual tree traversal is easiest, so it is still important to know the API.

Every tag in a document is represented by class called a Node. Each node in the tree has 0 or more children, 0 or 1 parent, 0 or more siblings, and 0 or more attributes. Nokogiri provides methods for accessing all of these things on any particular node. We can access any of those relative nodes like so:

	node.parent           #=> parent node
	node.children         #=> children nodes
	node.next_sibling     #=> next sibling node
	node.previous_sibling #=> previous sibling node

These node access methods can be used for manually traversing a tree, but I tend to leave the hard work to XPath or CSS queries and only use manual tree access when I have to.

When it comes to accessing attributes of a tag, the node may be treated like a normal Ruby Hash. We can get and set attributes on a node like so:

	node['class']           #=> the value of the class attribute
	node['class'] = 'foo'

We can even get a list of attributes or values of attributes like so:

	node.keys   #=> list of attribute name
	node.values #=> list of attribute values

For more information on things you can do with Nodes, check out the Node Documentation and also the Nokogiri tutorials section.

Conclusion

I hope this article has you on your way to HTML and XML parsing nirvana. Remember the tree data structure, and remember that XPath and CSS can be performed on HTML documents and XML documents.

Make sure to check out our documentation, and if you have any problems make sure to join the mailing list!

Share your thoughts with @engineyard on Twitter