Cassandra and Ruby: A Love Affair? (Key-Value Stores Part 3)

Most of today’s up and coming key-value stores are more than just simple key-value stores. You saw this when we looked at Tokyo Cabinet which, in addition to simple key-value capabilities, adds more sophisticated abilities, such as database-like tables. In this post we’ll look at Cassandra – a modern key-value store that continues this trend. Cassandra was originally developed by Facebook and released to open source last year. The Facebook team describes Cassandra as (Google) BigTable running on top of an Amazon Dynamo-like infrastructure.

Cassandra is implemented using Java, and unlike Tokyo Cabinet, is designed to be distributed. One key feature of its distributed architecture is that it is an eventually consistent design. For Cassandra, scalability isn’t about absolute speed, but about adding system capacity at a reasonable cost, while retaining reasonable speed. A data store that promises immediate consistency sacrifices either availability or the ability to survive network partitioning, and when you write internet applications that need to scale, those are the two properties that are generally the most desirable.

For example, consider Twitter. As a Twitter user, which of the following options would you select?

1. When you view your timeline, it is always correct, BUT sometimes it can’t be viewed at all
2. You always view your timeline, but it sometimes takes time before the timeline reflects new posts

From the loud griping that the Twitter fail whale causes, I think most people prefer availability to immediate consistency. Eventual consistency within a reasonable time period is sufficient, and that’s exactly what Cassandra provides.

With Cassandra, a write will always succeed, but a read will not always immediately reflect the result of that write. The benefit is that you can expand the capacity of your Cassandra based storage system just by adding more nodes to it.

In addition to being truly scalable and decentralized (which also means that your Cassandra installation can easily be built in such a way that it spans data centers, and keeps you up and running in the event of a large space rock hitting one of them), Cassandra also sports a few other neat features. It goes beyond a simple key-value data store to offer a table-like store. The schema for those tables, just like with Tokyo Cabinet, is flexible. You can add or remove fields (which are called columns in Cassandra parlance) on the fly. Cassandra also lets you do ranged queries on the keys, and permits the use of table columns as lists. It’s packed with features that resonate for the implementer of large scale applications.

If you think that Cassandra might be worth a look, installing it is simple. You can find the source code or precompiled binaries to download. There is a simpler approach, however: sudo gem install cassandra

If you’re using OS X, there is an additional complication—Cassandra requires the 1.6 version of the JDK, and even if you have kept up on your Apple system updates and have 1.6 installed, it is still not the default. Make sure you have jdk 1.6 installed, and then (assuming a bash-like shell):

export JAVA_HOME=/System/Library/Frameworks \\
/JavaVM.framework/Versions/1.6
export PATH=/System/Library/Frameworks/JavaVM.framework \\
/Versions/1.6/Commands/:$PATH

If you installed the Cassandra gem, you can start an instance of Cassandra with: cassandra_helper cassandra

Cassandra uses a multidimensional data model. At the top there is a keyspace, which is referred to as a table in the storage-conf.xml file. The keyspace defines a high level grouping for the data, and there’s typically one key space per application. Keyspaces must be defined in the storage-conf.xml file before startup.

Below the keyspace lies the column family, which is the basic unit of data organization within a keyspace. In a row oriented database, data is stored by row, with all columns grouped together. In a column oriented database, data is stored by column, with all rows grouped together. Cassandra’s use of column families allows a hybrid approach. A column family allows a set of columns for a given row to be stored together. This allows you to optimize your column design in order to group commonly queried columns together. Like keyspaces, column families must be defined in the storage-conf.xml file before startup.

Next comes the key. This is the unique, permanent identifier for a records. Cassandra will index this for you.

Below this level Cassandra provides a couple of options that allow either one or two additional dimensions of data organization. The first of these is the column.

It is at the column level that Cassandra’s kinship with simpler key-value stores becomes apparent. Columns are where a record’s data is stored, and a column is expressed as a basic key-value relationship. 'birthday' => '1998-08-22' Columns can be stored sorted alphabetically, or by timestamp (all column entries are timestamped). Columns can be defined on the fly.

The final tier of organization is an optional tier called the super column. This can be somewhat confusing, but a super column is really just a group of columns. Users cannot mix columns and super columns at the fourth tier of organization. Once again, users must define which column families contain super columns, and which contain standard columns, in storage-conf.xml before startup. Super columns allow you to group sets or related, sorted column data under a single name.

But enough with the exposition: it’s time to see how it works in code!

$ irb
>> require 'rubygems'
>> require 'cassandra'
>> include Cassandra::Constants
=> Object
>> store = Cassandra.new('Twitter')
=> # nil, :Users => nil,
   :StatusRelationships => nil, :UserAudits => nil,
   :Statuses => nil, :UserRelationships => nil,
   :StatusAudits => nil}, @host="127.0.0.1", @port=9160>

The Cassandra gem is brought to you by Evan Weaver, of Twitter, so there is a certain bias in the default storage-conf.xml configuration that he bundles the gem with. He provides several good schemas, though, which we can look at to understand how Cassandra really works.

>> store.insert(:Users, '12345', {'screen_name' => 'wyhaines'})
=> nil
>> store.insert(:Users, '67890', {'screen_name' => 'wayneeseguin'})
=> nil
>> store.insert(:Statuses, '1', {'user_id' => '67890', 'text' =>
?> 'Hey, what is Cassandra like?'})
=> nil
>> store.insert(:Statuses, '2', {'user_id' => '12345', 'text' =>
?> '@wayneeseguin, It is great!'})
=> nil
>> store.insert(:Statuses, '3', {'user_id' => '12345', 'text' =>
?> 'It is a key/value store with a lime twist.'})
=> nil

Using the Twitter schema, a couple of users were created, and then some status messages were created, with one field containing the user id, and another containing the text of the status message. Each status message has a unique ID.

>> store.insert(:UserRelationships,
?> '67890', {'user_timeline' => {UUID.new => '1'}})
=> nil
>> store.insert(:UserRelationships, '12345',
?> {'user_timeline' => {UUID.new => '2'}})
=> nil
>> store.insert(:UserRelationships, '12345',
?> {'user_timeline' => {UUID.new => '3'}})
=> nil

Using a column based database like Cassandra takes a bit of a mental shift from a simple key-value store or a typical row-oriented relational database. Recall the hierarchy of storage—Column Family, Key, Column/Value. If each status message has a unique key, I can’t just ask for all keys where column family == ‘:Statuses’ and column user_id == ‘12345’. UserRelationships is a super column. It’s defined like this in storage-conf.xml.

This says that UserRelationships is a super column, and that the sort order of its subcolumns is a TimeUUIDType; that is, a time based UUID. By inserting rows keyed by the user id into UserRelationships, with values that are a column, user_timeline and subcolumns composed of a time based UUID pointing to a message key, you build a structure that provides an easy path to query all of the messages from a given user, in time sorted order.

>> my_message_relationships = store.get(:UserRelationships,
?> '12345', 'user_timeline', :reversed => true)
=> #=>"3",
=>"2"}>

This query asks for the UserRelationships for key 12345, sorted by user_timeline, in reverse order. What is returned is a ordered hash keyed by the UUID timestamps, and keyed by message ids (i.e. exactly what was inserted earlier). You can use this to pull a list of recent messages.

>> my_message_relationships.values.each do |message_id|
?> puts store.get(:Statuses, message_id).inspect
?> end

#"It is a key/value store with a lime twist.",
  "user_id"=>"12345"
}>
#"@wayneeseguin, It is great!",
  "user_id"=>"12345"
}>
=>  \["3", "2"\]

As you can see, using Cassandra is more complicated than using a simple key-value store, even one like Tokyo Cabinet which builds a table model on a row based key-value system. However, just like the first time you tried to learn recursion, once your perspective shifts so that you can grok it, Cassandra’s structure naturally lends itself to a whole class of otherwise tricky, high labor queries.

The other significant drawback to Cassandra is that although column schema is fluid, and can be changed at runtime, the higher levels of data organization—keyspaces, column families, and super columns—have to be defined in an XML configuration file, storage-conf.xml at startup. For example, if you wanted to start a new project using the Cassandra gem, you have to create your own set of configuration files (look at gems/cassandra-0.5.5/conf for your ruby installation to see how the sample packaged with the Cassandra gem is structured).

Consider the common example of a blog. Let’s say you want to be fancy and allow your blog to have user accounts so that users can see threads of their own blog comments over time. Your storage-conf.xml config might look like this:

Cassandra is fun to work with, and using the Cassandra gem eliminates most of the hassles of manually setting it up to run while you are getting your feet wet. It offers an interesting balance of performance (it’s surprisingly fast!) and features an architecture that is truly horizontally scalable.

Cassandra has a lot of promise, but it’s also quite young, and certainly isn’t bug free. The Ruby API is in a state of flux, and if you have a non-standard setup, it can be a hassle. I had a heck of a time getting it to run right on my OS X laptop, despite apparently having all prerequisites installed correctly. If you like what you see, get involved, maybe even write a DataMapper adapter for it. I think Cassandra is going to be around in the Ruby community for a long time.