The Art of Library

Honk if you love hacking! I know I do, and boy is it ever satisfying to turn a tough problem into a clever solution. But as you tumble down the slippery hacking slopes, picking up loose bits of code, you may find the itch to start writing code for other hackers. Good thought, but realize that implementation is just the tip of the iceberg.

Library What, Huh?

What makes libraries so special? They save time and effort by providing code you don’t want to write. Skip the nitty gritty of this authentication scheme or that request protocol and get on with the task at hand. There are many ways for a library to capture wily domain knowledge and not all are created equal. After working on some libraries and reading many more, what follows are my personal guidelines for creating lovely encapsulating bits.

First, Do No Harm

It should be easier to work with a library than without it. People will grudgingly use libraries that are painful, but will recommend and rave about libraries that feel good. You should start with some idea of what an ideal interaction with the library might feel like. Your first pass at implementation will tend to be rough work, but keep utopia in mind. Subsequent iterations can work to unify reality and awesome.

This happened when I wrote shindo, a test framework that I wrote because I wanted something short and sweet. I have a love/hate relationship with testing. I like the confidence that tests give me, but I’m just not a big fan of the process. Shindo lets me easily understand what’s actually happening when tests run. It looks like this:

(# Each test is a Ruby block, with success determined by return value
Shindo.tests('examples') do
  test('successful test') { true }
  test('failing test')    { false }
  test('pending test')    # no block provided
end

)

My first pass at implementation was basic, but then I started iterating. I wanted the runner to be more interactive than what I was used to. So I added a prompt system that halts the test run when it runs into an error and asks you what you want to do. This led to rapid iterations:

1. I needed backtraces, but I didn’t want to raise errors. So I wrote a nice backtrace object and added some code to view its output.
2. I wanted a better way to sift through files referenced in the backtrace, so I added the ability to choose a line from the backtrace and get an excerpt of the file containing it.
3. I wanted a better way to debug errors, so I added the choice to launch irb with the same binding as the test block. That’s my kind of digging in.
4. There was no need to run all the tests all the time, so I added tags to help narrow a run down to the relevant ones.
5. Finally, I added the option to reload files and restart the test run.

Keep It Simple

Libraries should be less complicated than the problems they address. Simple libraries are easier to understand, improve and maintain. Abstractions are a tempting place to start, but they are easier to build on a solid foundation. Start with the simplest thing that could work, staying close to the problem you are trying to solve. For instance, when wrapping public APIs, match the names of attributes and functions to the API documentation. It makes it easier to refer to things and provides documentation from the start.

When I found out I would be joining the Engine Yard Cloud team, it occurred to me that I should have more than a passing knowledge of AWS. I started by reading tons of docs. Equipped with a rough idea of what was involved, I wanted an easy way to experiment and code AWS. Such was the genesis of fog. AWS interactions can be super complicated, so I started by mapping the docs almost directly to code. Using it looks like this:

s3 = Fog::AWS::S3.new(
 :aws_access_key_id => id,
 :aws_secret_access_key => key
)

# get a list of buckets
s3.get_service

# create a bucket
s3.put_bucket('bucketname')

# get the contents of that bucket
s3.get_bucket('bucketname')

# delete that bucket
s3.delete_bucket('bucketname')

I’ll be the first to admit that its not the prettiest, but it worked! And when in doubt, the AWS documentation could remind me what in the world some method I wrote a month ago was supposed to do. Recently someone asked if I had generated this code somehow. I wrote it all by hand (unless you count copy/paste as code generation), but for once it was oddly comforting to be mistaken for a robot.

Stay Focused

Stick to capturing stuff for one problem at a time. If sub-sections start getting distracting, consider extracting them into their own libraries. Avoid monkey patching; use inheritance or something that is not monkey patching. Premature optimization is the root of all evil, iteration is the trunk of all good. Do not undermine your tree, and do furnish it with leaves of documentation so that it does not wither and die.

While working on shindo, I ended up adding lots of backtrace related code. Since I was not relying on raised errors, I needed some other way to control starting and stopping tracing. At first it was small and self-contained, like this:

class Annals

  attr_accessor :buffer, :max

  def initialize(max = 20, &block)
    @max = max
    start
  end

  def lines
    @buffer.map do |event|
      "#{event[:file]}:#{event[:line]} #{event[:method]}"
    end
  end

  def start
    @buffer = []
    @size   = 0
    Kernel.set_trace_func(
      lambda { |event, file, line, id, binding, classname|
        if event == 'call'
          unshift(:file => file, :line => line, :method => "in #{id}")
        end
      }
    )
  end

  def stop
    Kernel.set_trace_func(lambda {})
    @buffer.shift # remove Annals#stop from buffer
  end

  def unshift(line)
    @buffer.unshift(line)
    if @size == @max
      @buffer.pop
    else
      @size += 1
    end
  end

end

But it just kept growing. So I pulled it out into its own project, annals. After extraction I could proceed to add other handy backtrace related features, even ones I had no real need for just then. Small libraries like this make great building blocks for newer, crazier stuff.

Add Value

Once equipped with more knowledge about the problem than is healthy for any one person, abstraction is a go. Tackle low hanging fruit first. Ruby is, afterall, object-oriented, so consider providing some nice models that encapsulate the nitty gritty bits. Write mocks. No one knows better what mocks should do. Add some command line tools or an interactive mode. Focus on eliminating pain points, increasing usability and generally making the project awesome to work with.

With a good foundation of low level functionality in fog, it was time to start adding value. I added easy mocking, because running tests against AWS can be expensive and slow. Then I started building objects around the various entities. Here is how the objects replace the low level calls from earlier:

# this is optional and great for testing
Fog.mock!

# connection
s3 = Fog::AWS::S3.new(
 :aws_access_key_id => id,
 :aws_secret_access_key => key
)

# get a list of buckets
s3.buckets.all

# create a bucket
bucket = s3.buckets.create(:name => 'bucketname')

# get the contents of that bucket
bucket.objects.all

# delete that bucket
bucket.destroy

The new interface mimics DataMapper, an interface myself and many others are already familiar with. I did my best to abstract away the nitty gritty captured in the low level functionality while leaving the ability to get stuff done. I think it is better, but it is still a work in progress. I dedicate a little time every day to filling in the blanks, but libraries have a way of never truly being finished.

Lather, Rinse, Repeat

Making libraries that capture a domain and feel good to use is hard. Implement the simplest thing first, but never stop iterating. Be absurdly consistent. Keep fiddling with the look and feel. If something seems painful, it probably is, so fix it. Make your code a joy to use and people will reward you by using it, and submitting patches (hint, hint)!

What do you think makes a good library? How can I improve mine? Comment and let me know!