Custom Chef Recipes with Engine Yard Cloud

One of the power user features of Engine Yard Cloud is the ability to use custom Chef recipes to install or configure anything that can run on Gentoo Linux that we have not already automated as part of the platform. This allows for extensive customizations of your environments and empowers you to run virtually all custom software you might need.

Let’s start by talking about one of the tools you’ll want to strongly consider using as part of the Chef process: Redis.

As we blogged about a few weeks ago, Redis is a new key-value data structure server that has some very interesting capabilities. In a standard key-value server you just have a string key and a string as the value. With Redis, the value can be typed as a String or as a Set or List. This means that you can build up very powerful indexing structures and ways of accessing your data that are much more powerful then standard key-value storage.

Redis also has many atomic operations it can perform on the various value types. You can do common array manipulations on the List type and you can do set intersection and difference on the Set value type. Redis is somewhere between memcached and Tokyo. It’s as fast or faster than memcached (up to 110k ops/second) and offers higher level data structures then memcached or Tokyo. It also offers persistence, done in an asynchronous fashion. This also means that it is possible to lose data with Redis, so so you want to be sure you take this into consideration when designing your applications.

Redis does persistence with triggers; you define checkpoints that will trigger a save and when a save is triggered, Redis will fork itself and write its entire data set to the disk in one sequential write. This eliminates random I/O, so it’s not hard on the I/O subsystem.

Let’s assume you’ve decided to use Redis in your next application. Now how do you deploy and configure Redis in a 100% reproducible way on the Engine Yard Cloud? You write a custom Chef recipe and attach it to your environment so it runs every time you spin up a new server and automatically installs, configures and runs Redis.

To get started using custom Chef recipes on Engine Yard Cloud, you’ll need to get the ey-flex gem and your cloud credentials installed locally. There’s a good getting started article on the Engine Yard Docs site, if you need to pause to sort out how. But for now, I’ll move on.

The best way to get started with custom recipes is to fork our ey-cloud-recipes git repository.

Once you have the repository forked and checked out on your local machine the first thing we need to do is to generate a new cookbook recipe for Redis. Here are the steps:

~ $ cd ey-cloud-recipes
~/ey-cloud-recipes $ rake new_cookbook COOKBOOK=redis
(in /Users/ez/ey-cloud-recipes)
** Creating cookbook redis
mkdir -p /Users/ez/ey-cloud-recipes/cookbooks/redis/attributes
mkdir -p /Users/ez/ey-cloud-recipes/cookbooks/redis/recipes
mkdir -p /Users/ez/ey-cloud-recipes/cookbooks/redis/definitions
mkdir -p /Users/ez/ey-cloud-recipes/cookbooks/redis/libraries
mkdir -p /Users/ez/ey-cloud-recipes/cookbooks/redis/files/default
mkdir -p /Users/ez/ey-cloud-recipes/cookbooks/redis/templates/default

These commands create the basic structure of our Redis recipe for us; now we need to start adding the Chef resources to install and configure Redis.

Open this file in your editor of choice:

~/ey-cloud-recipes/cookbooks/redis/recipes/default.rb.

The first thing you need to do is ensure that the Redis package is installed on the system; luckily Engine Yard has built a Redis ebuild tuned for our Gentoo systems. Basically, you add a package resource to the top of the recipe that will take care of making sure Redis is installed the first time you run these recipes. After the first run (which installs the package), subsequent runs will just verify that the package is installed and will skip building it again if it is already on the system.

package "dev-db/redis" do
  action :install
end

You also need the Ruby client library for Redis installed, so you need to throw in a gem_package resource with github specified as the source to ensure you get the client library installed properly.

gem_package "ezmobius-redis-rb" do
  source "http://gems.github.com"
  action :install
end

Now that you have the Redis package installed, you need to make a template for the redis.conf config file. You want to create a Chef template resource to handle this.

Template resources in Chef use ERB and should be familiar to anyone who has used ERB templates before. The cool thing about template resources is that they’re smart enough to checksum the current template on disk with what the new rendered template looks like, and do nothing if the checksums match. If they don‘t match then Chef will make a backup of the old output file and then re-template the new version.

Template resources are fairly self explanatory: you provide a source template as well as the user, group and permissions to spit the file out with, and then you pass in a hash of variables that will be turned into instance variables in the actual template.

template "/etc/redis.conf" do
  owner 'root'
  group 'root'
  mode 0644
  source "redis.conf.erb"
  variables({
    :basedir => '/data/redis',
    :logfile => '/data/redis/redis.log',
    :bind_address => '127.0.0.1', # '0.0.0.0' if you want redis available to the outside world
    :port  => '6379',# change if you want to listen on another port
    :loglevel => 'notice',
    :timeout => 3000,
    :sharedobjects => 'no'
  })
end

You also need to add the actual ERB template to render in a file called ~/ey-cloud-recipes/cookbooks/redis/templates/default/redis.conf.erb

# Redis configuration file example
daemonize yes
pidfile /var/run/redis.pid
port <%= @port %>
bind <%= @bind_address %>
timeout <%= @timeout %>
save 900 1
save 300 10
save 60 10000
dbfilename redis_state.rdb
dir <%= @basedir %>
loglevel <%= @loglevel %>
logfile <%= @logfile %>
databases 16
# slaveof <masterip> <masterport>
# requirepass foobared
# maxclients 128
# maxmemory <bytes>
glueoutputbuf yes
shareobjects <%= @sharedobjects %>
shareobjectspoolsize 1024

Next you need to create a directory for Redis to keep its database files in (/data/redis). Use a directory resource to get this done. Note that node[:owner_name] refers to the main Unix username you entered when you created your environment.

directory "/data/redis" do
  owner node[:owner_name]
  group node[:owner_name]
  mode 0755
  recursive true
end

Chef is based on the idea of declarative idempotent resources. This means that every action Chef takes should either take an action to make the system look like what you have declared, or just verify that the resource is already in place and then skip it if so.

This has the effect of allowing you to run your Chef recipes over and over again on the same system and the system will always converge on the state that you have declared. This means that it will not hurt your system to run these recipes all the time.

Most of the built in resources that come with Chef are idempotent by default but some of them (like the execute resource) can run arbitrary shell commands, so there is no way to automatically make them idempotent. For these resources you will need to write your own not_if or only_if guard conditions to ensure that your resource is not destructive if run more then once.

You’ll need a resource that will start up Redis if it is not already running, and I’d suggest you use a not_ifguard in order to let this resource run multiple times without starting multiple copies of Redis. Use an execute resource to accomplish this: an execute resource is basically a shell command, which comes in handy for when there is not an out of the box Chef resource that accomplishes what you need.

Be careful whenever you make an execute resource to always use a guard condition or ensure that the command you run is itself idempotent, or you run the risk of ruining the whole reason for Chef to exist. Being able to run the recipes all the time is where the power comes from; it is almost like unit tests for your servers. If the recipes are well written and they run to completion without any errors you can be pretty sure that the state of your server looks like what you declared it to look like.

execute "ensure-redis-is-running" do
  command %Q{
    /usr/bin/redis-server /etc/redis.conf
  }
  not_if "pgrep redis-server"
end

Now that the complete recipe is in place, you need to make sure it gets run when you attach your recipes to your environment. This means you need to edit ~/ey-cloud-recipes/cookbooks/main/recipes/default.rb and make sure it only requires the recipes you want to run. For now let’s assume that you only want the Redis recipe to run, so make your file look like this:

require_recipe 'redis'

Attach this to your Engine Yard Cloud environment (I’ll assume you have an environment called redis_production).

First thing you need to do is to commit all your changes to the git repo, and then upload and attach the recipes to your environment.

$ cd ~/ey-cloud-recipes
$ git add .
$ git commit -m "adding the redis recipe"
$ ey-recipes --upload redis_production

Now the next time you run a deploy from the dashboard, your custom recipes will run as well. Or, if you would rather kick it off from the command line you can do that like this:

$ cd ~/ey-cloud-recipes
$ ey-recipes --deploy-main redis_production

That’s all there is to it!

With the power of custom Chef recipes you can run virtually anything that runs on Gentoo Linux on your Engine Yard Cloud account. The possibilities are endless! We’ve also got some recipes premade for things like thinking_sphinx, and there’s also a set of #nosql recipes for configuring a bunch of different key value databases on Engine Yard Cloud.

We’re working on new things every day, Chef recipes included, so keep an eye on the Engine Yard Blog and comment with your thoughts!