How to use Bats to test your command line tools

Bash Automated Test System or “Bats”, written by sstephenson, is a testing framework written for and in the Bash command language. The best thing about Bats is that it can be used for much more than testing your shell scripts!

One of the wonderful things about the Unix philosphy is the composability it encourages by suggesting we write progams to handle text streams. Since a lot of command line tools in myriad languages expect to be used this way, we can write tests for a lot of existing software.

We write interface tests for our web applications to ensure they behave as expected, Bats allows us to have the same high standards for our command line tools.

At its simplest, a bats test might just be:

#!/usr/bin/env bats

@test "Check that the mysql client is available" {
    command -v mysql
}

Here, bats simply tests the exit code of the command that is run.

Breaking this down a little:

The hashbang line at the top simply declares that this should be run by Bats.

The @test "Check that the mysql client is available" { is how a test definition is declared. Spacing is important here; there must be a space after the description string, before the curly brace. The closing curly brace must be on a new line.

Inside the quotation marks (single or double) is a description or name of the test, and is used in the output when executed.

Another example of a test might look like:

#!/usr/bin/env bats

@test "Check that we have a /tmp directory" {
	run stat /tmp
	[ $status = 0 ]
}

In this test, we’re checking that the exit code of stat /tmp is zero (i.e., “ran successfully”, in Unix terminology). If you’re a little hazy on exit codes, try echo $? after the next command you run in the shell. This returns the exit code for the last command you ran. By convention, anything other than zero is usually an indication of failure.

The run command allows us to encapsulate the output and exit code of the passed command and inspect them using the special variables $status (the exit code) and $output (the text output of running the program).

The [ and ] are standard test syntax for POSIX shell. An important point about bats is that it is all Bash, and has the full Bash syntax available within tests.

The output of this particular test is:

$ bats simple.bats
 ✓ Check that we have a /tmp directory

1 test, 0 failures

Neat!

Like any good testing framework, there are also setup and teardown functions available.

Sometimes, you’ll have multi-line output, and need to check that specific lines are coming up.

For this, Bats provides the special $lines array to complement $status and $output.

An example of testing multi-line output might look like:

#!/usr/bin/env bats

@test "Check that total is listed" {
	run ls -l
	[[ ${lines[0]} =~ "total" ]]
}

You might have noticed that this time I’ve used the [[ ]] syntax for the tests. This is a Bash specific (as opposed to POSIX shell) extension that allows for partial matching using the =~ operator (amongst other things).

In the example here, the actual output is something like total 352354 which will not match total using the single square bracket POSIX syntax, and is particularly useful for testing responses that contain variable output.

By convention, Bats tests are given the *.bats extension. This allows Bats to seek out tests in a directory and run a suite:

$ bats testdir
 ✓ Test that we get the word 'bar'
 ✓ Check that we have a /tmp directory
 - A test I don't want to execute for now (skipped: feature coming)
 ✓ Check that total is listed

4 tests, 0 failures

In the above example, a test has been skipped. You can use the skip command inside a test to skip over tests which are known to fail, while preventing it from failing the suite. This is useful in cases where functionality is expected, but not yet delivered.

A skipped test might look like:

#!/usr/bin/env bats

@test "A test I don't want to execute for now" {
	skip "feature coming"
	run foo
	[ $status -eq 0 ]
}

Another excellent use case for Bats is integration testing. Because its only dependency is Bash, it makes for an excellent addition to a full CI workflow.

Things which are simple to check over SSH, but make less sense in a unit testing suite, are good fits for this. Checking that a config value has been passed correctly to a running process by inspecting the output of ps or that a particular software package has been installed, for example.

In fact, this is something we make use of internally at Engine Yard for ensuring our stacks end up with the commands and configuration that we expect them to have via Chef. Bats allows us to test that stuff works where it counts, where the rubber meets the road.

Using Bats with Travis CI

Travis does not offer native support for Bats yet, but you can integrate it easily by adding something like the following to your .travis.yml file:

before_install:
  - sudo add-apt-repository ppa:duggan/bats --yes
  - sudo apt-get update -qq
  - sudo apt-get install -qq bats
script:
  - bats test/bats

A full .travis.yml using this can be seen here.

Test Kitchen is notable for including Bats and expecting that tests be written in Bats.

Gotchas

One “gotcha” that might come up after using Bats for a little while is testing the result of piped commands.

For example, here’s a command which echos a string, the slices it up on spaces, and picks the second value:

$ echo 'foo bar baz' | cut -d' ' -f2
bar

So you might expect a test like this to pass:

#!/usr/bin/env bats

@test "Test that we get the word 'bar'" {
    run echo 'foo bar baz' | cut -d' ' -f2
    [ $output = "bar" ]
}

But in reality it fails:

$ bats gotcha1.bats
 ✗ Test that we get the word 'bar'
   (in test file /Users/ross/bats/gotcha1.bats, line 5)

1 test, 1 failure

This can be a bit of a head scratcher at first, but it makes sense when you realize that the run command is just like any other Unix command, and that run (which has no output) is in fact being piped to cut.

The solution is to encapsulate the entire command being tested as a bash -c inline string:

#!/usr/bin/env bats

@test "Test that we get the word 'bar'" {
    run bash -c "echo 'foo bar baz' | cut -d' ' -f2"
    [ "$output" = "bar" ]
}

Which produces the desired result:

$ bats gotcha1.bats
 ✓ Test that we get the word 'bar'

1 test, 0 failures

For a comprehensive example of real world tests, see the tests/bats directory of this Python project.

The bats README.md is also quite a comprehensive document, and covers some ways of debugging output.