Introducing Gem Testers

Note: Our latest guest post is from community contributor Josiah Kiehl. Josiah is a software engineer at Oracle and Hacker in Residence at Philadelphia Startup Leaders. Josiah blogs at http://bluepojo.com/, and tweets as @bluepojo. He and Erik Hollensbe have been hard at work on Gem Testers.

A strong disposition toward testing is one of my favorite features of the Ruby community. This leads to solid gems and applications and a general warm fuzziness when writing patches for code that has complete test coverage. This is all well and good until a platform specific bug pops up, causing the above warm fuzzies to disappear in a hurry. Gems that work in 1.8 but not in 1.9, in MRI but not JRuby, or on OS X but not on Windows, are a hassle and get in the way of building cool stuff.

###What is Gem Testers?

Gem Testers was built to make it easier for gem developers to write gems that run everywhere. This is split into two pieces:

rubygems-test is a Rubygems plugin which automatically runs the test suite of a specified gem on install. The output of the test suite is then collected, along with machine configuration details (Ruby version, machine architecture, etc) and posted to gem-testers.org.

Gem-testers.org is a central repository for test data. Test results are organized primarily by Ruby version and operating system. This allows gem users to see quickly if a given gem version is expected to work for their particular machine configuration. Gem Testers also provides RSS feeds and JSON responders in case an application needs to read in test results for other applications that might need to slurp in the data.

Why was this built?

How it is now:

• The developer writes/publishes a gem
• The user installs the gem and hopes it works
• If the gem does not work, an experienced user might:
  • Google around to find if this is a known bug, or where to report it if it is not
  • Create an account to get access to report said bug
  • Write a description of the problem to best of knowledge
  • Copy and paste the error output from their terminal
  • Remember to reply to the gem author when they discover it’s a platform specific bug and ask for machine configuration details.
• What do most users do? Get frustrated because the gem doesn’t work; give up trying to find a solution.

How it is with Gem Testers:

• The developer writes/publishes a gem with tests and rubygems-test support
• The user (who has rubygems-test installed) installs the gem, tests are automatically run and reported
• The gem’s test results page is updated with the results of executing the test suite with all relevant output and machine specs needed by a gem author to fix a bug.

When a new version (such as 1.9.3), or implementation (such as Rubinius) of Ruby comes along, it’ll be immediately clear what gems work and which need some fixes to work with said new release.

Few gem developers have a Xen box capable of spinning up all possible system configurations for testing their gems. Getting test results from an OS X machine is valuable when a gem developer doesn’t own anything made by Apple.

Note that bugs will not be reported with this method if there is not sufficient test coverage where the bug exists, but this will serve as a smoke test for different machines and versions of Ruby, and will reward gem developers for comprehensive test coverage with more useful Gem Testers pages (so write better tests!).

Ultimate result? Ruby gets better gems that don’t break as often and are fixed quicker as bug reports require less effort and initiative from gem users.

###How you can get involved:

Install rubygems-test. Bonus points for configuring it to run and report test results automatically in your .gemrc:

	`test_options:
	  auto_test_on_install: true
	  install_development_dependencies: true
	  upload_results: true
	  force_install: true # set if you want to install gems even if they fail tests.
	  force_uninstall_on_failure: false`

Gem authors: be sure to update your gem to support rubygems-test by following the steps in the README.

The more gem developers that support rubygems-test, the more valuable the tool gets for users. The more users that install the rubygems-test gem and upload results, the more valuable the tool gets for gem developers.