An Update from Travis CI

Engine Yard is well known for sponsoring Open Source projects and pushing Ruby forward. It has sponsored the development of crucial building blocks such as Rubinius, JRuby, Rails, RVM, but also smaller and less well known projects, as part of their Open Source Software Grant Program.

Now, since a few weeks ago Travis CI is the next happy participant of this program.

Travis CI is a hosted continuous integration service, which is completely free for Open Source projects and which itself is open source. We run tests for over 20 thousands projects (over 2.5 million of tests run in total to date) and thus help increase the quality of open source software in general.

I happen to be the very lucky guy who is sponsored by Engine Yard to work on Travis CI.

When I was first asked if I could imagine to work on Travis CI at a conference I couldn’t believe it and the next day I was actually wondering if this was not a dream. I love working on open source projects, so this would be the best job I could get!

I participated in the Ruby Summer of Code 2010 (also co-sponsored by Engine Yard) in the past, working on improvements to Rails Engines in Rails 3. I also try to contribute to Open Source as much as I can in my spare time - not always easy when you have a stressful full time job, so being able to work on Open Source, with a fantastic team, on a really relevant project is even more awesome for me.

I started working on Travis CI before Engine Yard offered to sponsor me, so I already knew what it feels like and it feels just great.

Travis CI is quite challenging technically, because with so many projects on board we need to scale well and in order to do that Travis CI is going towards a distributed architecture. Travis CI also uses quite a few bits of cutting edge technology, which pushes web development forward - like Ember.js to power its main web client and up to date JRuby versions for running background services.

But working on Travis CI also means participating in something relevant that changes our everyday lifes for the better. Travis CI makes failure visible across the community. Thereby it uncovers a lot of bugs in crucial components of our development environment and over time improves stability in general. I think it’s fair to say that Travis CI has changed the way we build software. And being able to contribute to this just makes me happy. ###What have I been working on so far? I have implemented a few features, like support for encrypted env vars which allows to include sensitive data to your test suites (like credentials for accessing external services) and fixed a range of issues and bugs. But the biggest part of my work so far was helping get the new web client and API done and released.

Our client is written in Javascript using Ember.js framework. If you don’t know Ember.js, it announces itself as a framework for creating ambitious client-side web applications. When you look at the main Travis CI web client (link), you may wonder what’s so ambitious about it - it displays a few repositories on the left, the current build details in the middle and a bit more information, like status of workers and queues on the right. The ambitious part is that it is “live”: lots of data is being updated at all times through websockets. If you keep the site open you will see new repositories and builds coming in, build jobs being queued and worker statuses updated. And, maybe most importantly, you can watch the output of your test suite being streamed to the client live. ###How does it work? First thing worth noting is that our client and API are deployed under different domain names, respectively travis-ci.org and api.travis-ci.org. The major reason for such approach is isolation. Client and API are completely different apps, one written in javascript and the other in Ruby. It makes much more sense to develop, test and deploy them separately.

In order to allow cross domain ajax requests, we use Cross-origin resource sharing, usually referenced with CORS. CORS is automatically used in browsers which support it, when you make a request to the other domain. The part which needs additional work is the server. When browser sends cross domain ajax request it needs to actually do 2 requests:

1. OPTION request, which checks for existence of CORS headers
2. If response from the previous request returns proper CORS headers, the browser will send an actual request.

You can read more about CORS in the specification.

The obvious drawback of such flow is the fact that now we need to send 2 times more requests. Although it may seem like a problem, in practice it does not matter that much, because OPTION response is usually very fast. If you can use SPDY, the problem can be decreased even further.

What really matters to me as a developer is the ease of work with such setup. Most of the time I don’t need to have API running locally. I can just point the client to the production API and check new features or bug fixes against it. This is extremely valuable, because usually you can’t have production data on a local machine (for good reasons). That way I can catch errors more easily and check my code more thoroughly. ###How does Ember help us build Travis? As I mentioned earlier, although Travis may look as a simple app, we try to do a few things that make it more ambitious. Travis is a one page application, which means that you don’t need to reload a page while you browse and also that you should not need to reload to have a fresh data. A few of the challenges you face when you build such applications are:

• updating views whenever data changes
• keeping the data fresh
• managing nested views
• cleaning up unneeded stuff (so the app can be open for a longer amount of time)

Ember helps us with all of those problems. I will try to briefly explain how exactly does it work.

The first point is pretty much no brainer in Ember. Its view layer deals with updating your DOM elements whenever it’s needed.

Considering that you have a simple template that looks like that:

Hello !

and an object which is the context of this template is:

person = Em.Object.create({ name: 'Piotr' });

the output you get is Hello Piotr!

This is kind of expected and there is nothing interesting about it. The fun starts when you update name field on the object, in Ember it can be done with a set function:

(person.set('name', 'John');)

Now the output will be automatically updated to Hello John!

The second thing from the list is also quite easy to achieve, thanks to Ember and Pusher. Pusher is a service that allows us to handle live updates with websockets easily. Whenever something is updated on the server, we send an event to pusher, which is then delivered to the subscribed clients. When we update objects in Javascript, the templates are automatically updated.

Handling this case is also much easier thanks to Ember Data. Ember Data is the library written on top of Ember.js, which helps with data management and supports defining models and relationships between them. Ember Data isolates the models defined in the client from data source (such as an API), which means that after you define how your data looks in an application, you can easily change a way you get it from the server.

A quick example of the way it works may be our job queue and repository list. When a new job is created, we send the job:created event, which is handled on the client by creating a new Travis.Job record. Job queue displays all the jobs, which have not been started yet. When Travis.Job record is created, all of the collections that observe changes will be notified and refreshed - in this case, filter function attached to an observer will check if the new record has created state and update UI accordingly. The similar thing happens on the repository list. We display 30 repositories with most recent builds there. When the application receives the build:startedevent, it will update the associated repository and as a result, Ember will take care of updating and ordering the repository list properly.

The last, but not the least, of the hard problems connected with building one page applications is managing views and events. This is really important if your application is not “flat”, ie. it has nested views and if you want it to run for a long period of time without reloading it.

The thing that makes is hard is keeping track of what should we clean. In Travis, for example, we sometimes display jobs list in the build view. Jobs list is needed if a build contains more than one job, for example you may want to run your tests on different versions of Ruby. Without Ember, if we wanted to destroy a build view to display some other build, we would have to destroy the current build view and displabefore displaying the new view, we would have to destroy the previous one and make sure that we removed all of the attached event handlers. It could look something like

destroy: function() {
  this.jobViews.forEach(function(view) {
    view.destroy()
  });
}

If we ever forget about any of the nested views, we now have a memory leak. But that’s not all! We often need to use event listeners. We need to remember about cleaning them, too. So, our code would have to be even more complicated:

destroy: function() {
  this.element.removeEventListener('mousemove', this.onMouseMove);
  this.someLinkElement.removeEventListener('click', this.onSomeElementClick);

  this.jobViews.forEach(function(view) {
    view.destroy()
  });
}

You may imagine that with a complicated application it may get messy pretty quickly and what’s even more important, developers should not need to handle such boilerplate code.

Ember.js on the other hand keeps track of nested views and clears them up automatically. You also don’t have to deal with event handlers - there is only one handler for each event, which is delegated to the proper view.

Of course there are still edgecases and things that we have to handle manually, but Ember constantly gets better at letting you to focus on your application, and not the boilerplace code. ###What else will I be working on? We released a new client about a month ago, but I’m continuing development in this area, looking for a ways to make it smoother and faster, adding more features and port the old mobile client.

My next bigger task is to allow people to easily upload files produced during a build to some sort of storage. Those files are often called artifacts and they can be really useful in various different scenarios. Some projects with acceptance test suites can do screenshots, which can be later investigated if a build fails. Other projects, like Ember.js, may build latest library version or docs and upload them for people to use. And these are just a few use cases, I’m sure that people will come up with incredible ways to use this feature.

Currently nothing stops developers from building their own scripts to save build artifacts, but this is just reinventing the same thing over and over again. We would like to address it and make artifacts management as simple as adding a few more lines in a configuration file and provide access to many different storage providers or services like Github, which allow to upload files for projects hosted there.

I will also do community support, take care of our issues on GitHub and act as a contact person.