How To Dissect a Clojure Web Application and Deploy on Engine Yard

In this post, I’m going to build a simple Clojure Web app and deploy it to a Jetty application server running on Engine Yard’s cloud application management platform. In a follow-up post, I expand my Web app and show how to integrate it with a MySQL database server running in the Engine Yard environment.

To build my web app, I’m going to use the most common tools in the Clojure web stack:

• Ring, a thin abstraction on top of HTTP request and response handling, and
• Compojure, a small routing library for working with Ring.

Ring and Compojure are built on and interoperate with, the Java HTTP Servlet (API), so applications built with them can be deployed on any servlet container, such as Jetty or Tomcat.

In you already have a Clojure application, to deploy with Engine Yard, please jump to the packaging section.

If you are new to Clojure Web application development, read on.

What is Clojure?

The description at Clojure.org captures the essence of Clojure as a dynamic, functional language with a solid runtime foundation on the Java Virtual Machine:

Clojure is a dynamic programming language that targets the Java Virtual Machine (and the CLR, and JavaScript). It is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language - it compiles directly to JVM bytecode, yet remains completely dynamic. Every feature supported by Clojure is supported at runtime. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection.

Clojure is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system. Clojure is predominantly a functional programming language, and features a rich set of immutable, persistent data structures. When mutable state is needed, Clojure offers a software transactional memory system and reactive Agent system that ensure clean, correct, multithreaded designs.

What is Engine Yard?

Engine yard offers a cloud application management platform for Ruby, PHP, Node. and Java applications. Since we’re going to use it with a JVM language, how does Engine Yard’s Java deployment work?

When you create a Java application environment with Engine Yard, the application server is a cluster of Java stack instances created and provisioned by Engine Yard in virtual machines running in the cloud. Each server instance is an Ubuntu server with a Jetty or Tomcat server running in a Java Virtual Machine. You upload a .war file with class files compiled from your Java (or in this case, Clojure, code), including its dependent libraries, and Engine Yard deploys it to your Jetty or Tomcat app servers.

This post won’t teach you how to program in Clojure, or how to get the most from the Ring and Compojure libraries, or even scratch the surface of using Leiningen. I’d recommend Dmitri Sotnikov‘s excellent book Web Development with Clojure as a comprehensive introduction to these topics. The guestbook app in Dmitri‘s book is definitely the inspiration for the todo list app in this blog post series.

But, this post will give you a flavor of each of these topics and show you how to package and deploy your Clojure web app onto a cloud platform. That said, let’s jump in.

Prerequisites

You need:

• A Web application using the Ring and Compojure libraries. We’ll build a simple one from scratch, if you don’t have one already built.
• Your application must use Leiningen because we are going to package the app and all its dependencies into a war file with lein-ring commands.
• An Engine Yard account. Sign up here.
• A Java 7 JVM (Get the JDK - it’s got all the dev tools)

Steps

The steps we’ll go through are:

1. Create the todoclj project. If you are starting with a Clojure app already built, jump to the packaging in Step 2.
2. Package the project as a .war file
3. Create a Java application environment using Engine Yard and deploy the .war file
4. Test the application

Create the todoclj project.

We’re going to use Leiningen to manage our project. Leiningen allows us to create a project skeleton, build, test, package into a WAR file, and even run it’s own test server. I think it also makes coffee, but I haven’t found that option yet. Think of Leiningen as Maven for Clojure (but not half as annoying). Leiningen has a system of templates that allow for different application types to be created with the lein new command. We’re going to use the Compojure template to create a starter web app that will contain the structure we need.

Assuming we’ve got Leiningen installed as per the README, we just do this:

~/projects $ lein --version
Leiningen 2.3.4 on Java 1.7.0_40 Java HotSpot(TM) 64-Bit Server VM

~/projects $ lein new compojure-app todoclj
Retrieving compojure-app/lein-template/0.4.1/lein-template-0.4.1.pom from clojars
Retrieving compojure-app/lein-template/0.4.1/lein-template-0.4.1.jar from clojars
Generating a lovely new Compojure project named todoclj…

In this post, we’re going to walk through the skeleton code that the Leiningen template for Compojure creates, to get to know Clojure, Leiningen, Ring and Compojure a little.

Like Rails, Grails, Play and many other productive frameworks or project environments, Leiningen produces a runnable skeleton. So, now just do this to compile the generated code, start an embedded Jetty app server with the code deployed, and open a browser window pointing to your app’s homepage:

~/projects/todoclj $ lein ring server
Compiling todoclj.handler
Compiling todoclj.repl
Compiling todoclj.routes.home
Compiling todoclj.views.layout
todoclj is starting
2014-04-01 13:10:11.107:INFO:oejs.Server:jetty-7.6.8.v20121106
2014-04-01 13:10:11.173:INFO:oejs.AbstractConnector:Started [email protected]:3000
Started server on port 3000

Fitting Leiningen, Ring and Compojure together

Because Leiningen is so central to Clojure project workflow, its tutorial is essential reading. Leiningen has also helpfully included that full (markdown formatted) tutorial as part of its help. If you do this:

~/projects/todoclj $ lein help tutorial > ~/leintutorial.md

And now view the leintutorial.md file in your favourite markdown reader, such as Mou, Marked, or Draft.

The Compojure app template creates a leiningen project with this layout:

~/projects/todoclj $ find .
.
./.gitignore
./.lein-repl-history
./project.clj
./README.md
./resources
./resources/public
./resources/public/css
./resources/public/css/screen.css
./resources/public/img
./resources/public/js
./src
./src/todoclj
./src/todoclj/handler.clj
./src/todoclj/models
./src/todoclj/repl.clj
./src/todoclj/routes
./src/todoclj/routes/home.clj
./src/todoclj/views
./src/todoclj/views/layout.clj
./test
./test/todoclj
./test/todoclj/test
./test/todoclj/test/handler.clj

In the file layout for the entire todoclj project above, you see src directories for things you find in most Web applications, whether they are MVC-style or not: models, routes, views and in this Leiningen project, also handler.clj and repl.clj files.

What does Ring add? The project wiki for Ring, defines it clearly, as:

Ring is a low-level interface and library for building web applications in the Clojure programming language. It is similar to the Ruby Rack library, the WSGI in Python, or the Java Servlet specification.

The fundamental concept in Ring is a handler, which describes what to do with HTTP requests and responses. The Ring handler (e.g., todoclj.handler/app in the one generated in the ./src/handler.clj file) is just a function (sometimes referred to as ‘middleware’) that maps the request to the response, and is the entry point for the application. Compojure, meanwhile, manages routes.

Leiningen and Ring provide our project with a useful REPL (read-evaluate-print-loop), typical in any dynamic language environment for interactively testing code easily. So when we issue the command:

~/projects/todoclj $ lein repl
Compiling todoclj.handler
Compiling todoclj.repl
Compiling todoclj.routes.home
Compiling todoclj.views.layout
nREPL server started on port 51974 on host 127.0.0.1
REPL-y 0.3.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

Leiningen pulls down all the dependencies (from Clojars) and compiles all the Clojure source code in the project. We are now executing in a shell environment in the project’s namespace. So we can, for example, do this:

user=> (use 'todoclj.repl)
nil
user=> (start-server)
todoclj is starting
2014-03-31 10:56:29.802:INFO:oejs.Server:jetty-7.6.8.v20121106
2014-03-31 10:56:29.849:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:8080
Started server on port 8080
You can view the site at http://localhost:8080
nil
user=>

If you look at the todoclj/src/repl.clj source (listed below) we pulled into our REPL environment, we have just used, or executed, we can see the ring imports in the namespace defintion, and the implementation of start-server which we executed above, it kicks off a local jetty server, listening on port 3000 for HTTP requests and opens a browser window at the root context of the todoclj application.

(ns todoclj.repl
  (:use todoclj.handler
        ring.server.standalone
        [ring.middleware file-info file]))

(defn start-server
  "used for starting the server in development mode from REPL"
  [& [port]]
  (let [port (if port (Integer/parseInt port) 8080)]
    (reset! server
            (serve (get-handler)
                   {:port port
                    :init init
                    :auto-reload? true
                    :destroy destroy
                    :join true}))
    (println (str "You can view the site at http://localhost:" port))))

The todoclj/project.clj file generated by the Compojure template looks like this:

(defproject todoclj "0.1.0-SNAPSHOT"
  :description "todo demo app"
  :url "http://example.com/FIXME"
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [compojure "1.1.6"]
                 [hiccup "1.0.5"]
                 [ring-server "0.3.1"]]
  :plugins [[lein-ring "0.8.10"]]
  :ring {:handler todoclj.handler/app
         :init todoclj.handler/init
         :destroy todoclj.handler/destroy}
  :aot :all
  :profiles
  {:production
   {:ring
    {:open-browser? false, :stacktraces? false, :auto-reload? false}}
   :dev
   {:dependencies [[ring-mock "0.1.5"] [ring/ring-devel "1.2.1"]]}})

The :dependencies array in the project.clj tells us that, apart from clojure itself, we are depending on Compojure, Hiccup (a front-end templating library) and ring-server.

So, as we did before right after we generated the project, when you type:

~/projects/todoclj $ lein ring server

the :ring map in section of the project (i.e., the 3 lines below) is examined by the Leiningen executor and passes the :handler function to the Clojure runtime to execute the todolj.handler/app function.

:ring {:handler todoclj.handler/app
        :init todoclj.handler/init
        :destroy todoclj.handler/destroy}

So, looking at that todoclj/src/handler.clj file, we see definitions for that app handler function, and the init and destroy functions also included in the Ring map. As I mentioned earlier, Compojure builds on Ring to add routes to a Ring application. Routes return Ring handler functions for different HTTP commands, and this is what is going on in the app function below:

(ns todoclj.handler
  (:require [compojure.core :refer [defroutes routes]]
            [ring.middleware.resource :refer [wrap-resource]]
            [ring.middleware.file-info :refer [wrap-file-info]]
            [hiccup.middleware :refer [wrap-base-url]]
            [compojure.handler :as handler]
            [compojure.route :as route]
            [todoclj.routes.home :refer [home-routes]]))

(defn init []
  (println "todoclj is starting"))

(defn destroy []
  (println "todoclj is shutting down"))

(defroutes app-routes
  (route/resources "/")
  (route/not-found "Not Found"))

(def app
  (-> (routes home-routes app-routes)
      (handler/site)
      (wrap-base-url)))

We see the -> thread-first macro which is useful syntactic sugar, making Clojure heavily nested code more readable. In the app function above, the -> is threading the Compojure site function and wrap-base-url. The Compojure site function sets up Ring middleware for standard website aspects (such as sessions, cookies, multi-part params, etc.)

To define the routes for the :​todoclj.routes.home​ namespace, we need the ​defroutes function from Compojure​. Each route represents a URI to which your application responds, such as “/”, starting with the HTTP verb, such as GET or POST, followed by the parameters (none here) and the body (none here either).

(ns todoclj.routes.home
  (:require [compojure.core :refer :all]
        [todoclj.views.layout :as layout]))

(defn home []
  (layout/common [:h1 "Hello World!"]))

(defroutes home-routes
  (GET "/" [] (home)))

In this route for a GET on the root web context “/”, control is being passed to the home function which is calling the common layout, then using hiccup to add the :h1 "Hello World". You can see the common layout sets up the head, page, style and includes the body.

(ns todoclj.views.layout
  (:require [hiccup.page :refer [html5 include-css]]))

(defn common [& body]
  (html5
    [:head
     [:title "Welcome to todoclj"]
     (include-css "/css/screen.css")]
    [:body body]))

In the followup post of this series, we’ll see how to use more of hiccup to flesh out this todo list app layout.

Step: Package for deployment

As I mentioned above, Engine Yard doesn’t distinguish between JVM languages. It just deploys a .war file to a running Java web container (jetty or tomcat).

To build a WAR (or Web application ARchive) file using Leiningen, run:

~/projects/todoclj $ lein ring uberwar
Created /Users/richardwatson/projects/todoclj/target/todoclj-0.1.0-SNAPSHOT-standalone.war

You can then easily deploy the resulting WAR to Tomcat, Jetty or any other Java application server (just copy it to the webapps directory). We’re going to deploy this onto an environment we create on Engine Yard.

Handy Tip

A good rule of thumb for working with the JVM and Java environment in Engine Yard is: if an app and its dependencies can be packaged into a .war file and it deploys and runs fine on a local Tomcat or Jetty server, it should work fine on Engine Yard. But, of course, you can’t guess at (or hardcode) information such as hostnames, open ports, or arbitrary file access like you can on your local development environment. Any server-specific information like this that Engine Yard configures as it creates the environment will be exposed to your application through the Jetty (or Tomcat) Java web container (e.g., the JDBC database as JNDI [the next blog in the series will discuss this], other things like Cache server hostnames using ).

Step: Create a Java “application environment” on Engine Yard and deploy the app

Next we’ll walk through setting up an environment for deploying the Clojure app to Engine Yard. The steps we’ll go through to do this are:

1. Create a project and environment, choose an infrastructure provider, and select a blueprint
2. Customize the blueprint, selecting choices for language, database and application server
3. ‘Update’ the clusters of servers
4. Deploy the war file using the UI

We’ll select a blueprint that matches our use case from Engine Yard library Blueprints are complete descriptions of the application environment.

1. Create a project, choose an infrastructure provider, environment and select a blueprint

The first step is simply to login to your Engine Yard account.

When you are logged in, you are now in the organization page. In your organization, create a project, and an initial environment. For that environment, choose your cloud infrastructure provider (Amazon Web Services or Microsoft Azure). Next, select a blueprint. This is a description of an application’s environment. For our app, we’ll select a blueprint with 3 servers, one each for app server, load-balancer and database.

2. Customize the blueprint by selecting, language, application server and database

We will can now customize the blueprint with what we need for our Clojure application. Having picked our language (Java), we can change the default server locations, pick an alternate supported database or application server. We’ll go with Jetty, MySQL and servers in the AWS US-WEST-1 region.

3. Update the clusters of servers

Engine Yard builds and orchestrates the deployment of an application environment from its blueprint. This is called ‘updating’ in Engine Yard terminology.

Once the update has finished, each of the clusters of servers is up and running:

• one with haproxy and nginx to load balance between app server instances (should I add more) and to terminate SSL connections
• one server with a Jetty web container running in a Java 7 VM, and
• the database server with a running instance of mysql.

These healthy, running servers are indicated in the image below, by the green tick symbols in the server slots or placeholders in the environment.

4. Deploy the war file using the UI

The next step is to deploy the .war file we built with lein ring uberwar. Hitting the green “DEPLOY” button on the application cluster, Engine Yard provides a file selector, or you can drag and drop the .war file in the dialog box.

The deployer uploads the .war file, deploys it to the Jetty server, and the UI tells me the app is deployed.

Test your deployment:

The web module configured by the .war file will be deployed to the root context of the Java web container. To test the application, click the “VISIT YOUR APPLICATION” link in the top right of the application server cluster details page (above).

This link doesn’t point to port 80 (or 8080) on the application server as it would have to test your application on a local development environment. These servers in your cloud environment are all on the public internet (since I haven’t used a VPC). Engine Yard configures firewall rules and ports in this environment so that only explicitly required connections can be made to them and between them. The only externally facing open HTTP port is 80 on the Load balancer, which forwards HTTP requests for the app to the app server.

So, when we click “VISIT YOUR APPLICATION”, the hostname opened is that of the load-balancer. And, by doing that, we’re GETing the root context of my deployed web app, which hits the GET "/" [] home route in our Compojure config.

So, um, ok, another todo list app? ;) What’s the point? Let’s think about why you would do what we’ve just done.

Why would you build web applications using Clojure?

• I doubt if traffic to my todo list app is going to hockey stick, but handling web traffic on a real high volume site requires the application to manage concurrent processing of requests in some way. Different languages take different approaches to this. Clojure is built for concurrency by making the core data structures immutable (therefore sharable across threads). Writing real web apps based on purely immutable data structures is tricky, if not downright impossible, so Clojure also provides consistent, easy state changes, using, according to Clojure.org:

  The software transactional memory system (STM), exposed through dosync, ref, set, alter, et al, supports sharing changing state between threads in a synchronous and coordinated manner. The agent system supports sharing changing state between threads in an asynchronous and independent manner. The atoms system supports sharing changing state between threads in a synchronous and independent manner. The dynamic var system, exposed through def, binding, et al, supports isolating changing state within threads.

• The Clojure web stack is based on libraries, not frameworks. This approach offers plenty of freedom to mix and match libraries that perform various functions for web apps. This contrasts with the ‘kitchen-sink’ approach of a framework, such as Spring in Java, or Rails with Ruby. Clojure developers appreciate this just-enough approach that minimizes the code base for a project.

• Clojure offers expressiveness rivaling other dynamic languages like Ruby and Python, while interoperating easily with Java libraries and the rest of the Java ecosystem. In a Clojure application, you can invoke a public Java method, instantiate a class, or get the value of a public field using the special operator . also known as dot or host. Some of the simplest examples, to give you a flavor:

(import 'java.util.Date) ; import a single class
(def today (new Date))	  ;
(def today (Date.))		  ; alternatively use the dot operator
(def now (System/currentTimeMillis)) ; call a static method
		
(. foo bar 7 4)   ; call the method bar of the instance/class foo with arguments 7 and 4
(. alice bob)     ; return the value of public field bob of the instance/class alice

• Clojure is used by many innovative technology organizations for building full applications. In particular, SoundCloud and Engine Yard’s CI partners, CircleCI are doing interesting, mission critical and scalable things with Clojure.

Other Clojure Web App Approaches

Rather than package up the Leiningen project into a .war file, another popular approach for deploying Web apps in Clojure is to start jetty programmatically using ring.adapter.jetty/run-jetty and passing in Ring handlers. For people who don’t have significant customizations to make, then running “lein ring server-headless” is very typical.

Balancing the hype: Some bad press recently

Like any developing language eco-system, Clojure suffers from growing pains, that manifest themselves, for example, in the lack of out of the box bullet-proof security.

Why would you deploy Clojure code onto Engine Yard?

• Primarily because it’s easy. If it works in a local Tomcat or Jetty server, will a local MySQL or PostgreSQL database, it will probably work on Engine Yard unchanged. No need to change your code, or build process.
• Your Java components are maintained and kept up-to-date by Engine Yard. This include security updates to Java, which are now issued at least quarterly by Oracle, and Jetty/Tomcat updates, as well as JDBC library updates - all tested for compatibility by Engine Yard stack engineers.
• You can scale up by adding app servers to your cluster and the load balancer will round robin requests to each of your app servers (by default, other behavior is configurable).
• You can SSH into the server with the Jetty or Tomcat server to recover logs, customize logging, or reconfigure the app server
• Engine Yard will tune the JVM heap parameters according to the available memory on the VM you use](get docs reference
• You can choose to locate these app servers in different availability zones, different regions or even different cloud infrastructures (we currently support AWS and Microsoft Azure).
• Price and performance transparency: The virtual machine in which your JVM is running is not a container, or a slice of a larger VM. It is a full VM, and it’s all yours. For example, if you choose and pay for an EC2 medium, you can dedicate as much of the the 3.75Gb memory to the JVM as you wish.

Community Testing

Testing with developer community groups has always been important to Engine Yard. I want to thank Norman Richards and the Austin Clojure meetup for testing Engine Yard as part of their hack day in January. Thanks also to Norman for his talk, slides here which were really helpful for me understanding the state of the art in Clojure Web Apps.

The Irish Clojure user group meet at the Engine Yard offices in Dublin on 9th April for a functional kata session. I’ll be there helping people deploy their Clojure web apps onto Engine Yard.

Stay tuned for a follow-up post, in which I expand my skeleton todo list app with a way to add tasks, a data model, and show how to integrate it with a MySQL database server running in the Engine Yard environment.