Varnish: It’s Not Just For Wood Anymore

In previous posts, I’ve routinely mentioned a piece of software called Varnish. Varnish is a caching reverse proxy for web traffic, and if your job or your interests lean toward production web applications at all, you definitely want to get familiar with it.

This post isn’t going to try to make a case for using a caching reverse proxy, as I think that’s already sufficiently covered. Instead, it’ll focus specifically on an overview of Varnish; what you need to do with it out of the box to implement decent caching for a typical web application, and some of the more sophisticated capabilities you’ll want to get familiar with.

Varnish was written from the ground up to be a high performance HTTP accelerator. It leverages its host operating system’s own memory management abilities and threading abilities in order to provide a high capacity, high throughput caching system. It has many features that make it a nice tool, but it does avoid the massive feature bloat of many other caching proxy implementations.

Among these are load balancing and graceful handling of dead proxy back ends, a built-in Perl-esque configuration language that permits sophisticated behavior customization, url rewriting, and support for the most useful parts of ESI.

As previously mentioned, Varnish is threaded. Specifically, it manages a thread pool—or set of thread pools (as determined by configuration)—, and it uses one thread for each connection. This generally works well, but it does make you need to think about configuration a little bit. If you configure varnish to accept 20,000 concurrent connections, then it’ll be running 20,000 system threads. Make sure you’re on a system that isn’t going to have its manhood threatened by that situation before you do it in production.

Installing Varnish is straightforward. It’s available in most operating system package management systems, though it may not always be the most recent version; definitely check that you’re getting an acceptably recent version from your package manager. You can also easily build it from source if you want to ensure that you have the most recent version. Simply download the source from Sourceforge. To build:

./autogen.sh
./configure
make
make install

With Varnish, building it isn’t quite the end of the story. While it’ll run out of the box with its build defaults for all parameters, it doesn’t actually run very well that way. It’ll deliver great performance to a point, but the defaults allow it to be overwhelmed pretty easily. Running a durable Varnish instance requires a bit of configuration love, and the command line configuration options are legion.

varnishd -a :80 \
-b 127.0.0.1:81 \
-T 127.0.0.1:6082 \
-s file,/var/lib/varnish,100GB \
-f /etc/varnish/default.vcl \
-u nobody \
-g nobody \
-p obj_workspace=4096 \
-p sess_workspace=262144 \
-p listen_depth=2048 \
-p overflow_max=2000 \
-p ping_interval=2 \
-p log_hashstring=off \
-h classic,5000009 \
-p thread_pool_max=1000 \
-p lru_interval=60 \
-p esi_syntax=0x00000003 \
-p sess_timeout=10 \
-p thread_pools=1 \
-p thread_pool_min=100 \
-p shm_workspace=32768 \
-p srcadd_ttl=0 \
-p thread_pool_add_delay=1

The command line is outrageously long, but don’t hyperventilate. You won’t be typing this by hand in a production deployment anyway, because your startups are all scripted, right? I am not going to go over every one of these settings—the Varnish web site has lots of getting started documentation to guide you when things get confusing. However, let’s take a look at a few of the more interesting parameters that you should know about when configuring.

-a :80

The -a option provides a host and port for Varnish to listen to. If the host is omitted, the given port is listened to on all interfaces.

-b 127:0.0.1:81

This provides a single default backend for varnish to proxy to. It also accepts a HOST:PORT pair.

-s file,/var/lib/varnish/100GB

Varnish functions by allocating a system controlled area of memory to use to store the cached data. This can either be a malloc allocated area, specified by the keyword ‘malloc’, or an area backed by a file, specified by the keyword ‘file’. The file variant uses mmap, while the malloc variant, with a large cache, will make use of swap space and the swapping subsystem.

If using the malloc type of storage, the only option that one provides is the amount of memory to allocate for the storage area. This is a number, in bytes, or a number in bytes suffixed by:

• K or k for kibibytes
• M or m for mebibytes
• G or g for gibibytes
• T or t for tebibytes

This is pretty straightforward. Tune your cache to the amount of content that you have and the amount of space you have at your disposal. These cache spaces only persist for the life of the Varnish process, which means that if Varnish is killed and restarted, the cache must be populated anew. However, as of version 2.1.0, which was released on March 24th, there is now experimental support for persistent caches.

-p overflow_max=2000

When there are more accepted requests than there are threads to handle them, Varnish sticks them into an overflow queue. If the overflow queue fills up, and the listen queue (the size of which can be controlled with the listen_depth option) is full, then requests start getting dropped.

Requests that are just sitting there waiting to be handled take up space, so this parameter shouldn’t be set absurdly high with no reason. That said, it needs to have a bit of a ceiling to allow for traffic spikes to occur without detrimental effects. This also lets it survive things like someone pointing ‘ab’ at the proxy and saturating it with requests.

Increasing the size of this parameter is one of the crucial changes from the default configuration which is necessary to help ensure a production capable Varnish deployment. If you don’t change it, it’s quite easy to DoS Varnish with something as ubiquitous as Apache Bench.

-p thread_pool_max=1000

-p thread_pools=1

-p thread_pool_min=100

-p thread_pool_add_delay=1

Taken together, these three parameters describe the thread pooling behavior for Varnish. The thread_pools parameter is self describing. Generally, you probably want one pool per core.

The thread_pool_min parameter gives a bottom limit for the number of threads to maintain, per pool, regardless of traffic. Don’t keep this too low, or you may limit Varnish’s ability to rapidly respond to traffic spikes when it’s otherwise not very busy. At the same time, setting it too high just increases the amount of time the OS spends babysitting threads that aren’t doing anything, so practice moderation.

When Varnish doesn’t have enough threads in its thread pool(s) to handle the traffic, it creates new ones. In order to avoid swamping the system, there’s a delay between the launching of each thread. The parameter that controls this is thread_pool_add_delay. This defaults to 20 milliseconds, but that’s far too long to handle load spikes. The prevailing wisdom right now is to set it at one to two milliseconds.

Finally, Varnish has a limit on the total number of threads that it’ll spawn. This is thread_pool_max. Pay attention here. The semantic is different between thread_pool_min, which is the minimum per thread pool, while thread_pool_max is the maximum, collectively. So, for example, if one has thread_pools=4 and also has thread_pool_max=1000 then that means that the entire Varnish process is limited to 1000 threads; this is not a per pool attribute. At the same time, if one had thread_pool_min=100, then there would be a minimum of 400 threads running at all times; that is 100 per thread pool.

These command line options just scratch the surface of what one can do with Varnish. Some of the best features of Varnish come from the Varnish Configuration Language.

This language, commonly called VCL, is a domain specific language that is used to customize Varnish’s request handling and caching behaviors. In appearance, it is reminiscent of a Perl kept to the most C-like basics, but it is pretty easy to both read and use. VCL is also very fast because Varnish actually translates the VCL code into C and then compiles it into a shared object, on the fly, so even complicated logic expressed in VCL has little overall impact on Varnish performance.

Varnish runs with a simple, functional default VCL configuration, but you may add to the configuration by providing your own VCL file like so: -f /etc/varnish/default.vcl

In the sample command line, above, Varnish is given a single back end to proxy to. However, backends can be defined in a VCL file, and when doing so, additional information about the behavior of a backend can be encoded.

backend fast {
  .host = "fasthost.mydomain.com";
  .port = "http";
  .connect_timeout = 1s;
  .first_byte_timeout = 2s;
  .between_bytes_timeout = 1s;
  .probe = {
    .url = "/ping";
    .timeout = 1s;
    .window = 4;
    .threshold = 4;
  }
}
 
backend slow {
  .host = "slowhost.mydomain.com";
  .port = "http";
  .connect_timeout = 6s;
  .first_byte_timeout = 8s;
  .between_bytes_timeout = 3s;
  .probe = {
    .request ==
      "GET /ping HTTP/1.1"
      "Host: pinghost.mydomain.com"
      "X-Ping: true"
      "Connection: close";
    .timeout = 5s;
  }
}

Take note of those .probe sections. These are completely option, but if provided, Varnish will use them to perform health checks on the backend. The .window and .threshold options can be used to provide a health tolerance. That is, given a certain number of checks (the .window), how many have to have been successful (the .threshold) for the backend to be considered healthy.

Varnish also has some support for load balancing between backends. It currently only supports round-robin and random selection, but this behavior can be controlled via VCL, as well.

director plump robin {
  { .backend = fast; }
  { .backend = slow; }
  /* Yep, you can define them inline, too */
  {
    .backend = {
      .host = "alternate.mydomain.com";
      .port = "8080";
    }
  }
}

or

director grasshopper random {
  .retries = 3;
  {
    .backend = fast;
    .weight = 9;
  }
  {
    .backend = slow;
    .weight = 1;
  }
}

As additional features, you can define both access control lists and grace periods for cached content in Varnish. A grace period is simply a period of time after an object in the cache has expired during which it can still be returned in response to a request. You’d use this if there are objects in the cache that take a long time to generate, in order to avoid having a bunch of requests piling up waiting for the generation of the new object.

These parts of VCL are just scratching the surface of the power of VCL, though. VCL offers the ability to define subroutines for grouping your VCL code, several useful built in functions for regular expression matching and cache manipulation, and a whole host of built in subroutines which serve as hooks into the entire request/response cycle for Varnish, allowing you to customize any point of that cycle.

For example, let’s say that you are using a round robin director to load balance between backends, and you want to be sure that if a request for a given resource from one backend fails, no more attempts to that backend, for that resource, are made for a short period of time, to allow it to recover from whatever problem it is having. You can do that with VCL.

sub vcl_recv {
  set req.grace = 60s;
}
 
sub vcl_fetch {
  if (beresp.status == 500) {
    set beresp.saintmode = 15s;
    restart;
  }
  set beresp.grace = 60s;
}

As another example, consider the case from my last post, where I used a Ruby proxy to cache content from Redmine. Redmine isn’t particularly cache friendly, returning cache control headers that normally don’t allow any caching of content. If you wanted to, though, you could make Varnish do it, using VCL.

sub vcl_fetch {
  /* This just says that no matter what those cache control headers are saying, */
  /* insert the content into the cache with a TTL of 60s */
  if (obj.ttl < 60s) {
    set obj.ttl = 60s;
  }
}
 
sub vcl_hash {
  /* This causes varnish to use the cookie contents as part */
  /* of the key for storing and looking up content from the */
  /* cache. For Redmine, this would mean per-user contents */
  /* in the Varnish cache. */
  set req.hash += req.http.cookie;
}

That’s pretty nice. A few lines, and the behavior of the cache is significantly customized. However, remember earlier in the article when I mentioned that VCL is translated into C code and compiled on the fly into a dynamically linked shared object? Well, this means that you can embed arbitrary C code into your VCL:

C{
  #include
  #include
}C
 
sub vcl_mylibstuff {
  C{
    mylib_superfunction(VRT_r_req_request(sp);
  }C
}

The capabilities of VCL are far too expansive to cover well in a short post, but this should give you a taste of how flexible and powerful Varnish is. The Varnish wiki has expanded documentation (which should continue to improve) and a number of examples of using VCL to do useful real world work. Varnish is a true power tool for caching. Check it out, and leave questions and comments here!