Fine-Tuning Memcached

Are you getting memcached interruptions in your application logs? Does memcached say its been running but your app says something else?

###Diagnosing the Problem

The first step is to check memcached stats with Telnet. If you’re running on Engine Yard (we use Gentoo) you can run emerge -g netkit-telnetd on your memcached server. You can then telnet localhost 11211 and then type stats. You should see something like this:

stats
STAT pid 27631
STAT uptime 104
STAT time 1401932723
STAT version 1.4.5
STAT pointer_size 64
STAT rusage_user 0.004999
STAT rusage_system 0.010998
STAT curr_connections 10
[...]
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 4354
STAT bytes 0
STAT curr_items 0
STAT total_items 0
STAT evictions 23420009
STAT reclaimed 0
END

The two stats that we will be focusing on are conn_yields and evictions.

###Do you have conn_yields in your stats?

Connection yields are the amount of times the memcached server has throttled a specific client. If you have conn_yields you are likely using large multi-get requests. It is possible to lower the sizes of your multi-get operations or increase the -R value, or both.

The default value of -R is 50 connections from a client before throttling. Note that not all memcached clients respond the same way to the connection yield throttling. Some clients reconnect without an issue, while others may report that the memcached server is unavailable. You will likely want to increase these numbers until you no longer see the conn_yield statistic being throttled.

###Understanding Evictions

You can verify you have enough space on the server and aren’t over allocating memory. Running free -m will make sure you have memory to spare.

Likely your slabs aren’t in ideal allocations for the objects you’re storing. You can verify the status of your slab memory allocation by running telnet localhost 11211 and then stats slabs. This command allows you to view how your slabs are allocated. For more on slab statistics, please check out this article.

The breakdown of those statistics are per slab and as a whole at the bottom. If there are large amounts of active slabs, but very few active items in those slab classes, you will likely want to alter the default growth factor.

If you have large amounts of active slabs all with very high active usage and high eviction count you may wish to increase the amount of slabs you have available.

The default growth factor for memcached is 1.25. There isn’t a magical best number for every application. If you need more available slabs you would lower the growth factor (-f) number from 1.25. If you needed less slabs you would raise the number.

###Conclusion

Memcached is a great option for caching some objects. But out of the box, it isn’t going to be fine tuned for every application’s data set. You’ll want to alter your growth factor (-f) and -R settings to reduce conn_yields and evictions. These will better tune memcached for your application’s needs.

For more on memcached memory usage, check out this article. Management of your memcached memory is the key to having a better, more smooth running environment on Engine Yard.

P.S. Have you fine-tuned your memcached setup in the past? What tips or tricks do you have? Any gotchas to share?