Understanding Engine Yard Cloud Takeover

Two basic tenets of cloud computing are that 1) servers are dispensable; and 2) any two servers in the same role are interchangeable. When an instance fails or detects an impending failure, it must be replaced with another. When performed on one of your app servers, Engine Yard Cloud refers to this process as a “takeover.” A takeover is specifically the process by which an app master is replaced by an app slave.

In Engine Yard Cloud, your environment’s Elastic IP is attached to the ‘app master’ instance. During a takeover, an ‘app slave’ is promoted to ‘app master’ and takes control of the IP address. The practicalities of running Ruby applications require the app master to be special in a few other ways, such as running cron tasks and migrations. If an ‘app slave’ fails, no takeover is necessary, because you can simply boot a replacement slave.

During takeover, a new app slave instance will be booted up to replace the one promoted to master. With the recent introduction of Stack Change Management, the new instance will be on the same Engine Yard stack version as the rest of the environment.

On the newly-promoted app master, a few things will happen. These include reconfiguring HAProxy and spinning up a new application server with the snapshot of the failed app master’s /data file system.

There are a couple key implications of this takeover process:

1. In your environment, there should be at least one "spare" application server. This is in addition to the application servers that provide the capacity while a new application server is spinning up. For example, if you need a minimum of three application servers to serve the everyday traffic, there should be four application servers during normal operation. Without a spare instance, your site could fail or slow down under load during the takeover.
2. If, after an app slave is promoted to master, you find yourself missing the data you had written to the `/data` mount, you should strongly consider storing such data in a ‘cloud-friendly' store, such as S3 or your own service running on ‘utility' instances.

By understanding the takeover process and implementing your system according to best practices, you can take better advantage of the Engine Yard Cloud platform and minimize potential downtime or data loss.