Migrating Your Application to a New Host

Migrating to a new host can be a daunting task but with proper testing and preparation, it can be a seamless transition. As a member of the Professional Services and Deployment teams, I’ve handled many migrations to Engine Yard as well as between Engine Yard environments and wanted to share some best practices.

Before you plan the migration, it is very important to test out the application on the new environment. Some of the more painful migrations I have been involved with were a result of incomplete testing. Functionality should be tested but it is also important to put load on the new environment to ensure it can handle the expected traffic. A performance monitoring tool such as New Relic is crucial in planning your load testing strategy. In choosing endpoints to test, I like to pick a combination of controller actions that consume the most time and have the highest throughput. A performance monitoring tool can also help you decide how many concurrent users you should send at the application. Using a tool such as Apache benchmark or Blitz.io, you can ramp up to double or triple that number to figure out when or if your application reaches unacceptable performance levels.

With testing complete, it’s time to focus on the steps required to switch over to the new environment. A complete plan consists of a pre-migration plan, a migration plan and a rollback plan. No one expects to have to rollback but it is important to have the steps in place so that you are not scrambling if things go wrong. I like to focus on the migration plan first, as this will dictate what your rollback steps and pre-migration steps look like. A basic migration plan typically consists of the following steps:

1. Put up maintenance page on current environment
2. Stop Rails processes on current environment
3. Move database and other data to new environment
4. Start Rails processes on new environment
5. Test new environment
6. Switch DNS to new environment

The steps above are very generic and many applications will require more steps than that. I want to focus first on the most important step, the database migration. There are two ways to move your database data over to the new environment and the method you select will depend on the size of your data set. If you have a relatively small data set, a simple dump and restore will work. I would recommend doing a trial run to get exact timing. If the trial run takes longer than the amount of downtime you are willing to accept, then you should consider replication. This method is a bit more complex and requires configuration of an SSH tunnel but it minimizes downtime as you just need to promote the slave database in the new environment to master.

The second big consideration for many applications are assets. Ideally, your application is using a CDN or storage service but that is not always the case. The best way to handle moving assets is with rsync. Rsync will allow you to perform an initial sync when you setup the environment, then every subsequent rsync will only copy across the new or changed files. Depending on the size of your assets, you will probably want to run another rsync ahead of the migration to minimize downtime. I like to run an rsync 24 hours in advance, then again 2-3 hours in advance.

There are many other steps that you may need to consider. Some considerations I’ve encountered over the years are:

• Cron jobs - Some cron jobs can't be run on the new environment until it is live, these need to be moved over or uncommented during the migration.
• Redis data - If you are using Redis, you may need to move over a Redis dump.
• Background Jobs - You may need to let background jobs finish on the old environment or move the data store for these jobs over to the new environment.
• Cache - If you are caching on the filesystem, it may make sense to rsync that data over so the cache will not have to warm up again on the new environment.
• Search Indexing - After database data is moved over, any search indexes should be rebuilt.

The pre-migration plan will typically consist of tasks such as setting up replication, running rsync and lowering the TTL setting on DNS. This should take place about 24 hours in advance. For some people, the amount of downtime associated with the DNS switch is too much. In these cases, you can use a tool such as iptables to forward all traffic from the old servers to the new servers. This will catch any user that has not received the DNS change and forward them on to the new IP. This will not work in all situations because every forwarded request will have the same source IP, which will be that of your old servers. If your application does things like rate limited based on IP address or utilizes GeoIP for location tracking, you will not be able to use this method.

The rollback plan should basically be your migration steps reversed. The rollback plan should cover switching back even if live traffic has already been sent to the new environment. If an issue is uncovered before DNS is switched over, the rollback will be much simpler, so make sure your testing plan during the migration is thorough enough.

Hopefully the best practices outlined in this post will help you optimally prepare for a migration. Proper preparation and testing are crucial in minimizing downtime and ensuring that everything goes smoothly. Potential customers who would like assistance migrating to Engine Yard should reach out to the Professional Services team. With lots of experience migrating customers to Engine Yard, we have the experience necessary to make your migration seamless.

If you have any other questions about Professional Services, head on over here and learn more.