Troubleshooting Intermittent 500/404 Errors in Unicorn After Ruby Version Change

Overview

If you are experiencing issues with your environments after deploying changes on the version of Ruby the application uses, such as intermittent 500 / 404 errors for all pages in your app, the problem might be related to stale unicorn worker processes.

The error logs may show an error similar to the following:
Your Ruby version is XXX, but your Gemfile specified YYY

However, running ruby -v will show that the version has been successfully changed.

If you have attempted to change the Ruby version, but the change did not seem to take effect as expected, this article will guide you through the solution.

Note: this article is written based on Unicorn, but the same concepts apply for Puma and Passenger.

Solution

When changing the Ruby version, a cold restart of Unicorn is required for the new version change to take effect, as the workers will try to use the old version otherwise. However, this will result in downtime during deployment. To perform a zero-downtime Ruby version change, follow these steps:

  1. Disable app master takeover by editing the environment and setting 'Disable' on the 'Takeover Preference' option.
  2. Upload any Chef recipe changes, but don't apply them.
  3. Change the version of Ruby on the environment, but don't apply the changes.
  4. For each app instance, hide every other instance from deployment.
  5. Stop Nginx on the chosen instance(s), either using `ey-core ssh` or by logging in them.
  6. Stop Unicorn on the chosen instance(s), either using `ey-core ssh` or by logging in them.
  7. Hit 'Apply' only on the chosen instance(s), this will prompt Chef to install the newer version of Ruby.
  8. Deploy the application prepared to run the newer version of Ruby. This will start Unicorn, but not Nginx.
  9. Test the instance(s) for Unicorn working properly. Feel free to run any command, or redeploy. As long as Nginx is stopped, the instance will not get traffic.
  10. Once you are satisfied with the status, start Nginx on the instance(s) to bring them back into the LB rotation.

Summary

By following the steps outlined above, you should be able to successfully change the Ruby version without experiencing downtime or errors. Remember to test each instance thoroughly before bringing them back into the load balancer rotation.

FAQ

  1. What is a cold restart of Unicorn?
    A cold restart of Unicorn is a complete stop and start of the Unicorn server. This is necessary when changing the Ruby version as the workers will not restart correctly, otherwise.
  2. What is the 'Takeover Preference' option?
    The 'Takeover Preference' option allows you to disable app master takeover, which is necessary when changing the Ruby version.
  3. What does 'Apply' do in this context?
    'Apply' prompts Chef to install the newer version of Ruby on the chosen instance(s).

Comments

Article is closed for comments.