Overview

Engine Yard Cloud serves as a managed platform for deploying Ruby on Rails applications, operating on AWS EC2 instances. This involves configuring and deploying Rails applications to suit the needs of each project.

In order to ensure that an EC2 instance is configured to be used with a certain Rails application, Engine Yard will use AMIs (images) with a preset configuration, and further apply changes as required via Chef. In this article, we will review the role of Chef in Engine Yard and the process of a Chef run.

Contents:

• What is Chef?
  • The EY version of Chef
  • EY Cookbooks
• Preparing the Chef run
  • Stack differences
• Executing the Chef run
• Customizing Chef
• Summary

What is Chef?

Chef is a utility developed by OpsCode that works with the Ruby framework. It allows for configuring a Ruby environment in a programmatic way ("infrastructure as code"), which is leveraged by Engine Yard to ready its instances for a Rails app deployment.

Some important terms related to Chef are the following:

• Recipe: A set of instructions executed by Chef.
• Cookbooks: A set of Recipes. Each Stack has its own Cookbook, which can be found on GitHub for each version.
• S3: A managed storage service from Amazon (Simple Storage Service).

The EY version of Chef

As it happens with many other elements in Engine Yard, our usage of Chef is nonstandard:

• A normal Chef deployment has a Chef server, and instances will check periodically with the server for configuration changes.
• However, in Engine Yard, we use a subset functionality called Chef Solo, which runs on demand. In our case, it will be AWSM (EYC's back end) that prompts Chef to run.

This essentially means that most of the information relating to Chef available online is not applicable to Engine Yard cases.

EY Cookbooks

The Cookbooks used by Engine Yard are available on GitHub. Each version of the stack has a different set of supported Cookbooks:

• ey-cookbooks-stable-v5
• ey-cookbooks-stable-v6
• ey-cookbooks-stable-v7

Note that, by design, customers can modify Chef with custom Recipes and, in v6 and v7, the use of environment variables.

Preparing the Chef run

When an instance needs to be configured, it will be necessary to first prepare the environment to be able to run Chef. While the exact process depends on the stack, the first step will be to configure the system to run Enzyme:

1. Primer or MNML pull dna.json
2. The Ruby version is validated
3. The existing Gems (Ruby libraries) are validated.
4. A check is run to ensure that the required directories are in place.
5. The /etc/engineyard/dracul.yml file for Enzyme is created.
6. Finally, Primer or MNML executes Enzyme.

Afterward, Enzyme runs the following processes:

1. Enzyme gets the required information:
   • The dna.json.
   • The S3 URL to pull the default stack Recipes.
   • The S3 URL to pull the customer's custom Recipes.
2. Enzyme fetches all required Recipes and unpacks them in /etc/chef.
3. Enzyme runs Chef.

Stack differences

Please note that Chef is prepared differently depending on the version of the stack:

• v4: AWSM SSHs into the instance and executes Enzyme, and Enzyme runs Chef.
• v5: AWSM SSHs into the instance and runs the EY Primer, Primer runs Enzyme, and Enzyme runs Chef.
• v6 & v7: AWSM SSHs into the instance and runs MNML, MNML runs Enzyme, and Enzyme runs Chef.

Executing the Chef run

Once Enzyme has prepared all required data, Chef is executed. A typical Chef run is executed as follows:

1. Chef will check dna.json to identify the starting point of the execution list (ey-init::main).
2. The required Cookbooks are synchronized.
3. Cookbook Gem dependencies (Ruby libraries) are installed.
4. Cookbooks are compiled. If any syntax errors are present, an error will occur at this stage.
5. Recipes are executed. You can search for “Recipe:” in the logs to find references to the recipe being executed.

Note that Chef will exclusively execute scripts found inside the Recipes folders (/etc/chef/recipes/cookbooks/<recipe_name>/recipes/).

To better understand the way in which the Chef Cookbooks are processed in Engine Yard, please refer to our article on The Anatomy of Chef Cookbooks in Engine Yard. Troubleshooting information is contained in Debugging the Chef Run.

Customizing Chef

As mentioned before, by design, customers are intended to configure Chef runs to fit their needs. To do this, you must create and upload the custom Recipes. While these Recipes will overwrite whatever exists in the EY default Recipes, the recommended process is to modify the blank ey-custom Recipes. If you absolutely need to overwrite the default behavior of the core recipes, it should be done using the overlay method.

The process through which Enzyme adds the custom Recipes depends on the stack. On v5 and newer, it is as follows:

• Enzyme will get a compressed file with the Recipes for the stack.
• It will unpack the contents in /etc/chef.
• Then, it gets another compressed file with the custom Recipes uploaded through the UI.
• It unpacks the new file and overwrites the contents of /etc/chef.

On v6 and v7, environment variables are supported that will be parsed from json.dna during the Chef run (ey-base::custom). This will prompt Chef to install additional Recipes or modify existing ones, and are prefixed by 'EY_':

EY_<app_name>

The same set of Environment variables can be used to configure the applications. If not set, the instance size will determine their default values.

On v4 and older, custom Recipes are applied in a separate Chef run that is executed after the main Chef run, which is why there are two separate logs for Chef.

Please refer to Debugging the Chef Run for tips on identifying customizations to the Chef run.

Note: On v7, the core recipes are all prefixed with 'ey-'. You should avoid naming your recipes in such a way as to not generate confusion.

Summary

Engine Yard uses a customized version of Chef, which runs on-demand and follows a set of instructions called “Chef Recipes”. The preparation process involves checking the Ruby version, gems, folders, and pulling needed Recipes from S3. Once prepared, Enzyme runs Chef Solo, which executes the Recipes from the starting point identified in dna.json.