* This article refers to chef recipes on our V4 stack. If you are running stack V5 review our Custom Chef for V5 documentation.
The Engine Yard Cloud stack that we provide is carefully provisioned and controlled by Chef cookbooks. This main collection of Chef recipes will set everything up on your instances for you whenever any of the following events happen:
- You create a new instance.
- You add a new instance to a running cluster.
- You trigger chef using the dashboard’s “Upgrade” or “Apply” buttons.
- You use the engineyard gem’s “recipe” commands.
If there is something that you need to configure on one of your instances, or across an environment that is not provided in our stack by our main run, you have the ability to make changes yourself with root access to each instance. However, while it may be tempting to quickly install a package or modify a config file on your instances, you need to bear in mind that these changes will all be lost, should any of the aforementioned events occur. To give you the ability to modify your instances, but to also provide persistence over rebuilds, we carry out a second Chef run, in which you can add your own recipes. In the user interface we refer to these as Custom recipes. For detailed instructions on how to create, upload and apply a custom recipe, see our Custom Chef Recipes documentation. In the rest of this post, we’ll round out your knowledge of the custom Chef system, by showing you some examples that are commonly used and pointing out some of the pitfalls that you should be aware of. With great power, comes great responsibility!
Anatomy of a Chef Run
As you start to create custom cookbooks, you are likely to encounter errors as you perfect the recipes. Having a good idea of what happens when a Chef run is in progress can help you understand what might be going wrong. When an instance is created via the dashboard or a run is triggered through applying recipes to an environment, the following processes take place:
- The main Chef run begins
- The correct version of the stack (our main cookbooks) is pulled down to your instances and a
chef-solo
run is started, using our cookbooks. - All the parts of our stack that need to be setup for the environment, based on the options you chose when configuring via the dashboard are configured.
- As part of this configuration, various services will be reloaded or restarted.
- The instance communicates to the Engine Yard Cloud back-end and lets the dashboard know whether the run was successful. This is what defines whether the status light against your instance is red or green.
- If the main run successfully completes, the custom Chef run begins.
- The last version of the cookbooks that you uploaded with the
ey recipes upload
command is downloaded to your instance and stored into/etc/chef-custom/
- Another
chef-solo
run is started, using this different set of cookbooks. - The instance again communicates with the Engine Yard Cloud back-end on the status of this run. Again, the status light will change to red or green, depending on whether this run was successful or not.
The logs for each run can be accessed via the dashboard, or can be found in /var/log/chef.{main,custom}.log if SSH is more your thing. You’ll need to look through these if the status light is red, and find out what caused the error(s).
Common Custom Recipes
Changing the timezone
Our AMI sets our instances up in the 'PST8PDT' timezone by default. Based on your location, you may want timestamps in logs to match your timezone. Changing the timezone on the instances involves modifying a symlink, then restarting some services (e.g.: cron, syslog, etc) so that they take in the new time. It’s a pretty simple recipe to do this (using Great Britain in this example):
# Note that this is for the GB timezone. Look in
# /usr/share/zoneinfo for your relevant file.
service "vixie-cron"
service "sysklogd"
service "nginx"
link "/etc/localtime" do
to "/usr/share/zoneinfo/GB"
notifies :restart, resources(:service => ["vixie-cron", "sysklogd", "nginx"]), :delayed
not_if "readlink /etc/localtime | grep -q 'GB$'"
end
To use this recipe, create yourself a new cookbook, as per our docs, then use the above as an example for the default.rb recipe file. You can see the different timezones that are available for use by looking under the /usr/share/zoneinfo/ directory.
Ready-made cookbook
While the above details serve as useful instructions for changing the timezone, we have since added a cookbook to do this to our custom recipe repo: https://github.com/engineyard/ey-cloud-recipes/tree/master/cookbooks/timezone
Installing masked packages
Engine Yard Cloud instances are running a version of Gentoo Linux and as such, you are able to install extra packages that you can find available in any Gentoo distribution. The simplest way to install a package is via the dashboard, and any packages installed via this method will persist rebuilds as they are added in to the main Chef run. If you are familiar with Gentoo’s portage system, you are probably aware of the concept of packages being "masked". Packages can be masked for various reasons, but usually it just indicates that the package maintainer has not carried out sufficient testing on the package to be comfortable to mark it as stable. This often is no cause for concern, but do be aware that a masked package can introduce instability into an environment. You cannot install masked packages from the user interface so you need to use a custom chef recipe to accomplish this.
To install a masked package:
- Make sure you have a recent copy of the ey-cloud-recipes main cookbook.
- Add the enable_package and package blocks to your custom recipe.
For example, to enable version 2.7.8 of libxml2 you'd use:
enable_package "dev-libs/libxml2" do
version "2.7.8"
end
Then to install libxml, you'd use:
package "dev-libs/libxml2" do
version "2.7.8"
action :install
end
Custom MySQL configs
If you need to customize the MySQL configuration, we provide methods to do this as outlined in our docs. Often, if people are making these changes with custom recipes, they usually follow up with a restart of MySQL via an execute call to /etc/init.d/mysql restart
, or via a service method with the "restart" action, not realizing that Chef’s service notification methods can be used to conditionally restart MySQL, only if the custom configuration file has actually changed. The following snippet added into your custom recipe would make sure that MySQL is only restarted if your custom config template actually changed since the last run:
service "mysql" do
supports :restart => true
action :enable
end
if ['solo', 'db_master', 'db_slave'].include? node[:instance_role]
template "/etc/mysql.d/custom.cnf" do
owner 'root'
group 'root'
source 'custom.cnf.erb'
notifies :restart, resources(:service => 'mysql')
end
end
Pimp my prompt
If you run any large environments and use SSH heavily, you may find that you’re often scratching your head trying to figure out what instance you’re currently logged in to. This can easily be rectified by modifying your PS1 shell prompt to provide you with the details. However, if you rebuild your environment, then the details you added could well be stale. The following recipe is perfect for modifying your prompt to show you what instance you are currently on. It displays the instance role, the instance ID and the name of the environment. You could expand this recipe to make other changes to your Bash shell environment too. https://github.com/omgitsads/ey-cloud-recipes/tree/custom_prompt/cookbooks/custom_prompt
Kudos to our App Support Engineer, Adam Holt for that one.
Monitoring custom scripts with monit
If you have any custom daemon-like scripts or rake tasks that you need to have running on your instance(s), it’s a good idea to put them under the watch of monit, to make sure that they behave (i.e.: can’t consume too much RAM or CPU resource) and are restarted automatically, if they crash for any reason. Monit relies on PIDs and PID files to keep an eye on processes, and it’s a good idea to create a wrapper script for your script that will ensure that a PID file is created for this purpose. Here’s an example recipe that will create a wrapper script for you that drops PIDs and the relevant monit configuration file so that your script/task will be monitored: https://github.com/tjl2/ey-script-wrapper
Again, to use this recipe, just clone it, copy into your cookbooks directory, modify the variables in the default.rb file and require it in your main cookbook recipe file.
The Node Object
As you will see from some of these examples and from the documentation, you can pull out details about the instances & environments from the node
object. However, aside from the examples that are peppered throughout our custom recipe repo and the docs, how do you know what information you can get from node
? During both of the Chef runs, the node object contains a combination of everything Chef knows about the instance along with details we provide about your environment, instances and applications. You can find out what Chef knows about your instance by obtaining output from the ohai
command (you can then parse the resultant JSON file in IRB): sudo ohai > ~/ohai.json
Likewise, you can find the other Cloud-specific details that we make available via the /etc/chef/dna.json file. Make sure you copy the file with sudo and change the ownerships first, before parsing it in IRB:
sudo cp /etc/chef/dna.json ~/dna.json && sudo chown deploy:deploy ~/dna.json
The above assumes that you have left the deploy user with the standard name, "deploy". Please change as appropriate if required. To parse these files you can then do the following in IRB:
# Run these commands first:
# sudo ohai > ~/ohai.json
# sudo cp /etc/chef/dna.json ~/dna.json
# sudo chown deploy:deploy ~/dna.json
# Then start an irb session in ~
require 'json'
ohai = JSON.parse(File.read('ohai.json'))
dna = JSON.parse(File.read('dna.json'))
# Examples...
# Find the instance type:
puts ohai['ec2']['instance_type']
# Find the instance role:
puts dna['instance_role']
When running your scripts, the contents of both of the hashes in the above snippet will be available in the node object.
Warnings and Caveats
While the provision of the second Chef run allows you an almost unlimited scope for customizing your instances, it can also give you enough rope to hang yourself with! Bear the following in mind when you start to customize your environment:
- Be careful if you make changes to Nginx or HAProxy port numbers - this commonly occurs when people try to implement caching technologies which want to listen on the ports that are reserved for HAProxy or Nginx. This can often lead to problems when the main Chef run happens, as HAProxy or Nginx can’t restart properly as they try to bind back to their original ports. Use
netstat -nap | grep LISTEN
to see our current ports that you shouldn’t use before starting any recipe that will modify port numbers. - If you’re currently customizing a solo instance environment, always think through how your changes may differ if you grow your environment to a cluster. For example, make use of the
node[:instance_role]
to ensure that recipes that should only run on an application instance don’t try to run on a DB instance, for example. A common issue that occurs when customers upgrade to a cluster from a solo instance arises when they have been setting up custom Nginx server configurations. This can cause problems because clusters have HAProxy sat in front of Nginx and Nginx then listens on port 81 (and 444 for SSL connections), rather than 80 (and 443). Custom configs need to be adjusted accordingly. - If you're running a clustered environment and have been testing out custom recipes, always make sure you don't leave a production environment running with broken recipes. While the recipe may not be causing problems that interfere with the running of your application, if a takeover occurs in the meantime, this may fail when the custom recipes are encountered. This can leave your environment stuck in a takeover loop - continuously killing the app_master and trying to add another in its place due to the errors being thrown by Chef.
- When testing recipes that are reconfiguring something that is always first set up by the main run, make sure that you hit the 'Apply' button on the dashboard after you have run
ey recipes apply
(which only runs the custom recipes). Often, changes such of this work the first time they are run, but can often then cause problems that will only show up in the main run when it next happens. - Custom recipes are not run on a promoted application master in the event of a takeover (see https://support.cloud.engineyard.com/entries/20996781-application-master-takeover). Usually, this should not be an issue, as nothing that your recipes do will have been undone since the instance was last put through a Chef run, but if you have custom recipes that set up cron jobs on the application master, or you have recipes designed to do something to the application master only, then these will not be applied to the promoted instance automatically and you'll need to hit the 'Apply' button on the dashboard.
Professional Services
If you get stuck or you just want to give us the specs of the chef recipes you need, don’t forget that we offer the help of our professional services team to help you get them done. They can give you the extra expertise you need to get you from a red light to a green light.
Further Reading
For more information about... | See... |
---|---|
Opscode Chef resources | http://docs.opscode.com/resource.html. |
A starter repo for custom chef recipes on Engine Yard Cloud | https://github.com/engineyard/ey-cloud-recipes. |
Opscode public cookbooks | https://github.com/opscode-cookbooks. |
37 Signals cookbooks | https://github.com/jsierles/chef_cookbooks. |
running custom Chef recipes during deploy | Custom Chef Recipes During Deploy blog by Dr. Nic Williams. |
Comments
Article is closed for comments.