Overview

When working with Engine Yard, it is crucial to understand where files are located inside an application instance. This knowledge aids in carrying out investigations and addressing issues effectively.

This article aims to be a starting point when working with an Application Instance. We will go over the application instance's roles, its directory structure, and the location of the main logs that may be investigated.

Contents:

The application instance
- Important directories
  - Application deployment: /data/<app_name>/
  - Other directories and files
- Relevant logs
Summary

The application instance

Application instances in Engine Yard are used for hosting a web application, whether it be a webpage or API. They always run a web server (Nginx) and one of several application servers (Unicorn, Passenger, or Puma).

For a given application, there can only be one application master. This differs from other application instances in the following ways:

The application master has an elastic IP attached to it so traffic can be received. All incoming web traffic is directed to the app master and distributed to other instances through the load balancer.
Deployment is executed on the app master, so it contains all logs concerning deployment. Other instances may have deployment hook logs.

Important directories

When working with Engine Yard, the following directories will normally need to be accessed when investigating and addressing issues:

Application deployment: /data/<app_name>/

The application deployment directory is used in the deployment process, and contains the following relevant elements:

Subdirectories & files	Description
releases	Contains the directories for successful deployments made on the instance.
releases_failed	Contains the directories for failed deployments made on the instance.
current	It's a symlink to the current 'release' folder.
current/deploy	Contains the deploy hooks, which allow for customizing the deployment process and are managed by the customer.
shared	Contains files used by all versions of the code, to prevent duplication. These are managed by Chef.
shared/config	Contains shared config files. Env files within the folder contain the environmental variables set in the Engine Yard Web UI.
shared/log	Symlink identical to that in /data/<app_name>/current/log/, pointing at /var/log/engineyard/apps/<app_name>.
shared/tmp	Used by the Rails app to store temporary files.
shared/bin	Contains the control files used by the deployment process.
shared/pids	Has the PIDs of the applications being managed by Monit.
shared/cached-copy	Contains a copy of the codebase, so pulling the code from GitHub does not require downloading all files during deployment.
shared/bundled-gems	Contains the installed Ruby libraries ("Gems"), so that only new and modified files are installed on deployment.
shared/assets	Similar to bundled-gems, but contains the JavaScript assets installed by Yarn during deployment.

Other directories and files

Directories & files	Description
/lib/systemd/system/	Contains the systemd configuration files. systemd is used in place of Monit in newer versions of the stack, and is a native OS utility.
/etc/monit.d/	Contains the relevant Monit configuration files relevant to Engine Yard.
/etc/init.d/	Contains service initialization scripts for Nginx, MySQL, etc.
/etc/chef/recipes/	Contains the installed cookbooks. In the v4 stack, etc/chef-custom/recipes exists as well for custom recipes.
/etc/chef/dna.json	This file contains all information about the instance type and use: When the instance is created or 'Apply' is clicked, triggering a Chef run, AWSM creates a new version of the file based on the existing configuration and pushes it to the instance. The file contains data such as instance role, environment type, Ruby version, etc. Variables contained here are used by Enzyme as an input when running Chef.
/etc/engineyard/	Contains configuration files: dracul.yml: Contains the URLs from which recipes are downloaded, among other data. instance_api.yml: Contains an instance ID and token for talking back to AWSM. Used by collectd to send alerts to the UI. collectd.conf: Contains the collectd configuration file.
/engineyard/bin/	Contains the control scripts for application servers.

Relevant logs

The existing flows and applications in the EY stack log their operations and errors to a variety of logs. This is a list of the logs that should be accessed during troubleshooting:

Deployment logs

Logs related to the deployment of an application's codebase to the instance.

Location

Description

/home/deploy/

Contains the last deployment log; for older logs, the Engine Yard web UI must be used instead.

The application master contains all deploy-related logs, while other applications only contain the deploy hooks.

Configuration & Chef-related logs

Logs related to instance configuration and Chef runs.

Version	Location
v4	/var/log/chef.main.log /var/log/chef.custom.log /var/log/enzyme.log
v5	/var/log/chef.log /var/log/ey-enzyme.log /var/log/ey-primer.log
v6 & v7	/var/log/chef.log /var/log/ey-enzyme.log /var/log/ey-mnml.*.log

Note: Chef runs work slightly differently depending on the version of the stack:

v4: AWSM SSHs into the instance and executes Enzyme, and Enzyme runs Chef.
v5: AWSM SSHs into the instance and runs the EY Primer, Primer runs Enzyme, and Enzyme runs Chef.
v6 & v7: AWSM SSHs into the instance and runs MNML, MNML runs Enzyme, and Enzyme runs Chef.

Application logs

Logs related to the execution of the application and the relevant web stack.

Location

Description

/data/<app_name>/current/log/

Contains logs for the rails application:

<environment_framework>.*.log: Logs configured for the deployed application.
Unicorn, Puma, and Passenger logs are also located here.

/var/log/engineyard/

Contains the following logs:

Nginx logs
Potentially, any other logs used by the application: Passenger, PHP-FPM, etc.

System logs

Other logs concerning OS operations.

Location	Description
/var/log/collectd.log	Lists all activities from the monitoring and alerting stack, corresponding to the alerts shown in the web UI. collectd can be configured as required.
/var/log/monit.log	Internal Monit logs, showing its actions.
/var/log/messages.log	OS log with info from applications run in the user's space. While Engine Yard rarely uses it directly, applications in utility instances and other apps may write to it.
/var/log/syslog	OS-related logs, logging information for cronjobs, collectd errors, Monit activities, kernel messages, etc.
/var/log/auth.log	Shows user access to the system, including all SSH attempts.

Summary

Understanding the file locations in Engine Yard application instances is crucial for effective troubleshooting and issue resolution. This guide provides a comprehensive list of directories that are typically accessed when troubleshooting Engine Yard.

Application Instance Cheat Sheet