Overview

By default, Engine Yard offers a suite of DB operations services with its deployed instances. This article provides an overview of these concepts, how they are implemented in the platform, and the main procedures to address related issues.

Contents:

• Database Operations
  • Backup
  • Snapshots
  • Replication
• Summary

Database Operations

As part of the normal database setup in Engine Yard, Chef will enable and configure a series of services for its managed schemas. These include DB backups, snapshots of the EBS volume where the DB is located, and replication of a master DB by a slave replica.

Backup

Backups are logical copies of a DB schema. To manage backups, Chef will install a tool called ey-backup:

• When run, ey-backup will create a backup and upload it to an S3 bucket located under the AWS account owned by the customer.
• The backup schedule is configured through the settings selected in the Engine Yard UI, and enforced by a CronJob that executes the ey-backup tool.
• Another tool, ey-restore is in charge of restoring these backups from the copy stored in S3.
• Support can only provide a backup if the account owner makes the request.

Logs for backups are located under /var/log/eybackup.log. To check the CronJob running the backups, use the command crontab -l.

Snapshots

Snapshots are physical copies of the underlying EBS volume where the DB is stored. These are created through the use of ey-snapshot:

• When run, ey-snapshot will create a snapshot and upload it to an S3 bucket located under the AWS account owned by the customer.
• The backup schedule is configured through the settings selected in the Engine Yard UI, and enforced by a CronJob that executes the ey-snapshot tool.
• Snapshots can be used as a source when creating a new instance.
• Snapshots cannot be provided by Support.

<supportagent>

• Snapshots cannot be provided by Support unless the BU provides authorization. Note that customers who bring their own AWS account will have access to these snapshots in either case.

</supportagent>

Logs for snapshots are located under /var/log/ey-snapshots.log. To check the CronJob running the snapshots, use the command crontab -l.

Replication

When enabled, Chef will configure the replication process for the instance. By default, replica databases are read-only; changing this is considered to be in the Professional Services scope. 

While there are many reasons why replication can fail, a majority of the time this will be related to the running application. Sharing the logs with the developer may assist in resolving this type of issue. Other reasons why replication can fail are:

• The master or replica DBs are overloaded on CPU usage and are taking too long to catch up with the replication process.
• The master or replica DBs have run out of disk space.
• One of the instances has failed.

Since replication is monitored by collectd, the status is tracked under /tmp/check_mysql_status/. Note that the information is updated once every minute. Some of the error states shown here are as follows:

• For issues where the replication is too far behind the master, you should check whether the time is decreasing with every refresh of the file. Otherwise, consider Troubleshooting any Database Issues present.
• If replication is not working, you will need to check the error logs as explained in Troubleshooting any Database Issues.

Finally, the following commands can be used to show the replication status:

• MySQL: 

  mysql “show slave status\G;”

  Note that Slave_IO_Running and Slave_SQL_Running need to be "yes" for replication to run.
• PostgreSQL:
  On master, run: 

  select * from pg_stat_replication;

  On replica, run: 

  select * from pg_stat_wal_receiver;

Summary

This article provides an overview of how backups, snapshots, and replication are handled in Engine Yard, and the role Chef plays in configuring these services.