EY Instance Access Controls and Logging

Access to customer instances operates on multiple layers, starting with the booting of an instance. When a new instance is booted, a key unique to that instance is generated with the KeyMaster service, and is installed onto the instance via BootStrap and user-data. Up until this point the authorized_keys file has been empty. Once this key is installed, the Engine Yard infrastructure can access the system (via this unique key) in order to run Chef, or to allow Engine Yard staff to have shell access to the instance.

At this point in time only our internal systems, and Engine Yard staff who have been given access to our bastion host system, called Hadrian, can access the instance. After the KeyMaster key is installed, the Engine Yard infrastructure will log into the instance and initiate a Chef run. The Chef run handles all of the customization of the instance, and one of the first things that it does is address SSH keys.

When Chef runs, it installs the user’s keys to root and to the deploy user. It also installs an internal deploy key with whitelisted IPs. This will include all of the keys for the Owner of the account, and for all collaborators who are permitted access in the UI. At this same time, Chef also installs for emergency keys corresponding to our general support time zones - US-East, US-West, Asia Pacific, and the EU.

These four keys are installed so that if there is a failure in a key component of the access infrastructure, such as KeyMaster or Hadrian, Support will still have access to the systems. These are unique protected keys that exist only for this purpose, and they are closely held by the Engine Yard support management team.

The final step in key management involves appending extra customer provided keys to the authorized_keys file. This is a feature provided by the platform. If a customer creates a file named “extra_authorized_keys” and places it into the .ssh directory alongside the “authorized_keys” file, the contents of the “extra_authorized_keys” file will be appended to “authorized_keys”. This is performed for both the root and the user account on the instance, and Engine Yard does not control what is placed into that file.

Each Chef run will regenerate the authorized_keys file, so manual edits to that file will be lost when Chef runs again. To summarize, there are four basic groups of keys that are inserted into the “authorized_keys” file:

1. The unique KeyMaster key that both the Engine Yard platform and Hadrian uses.
2. The internal key, common to all slices that EY_Serviside uses to deploy. Whitelists only other instances in that environment.
3. The four emergency keys held by the Engine Yard support management team.
4. User keys from the dashboard as well as anything added manually via “extra_authorized_keys”

Of these various keys, only access through Hadrian is specifically logged. Hadrian is our bastion host system which is used for EY Staff to get shell access to a given instance. Access is controlled through individual SSH keys for each individual, in combination with a Two Factor Authentication key. All access attempts through Hadrian are logged, regardless, of whether they were successful or not. In addition, keystroke logging is performed on every Hadrian session, so that there is a log of everything that an EY staff member typed while logged into an instance.

These logs can be queried whenever there is a question about accesses, when they happened, or what was done when they occurred. Simply file a support ticket with your inquiry, and we will be happy to answer it.