Replace Degraded Instances

Overview

Read this page if you have received notification from Engine Yard that one of your instances is degraded and that you need to terminate the instance and boot another.

Engine Yard gets notification of degraded instances from Amazon and forwards the message so that you can take action.

When you get one of these notifications, it might state the time that the underlying hardware will be taken offline. If there is no time specified, replace the degraded instance as soon as you can.

If you don’t replace the degraded instance in time, it might become frozen. For information about frozen instances, see Fix frozen or crashed instances.

What is a degraded instance?

An instance is degraded when the host hardware that the instance is running on fails. The hardware failure is not catastrophic. However, Amazon needs to shut down the host to carry out maintenance. When this maintenance happens, all instances on the host are terminated.

To replace a degraded instance

If the degraded instance is:

  • single-instance (or solo) environment, then terminate and rebuild the whole environment (See ELT on notes below).
  • An application slave, then terminate the specific degraded instance by clicking TerminateInstance.png to the right of the instance on the dashboard, and then add a new instance (See ITR on notes below).
  • An application master, then promote an app slave to be app master.
  • database replica, then terminate the specific degraded instance by clicking TerminateInstance.png to the right of the instance on the dashboard, and then add a new instance (See ITR on notes below).
  • database master, then promote a database replica to be database master.
  • utility instance, then terminate the specific degraded instance by clicking TerminateInstance.png to the right of the instance on the dashboard, and then add a new instance, using the most recent snapshot (See ITR on notes below).

Notes:

  • You must know the difference between the "Environment Level Terminate" (ELT) and the "Instance Terminate reference" (ITR). The first one is identified by the button "Terminate" located at the top of the environment's page, above its name. The second one is identified by a "Terminate" link next to each instance ID.
  • In some cases (and depending on your Support level), Engine Yard might initiate an instance takeover on your behalf. In that case, Engine Yard Support will contact you.
  • The environment's settings for takeover preference and failed app master behavior can also affect the way app takeovers occur.

Additional information

Check following articles for more information on possible fixes or guidance to replace instances:

Comments

0 comments

Article is closed for comments.