HAProxy Health Check

The default load balancer for Engine Yard Cloud is HAProxy. This runs on all Application instances, but is utilised only on the Application Master instance, as DNS should be configured to resolve the application domain(s) to the Elastic IP address attached to that instance. Therefore all application traffic hits HAProxy on the Application Master instance is then load balanced out to Nginx across all Application instances in the Environment. The Chef configuration runs made when instances are provisioned, terminated and restarted ensure that the HAProxy configuration file is kept up to date with the required instances.

HAProxy uses a health check system to decide when instances are in a healthy state to accept traffic. By default this check is a simple TCP connection on the Nginx port, meaning that if Nginx is running the instance will be seen as healthy.

In certain cases, such as an instance restart or problem with the functionality, configuration or application code of a specific instance it may be the situation that Nginx is running but the application not responding. In such cases requests directed to the instance by HAProxy will fail to load successfully, timing out or returning an error response/message.

To prevent such cases we allow for the setting of a HAProxy Health Check Path. When configured, the HAProxy health check will instead utilise a HTTP request to the requested path. Any 2XX, 3XX or 503 (returned when the maintenance page is in place) response is considered healthy, whilst all other responses (including no response) are considered unhealthy and the instance pulled from the load balancing pool.

The HAProxy Health Check Path is configured at the Application level and can be set either when creating a new Application or by editing an existing Application. If editing an existing Application then an Apply will be required afterwards to update the HAProxy configuration file on the instances.

In order to do a holistic test of your application and database, it is recommended that the path should connect to your application and not be set to a static or cached page, though you should avoid any paths that are complex or slow to load, in order to avoid any unnecessary resource consumption. It is also recommended that if you use analytics that you exclude this path from analytics collections. 

Once the health check has been setup is it recommended to check the Nginx logs (SSL and non-SSL) on all application instances to inspect the health check response code. In these logs you will see regular GET requests to the health check path coming from all the local IPs of the application instances (as HAProxy runs on all application instances at all times in readiness, but is only ever utilised on the application master) and can see the HTTP response. A 200 response is desirable, and whilst a 3XX is considered valid the redirect will most likely come from Nginx, meaning the health check is only checking that Nginx is running rather than the application health. Therefore ensuring the health check path is not being redirected is recommended.

Note: On the stable-v5 stack SSL termination is handled by HAProxy and all requests are forwarded to Nginx as HTTP requests, including health check requests. For applications with HTTPS redirects in place the health check will receive a redirect response from Nginx and so pass the check regardless of the application state. Therefore it is required to exclude the health check path from redirect in order to have a valid health check of the application itself.