System Inaccessible After Database Restart

Problem

System became inaccessible after Database was restarted using the Dashboard by clicking the button Restart as in the Figure 1:

Dashboard.png

Figure 1

Subsequent accesses prompt error message screen as in Figure 2:

Screenshot_21.png

Figure 2

Solution

The restart button on the dashboard does not act as a system bounce (a typical shutdown/start) done at the server prompt. It actually physically stops and restarts the whole instance on a new underlying host. The result is that the public and private hostnames and IPs change, making inaccessible all previously accessed applications.

After such restart is done, it is necessary to run a chef "Apply" from the Dashboard and then either restart the web server (nginx) or perform a deploy so that the application can read the updated PostgreSQL configuration file (database.yml).

As a caveat, it must be said that restarting a relational database that is struggling should typically be left as an action of last resort. Databases rely on the OS filesystem cache to store data and index references and reduce disk access when responding to queries. When you restart or reboot, this cache gets wiped forcing you to rely on slower disk access for your subsequent requests.