Legacy Stacks stable-v1 and stable-v2 Instance Configuration and Application Deployment Failures

Engine Yard stacks stable-v1 and stable-v2 were EOLed several years ago and are no longer actively maintained. Their outdated software versions, and in particular their lack of support for TLS v1.2 causes compatibility issues and configuration failures which must be manually addressed in order to make the instances functional and capable of handling an application deployment.

IMPORTANT: Whilst this guide provides manual workarounds for known failures for platform related failures in Chef configuration and application deployment, it is likely that issues will also be seen with custom Chef recipes and application related deployment issues, which are the customers' responsibility to resolve. These workarounds are also not guaranteed to work long term, as 3rd parties further restrict compatibility or the support for, and availability, of old technologies. It is therefore highly recommend that a migration to a newer stack version is actioned as soon as possible.

This guide provides examples of commonly faced issues when configuring instances and deploying applications, along with their workarounds. The problems are listed roughly in order, and as one issue is resolved, the next is likely to be faced. Any given commands should be executed on all instances in an environment.

Environment Boot Failure

Problem: Environment boot attempt results in no instances being created.

Symptom: After shutting down an existing EC2-Classic environment, attaching a Network and attempting to boot the environment again with the same IP address the boot process silently fails and no instances are created. This is caused by the EIP still being associated with EC2-Classic and so not available for VPC usage.

Solution:

Boot the environment again, this time avoiding re-using the previous EIP by firstly making a note of the selected IP under External Address, then changing this selection to Add IP address.
Once the instance has booted navigate to Tools -> IP Addresses and make a note of the IP address with Status "attached to" for the newly booted environment.
At the bottom of the IP Addresses page click IP Migration Assistant.
In the assistant, set Source as the IP noted in step 1 and Destination the newly booted environment, leave all other options and click Migrate.
Once the migration completes the newly booted VPC version of the environment will be accessible via the same IP (and therefore same DNS records) as the previous EC2-Classic version.
As a final step, in order to avoid the costs associated with an unused EIP, visit the IP Addresses page and click the bin icon to release the IP noted in step 2.

Chef Configuration Failures

NOTE: As a first step, all Upgrades should be applied to the environment. This will ensure that the environment is running the newest available versions of any stack components along with the most up to date Chef configuration recipes, thus minimising the issues encountered.

NOTE: Once command line `chef-solo` Chef runs complete without issue, run a final Apply in the UI to have the instance(s) register the completed configuration run and be marked as healthy.

Problem: No Chef run occurs, dashboard Chef log is empty

Symptom: An Apply for the Environment triggered in the dashboard results in an empty Base Chef log.

Solution: Upgrade openssl in order to allow secure connections.

Method: On each instance run the command sudo eix-sync, then sudo emerge -av openssl, then click Apply in the dashboard again to re-run Chef.

Problem: RubyGems installation fails

Symptom: The dashboard Base Chef log shows:

gem install rubygems-update -v x.x.x --no-ri --no-rdoc --source https://rubygems.org/ returned 1, expected 0
---- Begin output of gem install rubygems-update -v x.x.x --no-ri --no-rdoc --source https://rubygems.org/ ----
STDOUT:
STDERR: ERROR: While executing gem ... (OpenSSL::SSL::SSLError)
SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A

Solution: Fetch gems via non-secure method to avoid SSL failures.

Method: On each instance perform the following:

Run the command sudo gem sources -r https://rubygems.org/ && gem sources -a http://rubygems.org/
Type vim /etc/chef/recipes/cookbooks/ruby/recipes/rubygems.rb, then use the arrow keys to navigate to line 6, press 'i' to enter INSERT mode, delete the 's' to change the gem_source to --source http://rubygems.org/, then press Escape to exit INSERT mode, then type :wq to save and quit.
Run the command sudo export PATH=$PATH:/usr/local/ey_resin/bin && sudo /usr/local/ey_resin/bin/chef-solo -j /etc/chef/dna.json -c /etc/chef/solo.rb in order to run Chef using the modified recipe.

Problem: Gem list fails

Symptom: The dashboard Base Chef log shows:

ERROR: gem_package[unicorn] (/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/chef-0.6.0.4/lib/chef/recipe.rb line 187) had an error:
gem list --remote failed

or:

ERROR: gem_package[passenger] (/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/chef-0.6.0.4/lib/chef/recipe.rb line 187) had an error:
gem list --remote failed

Solution: Fetch gems via non-secure method to avoid SSL failures.

Method: On each instance perform the following:

Run the command sudo gem sources -r http://gems.rubyforge.org/
Run the command sudo export PATH=$PATH:/usr/local/ey_resin/bin && sudo /usr/local/ey_resin/bin/chef-solo -j /etc/chef/dna.json -c /etc/chef/solo.rb in order to re-run Chef.

Problem: Unicorn gem installation fails

Symptom: The dashboard Base Chef log shows:

ERROR: gem_package[unicorn] (/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/chef-0.6.0.4/lib/chef/recipe.rb line 187) had an error:
gem install unicorn -q --no-rdoc --no-ri -v 4.1.1 returned 1, expected 0
---- Begin output of gem install unicorn -q --no-rdoc --no-ri -v 4.1.1 ----
STDOUT: 
STDERR: ERROR: Error installing unicorn:
rack requires Ruby version >= 2.3.0.

Solution: Manually install the Unicorn dependencies to avoid non-compatible versions attempting to be installed.

Method: On each application instance perform the following:

Run the command sudo gem install rack -v 1.6.13 --no-ri --no-rdoc && sudo gem install raindrops -v 0.17.0 --no-ri --no-rdoc
Run the command sudo export PATH=$PATH:/usr/local/ey_resin/bin && sudo /usr/local/ey_resin/bin/chef-solo -j /etc/chef/dna.json -c /etc/chef/solo.rb in order to re-run Chef.

Problem: Passenger gem installation fails

Symptom: The dashboard Base Chef log shows:

ERROR: gem_package[passenger] (/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/chef-0.6.0.4/lib/chef/recipe.rb line 187) had an error:
gem install passenger -q --no-rdoc --no-ri -v 3.0.21 returned 1, expected 0
---- Begin output of gem install passenger -q --no-rdoc --no-ri -v 3.0.21 ----
STDOUT: Building native extensions.  This could take a while...
STDERR: ERROR:  Error installing passenger:
rack requires Ruby version >= 2.3.0.

Solution: Manually install the Passenger dependencies to avoid non-compatible versions attempting to be installed.

Method: On each application instance perform the following:

Run the command sudo gem install rack -v 1.6.13 --no-ri --no-rdoc
Run the command sudo export PATH=$PATH:/usr/local/ey_resin/bin && sudo /usr/local/ey_resin/bin/chef-solo -j /etc/chef/dna.json -c /etc/chef/solo.rb in order to re-run Chef.

Problem: Portage update fails

Symptom: The output of the command line Chef run shows:

---- Begin output of ey-enzyme --report 'processing portage update' ----
STDOUT: 
STDERR: sh: ey-enzyme: command not found
---- End output of ey-enzyme --report 'processing portage update' ----

Solution: Set the correct PATH for ey-enzyme.

Method: On each failing instance perform the following:

Run the command sudo export PATH=$PATH:/usr/local/ey_resin/bin && /usr/local/ey_resin/bin/chef-solo -j /etc/chef/dna.json -c /etc/chef/solo.rb in order to update the PATH and re-run Chef.

Problem: npm fails with an SSL error

Symptom: The dashboard Base Chef log shows:

ERROR: bash[install ey_node_app_info] (/usr/local/ey_resin/ruby/lib/ruby/gems/1.8/gems/chef-0.6.0.2/lib/chef/recipe.rb line 187) had an error:
bash /tmp/chef-script.32592.0 returned 1, expected 0
---- Begin output of bash /tmp/chef-script.32592.0 ----
STDOUT: STDERR: npm http GET https://registry.npmjs.org/ey_node_app_info
npm http GET https://registry.npmjs.org/ey_node_app_info
npm http GET https://registry.npmjs.org/ey_node_app_info
npm ERR! Error: 140459789620992:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:s23_clnt.c:602:

Solution: Bypass the need to install non-required Node packages.

Method: Upgrade the stack to the latest available version via the Upgrade button for the environment in the dashboard.

Deployment Failures

Known Issues:

Remote end hangs up
Unsupported deploy key
Unsupported authentication for deploy method

Problem: Remote end hangs up

Symptom: Deploy log shows deployment fails due to:

no matching mac found: client hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 server hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
fatal: The remote end hung up unexpectedly

Solution: Upgrade the version of openSSH on the instances.

Method: On each instance perform the following:

sudo echo =net-misc/openssh-5.9_p1-r1 >> /etc/portage/package.keywords/local

sudo emerge -av =net-misc/openssh-5.9_p1-r1

sudo /etc/init.d/ssh restart

Then retry the deploy.

Problem: Unsupported deploy key

Symptom: Deploy log shows deployment fails due to:

ERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.

Or:

no hostkey alg
fatal: The remote end hung up unexpectedly

Or similar failure message from another git host.

Solution: Replace the deploy key on the repository with a newly generated compatible one.

Method: In the Engine Yard dashboard, navigate to the relevant application (via the breadcrumb trail at the top of the Environment page), then:

Click Edit Application.
Make a slight change to the Git Repository URI (e.g. append `1` to it) so that it is altered. It does not need to be a valid URI at this stage.
Click Update Application.
On the following Allow Engine Yard access to private repository page, ignore the given deploy key and use your browser's back button to return to the Edit Application page.
Change the Git Repository URI back to the correct one.
Click Update Application.
This time on the Allow Engine Yard access to private repository page, follow the instruction and copy the deploy key and add it to the repository.
Click My Deploy Key is in place.
Navigate the relevant Environment page and run an Apply.
Deploy the application.

Problem: Unsupported authentication for deploy method

Symptom: Deploy log shows deployment fails due to:

fatal: remote error:
The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.

Solution 1: Change the application repository access protocol.

Method: In the Engine Yard dashboard, navigate to the relevant application (via the breadcrumb trail at the top of the Environment page), then:

Click Edit Application.
Update the Git Repository URI to either the HTTPS format of https://github.com/_org_/_repo_.git for a public repository or the SSH format of git@github.com:_org_/_repo_.git for a private repository.
For a private repository follow the instructions to put the deploy key in place, then navigate to the relevant Environment page(s) the application is associated with and run an Apply.

Solution 2: Change the application's gems repository access protocol.

Method: If the error message in the deploy log relates to a required gem and its repository rather than the application repository, then the source in your Gemfile needs updating. If you are using github in your Gemfile this can be replaced with git: 'https://github.com/_org_/_repo_.git' and references to git: 'git:// should be replaced with git: 'https://

Once those changes are made, the app should be bundled locally to update the Gemfile.lock and then re-deployed.

Contents:

Environment Boot Failure

Problem: Environment boot attempt results in no instances being created.

Chef Configuration Failures

Known Issues:

Problem: No Chef run occurs, dashboard Chef log is empty

Problem: RubyGems installation fails

Problem: Gem list fails

Problem: Unicorn gem installation fails

Problem: Passenger gem installation fails

Problem: Portage update fails

Problem: npm fails with an SSL error

Deployment Failures

Known Issues:

Problem: Remote end hangs up

Problem: Unsupported deploy key

Problem: Unsupported authentication for deploy method

Comments