Error occurs when start an instance AWS OpsWorks

When I start a new instance on OpsWorks Layers, I faced this error.

================================================================================
Recipe Compile Error in /var/lib/aws/opsworks/cache.stage2/cookbooks/aws/resources/cloudwatch.rb
================================================================================

NoMethodError
-------------
undefined method `property' for #&lt;Class:0x007f511b6ee538&gt;

Cookbook Trace:
---------------
/var/lib/aws/opsworks/cache.stage2/cookbooks/aws/resources/cloudwatch.rb:1:in `class_from_file'

Relevant File Content:
----------------------
/var/lib/aws/opsworks/cache.stage2/cookbooks/aws/resources/cloudwatch.rb:

1&gt;&gt; property :alarm_name, String, name_property: true
2: property :alarm_description, String
3: property :actions_enabled, TrueClass
4: property :ok_actions, Array, default: []
5: property :alarm_actions, Array, default: []
6: property :insufficient_data_actions, Array, default: []
7: property :metric_name, String
8: property :namespace, String
9: property :statistic, equal_to: %w(SampleCount Average Sum Minimum Maximum)
10: property :extended_statistic, String

I got a panic….
Because it certainly worked fine when I start an instance before a couple of hours…

I had found a solution.
Finally I found it.

That’s why AWS cookbook updated.

My OpsWorks system used chef11.
When AWS cookbook updated 6.1.0, it has used new method, property.

I specified a version of AWS cookbook on my Berkshelf

Finally it works well.

AWS의 Opsworks에서 새로운 instances를 만들다가 만난 에러를 정리해둡니다.

AWS의 Opsworks 는 인스턴스를 새로 생성하면 등록되어있는 Chef Recipe 들을 자동으로 실행해주는 툴입니다.

이번에 Instance 한개를 추가로 생성하고, 그 Instance에 하나의 App 을 배포하려고 기존 Layer에서 추가로 Instance 를 생성하게 되었습니다.

그런데 갑자기 에러가 발생해서 로그를 봤더니 이런 로그가 있었습니다.

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
execute[install-key 58118E89F3A912897C070ADBF76221572C52609D] (/var/lib/aws/opsworks/cache.stage2/cookbooks/apt/providers/repository.rb line 28) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D ----
STDOUT: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.YjqEGDmda9 --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D
?: p80.pool.sks-keyservers.net: Host not found
gpgkeys: HTTP fetch error 7: couldn't connect: Success
STDERR: gpg: requesting key 2C52609D from hkp server p80.pool.sks-keyservers.net
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
---- End output of apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D ----
Ran apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D returned 2

Resource Declaration:
---------------------
# In /var/lib/aws/opsworks/cache.stage2/cookbooks/ops-docker/recipes/install.rb

4: apt_repository 'docker' do
5: uri node['docker']['package']['repo_url']
6: distribution node['docker']['package']['distribution']
7: components ['main']
8: keyserver node['docker']['package']['repo_keyserver']
9: key node['docker']['package']['repo_key']
10: end
11:

Compiled Resource:
------------------
# Declared in /var/lib/aws/opsworks/cache.stage2/cookbooks/ops-docker/recipes/install.rb:4:in `from_file'

apt_repository("docker") do
action :add
retries 0
retry_delay 2
cookbook_name "ops-docker"
recipe_name "install"
uri "https://apt.dockerproject.org/repo"
distribution "ubuntu-trusty"
components ["main"]
keyserver "hkp://p80.pool.sks-keyservers.net:80"
key "58118E89F3A912897C070ADBF76221572C52609D"
cache_rebuild true
end

[2017-05-03T09:17:52+00:00] INFO: Running queued delayed notifications before re-raising exception
[2017-05-03T09:17:52+00:00] ERROR: Running exception handlers
[2017-05-03T09:17:52+00:00] ERROR: Exception handlers complete
[2017-05-03T09:17:52+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out
[2017-05-03T09:17:52+00:00] ERROR: apt_repository[docker] (ops-docker::install line 4) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[install-key 58118E89F3A912897C070ADBF76221572C52609D] (/var/lib/aws/opsworks/cache.stage2/cookbooks/apt/providers/repository.rb line 28) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D ----
STDOUT: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.YjqEGDmda9 --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D
?: p80.pool.sks-keyservers.net: Host not found
gpgkeys: HTTP fetch error 7: couldn't connect: Success
STDERR: gpg: requesting key 2C52609D from hkp server p80.pool.sks-keyservers.net
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
---- End output of apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D ----
Ran apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D returned 2
[2017-05-03T09:17:53+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

해서 검색을 시작했습니다.
저는 OpsWorks나 Docker에 서툴러서 뭔 실수를 했나 싶었는데 결과는…
그냥 단순히 서버 다운… OTL.

p80.pool.sks-keyservers.net: Host not found
gpgkeys: HTTP fetch error 7: couldn’t connect: Success

그냥 저 서버가 죽은거 였네요.

저 서버가 Docker설치시 기본으로 바로보는 PGP key server인 듯..
아래 링크가 이 에러와 관련된 Thread입니다.

Key server down for get.docker.com · Issue #13555 · moby/moby · GitHub

여기 가보면 kris라는 사람이
기존 Docker가 사용하는 서버주소를
From,

p80.pool.sks-keyservers.net

To,

hkp://keyserver.ubuntu.com:80

로 변경해주라고 답을 달아놨더군요.

결론

아무튼 이 버그의 결론은,
잠시뒤에 다시 시도 해본다;;;;

서버가 살아날 시간을 주는 것입니다;;