AWS의 Opsworks에서 새로운 instances를 만들다가 만난 에러를 정리해둡니다.
AWS의 Opsworks 는 인스턴스를 새로 생성하면 등록되어있는 Chef Recipe 들을 자동으로 실행해주는 툴입니다.
이번에 Instance 한개를 추가로 생성하고, 그 Instance에 하나의 App 을 배포하려고 기존 Layer에서 추가로 Instance 를 생성하게 되었습니다.
그런데 갑자기 에러가 발생해서 로그를 봤더니 이런 로그가 있었습니다.
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
execute[install-key 58118E89F3A912897C070ADBF76221572C52609D] (/var/lib/aws/opsworks/cache.stage2/cookbooks/apt/providers/repository.rb line 28) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D ----
STDOUT: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.YjqEGDmda9 --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D
?: p80.pool.sks-keyservers.net: Host not found
gpgkeys: HTTP fetch error 7: couldn't connect: Success
STDERR: gpg: requesting key 2C52609D from hkp server p80.pool.sks-keyservers.net
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
---- End output of apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D ----
Ran apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D returned 2
Resource Declaration:
---------------------
# In /var/lib/aws/opsworks/cache.stage2/cookbooks/ops-docker/recipes/install.rb
4: apt_repository 'docker' do
5: uri node['docker']['package']['repo_url']
6: distribution node['docker']['package']['distribution']
7: components ['main']
8: keyserver node['docker']['package']['repo_keyserver']
9: key node['docker']['package']['repo_key']
10: end
11:
Compiled Resource:
------------------
# Declared in /var/lib/aws/opsworks/cache.stage2/cookbooks/ops-docker/recipes/install.rb:4:in `from_file'
apt_repository("docker") do
action :add
retries 0
retry_delay 2
cookbook_name "ops-docker"
recipe_name "install"
uri "https://apt.dockerproject.org/repo"
distribution "ubuntu-trusty"
components ["main"]
keyserver "hkp://p80.pool.sks-keyservers.net:80"
key "58118E89F3A912897C070ADBF76221572C52609D"
cache_rebuild true
end
[2017-05-03T09:17:52+00:00] INFO: Running queued delayed notifications before re-raising exception
[2017-05-03T09:17:52+00:00] ERROR: Running exception handlers
[2017-05-03T09:17:52+00:00] ERROR: Exception handlers complete
[2017-05-03T09:17:52+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out
[2017-05-03T09:17:52+00:00] ERROR: apt_repository[docker] (ops-docker::install line 4) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[install-key 58118E89F3A912897C070ADBF76221572C52609D] (/var/lib/aws/opsworks/cache.stage2/cookbooks/apt/providers/repository.rb line 28) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D ----
STDOUT: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.YjqEGDmda9 --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D
?: p80.pool.sks-keyservers.net: Host not found
gpgkeys: HTTP fetch error 7: couldn't connect: Success
STDERR: gpg: requesting key 2C52609D from hkp server p80.pool.sks-keyservers.net
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
---- End output of apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D ----
Ran apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv 58118E89F3A912897C070ADBF76221572C52609D returned 2
[2017-05-03T09:17:53+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
해서 검색을 시작했습니다.
저는 OpsWorks나 Docker에 서툴러서 뭔 실수를 했나 싶었는데 결과는…
그냥 단순히 서버 다운… OTL.
p80.pool.sks-keyservers.net: Host not found
gpgkeys: HTTP fetch error 7: couldn’t connect: Success
그냥 저 서버가 죽은거 였네요.
저 서버가 Docker설치시 기본으로 바로보는 PGP key server인 듯..
아래 링크가 이 에러와 관련된 Thread입니다.
Key server down for get.docker.com · Issue #13555 · moby/moby · GitHub
여기 가보면 kris라는 사람이
기존 Docker가 사용하는 서버주소를
From,
p80.pool.sks-keyservers.net
To,
hkp://keyserver.ubuntu.com:80
로 변경해주라고 답을 달아놨더군요.
결론
아무튼 이 버그의 결론은,
잠시뒤에 다시 시도 해본다;;;;
서버가 살아날 시간을 주는 것입니다;;