Kubernetes on AWS + kops

Kubernetes on AWS + kops

Prerequisite

  • kops 1.9.0
  • kubectl client v1.10.1, server v1.9.3

설치

아래 링크대로 따라서 설치
Following this ref

그리고 어김없이 찾아오는 에러…

Error

kops 로 클러스터 구성(after executing, kops update cluster —yes $NAME) 후에 아래와 같은 output 을 볼수 있습니다.

...

Suggestions:
 * validate cluster: kops validate cluster
 * list nodes: kubectl get nodes --show-labels
 * ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.useast1.k8s.example.com
 * the admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
 * read about installing addons at: https://github.com/kubernetes/kops/blob/master/docs/addons.md.

아래 명령을 실행합니다.

kops validate cluster

Output:

tkwon-macbook:~/projects/k8s ktg$ kops validate cluster
Using cluster from kubectl context: useast1.k8s.example.com

Validating cluster useast1.k8s.example.com

INSTANCE GROUPS
NAME            ROLE    MACHINETYPE MIN MAX SUBNETS
master-us-east-1a   Master  m5.large    1   1   us-east-1a
nodes           Node    c5.large    2   2   us-east-1a

NODE STATUS
NAME    ROLE    READY

VALIDATION ERRORS
KIND    NAME        MESSAGE
dns apiserver   Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.

Validation Failed

뭔지 모르겠지만 실패했습니다.
ssh로 master node 에 접속 시도 해봅니다.

ssh -i ~/.ssh/id_rsa admin@api.useast1.k8s.example.com

ssh 접속이 되지 않았습니다. ..

route53에서 확인해보니 도매인의 ip가 master node ip로 변경되어 있지 않았습니다.

Ec2 dashboard 를 확인하니 정상적으로 instance가 실행중입니다.

ip로 ssh 접속 시도합니다.

ssh -i ~/.ssh/id_rsa admin@1.1.1.1

정상적으로 접속됩니다.

sudo docker ps

Output:

protokube                                          1.9.0

protokube 라는 이미지가 실행중입니다.
docker log 를 확인해보니 아래와 같은 에러로그가 반복되고 있습니다.

...
Mar 30 22:30:38 ip-172-31-1-13 docker[1546]: I0330 22:30:38.659365    1576 aws_volume.go:318] nvme path not found "/rootfs/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0f1a04a36c6baaaae"
Mar 30 22:30:38 ip-172-31-1-13 docker[1546]: I0330 22:30:38.659373    1576 volume_mounter.go:107] Waiting for volume "vol-0f1a04a36c6baaaae" to be attached
Mar 30 22:30:39 ip-172-31-1-13 docker[1546]: I0330 22:30:39.659499    1576 aws_volume.go:318] nvme path not found "/rootfs/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0f1a04a36c6baaaae"
Mar 30 22:30:39 ip-172-31-1-13 docker[1546]: I0330 22:30:39.659519    1576 volume_mounter.go:107] Waiting for volume "vol-0f1a04a36c6baaaae" to be attached
Mar 30 22:30:40 ip-172-31-1-13 docker[1546]: I0330 22:30:40.659641    1576 aws_volume.go:318] nvme path not found "/rootfs/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0f1a04a36c6baaaae"
Mar 30 22:30:40 ip-172-31-1-13 docker[1546]: I0330 22:30:40.659660    1576 volume_mounter.go:107] Waiting for volume "vol-0f1a04a36c6baaaae" to be attached
...

검색해보니,
Cluster hung up starting, waiting for etcd volumes to attach · Issue #4844 · kubernetes/kops · GitHub

위의 링크에 원인이 나와있었습니다.
thumb up 한번 눌러드리고 수정합니다.

$ kops edit ig master-us-east-1a
$ kops edit ig nodes

위 명령을 실행해서 instance type m5, c5 에서 m4, c4로 바꿉니다.

kops rolling-update cluster --cloudonly --yes

실행하고… 기다립니다.

kops validate cluster

Output:

tkwon-macbook:~/projects/k8s ktg$ kops validate cluster
Using cluster from kubectl context: useast1.k8s.example.com

Validating cluster useast1.k8s.example.com

INSTANCE GROUPS
NAME            ROLE    MACHINETYPE MIN MAX SUBNETS
master-us-east-1a   Master  m4.large    1   1   us-east-1a
nodes           Node    c4.large    2   2   us-east-1a

NODE STATUS
NAME                ROLE    READY
ip-10-110-42-179.ec2.internal   node    True
ip-10-110-45-25.ec2.internal    master  True
ip-10-110-47-243.ec2.internal   node    True

Your cluster useast1.k8s.example.com is ready

이제 kubernetes cluster 를 사용할 수 있습니다!

Author :
github: https://github.com/9to6

카테고리

Count per Day

  • 27893Total visitors:
  • 22Visitors today:
  • 59Visitors yesterday:
  • 465Visitors last week:
  • 63Visitors per day:
  • 2017년 4월 7일Counter starts on: