Kubernetes with CoreOS on AWS

This article provides reproducible steps for quick start of kubernetes cluster on AWS. The kube-aws part is roundup of CoreOs guide for kube-aws and first address for more details on that. Let start…

Setup kube-aws

I’m using kube-aws on linux and the version is 0.8.3. To install we need to execute:

wget https://github.com/coreos/coreos-kubernetes/releases/download/v0.8.3/kube-aws-linux-amd64.tar.gz
tar zxvf kube-aws-linux-amd64.tar.gz
sudo mv linux-amd64/kube-aws /usr/local/bin

Don’t forget to verify¹ your binaries.

AWS CLI and Authentication

Before start using kube-aws, you need to have AWS CLI installed² and be authenticated to the AWS with it.

To check if you are authenticated use aws ec2 describe-instances command. If negative, you need to provide credentials alongside with the AWS region you want to use. Here is an example of how it might look like:

aws configure

AWS Access Key ID [None]: AKID1234567890
AWS Secret Access Key [None]: MY-SECRET-KEY
Default region name [None]: us-west-2
Default output format [None]: text

Here AWS Access Key ID and AWS Secret Access Key are security credentials of your user that have enough right to set up all that³.

Preconfigure AWS Resources

Following AWS Resources are mandatory, so you need to prepare them in your AWS as well.

• EC2 key pair that will be used.
• KMS key (Encryption key)
• External DNS name

EC2 Key pair

Existing Key pairs can be found under EC2 section of AWS Web UI. Alternatively, you can create a new KeyPair by:

aws ec2 create-key-pair --key-name MyKeyPair

Of course, you can upload the existing public key as well.

AWS KMS

Now KMS. Probably you’ve not used an encryption key before or want to use a separate one for Kubernetes. You can create one by:

aws kms --region=<your-region> create-key --description="kube-aws assets"

which will give you the arn of the new key.

DNS name for cluster access

You need to define DNS hostname where the cluster API will be accessible. ==This hostname will be used to provision the TLS certificate for the API server==

When the cluster is created, the controller will expose the TLS-secured API on a public IP address. You will need to create an A record for the selected DNS hostname you want to point to this IP address.

Initialize an asset directory

Now everything is ready to create assets.

kube-aws init \
--cluster-name=kube-dev \
--external-dns-name=kube.mydomain.com \
--region=eu-central-1 \
--availability-zone=eu-central-1a \
--key-name=id_aws \
--kms-key-arn="arn:aws:kms:eu-central-1:588652342245:key/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

This will create cluster.yaml file-based on initial values. Check default values maybe you want to improve some of them. Unfortunately at the moment, they are not that well documented.

Generate contents of the asset directory

Now you can generate assets based on cluster.yaml file.

kube-aws render
#prints
WARNING: The generated client TLS CA cert expires in 3650 days and the server and client cert expires in 365 days. It is recommended that you create your own TLS infrastructure for revocation and rotation of keys before using in prod
Success! Stack rendered to stack-template.json.

Next steps:
1. (Optional) Validate your changes to cluster.yaml with "kube-aws validate"
2. (Optional) Further customize the cluster by modifying stack-template.json or files in ./userdata.
3. Start the cluster with "kube-aws up".

As you see I’ve generated assets using new random CA, it’s Ok for trying but maybe not that convenient for production, so consider using your own PKI stack.

However, validate results.

kube-aws validate
#prints:
Validating UserData...
UserData is valid.

Validating stack template...
Validation Report: {
  Capabilities: ["CAPABILITY_IAM"],
  CapabilitiesReason: "The following resource(s) require capabilities: [AWS::IAM::Role]",
  Description: "kube-aws Kubernetes cluster kube-dev"
}
stack template is valid.

Validation was OK!

Create everything in the AWS

kube-aws up

by that generated CloudFormation is applied to the AWS Cloud. It took about 5 minutes in my case.

Using new cluster

To start using the new cluster you need to set up kubectl tool first. Since it part of Kubernetes release we need to download it first from here

wget https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#downloads-for-v145

then unzip and setup

tar xzf kubernetes-client-linux-amd64.tar.gz
sudo cp kubernetes/client/bin/kubectl /usr/local/bin/kubectl
sudo chmod +x /usr/local/bin/kubectl

Now kubectl is installed, being in the directory of the cluster data you can use it to print node list of the cluster:

kubectl --kubeconfig=kubeconfig get nodes

NAME                                          STATUS                     AGE
ip-10-0-0-125.eu-central-1.compute.internal   Ready                      1h
ip-10-0-0-50.eu-central-1.compute.internal    Ready,SchedulingDisabled   1h

Enjoy!

Clean Up

You can destroy your Kubernetes cluster by executing.

kube-aws up

Summary

My experience with aws-kube is kind of bifid so far.

Pro

• Like a lot ==CoreOS== as Linux distribution for containers.
• Worked well with defaults (Separate VPC).

Contra

• Didn’t use my existing VPC. (Should be possible meanwhile, but didn’t found out)
• I miss a list of resources being created by Cloud Formation. It would be nice to see them all as a list.
• Documentation is not on the level I’ve expected it to be for production readiness. This is a kind of walk-through only.
• Didn’t found configuration options supported
• Didn’t see any advice on what to do on typical use cases: scale-up/down, emergency, typical trouble…
• Feature set is kind of limited so far. E.g. Auto Scaling is not used?.
• While destroying the cluster some resources could not be deleted including Security groups and VPC.
• Hard to reuse pre-configured resources, not possible or not documented.

That leads me to think of kube-aws not production-ready at the moment, while still interested to keep an eye on the project mainly because of CoreOS base.

Disclaimer: This is my personal view so far as someone who is not experienced in AWS Cloud Formation or Kubernetes details so far. My expectation here is of a typical newcomer.

P.S. Now it looks like the best way, in the long run, is to install Kubernetes “manually” and be aware of all the details. Manually can be automated of course in a declarative way, with Ansible for example.