Kubernetes with CoreOS on AWS

This article provides reproducible steps for quick start of kubernetes cluster on AWS. The kube-aws part is roundup of CoreOs guide for kube-aws and first address for more details on that.
Let start...

Setup kube-aws

I'm using kube-aws on linux and the version is 0.8.3. To install we need to execute:

wget https://github.com/coreos/coreos-kubernetes/releases/download/v0.8.3/kube-aws-linux-amd64.tar.gz  
tar zxvf kube-aws-linux-amd64.tar.gz  
sudo mv linux-amd64/kube-aws /usr/local/bin  

Don't forget to verify1 your binaries.

AWS CLI and Authentication

Before start using kube-aws, you need to have AWS CLI installed2 and be authenticated to the AWS with it.

To check if you are authenticated use aws ec2 describe-instances command. If negative, you need to provide credentials alongside with the AWS region you want use.
Here is example of how it might look like:

$aws configure

AWS Access Key ID [None]: AKID1234567890  
AWS Secret Access Key [None]: MY-SECRET-KEY  
Default region name [None]: us-west-2  
Default output format [None]: text  

Here AWS Access Key ID and AWS Secret Access Key are security credentials of your user that has enough right to setup all that3.

Preconfigure AWS Resources

Following AWS Resources are mandatory, so you need to prepare them in your AWS as well.

  • EC2 key pair that will be used.
  • KMS key (Encryption key)
  • External DNS name
EC2 Key pair

Existing Key pairs can be found under EC2 section of AWS Web UI. Alternatively you can create new KeyPair by:

aws ec2 create-key-pair --key-name MyKeyPair  

Of course you can upload existing public key as well.

AWS KMS

Now KMS. Probably you've not used encryption key before or want to use separate one for kubernetes. You can create one by:

aws kms --region=<your-region> create-key --description="kube-aws assets"  

which will give you the arn of the new key.

DNS name for cluster access

You need to define DNS hostname where the cluster API will be accessible. This hostname will be used to provision the TLS certificate for the API server

When the cluster is created, the controller will expose the TLS-secured API on a public IP address. You will need to create an A record for the selected DNS hostname you want to point to this IP address.

Initialize an asset directory

Now everything is ready to create assets.

kube-aws init \  
--cluster-name=kube-dev \
--external-dns-name=kube.mydomain.com \
--region=eu-central-1 \
--availability-zone=eu-central-1a \
--key-name=id_aws \
--kms-key-arn="arn:aws:kms:eu-central-1:588652342245:key/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

This will create cluster.yaml file based on initial values.
Check default values maybe you want to improve some of them. Unfortunately at the moment they are not that good documented.

Generate contents of the asset directory

Now you can generate assets based on cluster.yaml file.

$kube-aws render
WARNING: The generated client TLS CA cert expires in 3650 days and the server and client cert expire in 365 days. It is recommended that you create your own TLS infrastructure for revocation and rotation of keys before using in prod  
Success! Stack rendered to stack-template.json.

Next steps:  
1. (Optional) Validate your changes to cluster.yaml with "kube-aws validate"  
2. (Optional) Further customize the cluster by modifying stack-template.json or files in ./userdata.  
3. Start the cluster with "kube-aws up".  

As you see i've generated assets using new random CA, it's Ok for trying but maybe not that convinient for production, so consider to use your own PKI stack.

However validate results.

$kube-aws validate
Validating UserData...  
UserData is valid.

Validating stack template...  
Validation Report: {  
  Capabilities: ["CAPABILITY_IAM"],
  CapabilitiesReason: "The following resource(s) require capabilities: [AWS::IAM::Role]",
  Description: "kube-aws Kubernetes cluster kube-dev"
}
stack template is valid.

Validation OK!  

Create everything in the AWS

kube-aws up  

by that generated CloudFormation is applied to the AWS Cloud. It took about 5 minutes in my case.

Using new cluster

To start use new cluster you need to setup kubectl tool first. Since it part of kubernetes release we need to download it first from here

wget https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#downloads-for-v145  

then unzip and setup

tar xzf kubernetes-client-linux-amd64.tar.gz  
sudo cp kubernetes/client/bin/kubectl /usr/local/bin/kubectl  
sudo chmod +x /usr/local/bin/kubectl  

Now kubectl is installed, being in the directory of the cluster data you can use it to print node list of the cluster:

$kubectl --kubeconfig=kubeconfig get nodes

NAME                                          STATUS                     AGE  
ip-10-0-0-125.eu-central-1.compute.internal   Ready                      1h  
ip-10-0-0-50.eu-central-1.compute.internal    Ready,SchedulingDisabled   1h  

Enjoy!

Clean Up

You can destroy your kubernetes cluster by executing.

kube-aws up  

Summary

My expirience with aws-kube are kind of bifid so far.

Pro

  • Like a lot CoreOS as linux distribution for containers.
  • Worked well with default's (Separate VPC).

Contra

  • Didn't used my existing VPC. (Should be possible meanwhile, but didn't found out)
  • I miss a list of resources being created by Cloud Formation. It would be nice to see them all as a list.
  • Documentation is not on level i've expect it to be for production readiness. This is kind of walk-through only.
    • Didn't found configuration options supported
    • Didn't see any advises what to do on typical use cases: scale-up/down, emergency, typical trouble...
  • Feature set is kind of limited so far. E.g. Auto Scaling is not used?.
  • While destroying cluster some resources could not be deleted including Security groups and VPC.
  • Hard to reuse pre-configured resources, not possible or not documented.

That leads me think of kube-aws not production ready at the moment, while still interested to keep an eye on the project mainly because of CoreOS base.

Disclaimer: This is my personal view so far as someone who is not experienced in AWS Cloud Formation or kubernetes details so far. My expectation here are of a typical newcomer.

P.S. Now it looks like the best way in the long run is to install kubernetes "manually" and be aware of all the details. Manually can be automated of course in declarative way, with Ansible for example.


  1. CoreOS team is keen to provide security chain seamless. So please verify downladed binaries as described in kube-aws gude before you go to serious with this.

  2. AWS CLI -Official Guide

  3. can be created under Identity and Access Management > Users > > Security Credentials if you use web frontend. Official Guide to IAM