AWS for Docker containers: first impressions

During the past weeks, I was involved in a service migration project. We are migrating some mostly java-based services from legacy infrastructure to the Cloud. Amazon Web Services (AWS) was chosen on the strategic level and we started with the migration of one service.

Level 0: AWS basics

Before we took the first service, we created new Virtual Private Clouds (VPC), established VPN connections to our legacy Data Center. We created sub-networks, routes, and security groups. And it felt straightforward to do even for team members not having any considerable networking or operations background. Credits to AWS here. Also famous and matured EC2 service that is responsible for managing Virtual Machines on IaaS level, convinced the team quickly. No big surprises here. I’ve tried the first automation steps with ansible on the EC2 level and it worked as expected as well (even if automation is a big topic to talk about)

Level 1: Docker

So we turn to our systems. It was kind of obvious that migrating existing microservices can be eased with Docker since docker provides standardized API for managing containers and thereby for packaged services. We (re)packaged the first services as Docker containers and uploaded them to the AWS registry. Packaging application into the container was straightforward and was seen as advantageous by the whole team even there are still some corners like currently unsolved service discovery. However, we managed to play with EC2 instances and Docker standard tools and gained the first experience with containers in the cloud.

Level 2: Cluster management

As long you deal with one container the world is easy. As soon as you start to integrate several services and deploy them in different environments you suddenly start to feel a new set of problems that just arise with the nature of distributed applications.

• Service Discovery / Service Registration
• Convenient and scalable logs management
• “cloudy” Monitoring
• Deployment, Container Scheduling.
• Abstraction level you want to deal with IaaS vs. PaaS/CaaS
• Isolation, Security

Amazon EC2 Container Service (ECS)

A ==strategic goal given to us is to abstract as much as possible from IaaS¹== So we looked at the ECS. ECS is positioned by Amazon as the only serious vehicle to run applications that are packaged as Docker containers. ECS introduces a set of abstractions above EC2 instances, Load Balancers, Auto scaling groups, etc., and utilizes them. Basically, it defines a cluster and runs container on resources managed by the cluster. It works with two key abstractions, that as I guess are taken from Docker Swarm:

• Task Definition carries a Docker container(s). Is an atomic unit of scheduling. Similar to Kubernetes Pod
• Service combines one task with additional configuration and resources like Load Balancer. Service ensures that desired number of tasks run in the cluster. Restarts and reschedules tasks if needed.

So ECS solves scheduling, potentially a lot of the load balancing. However, the ==lack of Service Discovery mechanisms is a big disadvantage== here. This could mean we need to install and maintain that by ourselves and per cluster.

At the moment I cannot compare it to a container cloud management stack like Kubernetes in detail, but regarding Service discovery and port management Kubernetes wins clearly.

ECS limitations

Let me try to summarize it with the following points:

• Lack of Service Discovery
• Vendor-lock
• Looks like ECS is developed slowly and feels already not to be state of the art
• Exposed service needs to run behind AWS Load balancer. AWS Load Balancers cost money, there is some reusage possible but per ELB you pay ca. 20€ a month. (The new Application Load Balancer, just launched in August 2016, might help here with path-based routing across services — haven’t tried it yet though.)
• There is nothing like health checks on the container level. :( (Docker 1.12 HEALTHCHECK is technically supported by ECS, but it only restarts the container — no integration with load balancer or auto-scaling yet.)
• As far as I see it is not possible to use shared volumes across several containers or at least mount something independently of the underlying EC2 instance easily.

AWS Elastic Beanstalk with Multi-container Docker Environments

Since ECS is somehow in between IaaS and PaaS layers we’ve also tried Elastic Beanstalk with Docker (Non-Docker solutions are too limited or mean too much vendor lock). This is indeed interesting, in a sense of cluster provisioning! Really nice. But there is one thing that is strange for microservices. For the moment Beanstalk Service is Application-centric, which means every application can have many environments (clusters). This is something we see differently in our case… We need few clusters and be able to deploy a lot of services independently. They often belong to different “applications”. But AWS Elastic Beanstalk only allows you to deploy a whole set of services as an application at once - something we don’t like, something will not work for us.
(Worth noting: Beanstalk Multi-container Docker environments actually run on ECS under the hood, so the cluster infrastructure is the same.)

Alternatives

Well, in my ideal vision, every container that is started in a particular environment, e.g. “production cluster”, needs to be automatically configured or provisioned with dependent resources like Databases, referenced service access URLs, certificates, passwords, etc. This is where Service Discovery comes to play and this is what is missed by AWS so far out of the box.

So in the next weeks, we might end up deciding to build some kind of basic provisioning of our ECS machines that run something like Hashicorp’s consul, or we even look at Kubernetes. (Running Kubernetes on AWS is still a quite heavy lift - tools like kops are in early stages. The feature set is attractive but the operational overhead is real.)

Or maybe you have ideas to share. You are welcome to comment!