During the past weeks, I was involved in a service migration project. We are migrating some mostly java-based services from legacy infrastructure to the Cloud. Amazon Web Service (AWS) was chosen on the strategic level and we started with the migration of one service.

Level 0: AWS basics

Before we took the first service, we created new Virtual Private Clouds (VPS), established VPN connections to our legacy Data Center. We created sub-networks, routes, and security groups. And it felt and straightforward to do even for team members not having any considerable networking or operations background. Credits to AWS here. Also famous and matured EC2 service that is responsible for managing Virtual Machines on IaaS level, convinced the team quickly. No big surprises here. I’ve tried the first automation steps with ansible on the EC2 level and it worked as expected as well (even if automation is a big topic to talk about)

Level 1: Docker

So we turn onto our systems. It was kind of obvious that migrating existing microservices can be eased with Docker since docker provides standardized API for managing containers and thereby for packaged services. We (re)packaged the first services as Docker containers and uploaded them to the AWS registry. Packaging application into the container was straightforward and was seen as advantageous by the whole team even there are still some corners like currently unsolved service discovery. However, we managed to play with EC2 instances and Docker standard tools and gained the first experience with containers in the cloud.

Level 2: Cluster management

As long you deal with one container the world is easy. As soon as you start to integrate several services and deploy them in a different environments you suddenly start to feel new set level op problems that just arise with the nature of distributed applications.

  • Service Discovery / Service Registration
  • Convenient and scalable logs management
  • “cloudy” Monitoring
  • Deployment, Container Scheduling.
  • Abstraction level you want to deal with IaaS vs. Pass/Caas
  • Isolation, Security

Amazon EC2 Container Service (ECS)

A ==strategic goal given to us is to abstract as much as possible from IaaS1== So we looked at the ECS. ECS is positioned by Amazone as the only serious vehicle to run applications that are packaged as Docker containers. ECS introduces a set of abstractions above EC2 instances, Load Balancers, Auto scaling groups, etc., and utilizes them. Basically, it defines a clusters and runs container on resources managed by the cluster. It works with two key abstractions, that as I guess are taken from Docker Swarm:

  • Task Definition carries a Docker container(s). Is an atomic unit of scheduling. Similar to Kubernetes Pod
  • Service combines one task with additional configuration and resources like Load Balancer. Service ensures that desired number of tasks runs in the cluster. Restarts and reschedules tasks if needed.

So ECS solves scheduling, potentially a lot of the load balancing. However, the ==lack of Service Discovery mechanisms is a big disadvantage== here. This could mean we need to install and maintain that by our self and per cluster.

At the moment I cannot compare it to a container cloud management stack like Kubernetes in detail, but it regarding Service discovery and port management Kubernetes wins clearly.

####ECS limitations Let me try to summarize it with the following points:

  • Lack of Service Discovery
  • Vendor-lock
  • Looks like ESC is developed slowly and feels already not to be state of the art
  • Exposed service needs to run behind AWS Load balancer. AWS Loadbalancers const money, there is some reusage possible but per ELB you pay ca. 20€ a month.
  • There is nothing like health checks on the container level. :(
  • As far I see it is not possible to use shared volumes across several containers or at least mount something independently of the underlying EC2 instance easily.

AWS Elastic Beanstalk with Multi-container Docker Environments

Since ECS is somehow in between IaaS and PaaS layers we’ve also tried Elastic Beanstalk with Docker (Non-Docker solutions are too limit or mean too much vendor lock). This is indeed interesting, in a sense of cluster provisioning! Really nice. But there is one thing that is strange for microservices. For the moment Beanstalk Service is Application-centric, which means every application can have many environments (clusters). This is something we see different in our case… We need few clusters and be able to deploy a lot of services independently. They often belong to different “applications”. But AWS Elastic Beanstalk only allows you to deploy a whole set of services as an application at once - something we don’t like, something will not work for us.

Alternatives

Well, in my ideal vision, every container that is started in a particular environment, e.g. “production cluster”, needs to be automatically configured or provisioned with dependent resources like Databases, referenced service access URLs, certificates, passwords, etc. This is where Service Discovery comes to play and this is what is missed by AWS so far out of the box.

So in the next weeks, we might end up deciding to build some kind of basic provisioning of our ECS machines that run something like Hashicorp’s consul, or we even look at Kubernetes.

Or maybe you have ideas to share. You are welcome to comment!

  1. Some strategists believe this is the only way to reduce maintenance efforts. Personally, I’m doubting this dogmatism. It’s not about the level of abstraction only, it’s about clean concepts and all that stuff Software architecture is about. It can imagine, there is something that could be done on the IaaS level, that might save you a lot of maintenance efforts elsewhere. The good thing, big cloud providers like AWS and GCE allows you to choose the level of abstraction to some degree. ↩︎