During the past weeks i was involved into service migration project. We are migrating some mostly java-based services from legacy infrastructure to the Cloud. Amazon Web Service (AWS) was chosen on the strategic level and we started with migration of one service.
Level 0: AWS basics
Before we took first service, we created new Virtual Private Clouds (VPS), established VPN connections to our
legacy Data Center. We created sub networks, routes and security groups. And it felt and straightforward to do even for team members not having any considerable networking or operations background. Credits to AWS here.
Also famous and matured EC2 service that is responsible for managing of Virtual Machines on IaaS level, convinced the team quickly. No bog surprises here. I've tried first automation steps with ansible on EC2 level and it worked as expected as well (even if automation is a big topic to talk)
Level 1: Docker
So we turn onto our systems. It was kind of obvious that migrating existing
microservices can be eased with Docker, since docker provides standardised api for managing container and thereby for packaged services. We (re)packaged first services as Docker containers and uploaded them to AWS registry. Packaging application into container was straightforward and was seen as advantageous by the whole team even there are still some corners like currently unsolved service discovery. How ever we managed to play with EC2 instances and Docker standard tools and gained first experiences with containers in the cloud.
Level 2: Cluster management
As long you deal with one container the world is easy. As soon as you start integrate several services and deploy them in different environment you suddenly start to feel new set level op problems that just arise with the nature of distributed applications.
- Service Discovery / Service Registration
- Convenient and scallable logs management
- "cloudy" Monitoring
- Deployment, Container Scheduling.
- Abstraction level you want to deal with IaaS vs. Pass/Caas
- Isolation, Security
Amazon EC2 Container Service (ECS)
A strategic goal given to us is to abstract as much as possible from IaaS1
So we looked at the ECS. ECS is positioned by Amazone as the only serious vehicle to run applications that are packaged as Docker containers. ECS introduces set of abstractions above EC2 instances, Load Balancers, Auto scaling groups etc. and utilizes them. Basically it defines a clusters and run container on resources managed by the cluster.
It works with two key abstractions, that as i guess are taken from Docker Swarm:
- Task Definition carries a Docker container(s). Is an atomic unit of scheduling. Similar to Kubernetes Pod
- Service combines one tasks with additional configuration and resources like Load Balancer. Service ensures that desired number of task runs in the cluster. Restarts and reschedules tasks if needed.
So ECS solves scheduling, potentially a lot of the load balancing. However the lack of Service Discovery mechanisms is a big disadvantage here. This could mean we need to install and maintain that by our self and per cluster.
At the moment i cannot compare it to container cloud management stack like kubernetes in details, but it regarding Service discovery and port management kubernetes wins clearly.
Let me trying to summarize it with the following points:
- Lack of Service Discovery
- Looks like ESC is developed slowly and feels already not to be state of the art
- Exposed service needs to run behind AWS Load balancer. AWS Loadbalancers const money, there is some reusage possible but per ELB you pay ca. 20€ a month.
- There is nothing like health-checks on the container level. :(
- As far i see it is not possible to use shared volumes across several containers or at leas mount something independently of underlying EC2 instance easily.
AWS Elastic Beanstalk with Multi-container Docker Environments
Since ECS is somehow in between of IaaS and PaaS layers we've also tried Elastic Beanstalk with Docker (Non-Docker solutions are to limited or mean to much vendor lock). This is indeed interesting, in a sense of cluster provisioning! Really nice.
But there is one thing that is strange for micro services. For the moment Beanstalk Service is Application centric, means every application can have many environments (clusters). This is something we see different for our case... We need few clusters and be able to deploy a lot of services independently. They often belong to different "applications". But AWS Elastic Beanstalk only allows you do deploy a whole set of services as application at once - something we don't like, something will not work for us.
Well, in my ideal vision, every container that is started in a particular environment, e.g. "production cluster", need to be automatically configured or provisioned with dependent resources like Databases, referenced service access URLs, certificates, passwords etc.
This is where Service Discovery comes to play and this is what is missed by AWS so far out of the box.
So in the next weeks we might end up decide to build some kind of basic provisioning of our ECS machines that run something like Hashicorp's consul, or we even look at Kubernetes.
Or maybe you have ideas to share. You are welcome to comment!
Some strategist believe this is the only way to reduce maintenance efforts. Personally i'm doubting this dogmatism. It's not about the level of abstraction only, it's about clean concepts and all that stuff Software architecture is about. It can imagine, there are some thing that could be done on IaaS level, that might save you a lot of maintenance efforts elsewhere. The good thing, big cloud providers like AWS and GCE allows you to choose the level of abstraction to some degree. ↩