Friday, December 26, 2014

Orchestrating Docker - Docker Machine/Swarm/Compose

Orchestrating Docker

While docker link enables containers to communicate on a single host, but when we are deploying a large scale application, we’ll need a better tool when containers are deployed to multiple hosts.
Docker has announced 3 new orchestration services: Docker Machine, Docker Swarm and Docker Compose. One key benefit for the 3 new services is that they are implement the same docker API as the docker engine, so integrating with tools that’s working with docker engine will work seamlessly.

Docker Machine

With Docker Machine, there’s no need to log in to a host to installer docker, provising Docker daemon can be done remotely with a single command and we can manage the docker hosts with the same docker command.
It makes creating docker hosts either on local hypervisors (currently only support virtual box) and cloud provider (current supports Digital Ocean & Microsoft Azure). It has plan to support openstack, vmware and amazon web services (there’s a driver in github) but it’s not available at the moment (it might be possible for us to develop the adapter for vmware, there’s a Go library for vSphere, https://github.com/vmware/govmomi).

Docker Swarm

Now that we have provisioned several docker hosts, we can join multiple docker hosts (nodes) to a cluster and treat that as a single virtual host and schedule docker containers to run on top, automatically managing workload placement, failover, and maintaining cluster state. It ensures an application’s distributed containers are automatically “getting fed the right resources.”
By default, the default scheduler uses the resource requirements of the docker container workloads and resources availabilities of the hosts in the cluster and then uses bin pack to automatically optimize placement of workloads. So, we can schedule a container requiring certain memory/CPU/Ports (more to come). Also, we can add contraints like hosts with SSD storage, certain operating system, storage driver or custom label.
Swarm continuously health-checks the Docker daemon’s hosts and should one suffer an outage, automatically rebalances by moving and restarting the Docker containers from the failed host to a new one.
Docker Swarm has a pluggable architecture with default scheduler. It can swap in a more powerful backend, like Mesos, for large scale production deployments.


Docker Compose

What if we want to deploy 2 containers together on the same docker host? E.g. nodejs and redis to be deployed together so both containers can talk to each other using docker link (so docker will add environment variable and update hosts file for the linked container.) We can then use Docker compose to create a multi-container distributed app to run on top of the cluster created with Docker swarm. It’s just a simple YAML file.
And it can be started with a simple “docker up” command.


There’s some test builds on the docker github (compose itself is not a separate project, but adding the docker up commend to docker) but not sure how much is merged.

Pros and Cons of using Docker Orchestrating Services

Pros
  • Integrate with Docker
  • All using docker API
  • Swarm with Mesos protentially can solve load balancing / HA with HAproxy (Mesos can update HAproxy’s config) but still keeping the docker API
Cons
  • Alpha release
  • Machine doesn’t support vmware yet
  • Unclear if Swarm with Mesos as backend is released