Docker is a fantastic tool for encapsulating the runtime environment and deployable code of an application. However, it does not provide a capability for containers to communicate in a distributed architecture. There is Docker links but that only helps for containers that are hosted on a single node. As such a number of patterns and implementations have sprung up to bridge this gap. In this blog post I outline my own solution to distributing Docker containers.

The technologies that I use are:

Docker; Of course.

Etcd; A highly-available key value store for shared configuration and service discovery.

Skydns; A DNS service discovery for etcd.

HA Proxy; A reliable, high performance TCP/HTTP load balancer.

Custom Scripts; A set of scripts to automate service announcement, service registration, service discovery and automated configuration.

The application stack that I am building for this example is shown in the following diagram.

alt

Docker Containers

As well as the server process there are a number of additional processes in each of my Docker containers. As such I use Supervisord as a process manager. The following diagram shows the structure of the containers.

alt

As can be seen from the above diagram there is a base container that the others inherit from. This base container includes SSHD. Although Docker containers generally only have the server process running I find having SSH included to be invaluable for debugging issues.

The announce-service and listen-path daemons are described below.

announce-service;
This script announces details of a provided service in Etcd.

listen-path;
This script listens to one or more Etcd paths and starts an instance of the listen-service script for each one.

Announcing a Service

The announce-service script expects to find a number of environment variables. These variables are as below.

ANNOUNCE_PATH; The path within Etcd that the service will write details of itself to. This path will be prefixed with /announce.

PORT; The service port exposed within the container.

TYPE; The type of service. Can be either HTTP or TCP.

Below is an example of launching the PostgreSQL container so it will announce itself in Etcd.

docker run -d -t -P -e ANNOUNCE_PATH=/internal/app1/dev/db/ghost.example -e PORT=5432 -e TYPE TCP --name postgresql registry/postgresql

When the container starts a JSON string will be written to following path in Etcd.

/announce/internal/app1/dev/db/ghost.example/444a99dbf0c6

The last component of the above path is the host name that Docker has assigned to the container.

When the service is announced it will be given a TTL of 15 seconds. The announce-service script will attempt to update the path every 5 seconds. If it is unable to do so then the path will automatically be removed.

Below is an example of the JSON string that is written to the path.

{"ID":"444a99dbf0c6",
"APP":"app1",
"ENV":"dev",
"SERVICE":"db",
"NAME":"ghost.main",
"PORT":"5432",
"TYPE":"TCP"}

ID; The hostname Docker has assigned to the container.

APP; The application name.

ENV; The environment name.

SERVICE; The service name.

NAME; An optional name given to the environment. As you will see later this offers tremendous flexibility in creating multiple named environments within a main environment, such as DEV.

PORT; The service port exposed within the container.

TYPE; The type of service.

The following diagram shows the process of announcing the service.

alt

Registering a service

Once a service has been announced it needs to be registered. The reason for this extra step is that the container does not know of either the external IP address of the host or the external port that Docker has chosen for it.

There is a process called register-service that runs on each of the hosts that contain the Docker daemon and are within the same Etcd cluster. If a change is detected on the /announce path then the host running the container will write a new JSON string to the /services path. This can be seen in the following diagram.

alt

The example path below this time uses the Django container.

/services/internal/app1/dev/web/ghost.example/0ce60d000eac

As before the service is registered for a TTL of 15 seconds. Each time the announce-service script runs in the container it will cause the register-service script to re-register the service to the /services path.

Below is an example of the JSON string that is written to the path.

{"ID":"0ce60d000eac",
"APP":"app1",
"ENV":"dev",
"SERVICE":"web",
"NAME":"ghost.example",
"IP":"10.250.136.243",
"EXT_PORT":"49178",
"TYPE":"HTTP"}

IP; Is the IP address of the eth01 adapter of the host.

EXT_PORT; Is the external port Docker assigned to the exposed container port.

As well as registering the service in the /services path, the register-service script will also create a DNS entry for the service by writing to the /skydns path.

Using Django as an example the following path will be written to.

/skydns/internal/app1/dev/web/example/ghost

Below is an example of the JSON string that is written to the path.

{"host":"x.x.x.x"}

This will register the following DNS entry.

ghost.example.web.dev.app1.internal

Note: In the current version of my scripts the IP address for the DNS entry is hardcoded to be the elastic IP of the AWS server that contains the web frontend HA Proxy container. This will be changed to be more dynamic in a future revision.

Listening for a Service

If a container is running the listen-path script then it will listen for changes on one or more Etcd paths. The paths are defined using an environment variable, as below.

LISTEN_PATH; A comma delimited set of Etcd paths.

Below is an example of launching a web frontend HA Proxy container to service requests to named environments in both DEV and SIT.

docker run -d -t -p 80:80 -p 2200:22 -e LISTEN_PATH=/services/internal/app1/dev,/services/internal/app1/sit --name web_frontend registry/haproxy

The listen-path script will start a listen-service script for each path passed in the LISTEN_PATH environment variable.

alt

The listen-service script is responsible for configuring a local HA Proxy server. It watches for either a SET or an EXPIRE action on the path it is listening to. If such a change is detected then the relevant JSON string is read and the HA Proxy configuration files are created / modified / deleted.

Below is an example of starting a Django container that will both announce itself and listen for its PostgreSQL container.

docker run -d -t -P -e ANNOUNCE_PATH=/internal/app1/dev/web/ghost.example -e PORT=80 -e TYPE=HTTP -e  LISTEN_PATH=/services/internal/app1/dev/db/ghost.example --name django registry/django

As the Django container is announcing itself on the path /internal/app1/dev/web/ghost.example it will be detected by the HA Proxy container we launched earlier, as one of the paths the HA Proxy container is listening to is /services/internal/app1/dev.

The launch-service script will then modify the http-in frontend block in the haproxy.cfg file so that the environment the Django container is announcing is known, as below.

acl ghost.example.web.dev.app1.internal hdr_dom(host) -i ghost.example.web.dev.app1.internal

use_backend ghost.example.web.dev.app1.internal if ghost.example.web.dev.app1.internal

The launch-service script will also create a configuration file containing the backend block for the environment, as below.

backend ghost.example.web.dev.app1.internal
  log global
  server 0ce60d000eac 10.250.136.243:49176

If additional Django containers are launched announcing to the same path then they will also be added to the backend block, as below.

backend ghost.example.web.dev.app1.internal
  log global
  server 0ce60d000eac 10.250.136.243:49176
  server 7874ghGD121s 10.250.136.221:49188

The previous example describes configuring a HTTP service, but it works equally well for TCP too. An example of which is where the Django container is listening for its PostgreSQL container. Once the PostgreSQL container is detected it is made available on 127.0.0.1:5432 of the Django container. This means that that the Django container can move between any environment without modification. The only change being the values of the environment variables.

Benefits

Once Docker containers are able to communicate using service discovery many benefits can be realized. Some of which are below.

  • The Docker containers can be distributed across many hosts.

  • The same Docker containers can flow between all environments by changing just a few environment variables.

  • An entire multi-tiered environment can be started in seconds.

  • There can be many named environments within a main environment. This would allow each developer to have their own environment or maybe even an environment created for every branch commit.

After service discovery for Docker containers has been solved then the next requirement is to orchestrate the containers start / stop across the hosts. This is where CoreOS Fleetd comes into play. Maybe that will be the subject of a future article :-)