Table of Contents
Deploying an application with Docker Compose typically requires you to follow these steps : pull the new image, destroy the container and recreate the container with the new image. However, the downside, especially if your application takes a few seconds to boot, is that you can have a moment of downtime between destroying and creating a new container.
A zero downtime deployment, commonly referred to as a "blue-green" deployment, involves creating two instances of your application : the old version and the new version. A reverse-proxy is used to seamlessly route traffic from the old app to the new app.
About the example
The application used in this example is a simple FastAPI app that performs geolocation on IP addresses — you can view the API docs here. This is a stateless application so, it's easy to Dockerise and demonstrate.
In this example, we'll use Caddy for a reverse proxy server, but you could also use NGINX, Traefik, HAProxy, etc. If you want to take a look at the full source, you can find the project on GitHub.
Overview
By default, traffic flows to both the Blue and Green instances using a simple load-balancer.
When deploying, we modify our reverse-proxy configuration to route traffic to only the Green instance. With a soft shutdown, we make sure that all requests have been processed and that the Blue instance is idle. Then we can destroy and recreate the Blue instance with our new image.
When the Blue instance is up-to-date and ready, we can route the traffic to it and recreate the Green instance performing the same steps as before.
Once fully deployed, we can resume traffic to both instances and the app is fully updated.
Docker Compose
In our compose.yaml file, we can define both Blue and Green instances of our application. Using YAML anchors, we can avoid repeating the same properties.
There is also a custom Caddy image with our configuration exposed to port 8080. In this example, SSL termination is provided by an NGINX server outside of Docker, but you could use expose Caddy on port 443 and perform SSL termination there instead.
services:
zerodowndeploy-blue: &app
container_name: zerodowndeploy-blue
image: ghcr.io/jamie-mh/zerodowndeploy:latest
restart: always
networks:
- zerodowndeploy
zerodowndeploy-green:
<<: *app
container_name: zerodowndeploy-green
zerodowndeploy-caddy:
container_name: zerodowndeploy-caddy
image: ghcr.io/jamie-mh/zerodowndeploy-caddy:latest
restart: always
ports:
- 8080:8080
networks:
- zerodowndeploy
networks:
zerodowndeploy:
driver: bridge
In our Caddyfile, we can simply expose the server on port 8080 and perform a simple load balance on both instances of the app.
:8080
reverse_proxy {
to zerodowndeploy-blue:8000 zerodowndeploy-green:8000
lb_policy round_robin
}
To make this self-contained, a custom Caddy image is used. This also includes cURL that we will use to interact with the admin API later.
FROM caddy:2.8-alpine
LABEL org.opencontainers.image.source=https://github.com/jamie-mh/zerodowndeploy
RUN apk --no-cache add curl
COPY Caddyfile /etc/caddy/Caddyfile
Healthchecks
Optionally, a healthcheck configuration can be added to the app within the Docker Compose file. By pinging the /health endpoint we ensure that the application is running and ready to accept requests.
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:8000/health" ]
interval: 30s
timeout: 10s
retries: 10
start_interval: 10s
start_period: 30s
GitHub Actions
To deploy the app, we can use a simple workflow that runs based on a tag. In this case, when a commit is tagged with v[something], the images will be created and the app will be deployed.
name: Build and Deploy
on:
push:
tags:
- "v*"
jobs:
...
For this workflow to work, you need to grant write permissions using the runner's GITHUB_TOKEN:
Build job
Our first job builds our app and Caddy images, the docker/build-push-action can achieve this is a single step.
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push app image
uses: docker/build-push-action@v5
with:
push: true
context: .
file: app.Dockerfile
tags: ghcr.io/jamie-mh/zerodowndeploy:latest,ghcr.io/jamie-mh/zerodowndeploy:${{ github.ref_name }}
- name: Build and push Caddy image
uses: docker/build-push-action@v5
with:
push: true
context: .
file: caddy.Dockerfile
tags: ghcr.io/jamie-mh/zerodowndeploy-caddy:latest
Deploy job
In order to deploy remotely, we need to set up SSH for our target server. This assumes you have a deploy user on your server that belongs to the docker group and that you've set the DEPLOY_KEY and DEPLOY_HOST secrets.
deploy:
runs-on: ubuntu-latest
needs: build
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up SSH
run: |
mkdir ~/.ssh
echo "${{ secrets.DEPLOY_KEY }}" > ~/.ssh/deploy.key
chmod 700 ~/.ssh
chmod 600 ~/.ssh/deploy.key
cat >>~/.ssh/config <<END
Host remote
HostName ${{ secrets.DEPLOY_HOST }}
User deploy
IdentityFile ~/.ssh/deploy.key
StrictHostKeyChecking no
ControlMaster auto
ControlPath ~/.ssh/control-%C
ControlPersist yes
END
- name: Deploy
run: |
...
In the deploy step, the next script will make our blue-green deployment happen. To prepare for the deployment, the following steps will be executed:
- Prepare the Docker agent by setting the target to our remote host and logging in using our ephemeral token
- Adjust the
compose.yamlfile to deploy our created tag instead oflatest. This allows us to rollback easily by retrying the deploy job of a previous workflow. - Pull the images and create the services that don't already exist (for a first deployment for instance)
Once ready, the next step is to create our containers accordingly:
- Route traffic to the Green instance by patching the Caddy config dynamically
- Recreate the Blue instance with
downandup. Since the images were pulled previously, this will use the new tag. - When Blue is ready to accept requests, route traffic to it and restart Green
- When Green is ready, route traffic back to both
- Cleanup old images
prepare() {
export DOCKER_HOST=ssh://remote
echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
# Deploy current tag
sed -i 's/zerodowndeploy:latest/zerodowndeploy:${{ github.ref_name }}/' compose.yaml
docker compose pull
docker compose up -d --no-recreate
}
route_traffic() {
echo "Routing traffic to $1"
case $1 in
"blue")
upstreams='[{"dial":"zerodowndeploy-blue:8000"}]'
;;
"green")
upstreams='[{"dial":"zerodowndeploy-green:8000"}]'
;;
"bluegreen")
upstreams='[{"dial":"zerodowndeploy-blue:8000"},{"dial":"zerodowndeploy-green:8000"}]'
;;
esac
docker compose exec zerodowndeploy-caddy curl \
-H "Content-Type: application/json" \
-d "$upstreams" \
-X PATCH http://localhost:2019/config/apps/http/servers/srv0/routes/0/handle/0/upstreams
}
restart() {
echo "Restarting $1"
container="zerodowndeploy-$1"
docker compose down "$container"
docker compose up -d "$container"
}
wait_healthy() {
echo "Waiting for $1 to be healthy"
container="zerodowndeploy-$1"
if docker compose exec "$container" \
curl -I --retry 30 --retry-max-time 0 --retry-all-errors --fail-with-body http://localhost:8000/health
then
echo "$1 is healthy"
else
echo "$1 failed to start!"
exit 1
fi
}
cleanup() {
docker image prune -f
}
prepare
route_traffic green
sleep 10
restart blue
wait_healthy blue
route_traffic blue
sleep 10
restart green
wait_healthy green
route_traffic bluegreen
cleanup
Optional: clean job
After a while, it makes no sense to keep old images. The following job will clean up old images but keep at least 3 versions in order to rollback if necessary.
clean:
runs-on: ubuntu-latest
needs: deploy
steps:
- name: Delete old images
uses: snok/container-retention-policy@v3.0.0
with:
account: user
token: ${{ secrets.PAT }}
image-names: "zerodowndeploy*"
image-tags: "!latest v*"
cut-off: 2d
keep-n-most-recent: 3
A secret called PAT must be defined with registry write permissions to work.
Testing
Using K6 to perform a load test during a deployment with a few hundred requests per second yields the following results :
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: k6.js
output: -
scenarios: (100.00%) 1 scenario, 200 max VUs, 1m30s max duration (incl. graceful stop):
* default: 200 looping VUs for 1m0s (gracefulStop: 30s)
data_received..................: 5.9 MB 97 kB/s
data_sent......................: 545 kB 8.9 kB/s
http_req_blocked...............: avg=5.27ms min=0s med=0s max=525.06ms p(90)=1µs p(95)=1µs
http_req_connecting............: avg=601.74µs min=0s med=0s max=50.76ms p(90)=0s p(95)=0s
http_req_duration..............: avg=53.13ms min=35.55ms med=48.82ms max=180.61ms p(90)=71.31ms p(95)=80.24ms
{ expected_response:true }...: avg=53.13ms min=35.55ms med=48.82ms max=180.61ms p(90)=71.31ms p(95)=80.24ms
http_req_failed................: 0.00% ✓ 0 ✗ 11417
http_req_receiving.............: avg=132.29µs min=6µs med=57µs max=16.13ms p(90)=269µs p(95)=371.19µs
http_req_sending...............: avg=186.48µs min=11µs med=103µs max=21.89ms p(90)=352.4µs p(95)=526.19µs
http_req_tls_handshaking.......: avg=4.64ms min=0s med=0s max=473.65ms p(90)=0s p(95)=0s
http_req_waiting...............: avg=52.81ms min=25.93ms med=48.5ms max=180.49ms p(90)=70.95ms p(95)=79.88ms
http_reqs......................: 11417 187.095807/s
iteration_duration.............: avg=1.05s min=1.03s med=1.04s max=1.56s p(90)=1.07s p(95)=1.08s
iterations.....................: 11417 187.095807/s
vus............................: 5 min=5 max=200
vus_max........................: 200 min=200 max=200
With 187 requests per second on average, the median response time is 48.82ms and none of the requests failed to complete. The deployment works as expected with no downtime.
Conclusion
For many applications, Docker Compose is sufficient for running and deploying. This article demonstrates that with a little scripting, it's easy to deploy a containerised application to a server with no downtime. If your needs are more complex however, you might need to consider a container orchestration tool such as Kubernetes.
