3 minutes, 54 seconds

At work we’ve written some monitoring and alerting software. We’ve also automated releases and we wanted to take it to the next level so that the instant there was a new release, we’d deployed to production. This automated releasing of software upon a new release is called Continuous Deployment, or “CD” for short. This post documents the simple yet effective approach we took to achieve CD using just 20 lines of bash.

The 20 lines of bash

Super impatient? I got you. This code will mean a lot more if you read the post below, but I know a lot of engineers want more copypasta and less preachy blog posts:

#!/usr/bin/env bash

# Checks for local version (current) and then remote version on GH (latest)
# and if they're not the same, run update script
#
# uses lasttversion:  https://github.com/dvershinin/lastversion

current=$(cd /root/cht-monitoring;/usr/bin/git describe --tags)
latest=$(/usr/local/bin/lastversion https://github.com/medic/cht-watchdog)

update(){
	cd /root/cht-monitoring
	git fetch
	git -c advice.detachedHead=false checkout "$latest"
	/root/down-up.sh	
}

announce(){
	/usr/bin/curl -sX POST --data-urlencode "payload={\"channel\": \"#channel-name-here\", \"username\": \"upgrade-bot\", \"text\": \"Watchdog has been updated from "$current" to "$latest". Check it out at https://WATCHDOG-URL-HERE\", \"icon_emoji\": \":dog:\"}" https://hooks.slack.com/services/-SECRET-TOKEN-HERE-
}


if [ ! "$current" = "$latest" ];then
	$(update)
	$(announce)
	echo "New version found, upgraded from $current to $latest"
else
	echo "No new version found, staying on $current."
fi

Why use CD?

There’s been a lot written about CD, like this quote:

Engineering teams can push code changes to the main branch and quickly see it being used in production, often within minutes. This approach to software development [allows] continuous value delivery to end users.

– rando GitHub blog post

I’ll add to this and say, as an engineer, it’s also extremely gratifying to have a pull request get merged and then minutes later see a public announcement in Slack that a new release has just automatically gone live. There’s no waiting even to do the release manually yourself, it’s just done for you, every time, all the time! Often when engineers are bogged down by a lengthy release process and slow customer adoption where it can take weeks or months (or years?!) to see their code go live, CD is the antidote to the poison of slowness.

Tools Used

The script uses just one custom binary, but otherwise leans on some pretty bog standard tools:

• git – Since our app is in a local checked out git repository, we can use trivially find out what version is running locally with git describe --tags
  
• curl – We use curl to do the POST to slack using their webhook API. It’s dead simple and requires just a bearer token
  
• lastversion – I found this when searching for an easy and reliable way to resolve what the latest release is for our product. It’s on GitHub and it just made my day! It solved the exact problem I had perfectly and it really Does One Thing Well (well, admittedly also downloads)
  
• down-up.sh – this is the cheating part of this solution ;) This is an external bash script that makes it easy tokeep track of which docker compose files we’re using. Every time we create a new compose file, we add it to the compose down and compose up calls in the script. This ensures we don’t exclude one by accident. It’s just two lines which are something like:
  
  docker compose -f docker-compose.yml -f ../docker-compose-extra.yml down
docker compose -f docker-compose.yml -f ../docker-compose-extra.yml up --remove-orphans -d
  
  
• Inside the repository itself, we’ve hooked up Semantic Release which automatically cuts a new release based on Semantic Versioning (aka “SemVer”)

New release process

With all the tools in place, here’s how we cut a new release, assuming we’re on version 1.10.0:

1. An engineer will open a pull request (PR) with some code changes.
2. The PR will be reviewed by another engineer and any feedback will be addressed.
3. Once approved, the PR is merged to main with an appropriate commit message.
4. A release is automatically created: A fix commit will release version 1.10.1. A feat (feature) commit will release version 1.11.0. A commit citing BREAKING CHANGE will release version 2.0.0
5. Every 5 minutes a cronjob runs on the production server to compare the current local version verses the latest version on GitHub
6. If a new version is found, git is run to check out the latest code
7. The down-up.sh script is called to restart the docker services. We even support updates to compose files citing a newer tag of a docker image in which case docker downloads new releases of upstream. Easy-peasy!
8. A curl command is run to make a POST to the Slack API so an announcement is made in our team’s public channel: