At work we’ve written some monitoring and alerting software. We’ve also automated releases and we wanted to take it to the next level so that the instant there was a new release, we’d deployed to production. This automated releasing of software upon a new release is called Continuous Deployment, or “CD” for short. This post documents the simple yet effective approach we took to achieve CD using just 20 lines of bash.
The 20 lines of bash
Super impatient? I got you. This code will mean a lot more if you read the post below, but I know a lot of engineers want more copypasta and less preachy blog posts:
#!/usr/bin/env bash
# Checks for local version (current) and then remote version on GH (latest)
# and if they're not the same, run update script
#
# uses lasttversion: https://github.com/dvershinin/lastversion
current=$(cd /root/cht-monitoring;/usr/bin/git describe --tags)
latest=$(/usr/local/bin/lastversion https://github.com/medic/cht-watchdog)
update(){
cd /root/cht-monitoring
git fetch
git -c advice.detachedHead=false checkout "$latest"
/root/down-up.sh
}
announce(){
/usr/bin/curl -sX POST --data-urlencode "payload={\"channel\": \"#channel-name-here\", \"username\": \"upgrade-bot\", \"text\": \"Watchdog has been updated from "$current" to "$latest". Check it out at https://WATCHDOG-URL-HERE\", \"icon_emoji\": \":dog:\"}" https://hooks.slack.com/services/-SECRET-TOKEN-HERE-
}
if [ ! "$current" = "$latest" ];then
$(update)
$(announce)
echo "New version found, upgraded from $current to $latest"
else
echo "No new version found, staying on $current."
fiWhy use CD?
There’s been a lot written about CD, like this quote:
Engineering teams can push code changes to the main branch and quickly see it being used in production, often within minutes. This approach to software development [allows] continuous value delivery to end users.
– rando GitHub blog post
I’ll add to this and say, as an engineer, it’s also extremely gratifying to have a pull request get merged and then minutes later see a public announcement in Slack that a new release has just automatically gone live. There’s no waiting even to do the release manually yourself, it’s just done for you, every time, all the time! Often when engineers are bogged down by a lengthy release process and slow customer adoption where it can take weeks or months (or years?!) to see their code go live, CD is the antidote to the poison of slowness.
Tools Used
The script uses just one custom binary, but otherwise leans on some pretty bog standard tools:
git– Since our app is in a local checked out git repository, we can use trivially find out what version is running locally withgit describe --tagscurl– We usecurlto do thePOSTto slack using their webhook API. It’s dead simple and requires just a bearer tokenlastversion– I found this when searching for an easy and reliable way to resolve what the latest release is for our product. It’s on GitHub and it just made my day! It solved the exact problem I had perfectly and it really Does One Thing Well (well, admittedly also downloads)down-up.sh– this is the cheating part of this solution ;) This is an external bash script that makes it easy tokeep track of which docker compose files we’re using. Every time we create a new compose file, we add it to thecompose downandcompose upcalls in the script. This ensures we don’t exclude one by accident. It’s just two lines which are something like:docker compose -f docker-compose.yml -f ../docker-compose-extra.yml downdocker compose -f docker-compose.yml -f ../docker-compose-extra.yml up --remove-orphans -d- Inside the repository itself, we’ve hooked up Semantic Release which automatically cuts a new release based on Semantic Versioning (aka “SemVer”)
New release process
With all the tools in place, here’s how we cut a new release, assuming we’re on version 1.10.0:
- An engineer will open a pull request (PR) with some code changes.
- The PR will be reviewed by another engineer and any feedback will be addressed.
- Once approved, the PR is merged to
mainwith an appropriate commit message. - A release is automatically created: A
fixcommit will release version1.10.1. Afeat(feature) commit will release version1.11.0. A commit citingBREAKING CHANGEwill release version2.0.0 - Every 5 minutes a cronjob runs on the production server to compare the current local version verses the latest version on GitHub
- If a new version is found,
gitis run to check out the latest code - The
down-up.shscript is called to restart the docker services. We even support updates to compose files citing a newer tag of a docker image in which case docker downloads new releases of upstream. Easy-peasy! - A
curlcommand is run to make aPOSTto the Slack API so an announcement is made in our team’s public channel:
Wrap up
You may become overwhelmed in the face of needing to store SSH private keys as a GitHub secret or having a more complex CD pipeline with a custom GitHub Runner. Don’t be overwhelmed! Sometimes the best solution is just 20 lines of bash you copied off a rando blog post (that’s me!) which you tweaked to work for your deployment. You can always get more complex at a later date as needed, but getting CD up and running today can be incredibly empowering for the engineers working on your code!
