The Problem
We have all run into this issue - you’re mid getting a service ready to deploy or update, and you make an on-the-fly change to a compose file, to update a health-check, change a deploy parameter or modify the update interval, and then, when you go to deploy the service…
yaml: line 12: did not find expected key
Dang! What a mess up, now we have to go back, fix the issue (hopefully test it a bit better) re-commit to the code base, potentially re run test steps, and trigger redeployments.
I had run into this issue one too many times, so I set about to find a fairly quick and dirty way to have some sort of validation done on docker-compose files, when or before they are committed to the git repo, so that I can hopefully catch some potential fat-finger errors, sooner rather than later.
The Solution
Now, the problem here, is that, yes, we could do a normal yaml lint on the compose files, but generally, this would not catch our issues, as the issues I run into tend to be things that a regular yaml lint would not complain about. For instance, having stop_grace_period at the wrong indentation level in your compose file, is not detectable by a traditional yaml lint, but certainly got me thinking, is there some way to detect this?
Thankfully, I found the docker-compose config command, which validates your compose file against the docker compose format, and returns you any errors that it may have, right there in your terminal.
Combining the power of that command, with the power of git, and it’s pre-commit hook, I managed to come up with a simple bash script that does exactly this. Let’s dive into it a bit, and see what it does.
The Implementation
The script itself is available in the .githooks folder in my repo, available here: https://github.com/devinsmith911/docker-compose-verifier - there are also some instructions in the repo on how to get this working, and some notes around it.
Looking at the pre-commit hook file itself, it is a fairly simple script, written in bash, that runs whenever one runs git commit a very common command to check files into a codebase.
The first thing the script does, is grab the list of files being committed to the repo using the command arr=($(git diff-tree --name-only --no-commit-id -r HEAD)). This simply returns a list of file names which have been modified in your commit and pushes them to a list.
Once we have the list of files that we need to test, things are pretty simple, we loop over each file name and check if there is docker-compose in the file name: [[ "$x"==*"docker-compose"* ]] if there is we then run the test:
test=`cat $arr | sed '/env_file/N; //d' | docker-compose -f /dev/stdin config -q --no-interpolate`
In this line, there is some weirdness, and some trickery. The primary reasoning here is that if you run docker-compose config against a compose file which has env_file declared somewhere in it, then the config command fails, as it will look for the env_file that is declared, and complain if it can’t find it. Unfortunately, in my case (and I’m sure many others) our CI tooling generates this file, and thus it is never available when committing to the repo itself.
Therefore, this command takes the file, reads it to stdin, and removes any declarations of env_file in the file, and then runs that through the docker-compose config command, returning the relevant output and exit code.
From here, we simply return 1 or 0 depending on whether the test passed or failed, if it failed (returns 1) then we simply exit 1 in the script, so that the commit does not go through to git. This ensures that a faulty or malformed compose file can not be committed to the repo, and thus this should vastly help with catching fat-finger errors before they are committed to the repo.
The Output
What does all this stuff actually result in? Well…
git commit -m 'test123'
docker-compose.yml
Checking for any docker-compose files that are being committed...
Testing compose file named: docker-compose.yml
WARNING: Some services (docksockprox) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
File docker-compose.yml passes compose check
As we can see above, the moment that I run a git-commit that has a docker-compose file present in it, we can see the check command run and display it’s output. At the end of the output we get File docker-compose.yml passes compose check and the commit goes through.
On a commit with an invalid compose file, we get the following output:
docker-compose.yml
Checking for any docker-compose files that are being committed...
Testing compose file named: docker-compose.yml
ERROR: Top level object in '/dev/stdin' needs to be an object not '<class 'NoneType'>'.
docker-compose.yml does not pass docker-compose config check
Not commiting...
As we can see here, we are getting an error from the compose file validation, and it has stopped the git commit process from completing successfully, which will occur until I modify the file to be correct.
Conclusion
There is a lot of room for improvement here! However, this was a pretty fun exercise to quickly do some validation on my compose files, and keep things as sane as possible before they hit the CI pipelines for deployment.
In the future, I’d like to extend this to do additional things such as do an actual yaml lint, as well as require that services in the compose file have certain things, such as health-checks or startup validation periods.
I purposefully kept this project fairly lightweight and simple, requiring no dependencies apart from having the git config set in the repo, and having docker-compose installed on your local machine.
Instructions on how to install docker compose can be found here: https://docs.docker.com/compose/install/ for all system types.