Why I Prefer Docker

Every couple days you see a discussion on Hacker News like this. To summarize, every time a Docker related article gets linked, there are a chorus of people complaining about the hype engine, usually with some people talking about how Docker is good for production but interferes with dev... and some people saying it's great for dev but interferes with production :)

The truth of the matter is that Docker does a very good job of abstracting away the underlying hardware you are messing with, no matter what that hardware may be. Let me explain the path I have taken here. When I joined my most recent company, I took over the DevOps function and began working on our deployment system. We were using Chef, and it was rough. Our stack has a large number of dependencies- Elasticsearch, Postgres, Mysql (yes both, Postgres is a stand in for Amazon Redshift for our dev environment), Redis, Nginx, uWsgi, Celery workers (and a Celery Beat service), an FTP and SFTP server, and our Flask application. Add on to that NPM, Grunt, Yeoman, and who knows how many other dependencies that were involved with building our Angular app.

So, not too bad right? Then let's consider a couple different environments. We need to be able to run everything on our local machines for dev- so we had a Vagrant box setup to run the whole mess. We had a 'dev' environment where we would deploy each new version of the code to AWS- this was a single machine, so used the same Chef recipes... but the process of getting chef actually installed on that EC2 instance was different than the process for Vagrant, so needed some love in deploy scripts (There are a bunch of tools to do this, I know this :) ). Further, we had a 'demo' environment that was the same as 'dev', but deployed less often and had a special set of test data loaded. Finally, there was a 'staging' and 'release' environment- these run the same code, but have to be able to scale horizontally... they use AWS data stores (RDS, Redshift, Elasticache), and had to be idempotent (one annoying problem, for example, is that in order to ensure I was running exactly one Celery Beat service, I had to track the status of that service somewhere...).

Another problem- we wanted everything to be fully 'Phoenix Server' capable- if we woke up tomorrow and EC2 had dropped all of our machines, we should be able to run our script and get everything working again. This is great, and Chef is a big help for that... but Chef from bare metal was taking around an hour to bootstrap our EC2 instances. To make that even more fun, when one of the Chef recipes would start failing (someone changed the download link for an Elasticsearch plugin, for example), it generally meant another hour to test the fix. So, for an auto-scale group or anything like it to work, we had to start at least partially provisioned- we would run Chef, get an AMI, and then re-run it on launch to update to the latest. This provided a speedup (though Chef still used like 5 minutes on a no-op) ... but it meant that the 'bare-metal to ami' script got tested less frequently. This meant that getting a new laptop almost always meant having to fix the Chef script before you could actually do any development, since there was no way to get going when it was broken.

Docker solved these problems for us- in less time then it took us to get to the point where we realized all of the problems we were going to have to overcome. The biggest problem with Chef is that cookbooks are supposed to play together, so that installing two of them doesn't break either one- this makes them super complicated and fragile. Dockerfiles, on the other hand, only have to go from one known state to another known state. This makes is drastically easier to get them working properly. Plus, you get build caching and imaging (so you don't have to restart from scratch every time while developing them). This means that running our tests (docker-compose build, docker-compose run test) does

• smart refreshing of all of our infrastructure - only run pip install if requirements.txt has changed
• pull down images of our dependencies - we package up our mysql and postgres images with test data pre-populated
• fully clean up after themselves
• and only with seconds of overhead when only code has changed

Our jenkins unit-test job runs in less than a minute now... whereas the Chef no-op used to take 5. I never have to destroy and re-create jenkins slaves anymore because something wrong with the Chef script left it in an unrecoverable state... the only software even installed on them is Java, Docker, and Make. And new machines are running the unit tests and a fully functional dev environment a half hour after cloning our repo (it DOES take a while to download all those images!).

Further, while all of that is great for local, it ALSO fixed our other environments. Both the local virtual machine and the remote dev/demo environments get started the same way, with docker-machine. A docker-compose file lets you launch the full stack on any of the three with a single command. The real challenging part in Chef (that I hadn't solved before we ditched it) was going to be orchestrating the horizontal scaling with different numbers of each service though. And Docker, using ECS, solved that for me in about a day. We launch ECS servers using a 10 line cloud-config script from the official ECS AMI (we install New Relic servers and set up the private Dockerhub credentials), and ECS lets you scale each service individually behind a load balancer. If I want to launch 10 more Celery Workers, I do that with a single command and they are distributed across any excess server capacity we have. It DID take a little setup to get it right, especially since we had to write all the ECS commands in JSON because we started using it way before they came up with the compose based interface. But ultimately, it wasn't too hard. We have a manual Jenkins job that updates each ECS task definition, and then updates each Service to use the new one (and then builds/pushes updated compiled Angular code to S3). Staging and Release share the same cluster. The task just sets a different environment variable in the containers, and the code knows what to do from there.

I KNOW everything we did could be accomplished in a lot of different ways. That is probably the worst thing about DevOps these days, there are so many cool tools. But I am a strong believer in Docker. It lets me try stuff without installing it on any machine. It lets me easily create installable images that will run on all the environments I'm interested it. It helps me move those images around instead of having to think about how to transfer the files. It even makes it easy to run things in different configurations - read some of the interesting entry point files people have created for the Dockerfiles, and you'll see how a single image can be run in a ton of different ways.