Disk Page Caching on Amazon EBS

It can be hard sometimes to admit you don’t understand something- especially when you don’t even realize it exists. We solved a problem in our system stack recently that fell into this category. A thing that I didn’t even know existed was broken! I wanted to share the story because there are probably other people out there who don’t know that this is a thing and who it would help.

The odyssey started when our system started routinely restarting application instances. A couple times a day, one of our applications would suddenly start consuming drastically more Disk IO, exhaust the IOPS balance on the instances EBS volume, and then the whole thing would grind to a halt. This would cause our monitoring to kill and retry the app, which fixed it- the new app on the same machine would work fine. And then six or eight hours later, the same thing would happen somewhere else.

The app in question is deployed in a docker container using Amazon Elastic Container Service. It consists of a single container using a supervisor, an Nginx process, and a uWsgi process. Nginx and uWsgi both run multiple worker processes. We do not have any swap of any kind enabled. My naive assumption was that it had to be a swap issue of some sort- by carefully looking at the monitoring we could see that the RAM usage had approached the container limit right before things blew up. I spent a lot of time looking into issues with how Docker specified swap to be used and how that interacted with ECS. It looked to me like we had everything configured properly- no swap should be used, and all the processes said that no swap was enabled. So why was my process clearly swapping to disk?

This is where I learned something new. It turns out that Linux file systems do a lot of magic. The important part here is something called a page cache. When you open a file and read the data, the file system caches that data using a portion of your available ram. If you read the same file again, it will come from the cache instead of the actual disk. When your system starts to approach the RAM limit the file system will reduce the size of the cache to allow the system to use 100% of the available RAM.

The reason our system was failing was a combination of factors. Because we had a lot of little processes, we were approaching the memory limit very gradually. Instead of shooting over it (and the OOM killer getting involved) it would slowly get close to the limit, causing the disk cache to get flushed. At this point, the few disk accesses our system made (reading a few jinja templates) were going to the actual underlying hardware, dramatically increasing the IOPS required.

Under normal operation, with plenty of memory available, we would see exactly 0 disk access- all of the reads were coming from a couple files that were already cached! So as we approached OOM we would see disk access go from 0 to very high. This wasn’t swap though- just the disk cache. Because we are using Elastic Block Store for the file system, this would work for a little while, at which point the IOPS for the block device would be exhausted and whatever process was reading from those files ground to a halt.

I had two key learnings from this adventure. The first is that page cache is a way of caching data that is technically on disk and moving it into memory. This means that accessing the same file repeatedly is actually much faster than I would have expected, and that seeing 0 file system activity most of the time does not mean that your application isn’t using the file system. Second, the memory in use does not actually include everything stored in memory. There is another chunk of memory used for the disk cache, and if enough memory is not kept free for it to work properly, it can severely affect the performance of your application.