uWsgi and Nginx Connection Queues
I have been using Flask for quite some time now to serve as my web framework of choice. It requires little/no setup, and I already have code examples to get through most of the pain points. I have pretty much always used uWsgi as the application container and Nginx as a frontend reverse proxy. Life was good with fewer decisions. However, there was something that I fundamentally misunderstood that was leading to performance problems.
Our primary application at BAMx handles many very simple transactions- data capture, session pixels, things like that. These respond in 1–10 ms routinely, and there’s no problem. The heavier requests- ones that wind up hitting a database, for example, are much fewer, and we can handle several times our current peak load with only a few machines. I had always relied on Nginx to handle connection pooling- if one request blocks my uWsgi workers for an unusual amount of time, I figured that Nginx would just hold the other requests until the worker was ready again.
However, this is not what I was seeing! When things slowed down, requests would be dropped and our logs would be filled up with
failed (11: Resource temporarily
unavailable) while connecting to upstream
instead of just seeing increased response times as requests waited in the queue. Increasing the number of Nginx workers or uWsgi workers didn’t really seem to help with the problem… whenever we managed to slow down our database with some poor coding decision, we would wind up dropping requests.
Eventually, after a lot of pulling out what little hair I have, I started to understand some things. The main one here is that Nginx queues up requests that are waiting on Nginx to handle them! However, these requests aren’t in that group. What it was actually doing was accepting the connection, then forwarding that connection to the unix socket that uWsgi was listening on, and considering matters closed. IT had handled the request, it wasn’t its fault that the server couldn’t handle things!
The design is that the uWsgi head connects all of the workers to the same socket. Each worker wakes up, takes a connection/request off the queue, and then processes it. So it was actually the linux kernel that was handling queueing these requests up, not either of the pieces of software I was using. It turns out that the kernel setting net.core.somaxconn controls how many connections a socket is willing to queue up, and the default is set to 128, which on a busy server gives you only a couple seconds of slack. This got us most of the way there- a simple test server verified that the number of concurrent requests a uwsgi server could handle was limited by workers + somaxconn.
However, there were a couple more obstacles in the world of Docker on ECS in EC2. Actually changing the setting for this was a simple tweak to our deploy script. Making our uWsgi servers use the higehr limit is simple by setting the --listen=16384 flag on the server. However, when we tried to deploy it, it didn’t work!
There is one more bit of nuance for my unsophisticated self. Some of the kernel networking settings are name-spaced, so when Docker sets up a new networking bridge, that got its own settings. Painfully, 128 was hardcoded as the value for that setting by Docker. Version 1.12 added the ability to override this setting on a per-container basis… but ECS was still on 1.11 (and we would have had to wait for them to support the new flag anyway). This left two options- run my web servers in privileged mode and have the entrypoint script running kernel tuning commands, or switch to using net=host for our web servers. The second didn’t seem like a terrible idea (with ECS, we already couldn’t run two copies of the same container since it forces you to bind specific ports for the load balancer…) and should even result in a bit of performance gain. So that was the solution- set the host networking limit much higher, then have our ECS service use the host network stack (and adjust that service to have the pods communicate with each other over localhost instead of the dns names the docker network provided :( )
Either way though- our app could now handle temporary slowdowns without dropping connections for MUCH longer periods of time!