Nginx has had some strange bugs under heavy load. Way back in the day, reddit used Nginx as its frontend webserver/load balancer. All of a sudden one day I noticed that it was sending 99% of the traffic to the app server listed first in the config, then 99% of the remaining traffic to the next one, and so on down the line, such that the last app server in the list (only about 10 at the time) was essentially getting no traffic. If I changed the config, the traffic pattern was still based on the config order, so it wasn't a bug in the hashing or the numerical values of the IPs of the app servers or anything like that.
Never did figure out what caused that problem. We just switched out Nginx for Haproxy and never had a problem again. :)
Never did figure out what caused that problem. We just switched out Nginx for Haproxy and never had a problem again. :)