I ended up reducing the number of clients. In my case I had a thousand servers, and I was able to change the application structure and merge them into a dozen big application servers. Now with connection pooling each server has less than 100 connections.
As a more permanent solution for scaling, I'm moving out of mysql into something more distributed.
As a more permanent solution for scaling, I'm moving out of mysql into something more distributed.