Hmmm... 39M requests per month = 15 requests per second. "20.3MM out of 39MM" of those are hitting a cache, leaving 7.2 requests per second that actually need to be run. The article says requests typically take 2.4s but it's a scraping API and many of those seconds are going to be I/O rather than CPU time. Assuming that there's say 1.0s of CPU time used per request, you should only need an average of ~8 cores/hyperthreads for this.
I suspect you could fit this into a pair of dedicated server s from somewhere like OVH or Hetzner for <$150/month (two servers to provision for peak load, which I'm assuming is ~3x as high as the troughs).
I suspect you could fit this into a pair of dedicated server s from somewhere like OVH or Hetzner for <$150/month (two servers to provision for peak load, which I'm assuming is ~3x as high as the troughs).
Is there any reason why this wouldn't work?