Cloudflare has almost no cold boot and I think their ml models are prefetched within the same DC, so loading the model the first time should have no noticable overhead either.
Correct me if I'm wrong, but that's how I interpreted it when they first started with ML models on their CPU's.
Correct me if I'm wrong, but that's how I interpreted it when they first started with ML models on their CPU's.