Because it’s a waste for anything other than proof of concept/handful of users.
It’s really simple to take some python inference code and wrap FastAPI around it.
However, inference servers exist for a reason. You’ll quickly find that performance, VRAM usage, model management, etc isn’t practical with the FastAPI approach.
Speaking personally inference server implementations like Nvidia Triton bring a model to performance metrics that are absolutely night and day vs the FastAPI approach - in many cases orders of magnitude higher performance in terms of response time and requests per second.
It’s really simple to take some python inference code and wrap FastAPI around it.
However, inference servers exist for a reason. You’ll quickly find that performance, VRAM usage, model management, etc isn’t practical with the FastAPI approach.
Speaking personally inference server implementations like Nvidia Triton bring a model to performance metrics that are absolutely night and day vs the FastAPI approach - in many cases orders of magnitude higher performance in terms of response time and requests per second.