Comment by hexomancer

I think very soon big LLM providers (OpenAI et. al.) will provide a service like this. With the LLMs becoming faster, the agentic task completion bottleneck will soon move to the tool calls (both execution time and round-trip latency), so it makes sense to have a server host the project very close to the actual LLM doing the inference in order to minimize latency.