Comment by sebazzz

Those local APIs already exist. It is called Microsoft Foundry Local: https://learn.microsoft.com/en-us/azure/foundry-local/get-st...

Supports GPU, NPU and CPU.