Comment by convivialdingo

I spent a few months working on different edge compute NPUs (ARM mostly) with CNN models and it was really painful. A lot of impressive hardware, but I was always running into software fallbacks for models, custom half-baked NN formats, random caveats, and bad quantization.

In the end it was faster, cheaper, and more reliable to buy a fat server running our models and pay the bandwidth tax.