> This is an excellent use case for completely local, small model inference,
How small is this 'small model' inference?
Browsers especially are expected to operate on a very wide number of devices. It makes perfect sense to me to start off with a cloud driven approach and later allow people to configure it to run locally if they have the hardware to do so. Anyone with the hardware to do so is already signaling that they'd have the technical interest to make that change while having the default 'works for everyone' mode be one that doesn't require a powerful local device.
I say this as someone using a 6 year old computer and only upgraded to 8GB of ram when the latest WoW expansion was released.
How small is this 'small model' inference?
Browsers especially are expected to operate on a very wide number of devices. It makes perfect sense to me to start off with a cloud driven approach and later allow people to configure it to run locally if they have the hardware to do so. Anyone with the hardware to do so is already signaling that they'd have the technical interest to make that change while having the default 'works for everyone' mode be one that doesn't require a powerful local device.
I say this as someone using a 6 year old computer and only upgraded to 8GB of ram when the latest WoW expansion was released.