kaycey2022
Joined 3 years ago
Joined 3 years ago
For instance recently, media downloads on my NAS became quite a bit slower. I asked it to investigate and it quickly got back to me saying the issue was the ethernet link had been downgraded to 100Mbps from 1Gbps. It even went through journalctl to tell me when this had happened. To fix it I just had to plug an ethernet cable out and back in. Now this is the kind of thing, earlier I would have to spend an evening on. But given an open system like Linux, a well made safe model just makes it so much more accessible than having to RTFM for one off things I am sure to forget in a month.
I dont really let it run loose through my systems, and keep an eye on the thinking traces it puts out and the permissions it asks for. But now I dont have to slog through manuals and deal with gruff people on the internet. I can choose to do so after the fact, depending on how much free time and curiosity I have.
A serious router therefore needs to build up a dataset of how different models responded, end to end, to different tasks on different contexts. I won’t comment on whether the current frontier models can reliably judge these outputs, but I am sceptical about that. And moreover look at the state of evils! They are gamed to hell and keep losing credibility.
A more difficult problem for router builders is that they are working on an opaque system behind an external API. How can you reliably guarantee model behaviour when model behaviour has been shown to deteriorate under arbitrary conditions that have nothing to do with the task given? So much investment only to be an AI company that can get rug pulled by the real AI companies at any given time.
I would think the only people who can come up with good routers for a collection of models are the inference providers themselves. Because theoretically they have full control of how their models are served. And inference is not zero cost or cheap for them either. And going by OpenAI’s experience routing is not an easy problem for them to solve either. And they don’t have the incentive to route you to cheaper models and reduce costs for customers at the same time. Routing objective for them is to increase their own profits.