ubutler

Rule #1 of rolling your own RNG: don’t.

Rule #2? Please don’t.

If anything, the latest generation of AI models, Astra and Fable, are prime example of overfitting—whereas benchmarks suggest they’re AGI-tier, users (including myself) report the same old gaslighting, hallucination, context rot, cheating, incomprehensibility patterns as with prior models, sometimes even more pronounced.

Fable and Opus 5, I suspect, will become textbook examples of RL collapse.

Just a word of advice, it could go a long way to have a human write the README rather than an LLM.

Right now, it’s a little hard to follow what the actual findings are, why they are what they are, and why that’s important.

> The six solid objects discovered on Forrest Beach, to the north of Townsville, are thought to be space debris, and the Australian Space Agency (ASA) is now trying to determine where they came from. The BBC has approached the agency for comment.

Off-topic but this is the first time I’m learning we (Australia) have a space agency :O

> Sounds nice except that these are 1 very small scale model, 1 reranker, and 1 embedding model that are far from frontier LLM level.

We've tried to take a first-principles approach to our end goal of 'legal superintelligence' that has involved identifying the areas of our domain most in need of improvement and releasing models that raise the bar on quality in those areas.

We've been around for a couple months now and ended up starting with retrieval and enrichment. The models we've released to tackle those problems have indeed been smaller in size than their competitors, yet they still rank ahead on open-source benchmarks.

Them being so small also helps with their accessibility — as I mention in our post, our models can be deployed on ordinary hardware, not a supercomputer.

Next on our roadmap is reasoning and research, which will require more infrastructure to support, but again, we aim to be judged by performance at the time of release.

> As much as I agree with the message, this reads like marketing copy trying to make a big deal out of a tiny model being hosted privately.

The point of this point is really just to reaffirm our commitment to sovereignty and accessibility and contrast our approach with that of major AI labs. It _is_ possible to commercialize LLMs while still keeping them accessible. A customer using a self-hosted deployment today does not need to worry about our models no longer being available tomorrow. We think that's a good thing. And moving forward, we want to keep that option available for anything we do, instead of trying to pull up the ladder while we're ahead.

Our models can be deployed on premises as well, though that is more of a bespoke offering at the moment. We've also been fortunate enough to have trained most of our models on our own private infrastructure. Your question raises a broader question, though, about sovereign risk attached to cloud services in general. For some, particularly governments, sensitivity is so high that certain workloads must run fully on their own hardware. I don't see that changing for now, and, in fact, in certain jurisdictions, the trend is turning against cloud services.

ChatGPT already does this, albeit in limited circumstances, through the use of its sandbox environment. Asking GPT in thinking mode to, for example, count the number of “l”s in a long text may see it run a Python script to do so.

There’s a massive issue with extrapolating to more complex tasks, however, where either you run the risk of prompt injection via granting your agent access to the internet or, more commonly, an exponential degradation in coherence over long contexts.