h14h
Joined 1 year ago
Joined 1 year ago
If you pay API prices for inference in your product stack & have specific use-cases w/ custom evals, these smaller models provide an additional cost/speed lever to adjust. Ultimately, I think the point is to compete on cost for use-cases where Sol on the lowest reasoning is overkill.
If you have a fairly straightforward usecase where Terra performs just as well as Sol, but is faster and cheaper, why not use it?
Built this little vibe-coding app as an experiment/cost exploration: https://glare.run
It's awesome how cheaply you can run an entire agentic coding loop (including compilation & live reload) in their v8 isolate sandbox.
The onboarding experience of answering a WhatsApp message asking simply "what have you been putting off?" was pretty slick. Now I just want to see a version I can run on my own hardware so I don't have to worry about some sketchy company hoarding my data.
I know OpenClaw and Hermes are supposedly answers to this, but I find that they've each become so bloated with features that they're far more difficult to use and bug prone than I'm happy with. I'd prefer something more opinionated that limits the surface area for bugs as well as the number of decisions I need to make as an operator.
1. AI Agents (which are at the center of Omarchy) arguably make just about anyone a "developer" in the conventional sense, and potentially more willing to use a dev-forward desktop 2. Converting the types of devs who've been stuck on MacOS for years because they're neurotic about UI/UX "polish" could be a great way to smooth out Linux's many rough edges
IMO the biggest hurdle with building "a Linux desktop for everyone else" is QA. Discovering and fixing the random little bugs that only surface from millions of slightly different hardware/software configurations is a massive undertaking that I'm not sure can be easily brute-forced.
Alternatively, I see a lot of value in building an exciting desktop that attracts a critical mass of folks who are comfortable diagnosing bugs, submitting clear diagnostic reports, or even fixing said bugs themselves, and willing to do that work (or delegate it to an AI agent) for free.
Even if Omarchy doesn't end up being the desktop "everyone else" actually uses, it could be a great first step towards building out the kind of community linux would need to ensure such a desktop meets the requisite user-friendliness standards.
Honestly I'm probably more likely to build my own OpenClaw-mini with a custom client app. Shoehorning client connectivity into an existing messaging app borderline feels more complicated than just having an LLM build a minimal client app from scratch with full awareness of the backend capabilities.
It's not trivial to do, and I never managed to get bot-to-bot communication working. Even with my janky setup, the experience is honestly pretty great. Grok Bot simplifies the setup for this about as far as I imagine is possible, and frankly it's a pretty slick experience.
I fully expect this paradigm to catch on quickly.
Recently picked up a 2013 Mac Pro (the trashcan one) on eBay for $150, and the official 6-bit quant of 2.6B runs at 25-30 tok/s under llama.cpp on one of the D500 GPUs. After installing Ubuntu LTS I have myself a lovely (if somewhat inefficient) little homelab server.
https://artificialanalysis.ai/methodology/intelligence-bench...
Edit to provide AA's article explaining it:
https://artificialanalysis.ai/articles/artificial-analysis-i...
Whether authoring such a feature is itself a criminal act is an outrageous question, to which the answer should be an obvious, and emphatic "no".
I have high hopes on this topic, given token efficiency seemed to be the primary (only?) goal of the K2.7 Code release.
Excited to see the signals that come out of the big eval/benchmark sites.
Terra 6.1 on low reasoning outperforms Luna 6 on max reasoning, while costing half as much per task and completing each task over 10x faster (per AA v5).
Luna 6.1 will likely close the intelligence & cost gaps, but that speed advantage will remain.