a11r
Joined 11 years ago
Karma
632
Submissions
78
Member since
Feb 26
Recent submissions
Comment · 14 days ago
A negative datapoint - We have been evaluated prior Deepseek Flash models (both the original 4.1 flash and the smaller 4.0 flash) and they have been overthinkers. Our workload is well defined but moderately difficult coding tasks and among open weight models GLM5.3 flash performs best with Qwen3.8 flash next a close second. DeepSeek 4.1 flash fails at lower thinking levels and spends a huge amount of tokens (an order of magnitude more) at the highest thinking levels where it is able to successfully complete the tasks. I will evaluate this newest release since they specifically claim "the model reduces task execution time and output length", but keeping my hopes low.
Comment · 15 days ago
Artificial analysis shows this: underperforming Qwen3.8 Flash Next and GLM 5.3 Flash, both of which you can run on a single B300 . I think Meta has a bit of catching up to do in the open-weight space.
As an aside, it's really unfortunate that artificial analysis only provides comparison at the highest reasoning level, which is mostly unusable for high-volume production use cases. I would really like them to start providing comparisons at low and medium reasoning levels across these models because those tend to be the ones that actually get used in production.
Comment · 17 days ago
I seriously doubt their ability to double API prices on Dec 1. Token prices are in a clear downward trend and Google should maximize revenue by competing for volume. Currently OpenAI is the only US frontier lab trying to seriously compete with open-weight models that you can run yourself for an order of magnitude lower than what frontier labs want to charge
Comment · 26 days ago
I am renting spot VMs from Nebius. I've tried a variety of other providers like Vast and Spheron. Vast worked well for renting 5090s but I like the large memory and pricing I'm getting at Nebius for RTX Pro 6000. The extra RAM really matters because I need PLE to offload the ngram to RAM.
Comment · 26 days ago
Yes, you can probably get the whole thing up and running in about an hour the first time. If you pause and restart, it takes about 15 minutes to load the models from disk into GPU memory, so budget for cold startup time.
Comment · 27 days ago
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
Comment · 27 days ago
We recently moved from 27B to Flash Next. The quality is superior for coding. Our workload is primarily well-defined coding tasks that need to be attempted a few times before the model gets it just right. FlashNext is also better at finding issues in generated code than Gemini 3.8 Flash.
Comment · 1 month ago
First they ignore you. Then they ridicule you. Then they attack you and want to burn you. And then they build monuments to you. (Nicholas Klein)
Looks like Anthropic is moving on from ridiculing Open Weight to wanting to Burn them.
GLM-5.3 flash has been great for us and I am going to now invest serious effort in evaluating the full fat GLM-5.3 given this ringing endorsement.
Interestingly Z.ai does not train on user prompts, unlike Anthropic. (source: https://openrouter.ai/z-ai/glm-5.3#providers )
Comment · 1 month ago
One key factor not directly mentioned in the article is the simple alignment of financial incentives when electricity distribution was privatized. Before privatization, the agencies responsible for collecting electricity bills had little incentive to actually perform the collection. When coupled with rampant corruption, you get a situation where a lot of bills are either getting unpaid or not even being generated and just being written off as transmission and distribution losses. Switching to private distribution immediately aligns the incentives of the distributor to actually bill their customers for the electricity they are using. None of this is meant to take away from the value of the technical and policy work that led to improvements, and I'm sure all of the factors called out in the article also contributed to this change.
Comment · 1 month ago
That whitepaper puts the cost of a 40 MW cluster at $167m on land and $8.2m in space, inclusive of launch costs.
Comment · 1 month ago
Pretty impressive to see the amount of performance they can squeeze out of the same hardware. I suspect the same process will play out for all combinations of LLMs, inference providers and hardware stacks. This should bring down the cost of inference for the providers by an order of magnitude in the next year and lead to fantastic margins for inference providers.
Comment · 1 month ago
Back in 2008 Fujitsu has one of the best performing 10Gbps Switches. We were building 40 Gbps packet sniffers at Google (4x 10 Gbps NICs) and needed switches that could do things like mirror traffic across ports at line rate. Fujitsu was way ahead of the pack. I always wondered what held them back from building a meaningful networking business in the US.
Comment · 1 month ago
It is great to see another player introduce a fully open stack. Nvidia's Nemotron is the only other prominent one I know of.
All that said, the headline claims do not match the self-reported performance. For example, the dense 32B model is significantly behind Qwen3.8 27B (chart towards the bottom of https://ifm.ai/blog/k2). Gemma4 31B is not in the comparison set. This is the most important sweet spot for self hosted open-weight models today and real competition here will be very welcome.
Comment · 1 month ago
Looks like the strategy of regular updates with incremental improvements is working out well. Interestingly, the biggest jump in Artificial Analysis Intelligence Index score is for reasoning level Medium ( 3.7 was 51, 53, 57 for Low, Medium and High, 3.8 is 52,57, 59 respectively). I think scores at lower reasoning levels are more indicative of model capability since higher reasoning levels are focussed on benchmaxxing. We use the lowest reasoning level in production with good results.
Comment · 2 months ago
Quoting RGFusion from Reddit: LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.
The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.
Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.
https://www.reddit.com/r/LocalLLaMA/comments/1vy6smx/comment...
https://morph.systems https://twitter.com/_a11r_ https://www.linkedin.com/in/1abhishekkumar/