sjt-at-rev

Yeah, I've had good experience with Kimi too. Good for the price point, for sure.

Anthropic models are still the best for me -- as long as you don't ask them to do something they don't want to -- but also, way too expensive for bulk pipeline processing. So I keep it to coding and Coworking...

I think the other side of that coin is how much effort it takes to get it to do what you need. Our pipeline is a sequence of very precise tasks where subtle contextual cues matter a lot, and there are a large classes of related error modes.

So yes, while we can work with any of these models to get them to do what we need eventually -- e.g. with prompt tuning to their particular style, adding more examples, or breaking tasks into smaller steps, etc. -- their instruction following has a huge impact on how quickly we can move as a team.

When I say "stinks", for me, if we do three rounds of optimization and testing and a model is still performing inconsistently across a class of related traps then using that model is going to slow us down, and I think it stinks.

In my experience, gemini3.1pro tends to work very consistently with light nudging, GLM with 2-ish rounds of optimization, and for GPT5.4, well it provided no improvement over prior models and would slow us down over the others meaningfully ... and costs too much for the effort.

So, meh, so I still think it stinks, skill level considered.

We've been testing it extensively and its performance is no better than prior versions, and in many cases worse than open weight models like GLM. Gemini3.1 Pro is so significantly better.

To me, the play is: open weight on a provider like BaseTen (solid performance, low price point), or pay up for Gemini3.1 Pro if you need it.

But at their high price and low-ish quality, OpenAI models just aren't in the conversation right now without heavy incentives, e.g. via Azure.

Crazy, TBH. Curious if others find the same thing?

Pulling KPIs from SEC filings is trickier and more error prone than it looks at first blush, and the risk of plausible land-mine hallucinations is still a deal breaker for a lot of serious finance folks. Our solution is a carefully engineered pipeline to process them offline, then MCP/API servers to flow them into agents on demand. Every data point links back to the specific line in the filing it came from. Supports derived metrics (implied Q4s, per-unit economics, etc.). Honestly, results are pretty amazing, e.g. with Claude...