chriddyp
Joined 6 years ago
Karma
131
Submissions
50
Member since
Apr 16
Recent submissions
Comment · 6 days ago
In my tests [1], GPT-6.1 Terra joins the Pareto Frontier in between GPT-6 Luna and GPT-6.1 Sol. So, exactly as advertised!
It's also a very fast model, of all 33 models we've tested so far, only Haiku 5.5 and Nemotron 3.5 Super VL have passed the exam faster but both scored lower. It also wrote the _least_ output in its responses.
[1] Benchmarks for data analytics tasks: https://plotly.com/blog/gpt-6-1-terra-plotly-data-analytics-...
Comment · 9 days ago
Big improvement over Kimi K3 in our data analytics benchmark:
it scored a 98% vs a 81% in July.
Quite slow in our exam. A lot of that was due to the number of tool calls that it's making. In one of our questions about finding data issues in a dataset, it made 73 queries. Other models like 6.1 Sol only made 18 for that question.
Full deep dive here: https://plotly.com/blog/kimi-k3-1-pro-plotly-data-analytics-...
Comment · 13 days ago
In our data analytics benchmark, it got the best score of all of the Chinese models and tied Opus 5.5 & Argon (41/43 questions correct). It's also 1/3rd the price of both of those models. However, it seems to use A LOT of tokens and it's the slowest model that we've tested for the exam; it's slow because of the number of tokens it uses, not the inference speed.
We also tested Space Bunny (its stealth version) a few weeks ago and the behavior changed quite a bit. Went from 30s / question to 3 minutes / question and the accuracy improved quite a bit.[1]
This benchmark is short-horizon data analytics questions, so not fully representative of software engineering tasks. YMMV.
[1] https://plotly.com/blog/minimax-m3-1-flash-plotly-data-analy...
Comment · 18 days ago
Awesome model. Tested it for data analytics and found it's in the class of Opus 5.5 in terms of performance and cost.
Compared to Muse Spark 1.3, found that it's more than 2x faster about 2x more expensive. I normally equate accuracy to size, and size to speed, so nice surprise to see a model get better _and_ faster.
Like Argon 4, models in this class are a bit over powered for short-horizon data analytics tasks. GPT-6.1 Sol, Haiku 5.5, GLM all perform a similar grade at 1/10th the cost.
Full results here: https://plotly.com/blog/muse-wonder-plotly-data-analytics-be...
Comment · 18 days ago
Ran on our Data Analytics benchmark: https://plotly.com/blog/gemini-4-argon-plotly-data-analytics...
Excellent model. Same price and score as Opus 5.5 in our 23 question exam.
This size / class of model is now a bit overpowered for these short-horizon data analytics tasks. OpenAI 6.1-Sol, Haiku 5.5, GLM 5.3 Flash all achieve about the same results but for ~1/10th the cost.
In other words, we're getting to the stage where models are "good enough" for certain tasks like data analytics and competitiveness is going towards size & speed.
Comment · 23 days ago
I've been testing the models against data analysis tasks, and found that the recent Luna & Haiku releases from the last month have really reset the narrative around price x quality.
DeepSeek 4.1 Flash is really is 10x cheaper than Opus, but it's in the same ballpark as Luna & Haiku in terms of quality, cost, and speed for data analysis. (Luna & Haiku actually perform marginally better, cheaper, and faster).
Here's a breakdown: https://plotly.com/blog/deepseek-v4-1-flash-data-analysis-be...
Note that this benchmark is for data analysis, not general software development. So the tasks are shorter and more constrained.
Comment · 24 days ago
It's a valid concern. We're not releasing the dataset, the answers, or the full set of questions to help prevent this. At the same time, I like to share where it gets things wrong in a bit more detail, which involves sharing a bit of the exam. I expect that we'll create a new benchmark with a new dataset in 6 months.
Comment · 25 days ago
Ran our DataAnalyticsBench benchmark on it: https://plotly.com/blog/claude-haiku-5-5-plotly-data-analyti...
9x cheaper than Haiku 4.5 and 2 letter grades better. It's also now the fastest model (using the default speeds, not trying any of the other models "Fast" mode) to complete the exam.
Similar ballpark to Luna in price, cost, and accuracy. These are very cheap models: $0.38 to answer 40 in-depth data analytics questions (compared to $15 for Opus 5.5 or $20 for Astra).
Overall very good at data analysis - handling all of the straightforward data analytics questions correctly. It fell short answering some of the questions that required some deeper statistical analysis like looking into other variables. In other words, it's not as persistent as other models in its analysis, which I think we'd expect from how they're positioning the model.
Compared to OpenAI: GPT-6 Luna did a bit better and was about 30% the cost of Haiku 5.5. GPT-6.1 Sol got all answers correct, but was 10x more expensive.
Comment · 25 days ago
That’s correct. About the same in price, but Qwen does get more answers correct in this particular benchmark.
Comment · 26 days ago
Just ran this through our data analytics benchmark (I work at Plotly).
It's 10x cheaper than Mistral Medium 3.5 from April and goes from 58% to 74% correct. Definitely a generational shift.
It's not on the Pareto curve yet, but it's good enough for data analytics, and at this rate I suspect it'll be excellent in another few months.
Full write up: https://plotly.com/blog/mistral-large-4-plotly-data-analytic...
GitHub: https://github.com/chriddyp Personal website: https://chris-parmer.com