lebovic
Joined 7 years ago
Karma
1,931
Submissions
143
Member since
Apr 2
Recent submissions
Comment · 2 months ago
I formed a PBC and worked at a well-known PBC. Personally, I opted for a PBC because I liked that I could balance a specific cause with shareholder benefit.
In most cases, it doesn't really matter. The board + management is still in charge, and they have significant legal leeway regardless of the structure. But there's little additional cost to opt for a PBC, and it does give you more legal defensibility to be truly mission driven. Standard C Corps weren't really intended for mission driven companies (see the shareholder primacy norm).
I think it's popular for AI startups, because many great researchers understand the risks involved, and they don't want what they build to be controlled solely for shareholder benefit.
While non-profits are also an option for a mission driven org, it's harder to raise the large amounts of cash that some AI startups need, and laws around deferred compensation and private inurement (e.g. options-like structures) make employee compensation harder.
Comment · 3 months ago
> all frontier models benefit from more tokens not just Kimi K3
Past a point, that doesn't hold and the score plateaus.
Token hungry models tend to plateau at a much higher token count. Because Kimi K3 is a token hungry model – and 100M tokens (including cache hits) seems at the edge of the plateau for these evals – it could disproportionately benefit from a higher token budget.
For Kimi K3 specificially, policymakers are interested in whether it can find and exploit the same scope of vulnerabilities as models like Mythos. In that context, an answer of "yes, but with quintuple the token budget" is materially different from "no, it performs significantly below the most recent frontier cyber-capable models".
(As an aside, I like the UK AISI and think they're the best example of that kind of group!)
Comment · 3 months ago
The UK AISI post is https://www.aisi.gov.uk/blog/preliminary-assessment-of-kimi-...
Comment · 3 months ago
> Kimi K3 performs significantly below the most recent frontier cyber-capable models
UK AISI cyber evals seem to under-elicit capabilities from quirky models [1]. Kimi K3 is a token-hungry model, and I suspect it hit the eval's 100M token limit well before saturating scores [2].
This gap was true for GLM 5.2 as well; they ranked it at Opus 4.5 level [3]. Both anecdotally and with a held-out eval, I've found GLM 5.2 to be better at security research than Opus 4.6 [4]. But it's a quirky model that degrades quickly at long context lengths.
Personally, I'd rank Kimi K3 above Opus 4.8 and lower than GPT 5.6 Sol in its ability to find vulnerabilities and exploit them. But it's not far from the frontier.
[1]: From the UK AISI: "Our setup likely slightly underestimates open weight models’ maximum capability: we didn’t pursue specific elicitation or optimisations which could have improved performance" (https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are...).
[2]: Their eval also counts cache hits towards the token budget; the 100M token budget is comparable to a ~5M token budget in other evals.
[3]: See GLM 5.2 eval scores in https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are...
[4]: https://dualuse.dev/posts/chinese-models-are-sometimes-bette...
Comment · 3 months ago
Almost exactly a year ago, we released Opus 4.1. It was definitely capable of finding vulnerabilities, and people were using custom harnesses to do so quite effectively.
The newer models are still more capable, but there were people doing this and writing about it (e.g. XBOW).
Comment · 3 months ago
Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though!
Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc).
You mentioned the UK AISI's evals. Their harness is like basic Claude Code with compaction, and it doesn't include any of these tricks (afaik). As a result, I interpret their evals as a lower-bound of capabilities.
Newer models are still more capable, and they require almost no harness to find and exploit vulnerabilities. They're also more capable of performing more complex long-horizon attacks. But we've been past the threshold of modes capable of autonomous hacking for a while now [1].
[1]: Opus 4.6 was used for https://www.noahlebovic.com/testing-an-autonomous-hacker/
Comment · 3 months ago
(This comment was originally on another merged post, and "this page" referred to https://www.qwencloud.com/pricing/token-plan)
Comment · 3 months ago
I'm haven't found an announcement page, but there's a banner on the website announcing Qwen 3.8 and redirecting to this page.
Looks like they're previewing the model only on their subscription plan.
Comment · 3 months ago
Kimi K3 only supports "max" reasoning effort right now, but they plan to enable other levels soon [1].
When I looked at traces from benchmarking, I saw a lot of backtracking and uncertainty while reasoning ("wait, but..."). This also happens with GPT 5.6 and Fable with xhigh/max thinking, albeit to a lesser degree.
I think that explains part of the token inefficiency. Hopefully it will improve with lower reasoning effort settings.
[1]: https://platform.kimi.ai/docs/guide/use-thinking-effort
Email is my [email protected]
Formerly at Anthropic, co-founder of Toolchest (a YC startup), and a bio startup