gamegoblin

Like if I've made a change that is causing some unit test to fail, it can just run off and figure out where I made an off-by-one error or whatever in my change.

Comment · 8 months ago

I routinely leave codex running for a few hours overnight to debug stuff

If you have a deterministic unit test that can reproduce the bug through your app front door, but you have no idea how the bug is actually happening, having a coding agent just grind through the slog of sticking debug prints everywhere, testing hypotheses, etc — it's an ideal usecase

Comment · 9 months ago

AI detectors in general are unreliable, but there are a few made by serious researchers that have only 1-in-10000 false positive rate, e.g. https://arxiv.org/pdf/2402.14873

Having worked in a bigcorp, I've read my fair share of management-speak, and none of it sounds quite as empty as the allegedly AI text.

The AI sounds like someone conjuring a parody emulation of management speak instead of actual management speak.

More broadly — and I feel this way about AI code at well as AI prose — I find that part of my brain is always trying to reverse engineer what kind of person wrote this, what was their mental state when writing it?

And when reading AI code or AI prose, this part of my brain short circuits a little. Because there is no cohesive human mind behind the text.

It's kind of like how you subconsciously learn to detect emotion in tiny facial movements, you also subconsciously learn to reverse engineer someone's mind state from their writing.

Reading AI writing feels like watching an alien in skinsuit try to emulate human face emotional cues — it's just not quite right in a hard-to-describe-but-easy-to-detect way.

Comment · 9 months ago

While reading the text, my mental AI alarm bells were going off, sent it all to pangram.com and it flags both the layoff post and his campaign website text as being 100% AI generated

Comment · 11 months ago

Weird to me that nobody ever posts the actual alleged false positive text in these criticisms

I've yet to see a single real Pangram false positive that was provably published when it says it was, yet plenty such comments claiming they exist

Comment · 11 months ago

I sent the text through an AI detector with 0.1% false positive rate and it was highly confident the Zig book introduction was fully AI-written

Comment · 11 months ago

The other pangram mention elsewhere in this comment section is also me -- I'm totally unaffiliated with them, just a fan of their tool

I specify the accuracy and false positive rate because otherwise skeptics in comment sections might otherwise think it's one of the plethora of other AI detection tools that don't really work

Comment · 11 months ago

pangram.com, the most accurate and lowest false positive AI detector

https://www.pangram.com/blog/third-party-pangram-evals

Comment · 11 months ago

Yes, it's a false claim

Comment · 11 months ago

It's AI-written FWIW

though maybe AI is getting to the point it can do stuff like this somewhat decently

Comment · 11 months ago

Pangram[1] flags the introduction as totally AI-written, which I also suspected for the same reasons you did

[1] one of the only AI detectors that actually works, 99.9% accuracy, 0.1% false positive