Recent submissions

Comment · 24 days ago
When Jev launched last month I assumed it was a novelty. Then we tried it on the job our AI receptionist finds hardest. A caller asks something oddly specific, like whether a free valuation ties them into selling with the agent, and the answer is buried somewhere on the business’s website.

We gave Jev a one-line description of every page on the site and asked it one choice question, which page answers this, with “none of these” as the last option. Across four of our customers’ sites (22 to 232 pages) and 168 caller questions, it put the right page first 88% of the time, against 70% for our production embeddings. The number I care about more is how often the LLM got handed the wrong page, 3% against 33%. When the site didn’t have the answer, Jev said so 95% of the time and our current search 23%. Each lookup took about a third of a second, and all 1,449 requests cost about 45 cents.

Fair warning, this piece is written for vibe coders and tech business owners rather than engineers, so it’s the plain English version. The method, charts and the bits that went wrong are in the companion write-up: https://sayora.ai/many-hands/jev-vs-embeddings/

Most of the failures were ours. Jev reads page descriptions literally. A plumber’s installation page mentioned water softeners in the body but not in its description, so Jev said “none of these” at 96%. Above 255 options or about 64k tokens you need embeddings to shortlist first. And we couldn’t find a published data processing agreement (DPA), which matters with UK callers’ data.

Happy to answer anything about the setup or where it didn’t work.

Comment · 26 days ago
Fair, and no, it doesn't prove that. On ranking alone it didn't beat a regular LLM. GPT-6 Luna picking from the same list put the right page first 86% of the time against Jev's 88%, and the interval on that difference (-4 to +7 points) says there's not actually a clear winner.

Where they did differ however! Was speed. And saying "none". Luna took 0.66s at at the median and 1.27s at the p95. Versus 0.28s and 0.42s with Jev, which really matters for live agents, like our use case, which needs to happen quickly and naturally on the phone. It also caught 32 of the 40 unanswerable questions vs Jev's 38, so it handed over a wrong page 8% of the time against 3%.

On a wider data set, also fair. But it's interesting enough for us to pursue, so I'll report back with more data soon hopefully!

Thanks for taking the time :)

Comment · 11 months ago
For me, the hardest part of Shape Up isn't understanding the principles, it's knowing when a pitch is actually done. What's the right level of detail? How do you know if you've shaped enough?

This is a reference I made for myself. Five required sections from the book, quality checks for each one, and one extension I added (Open Questions as a shapedness metric).

Would love to hear how others have adapted Shape Up for their context.

Comment · 1 year ago
True, but is that a strategy problem rather than an operational one? Have they lost the ability to ship, or are they misaligned on what people want? You need both.

I'm not sure I know the answer, but it's a very good point! Thanks

Comment · 1 year ago
I started researching how Michelin-starred restaurants coordinate their teams - not because I care about fancy food, but because they solve exactly this: small specialized teams creating something complex together under pressure.

The parallels to software are exact. Front of house/back of house coordination. Pre-service meetings. The "pass" as a coordination point. Station ownership. The quiet kitchen principle.

Then I discovered Basecamp's Shape Up is essentially their version of the brigade de cuisine system. 6-week cycles, betting table, appetite setting, cool-down periods. It maps perfectly.

I wrote this to work through what small bootstrapped software companies can learn from how these organizations actually operate. Would love feedback from others dealing with this coordination problem.

Comment · 1 year ago
Hmm, your ChatGPT subscription would come up as "OpenAI ChatGPT Subscription" in your bank account, or it does for me. That sounds more like OpenAI API use? Which would not include "ChatGPT" in the transaction.

If it doesn't sound right, maybe check you've not leaked any keys in public github repos etc? There's crawlers hunting for exposed API keys. Stay safe out there!

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton