Bocajmai
Joined 3 years ago
Joined 3 years ago
Where they did differ however! Was speed. And saying "none". Luna took 0.66s at at the median and 1.27s at the p95. Versus 0.28s and 0.42s with Jev, which really matters for live agents, like our use case, which needs to happen quickly and naturally on the phone. It also caught 32 of the 40 unanswerable questions vs Jev's 38, so it handed over a wrong page 8% of the time against 3%.
On a wider data set, also fair. But it's interesting enough for us to pursue, so I'll report back with more data soon hopefully!
Thanks for taking the time :)
This is a reference I made for myself. Five required sections from the book, quality checks for each one, and one extension I added (Open Questions as a shapedness metric).
Would love to hear how others have adapted Shape Up for their context.
I'm not sure I know the answer, but it's a very good point! Thanks
The parallels to software are exact. Front of house/back of house coordination. Pre-service meetings. The "pass" as a coordination point. Station ownership. The quiet kitchen principle.
Then I discovered Basecamp's Shape Up is essentially their version of the brigade de cuisine system. 6-week cycles, betting table, appetite setting, cool-down periods. It maps perfectly.
I wrote this to work through what small bootstrapped software companies can learn from how these organizations actually operate. Would love feedback from others dealing with this coordination problem.
If it doesn't sound right, maybe check you've not leaked any keys in public github repos etc? There's crawlers hunting for exposed API keys. Stay safe out there!
We gave Jev a one-line description of every page on the site and asked it one choice question, which page answers this, with “none of these” as the last option. Across four of our customers’ sites (22 to 232 pages) and 168 caller questions, it put the right page first 88% of the time, against 70% for our production embeddings. The number I care about more is how often the LLM got handed the wrong page, 3% against 33%. When the site didn’t have the answer, Jev said so 95% of the time and our current search 23%. Each lookup took about a third of a second, and all 1,449 requests cost about 45 cents.
Fair warning, this piece is written for vibe coders and tech business owners rather than engineers, so it’s the plain English version. The method, charts and the bits that went wrong are in the companion write-up: https://sayora.ai/many-hands/jev-vs-embeddings/
Most of the failures were ours. Jev reads page descriptions literally. A plumber’s installation page mentioned water softeners in the body but not in its description, so Jev said “none of these” at 96%. Above 255 options or about 64k tokens you need embeddings to shortlist first. And we couldn’t find a published data processing agreement (DPA), which matters with UK callers’ data.
Happy to answer anything about the setup or where it didn’t work.