turtlesoup

Thanks! It is part of the clustering step, I tell the model to make a judgement of whether something is inanimate or hallucinated (as defined by low support from only non-frontier models / judgement). I iterated on this a lot and made an eval set out of my LinkedIn contacts where I run GPT5.5 with web search and xhigh reasoning to determine pseudo ground truth. I tuned this to be higher recall (more things classified as non-hallucination) but it is definitely not perfect

Top percentile of people searched!

Whoops we had some scrolling bugs with sound, hopefully fixed

Interesting, that probably reveals something about the training set for most of these models

What tracking cookies are you seeing? The intention was just some cloudflare checks for spam identification

This is directional; models self-report confidence on their answers and the strength is a linear combination of the confidence plus a bonus for every model that got clustered in.

Models are notoriously uncalibrated especially for self-reporting confidence so I would treat it lightly. Hopefully I can study this a bit later on!

Sure thing! It is the same prompt for every model in the rollouts, here it is

  No tools are available. Do not imply that you searched, looked up, browsed, or verified anything externally. If the name is ambiguous, return distinct likely people or entities rather than blending them. Do not invent entries to fill the list. Return only JSON.

  Return fewer than 8 if fewer credible matches exist. Return {"results":[]} if you do not recognize any credible person or entity. Use this JSON shape:
  {
    "results": [
      {
        "rank": 1,
        "name": "Resolved person or entity name",
        "confidence": 0,
        "snippet": "Concise snippet supporting this result."
      }
    ]
  }

  Confidence is 0-100 for how strongly you recognize this specific person or entity. Snippet should be one short, complete search-result-style description (≤ 160 characters).

  The query is: Who is "<name>"?

The clusterer prompt is more intricate and I'm happy to share if of interest, but I have an invariant that every result showing up in a rollout must be clustered into one result (sometimes collapsed into the hallucinations section).

No purpose, just a fun hack and science experiment. Glad to see it getting a good reception!

Just disabled latest!

Just deployed a fix for this; removed latest and capped pagination.

Ah yeah, the "hallucinations" classification is optimized for recall (keeping as many results as I can) not precision. It is mostly based on small models being the only support for a claim. Certainly lots of hallucinations everywhere!