0x63_Problems

which URL is it? it's kind of a deep assumption since we're evaluating on ease-of-navigation from the homepage, and looking for /llms.txt etc. but there should be a way for me to add support

Behind an agent is a human that might use the product. I think customers/consumers will demand that their personal/subscription agent can access the product.

Each product is different (how much does it cost to serve a marginal user, how risky is it). But generally I think allowing ephemeral accounts like Cloudflare, or serving simulated/static accounts before a human claim can both work.

For a lot of products it can just be an extension of the free tier I think too.

Are you mainly concerned about scraping? or anti-agents in general?

The initial step of the scan looks at homepage content to try to figure out where to navigate next, it's getting confused by all the different products linked. Will fix this!

I think it's only recursive from the perspective of the humans, i.e. they design Astra, which itself as part of its deployment designs Galactica, etc.

So humans develop things one after the other, but when the thing itself starts developing new things, those are happening 'recursively' in its scope.

Very cool!

Yup I definitely see confusion in our responses around the product and brand names. We do another pass through an LLM specifically aimed at ‘canonicalizing’ the names, but we’ll need to get more sophisticated to catch most issues.

In that case you mentioned, the brand confusion is what accounts for the top three omission for QBO. Both OpenAI and Perplexity rank it #1, but Anthropic ranks the slightly different “Quickbooks” product as #1. Our overall ranking prioritizes products that appear in all three responses, so both are dropped down.

No specific reason, just started with these three, will add Gemini soon!

We are using sonar-pro

These are structured results from explicitly asking the LLM for a ranking in the given category, and we provide guidance in the prompt telling the LLM to 'use best judgment' when the topic doesn't clearly include products.

Also we include the 'key features' from each answer - you can see this by clicking the cell containing the rank (e.g. '1st' in the Anthropic column)

In this case, Anthropic said of 'Death during sleep':

Anthropic Analysis for Death During Sleep

    Painless and unaware experience
    No anticipatory anxiety
    Common with certain cardiac conditions
    Often described as 'peaceful'
    No suffering