13pixels

Facebook is honestly the least interesting crawler misbehaving right now. The real shift is GPTBot, ClaudeBot, PerplexityBot and a dozen other AI crawlers that don't even identify themselves half the time.

I've been monitoring server logs across ~150 sites and the pattern is striking: AI crawler traffic increased roughly 8x in the last 12 months, but most site owners have no idea because it doesn't show up in analytics. The bots read everything, respect robots.txt maybe 60% of the time, and the content they index directly shapes what ChatGPT or Perplexity recommends to users.

The irony is that robots.txt was designed for a world where crawling meant indexing for search results. Now crawling means training data and real-time retrieval for AI answers. Completely different power dynamic and most robots.txt files haven't adapted.

One underappreciated angle: the dot-com boom created new distribution channels that were broadly accessible. Anyone could put up a website and reach people. The AI boom is quietly reshuffling distribution in ways most companies haven't noticed yet.

We've been tracking how LLMs recommend products across categories. The correlation between Google ranking and LLM recommendation is near zero (0.08 across ~150 brands we tested). So a company that spent a decade building SEO authority can be completely invisible to ChatGPT, while a smaller competitor with better documentation and more mentions on Reddit or GitHub gets recommended instead.

That's a massive, silent redistribution of discovery -- and unlike the dot-com era, the companies being disrupted mostly don't even know it's happening yet.

The irony is that while AI slop is gaming Google, the AI models themselves (ChatGPT, Claude, Perplexity) are surprisingly resistant to it. We tested ~150 B2B tools and the correlation between Google rank and LLM recommendation was 0.08. Basically zero.

So you end up with two diverging discovery layers: Google, increasingly polluted by cheap AI content, and LLMs, which seem to weight structured data, documentation quality, and mentions on high-trust platforms (Reddit, GitHub, Stack Overflow) way more than traditional SEO signals.

The real question is whether Google fixes this before users just default to asking ChatGPT instead of searching.

The explicit ads angle is only half the story. Even without paid placements, these models already have implicit recommendations baked in.

We ran queries across ChatGPT, Claude, and Perplexity asking for product recommendations in ~30 B2B categories. The overlap between what each model recommends is surprisingly low -- around 40% agreement on the top 5 picks for any given category. And the correlation with Google search rankings? About 0.08.

So we already have a world where which CRM or analytics tool gets recommended depends on which model someone happens to ask, and nobody -- not the models, not the brands, not the users -- has any transparency into why. That's arguably more dangerous than explicit ads, because at least with ads you know you're being sold to.

The JS rendering point is critical. Even though bots like GPTBot technically have headless capabilities, they often fall back to text-only extraction for non-priority pages to save compute. We see a lot of "invisible" content in e-com especially because of this.

One other signal to check: internal linking structure. AI crawlers seem to respect semantic clusters more than traditional pagerank flow. If your "about" page isn't semantically linked to your "product" page in a way the LLM understands as a relationship, it often hallucinates the connection.

One angle thats underappreciated here is discovery. Even if the SoR survives as the backend, the interface layer is shifting to AI agents and assistants.

We've been looking at how LLMs actually recommend software tools (audited ~150 B2B SaaS across ChatGPT, Claude, Perplexity, Gemini) and there's a massive bias toward products with clean, parseable documentation and structured data. The "vibe-coded" replacements often win here not because they're better products, but because they're easier for an agent to read and integrate with than a legacy SoR hiding behind a login wall or a PDF.

If your product is invisible to the agent doing the recommending, the quality of your backend stops mattering. Has anyone else noticed this shift in how enterprise software gets discovered?

Interesting that you have "GEO ready" baked into the boilerplate. Curious what that covers specifically -- are you doing structured data / schema markup for LLM retrieval, or something more like llms.txt / meta tags aimed at AI crawlers?

We've been testing how different content structures affect whether AI assistants actually recommend a product, and the difference between well-structured docs vs not is pretty dramatic.

This is a great dataset. The 'cross-domain causality leap' is something we see constantly in brand monitoring—e.g. an LLM seeing a pricing page for 'Product A' and a feature list for 'Product B' and confidently asserting 'Product A has Feature B for $X'.

One edge case you might want to add: *Temporal Merging*. We often see models take a '2024 Roadmap' and a '2023 Release Note' and halluncinate that the roadmap features were released in 2023. It's valid syntax, valid entities, but impossible chronology.

Are you planning to expand this to RAG-specific failures (where the context retrieval causes the mix-up) or focusing purely on model-internal logic gaps?

Distinguishing 'AI Research' (crawling) from 'AI Referral' (user clicks) is the hardest part. Most agents (OAI-SearchBot, ClaudeBot) declare themselves in UA, but the actual click-through often strips the referrer or shows as direct/none. We've had some luck correlating 'time of crawl' with 'time of visit' to fingerprint AI traffic, but it's noisy.

Self-hosted is definitely the way to go for raw logs though. GA4 obfuscates too much of this.

Very cool to see a local-first approach to agent memory. The shift away from hosted vector DBs for single-agent use cases makes total sense.

Are you using `sqlite-vec` under the hood for the embeddings or a custom extension? Also, curious how you handle the 'hybrid' search part—is it just a linear combination of FTS5 bm25 and vector cosine similarity, or do you have a re-ranking step?

This matches our internal data too. The biggest shift we're seeing is that 'Zero-Click' searches on Google are becoming 'Multi-Turn' conversations on AI.

Users don't just search for a keyword; they describe a problem, get an initial answer, and refine it. Those mid-funnel queries (e.g., 'how does X compare to Y for Z use case') are invisible to traditional keyword tools because the volume per specific query is tiny, but the aggregate intent is huge.

The challenge is measuring this effectively. Are you finding any reliable way to group these long-tail queries into meaningful clusters, or is it still very manual?

The 'Vector DB vs Keyword Search' section caught my eye. In your testing for RAG pipelines, where do you draw the line?

We've found keyword search (BM25) often beats semantic search for specific entity names/IDs, while vectors win on concepts. Do you cover hybrid search patterns/re-ranking in the book? That seems to be where most production systems end up.

Interesting tool. We've been looking at this space (GEO) as well.

One big challenge we've hit is the lack of determinism. Unlike Google where rank #1 is fairly stable, ChatGPT/Gemini answers vary wildly based on user history, location, and random seed.

How do you define a 'rank' in a conversational interface? Is it just frequency of mention over N runs? Or do you try to map 'top' recommendations to positions?

The 'without SEO bias' point is the most interesting part here. We're already seeing users trust LLM synthesis more than direct search precisely because it (theoretically) filters out the affiliate spam.

But aren't we just moving the problem up a layer? Brands are aggressively optimizing for LLM visibility now ('Generative Engine Optimization'). If your framework relies on LLM training data or RAG, it's still downstream of whatever content dominates the web.

Curious if you've seen any 'hallucinated brands' or persistent bias towards certain vendors in your tests across models? e.g. does Gemini favor Google products in your laptop comparison?

Signal aggregation is definitely the right mental model. We've found that tracking 'Share of Model' over time (e.g. how often a brand appears in the top 3 recommendations for a category query) is much more stable than individual prompt outputs, which can vary wildly due to temperature.

It's similar to share-of-voice in traditional PR. You can't control every mention, but you can track the aggregate trend of whether the model 'knows' you exist and considers you relevant.