robhoeijmakers

Same ratio roughly. 80% Crawlers and agents, 20% human. Loads of the agents actually serve the content to humans, mostly in ChatGPT.

It is good to make a proper distinction, in the ChatGPT context, between crawlers and agents. The crawlers go for the content to build a new model, the agents serve content to users. The last one can be very useful.

I am on a low tier Ghost subscription, I could not rewrite some of the HTML. So I do this with Cloudflare and then cache it again.

Yes, the architecture setup is generated by ChatGPT but in itself it says what it needs.

I started blocking some of them. But for now I want to improve visibility before further blocking or optimising. The dashboard helps with this.

Thanks. It seems to be very local/incidental. The page works from the locations I can test, but I’ll check whether one edge cache or request path served a bad response.

Built a Cloudflare Worker that classifies all traffic by visitor type: humans, AI crawlers, SEO bots, residential proxies. On most days, humans are a minority. The article explains what the patterns reveal and why edge logs show things your analytics tool can't.

A short story on the decades long search for the blue LED, the missing piece in the production of screens and energy efficient white light.

Social media platform Twitter reverted to cropping images in the timeline. Not for the web and desktop app, but they do for iPhone.