addedlovely
Joined 14 years ago
Joined 14 years ago
Will be publishing a report on www.theweb.report soon - if you're interested.
That's a seed of about 13 million domains - pretty sure WordPress would be dominating the even longer tail.
( Also worth noting it's sooo easy to detect a site is WordPress, it screams it across every signal we gather, where-as some sites are just well made and have limited information leaks ).
Custom python crawler getting 240 sites a second crawled and classified. ( homepages and minimal probes, no headless browsers )
Be interested to test some false positives, if you have a URL I'll tell you what I see :)
I'm seeing about 6.6% block rate, but that does climb over time.
I've been doing the same bit wider scope, for the whole Crux list, pruned to apex domains, and looking for CMS signals - how's your throughput?
I'm not doing any headless browser stuff, or many requests, so hyper optimised for speed.
I do grab robots.txt - didn't really see much in llms.txt or humans.txt in the wild, does yours?
have found Shopify's AI implementation to be sane and really useful ( building flows and surfacing documentation correctly ).
I’m doing some really powerful stuff with the custom CLI commands, highly recommend the YouTube video on it. Makes bulk import / api enrichment and other things a breeze.
It’s licensed, and I’m happy to pay for a professional sustainable solution.
Replacing my iPhone was a nothing burger of choice, on paper the iPhone 15 pro was the best feature set for value vs buying a new iPhone 17, but Apple know that so don't sell the older models directly when the new models come out.
There's really limited impactful innovation when you get into the details.