dangerlego5

I kept rebuilding the same arXiv scraper at the start of every ML project. After the third time I wrote a dedup pipeline, I automated the whole thing.

The interesting part is that the pipeline is shared; if two people subscribe to the same topic, they share one crawl and one deduplicated record pool. Happy to talk through the pgvector dedup approach if anyone's curious.

The visual regression point is interesting. In my experience, the models that do best at "overlapping text/bad layout" catches are the ones being fed actual screenshots rather than DOM snapshots. If Fable is doing screenshot-based diffs natively, that would explain an improvement there, but I haven't verified it.