Comment by mrborgen

That was one of the challenges when building this. We ended up using Firecrawl. But even so there are a few pages we aren't able to read. In which case we use the Algolia API to get the HN comments and generate the video about the HN reaction on the article instead.