gaberivera

Huge topic, but briefly, we use a combination of RSS, scrapers, and official APIs to obtain article/post text. As far as clustering, it's a combination of text analysis, link analysis, and editorial input. I don't expect us to open source much of our code unless we find a new source of revenue!

Hi there, I wanted to flag a new feature we added this week: under the headlines on Techmeme we include links to "Forums" like Reddit, Ars OpenForum, Slashdot, etc. when we find threads discussing the news.

Included in the communities we check is Hacker News. In fact, currently the top story cluster links to Hacker News four times, and Hacker News appears a total of seven times on the page. (This will change of course as Techmeme evolves throughout the day.)

So this is a good way to sample the views this community has on the big tech stories of the moment, and often a handy way to see where they differ from other communities.

Ironically, we're using Reddit's search API to find all the subreddits linking to news on Techmeme. I'm pleased Reddit hasn't turned off that particular API!