Comment

Comment by kusmi

I used to, I now use zotero to save whole pages onto webdav, from there bunch of scripts peel the ads off, scrape the text, convert to PDF, store in cms and index for full text search on solr. Also hooked up Dropbox to do the same for one click archiving from mobile. Since Dropbox and the webdav are shared between my partners and I, it's a convenient way to build knowledge base. Experimenting hooking up Telegram and slack as well to integrate everything for no hassle user-end. The real pain in the ass is passing the URL itself, consistently, without insisting users use another third party app.

*Forgot to mention the best part: Backend pools these full-text documents, cleans and parses for NLP, then generates meaningful tags, and organizes documents in an auto generated folder hierarchy which is based on word2vec/doc2vec and content clusters. Whole thing runs on a dedicated server with two 1070 GTX video cards for the NLP work which is training and re-evaluating constantly as new content pours in.

Altogether it was 2-3 years of work.

parent kusmi 9 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton