About a decade ago, I took on a personal toy project to try and teach myself larger scale programming in java, as well as the APIs provided by multiple internet services (google, twitter, facebook, ....)
The project was to try and collect and make searchable my "internet self". something I called "personal search". i.e. the idea was to try to index every web page I look at, every e-mail I get / see. Social media content shared to me by my social graph (using said APIs), and further indexing pages shared.
The indexing itself wasn't the hard part (per se, as at the time, the APIs facebook, twitter et al were very expansive, much more limited these days, one can attempt to deal with it with intelligent dom scraping, but that's a never ending race where your sources are consistently changing things and you are chasing their changes), the question is how does this information really create significant value for myself? i.e. how often am I going to actually be searching this personal archive. I search google many times a day to find new things (or refind things I already found through it), but how often do I search for things that are within my personal index during a normal day? a handful at most I'd think (and many times, not even once).
With that said, the concept that I then decided to try and teach myself was trying to write a browser extension that could do a similarity search (of sorts) between the documents in my personal index and the web page that I'm currently looking at + content related to that (ex: looking at a news article about current events, idea is that it should surface other articles your friends have shared on the topic and their comments on it). That ended up being an area i didn't have the time (or expertise) to really go far with, so it sort of ended there.
The project was to try and collect and make searchable my "internet self". something I called "personal search". i.e. the idea was to try to index every web page I look at, every e-mail I get / see. Social media content shared to me by my social graph (using said APIs), and further indexing pages shared.
The indexing itself wasn't the hard part (per se, as at the time, the APIs facebook, twitter et al were very expansive, much more limited these days, one can attempt to deal with it with intelligent dom scraping, but that's a never ending race where your sources are consistently changing things and you are chasing their changes), the question is how does this information really create significant value for myself? i.e. how often am I going to actually be searching this personal archive. I search google many times a day to find new things (or refind things I already found through it), but how often do I search for things that are within my personal index during a normal day? a handful at most I'd think (and many times, not even once).
With that said, the concept that I then decided to try and teach myself was trying to write a browser extension that could do a similarity search (of sorts) between the documents in my personal index and the web page that I'm currently looking at + content related to that (ex: looking at a news article about current events, idea is that it should surface other articles your friends have shared on the topic and their comments on it). That ended up being an area i didn't have the time (or expertise) to really go far with, so it sort of ended there.