snikolov
Joined 17 years ago
Karma
378
Submissions
147
Member since
Jan 25
Recent submissions
Comment · 12 years ago
a program can still generate the score that a musician plays sincerely
Comment · 14 years ago
People who do hard work tend to bite off more than they can chew, especially at MIT, and there is obviously a limit to how much you can handle. What MIT does right when it comes to stress relief is its housing system. You get to pick where you want to live, and you mostly get your choice (or you can move eventually). That way you can live within a supportive community. It really makes a difference during those hell weeks.
- MIT '11 MEng '12
Comment · 14 years ago
It is online learning, but as far as I understand, there is another side to it. Right now, if you get a new data stream, you have to play around to build the right models for it (mostly manually, with trial and error) before you do online learning. As the number of data streams grows, this doesn't scale well. Their algorithm adapts online to the data stream without you having to design any model.
Comment · 14 years ago
It sort of comes down to the question of what's really being learned here?
That's a great question. We are learning to recognize trends and non-trends based on previous examples. Since the Twitter trends algorithm gives us such examples, you could say we are learning to replicate the outputs of an arbitrary algorithm --- and you'd be right. But learning from examples is a very general thing, so the method has applications beyond detecting trending topics.
Are they modeling some inherent process of topics becoming popular
No, we don't model the process of something becoming popular. (To do this, one might suppose people spread popular topics in X way and unpopular topics in Y way, and try to estimate from the data whether the topic is popular or unpopular.) The beauty of this is that we never have to build a model, because we rely directly on the data. As a corollary, this approach is applicable out of the box for any domain with time-varying data (though I suppose you might have to take care to measure the right kind of time varying data).
Does that answer your question?
Comment · 14 years ago
Thanks! I'm afraid my background, like yours, is more in methods that are not specialized for time series, and so I couldn't credibly give any comprehensive references. My understanding is that a lot of methods designed specifically for time series draw heavily on the theory of stochastic processes. For example http://en.wikipedia.org/wiki/Autoregressive_model, http://en.wikipedia.org/wiki/Autoregressive%E2%80%93moving-a.... I once took a course called Signals, Systems, and Inference that covers some of these ideas (full course notes here http://ocw.mit.edu/courses/electrical-engineering-and-comput...), but that's about as far as I've gotten along that road.
Comment · 14 years ago
You bring up a great point. To classify something as a trend or not a trend, we have to use this artificial black box to supply ourselves with examples of what's a trend and what isn't. The nice thing, IMO (and this is something I admittedly gloss over at the very end) is that doing prediction/forecasting with this method is almost the same as doing classification, even though you don't have any labeled examples when doing prediction. To do classification, we compare current activity to past examples of activity, and decide if it looks like the positive examples or the negative examples. For prediction, we compare current activity to past activity, and see how similar-looking past activity continued to evolve over time.
Comment · 14 years ago
You can get the trending topics through the Twitter API https://dev.twitter.com/docs/api/1/get/trends/%3Awoeid
But I think what you are asking is how such a method would come up with its own trends, given just a stream of tweets. This is a supervised approach (http://en.wikipedia.org/wiki/Supervised_learning), so for now, you would need to train it (possibly online) by giving it examples of what should be a trend and what shouldn't. It would be interesting to make it semi-supervised (http://en.wikipedia.org/wiki/Semi-supervised_learning) so that you would only need to provide a small number of labels.
Comment · 14 years ago
Not in general. I was part of the VI-A program (http://vi-a.mit.edu/) at MIT, which allows you to do your thesis at a company.
Comment · 14 years ago
I agree with you --- this seems like a perfectly good way to tackle the problem of trend prediction directly. What we had in mind was something that would be more generally applicable to any kind of time series data, and we figured it would be interesting to test it on Twitter trends.
Comment · 14 years ago
Thanks for the interest! We collected tweets by sampling a small percentage of all tweets in a time window, to emulate what one might get from the streaming API (I did this as part of the VI-A http://vi-a.mit.edu/ masters program at MIT, and was an employee of Twitter). We did pick a fixed set of topics to track, but those were randomly sampled (though we did get rid of topics that trended multiple times in a large time window, like the name of a football player who scored in multiple matches, in which case we don't know which event we are trying to detect). One thing the algorithm doesn't do at the moment is come up with its own trending topics. It just tests prediction of trending/non-trending on a hold-out set taken from the original set of topics.
Comment · 14 years ago
My problem is often in the other extreme. I tell myself "I don't want to bother so and so, I still haven't tried x." But there's a point at which you should ask for help if you're no longer making progress. Especially if your question is expected to have a relatively quick answer ("take a look at y and do z") that you just don't happen to know yet.
Comment · 14 years ago
Dynamic Time Warping is a pretty neat application of dynamic programming that takes two time series and tries to squish one time series to look like the other, with some constraints that you can't do too much squishing. This is used a lot in speech recognition to normalize away variation like e.g. how long you spend on a vowel sound.
Comment · 14 years ago
Indeed, projection onto the principal subspace is a kind of regularization, which makes for better generalization.
This kind of makes sense intuitively. These slides go into more detail (the whole course is great), where it says "Projection regularizes!"
http://www.mit.edu/~9.520/spring09/Classes/class07_spectral....
my HN username at mit.edu