pcovington

Your intuition is correct - there are other ways to capture the non-stationary nature of this particular problem. We thought that the example age approach is neat because it is a general technique for removing bias inherent to any machine learning system. Since examples always come from the past, you often have to be careful to prevent any system from being overly biased towards historical behavior. You don't need any additional metadata about items (what's the age of a search query?) and it's more resilient to predicting in regions the model has never seen because you fix serving to the very end of the training window.

I tend to think the focus on recent behavior is an artifact of underfitting. Research into richer temporal modeling is needed and recurrent networks seem promising.

We debated internally whether to use the "deep" moniker - Alexnet was 8 layers, so maybe the threshold is 8? The depth seems sort of irrelevant since stacking layers is trivial once the basic architecture is in place.

Thanks! This model handles new users gracefully because it can fallback to demographic/geographic priors and gradually specialize as the user watches videos. New items are difficult because of the fixed output vocabulary and batch training. In practice, this model is best suited for the head of the distribution and other specialized recommenders handle extremely fresh/low viewcount items. Feature engineering is key for new content during the ranking phase.

Figure 3 illustrates that the variable sized watch history is combined with an average operation. This is partially why the embeddings need to be so large - in order to retain information after averaging, you need lots of dimensions to spread out disparate items.

This is of course not optimal, as the network should be able to learn how best to summarize the sequence. In the paper, however, we emphasize the importance of withholding certain sequential information from the classifier.

This is a very natural avenue and an active area of research at Google/Deep Mind. Stay tuned...

word2vec did inspire earlier iterations of the model, but the key insight is that embeddings are learned jointly with all other model parameters. There is no separate source of embeddings. This way, embeddings are specialized for the the specific task.

The video embeddings in the paper are learned purely based on observing what users co-watch in sessions. In this sense, they can be thought of as latent factors in more traditional collaborative filtering approaches. When we inspect them, nearby vectors have a surprising amount of semantic similarity.

Features about the videos such as titles and tags, as well as features derived from audio and video, are introduced in the ranking phase.

Yes, it's a reasonable proxy. It was challenging to set up similar experiments with the old system because it was trained to approximate a different "surrogate" problem. We've also found that recommendation systems are very difficult to evaluate offline.

Author here - happy to answer questions about the techniques in the paper. We're super excited to finally share this work externally. Feedback about YouTube recommendations in general also welcome.