mattj

I've built one of these based on the instructions and use it daily (and love it!), but the tone of the site is pretty reflective of the project overall. It's definitely a really impressive hack and I appreciate all the hard work that's gone into it, but could really use a little more user-facing empathy.

I'm not sure if I'd recommend it to someone else - and if I was doing it again I'd probably spend a few hundred more (than the gaggia + parts cost) and just buy an off-the-shelf machine with the same feature set.

I think the grandparent is probably referring to upstate in the Hudson Valley sense of the phrase. Plenty of cute towns, but Hudson / Woodstock / Kingston are definitely in the O(tens) of great restaurant options, comparable to a slice of any single Manhattan / Brooklyn neighborhood most tech people live in.

The other answers cover the math well, but I think the “why do you need attention?” statement is worth making (and answers the more engineering-y question of “how/when?”):

DNNs typically operate on fixed-size tensors (often with a variable batch size, which you can safely ignore). In order to incorporate a non-fixed size tensor, you need some way of converting it into a fixed size. For example, processing a sentence of variable length into a single prediction value. You have many choices for methods of combining the tensors from each token in the sentence - max, min, mean, median, sum, etc etc. Attention is a weighted mean, where the weights are computed based on a query, key, and value. The query might represent something you know about the sentence or the context (“this is a sentence from a toaster review”), the key represents something you know about each token (“this is the word embedding tensor”), and the value is the tensor you want to use for the weighted mean.

This is great - the like / heart button is really slick, and I love how it doesn't get in the way at all. I've used pinboard and others in the past, and the (relatively) heavier bookmarking flow would often stop me from saving things as I didn't want to break my flow.

Excited to see where this ends up!

I agree with you (and love your blog, btw), but I think you're skipping over at least a few benefits you can get out of a mature / well built a/b framework that are hard to build into a bandit approach. The biggest one I've found personally useful is days-in analysis; for example, quantifying the impact of a signup-time experiment on one-week retention. This doesn't really apply to learning ranking functions or other transactional (short-feedback loop) optimization.

That being said, building a "proper" a/b harness is really hard and will be a constant source of bugs / FUD around decision-making (don't believe me? try running an a/a experiment and see how many false positives you get). I've personally built a dead-simple bandit system when starting greenfield and would recommend the same to anyone else.

I think the change here is they're learning the embeddings alongside the feature weights (eg they're part of the same loss function).

This kind of language is very standard. The risks section pretty much always contains obvious platitudes ("An earthquake might destroy all our computers," "All our employees may quit").

Similar experience, but I focused on finding UI elements I liked from native apps or websites and attempted to clone them without looking at the source, then played around with the result to figure out how I could simplify it, how it behaved cross browser, etc etc.

heads up, your math is a little buggy - 1% of $90b is $900m, not $90m

What does this do? I've browsed through the site for a few minutes, and still have no idea what kind of tools you could build with this that you couldn't build before.

Is this for cross-language doc generation? Refactoring tools? Something else?

Are there any concrete examples of a tool built on top of this that would otherwise be impossible / very difficult?

These are both good points. I think it's worth calling out here that this post is really about the infrastructure to perform experiments, not necessarily the means of analysis (although the ui you see in the screenshots performs that analysis).

In terms of these issues, we handle them in a few ways. In particular, experiment review catches many of these issues. Think of it like code review for your experiments.

In order to run an experiment, we require you to have an "experiment helper" sign off on your change. This involves reviewing your group sizes, verifying that you have the statistical power you need to test the magnitude of change your hypothesis expects, verifying you interact with the framework correctly etc.. Training to become an experiment helper is generally not very easy, and involves a combination of shadowing existing reviewers, performing enough reviews across the stack, and taking a test to verify you understand potential errors (the test itself being composed of many experiments where we have made mistakes).

Changes to experiments (increasing group sizes, terminating an experiment, modifying an experiment etc.) all require this review.

Yup! My experience with redshift has actually made me curious to try out Postgres (I've always used MySQL before this). The stricter SQL dialect was a little odd at first, but I think I've become more comfortable with it over a few months.

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it.

One huge advantage of redshift over hive: you can connect with plain old Postgres libraries, so you can build redshift results into your admin interfaces, one off scripts, and anywhere else you're fine trading a few seconds of latency for extra data.