nil-sec

Sorry maybe I should have added more explanation. One way to think about attention, which is the main distinguishing element in a transformer, is as an adaptable matrix. A feedforward layer is a matrix with static entries that do not change at inference time (only during training). The attention mechanism offers a way to have adaptable weight matrices at inference time (this is implemented by using three different matrices, K,Q & V called keys query and value in case you want to dig deeper).

I guess I was more thinking about self attention, so yes. The more general case is covered by your notation!

Feedforward: y=Wx

Attention: y=W(x)x

W is Matrix, x & y Are vectors. In the second case, W is a function of the input.

I had the same experience a couple of years back when people started discussing AI. I try to keep this in mind but somehow I keep forgetting. It’s genuinely difficult to filter good from bad takes without expert level understanding of a topic unfortunately.

It’s quite nice to have something irreversible though. It gives you time. Also nobody really thinks QM is the end of it, assuming semi classical physics under the hood is just odd to me. There is something below QM that we don’t understand yet (AdS/CFT looks like a good start to me) and personally I think the whole interpretation of QM debate will look stupid in retrospect. Yeah collapse is odd, but it just shows us this isn’t it. Reality is much weirder than we thought and giving up on realism is just the beginning.

It’s funny because, for me, this was one of the major confusions when I moved from Europe to the US. In Europe, there are a lot of prominent subcultures, particularly in college. This was totally absent in the US in my experience. Even at places where you would expect it, e.g. underground, hard Techno warehouse raves in Baltimore. Instead the same mainstream people & opinions were there too.

Insane take, the west vs Russia in open confrontation ends in a nuclear winter.

I have no background in control theory but this sounds very similar to the identifiability problem in nonlinear ICA. Are those equivalent?

This is a good point which I think stems from wrongly equating human level intelligence to AGI in the popular literature. It’s not at all clear what a general intelligence should be and it’s much less clear that humans have general intelligence. In my view we have a set of very good innate priors (e.g. space/time continuity, intuitive physics) that have been optimized by evolution for thousands of years. These priors in turn allow us to learn fast from unlabeled data, but would anyone call such a system general intelligence? I’m not sure.

While I agree with the general point of this paper I don’t think it’s quite right to compare the current situation with the last AI spring. It’s not AGI but it’s very good narrow AI that has real commercial value right now. The systems back then did not to the same extent, and for this reason I don’t see funding dry up for current ML approaches.

For a given causal model it is (1) in my understanding.

For one it lets you avoid controlling for the wrong variables and causing e.g. spurious correlations by doing so. In fact this is one of the best examples of why a causal model is necessary, because without one you can easily end up with a correlation that doesn’t exist as is illustrated quite nicely in his book.

It is incomprehensible to you, because you just simply do not understand what your parent is talking about. You are the ignorant one here and indeed quite rude. Doesn't matter that genetics is not natural language. The point is we can train large transformers auto regressively and the representation it learns turns out to be useful for a) all kinds of supervised downstream tasks with minimal fine-tuning and b) interpreting the data by analysing the attention weights. There is a huge amount of literature on this topic and what your parent says is quite sensible.

Agreed, I trained a 3D version of b0-b2 on a classification task I worked on and besides being very slow to train they did not outperform a simple baseline VGG architecture. Interestingly training time was much improved by setting the cudnn benchmark flags in pytorch. Haven’t seen this reduction in training time for any other architecture but for b2 it went from about 12 sec per iteration to about 0.8. I guess there is more margin optimizing NAS models.