Comment by wild_egg

Last time I did anything with a BERT, you had to train or fine-tune. Is that not still true?

For me the cool bit is that it's all in-context learning or whatever so you can use it in any domain with zero setup.

Maybe bert and co. could do all the same things before, but the way in which you use them is quite different and that helps a lot.