octbash

>I found this part to be the most interesting. If this current proclamation is a prelude to the overhaul of H1B system in a way that would make it work like described above, then it is somewhat exciting for a couple of reasons.

I think most people who've dealt with the immigration system would think that that is naive. This administration has proven time and time again with regards to immigration that they will pay lip service to making improvements while almost always simply making life harder for immigrants and people on visas.

See: how they suspended H-1B premium processing for a while some time back, also claiming that it was in service of "overhauling" the H-1B system.

I seem to find myself in the minority, but I don't think distill.pub is a particularly ideal model for publicizing research.

distill.pub heavily favors fancy and interactive visualization over actually meaningful research content. This is not to say that the research publicized on distill.pub is not meaningful, but that it is biased to research that can have fancy visualizations. So you end up seeing a lot of tweakable plots, image-augmentations, and attention weights visualizations. It is also further biased towards research groups that have the resources to create a range of D3 plots with sliders, carved out of actual research time.

For instance, I don't think BERT could ever make it into a distill.pub post. Despite completely upending the NLP field over the last 2 years, it has no fancy plots, multi-headed self-attention is too messy to visualize, and its setup is dead simple. You could maybe have one gif explaining how masked language modeling works. The best presentation of the significance of BERT is "here is a table of results showing BERT handily beating every other hand-tweaked implementation for every non-generation NLP task we could find with a dead-simple fine-tuning regime, and all it had was masked language modeling."

To give another exmaple: I think it's one of the reasons why a lot of junior researchers spend time trying to extract structure from attention and self-attention mechanisms. As someone who's spent some time looking into this topic, you'll find a ton of one-off analysis papers, and next to no insights that actually inform the field (other than super-trivial observations like "tokens tend to attend to themselves and adjacent tokens).

How is that draconian? If you're issued a stay-home notice, STAY AT HOME.

Excuse me - what "draconian quarantine laws"?

One is a language generation model, the other is a fill-in-the-blank model. It sounds like they might be similar, but in practice they are different enough objectives (and in particular the "bi-directional" aspect of BERT-type models) that the models learn different things.

Those are question-answering and language-understanding benchmarks respectively, neither of which has been suitable for language generation mode evaluation since GPT-1 was roundly beating by BERT. GPT-2 didn't evaluate on them either.

Is that essentially repeating the position embedding? I'm surprised that works, since the model should have no way to distinguish between the (e.g.) 1st and 513th token. (If I'm understanding this correctly.)

How are you using GPT-2 with an expanded context window? I was under the impression that the maximum context window was fixed.

My counter-arguments (as a huge PyTorch fan) are:

1. GPT hasn't really been about model/architectural experimentation, just scale. GPT-2 and GPT were architecturally very similar. Scale, especially at the scale of GPT-*, is one avenue that TensorFlow does have an edge over PyTorch 2. Work on GPT-3 probably started quite a while ago.

Minimum=/=lowest effort.

Especially with the caveat of "with a programming background", it is far easier to reason and debug through PyTorch with just Python knowledge, compared to TensorFlow/Keras, which sooner or later requires you to learn a condensed history of TensorFlow/Keras development to understand why things are the way they are.

In my opinion,

  import lib
  lib.train("imagenet", "resnet50", epochs=10)
  lib.eval()

is NOT a good example of a beginner friendly library. It's a thin wrapper facade that hides all of the actual complexity behind "Train ImageNet in 3 lines of code!"

Besides vocabulary/characters, in what sense is Mandarin not simple?

Yes, the Reformer is basically trading off noisier for faster training / memory savings.

You keep insisting that they're the same when they're not, and then you try to subtly expand your original claim of "using less memory by hashing" to "to be more memory and compute efficient" (emphasis mine), just to force them into the same bucket.

Yes, obviously locality sensitive hashing is a form of hashing. The fact that it's locality sensitive is important for this application, but you'd rather ignore that and insist on labeling them as the same thing just because they're both hashing.

This sounds like some sort of perverse inverse of "This is good for bitcoin."

Making progress of NLP benchmarks? Must be a sign that we're moving even closer to an even longer and more bitter AI winter.