axiom92

Afaik PaLM (Google's OG big models) tried this trick, but it didn't work for them. I think it's because PaL used descriptive inline comments + meaningful variable names. Compare the following:

```python

# calculate the remaining apples

apples_left = apples_bought - apples_eaten

```

vs.

```python

x = y - z

```

We have ablations in https://arxiv.org/abs/2211.10435 showing that both are indeed useful (see "Crafting prompts for PAL").

Comment · 1 year ago

You can do this at grok.com.

There is a "start thread" option below every conversation. You can also read the responses aloud (helpful if you want to do something async).

Comment · 1 year ago

Yeah, that's the first formal reference I remember as well (although, BERT is probably the first thing NLP folks will think of after reading about diffusion).

I collected a few other text-diffusion early references here about 3 years ago: https://github.com/madaan/minimal-text-diffusion?tab=readme-....

Comment · 1 year ago

From last neurips https://automix-llm.github.io/automix/

Comment · 1 year ago

The demo was done live (as was everything else).

Comment · 2 years ago

Looks pretty cool https://www.youtube.com/watch?v=mogSbMD6EcY

Although, it seems it's only going to cover the first book (which makes sense, given how difficult the other two would be to film). The real magic for me was in book 3. It was inspiring to see someone think so far out, so boldly.

Comment · 2 years ago

Welcome to one of the most hated parts of the academia.

Comment · 3 years ago

The joke is that he doesn't own any OpenAI shares.

[1] https://www.cnbc.com/2023/03/24/openai-ceo-sam-altman-didnt-...

Comment · 3 years ago

Right, but no separate image encoder + half the size could be very helpful for many applications.

Comment · 3 years ago

> tasks that need deterministic outputs and the thing you need to create is already known statically

Wow, interesting. Do you have any example for this?

I've realized that LLMs are fairly good at string processing tasks that a really complex regex might also do, so I can see the point in those.

Comment · 3 years ago

> And basically all servers will have 8xA100

for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).