peaslock
Joined 4 years ago
Karma
37
Submissions
31
Member since
Jun 21
Recent submissions
Comment · 3 years ago
Maybe not a good idea to link a page that runs source code created by random people. Well CSS is very safe, but still.
Comment · 3 years ago
Not if you ask first.
Comment · 3 years ago
Neural nets often fail with (repetitive) gibberish output when the input is too different from the training data. This model appears to take in the entire text input at once or look ahead at the next input letters, so the unusual "bla bla" at the end can mess up outputs near the beginning.
Comment · 3 years ago
Can DRIZZLE help to achieve higher resolution? Though with hundreds of photos this will imply a lot of work:
Comment · 3 years ago
Geoffrey Hinton has recently been talking about how analog and "imperfect" computing with specialized hardware/circuitry may yield much cheaper neural nets, that could easily be as large as human brains, but would only cost a few dollars and would be extremely cheap to run. Not a new idea, but it is a fairly promising outlook, I think.
https://www.zdnet.com/article/we-will-see-a-completely-new-t...
Comment · 3 years ago
But they have highly likely internal prototypes with higher bandwidth and latency. Also, with distilled latent diffusion one can probably generate text(-images) much faster anyhow as it could produce long chunks of text at once, rather than needing recurrently feed back the new token to the inputs.
Comment · 3 years ago
Yes, but the losses in Figure 3 increase because the larger models see fewer data to keep the FLOP budget constant, not because of overfitting. Large models do not overfit very much, so the loss of a larger model will still be better compared to a smaller model when you keep dataset size constant.
Comment · 3 years ago
Though isn't it highly likely that core devs working at the big tech giants have access to 10x-100x faster compute, e.g. some secret TPU successor at Google?
Comment · 3 years ago
> if you want improved performance, you still need more data
Not true. See figure 2: https://arxiv.org/pdf/2203.15556.pdf#page=5
The loss decreases with greater model size at the same compute budget (i.e. stopping sooner regarding training data). Also some rehearsal/multi-epoch training improves the forgetting rate (thereby improving performance substantially), which hasn't been taken into account by Chinchilla et al. because they train <1 epoch.
Comment · 3 years ago
Not necessarily: https://arxiv.org/abs/2206.14486
Also, even with "Chinchilla laws", you still gain performance in a larger model, you just need a lot more data (if just as noisy) to reach the same level of convergence, but a larger model will have already partially converged to a superior model with the same amount data.
Comment · 3 years ago
Yeah, continuous online learning by fine-tuning seems like an obvious way of making these models recall information from outside the perceptible context. One could also prompt the model to (recursively) summarize code and prepend this summary to each prompt, and/or enable the model to interactively query function definitions or code summaries before outputting a final answer (trained by RLHF). But any such tricks might also quickly be outcompeted by an even more general model, e.g. one that directly controls the GUI and can communicate with coworkers...
Comment · 3 years ago
The model with the most similar name in this list is code-cushman-001 which is described as "Codex model that is a stronger, multilingual version of the Codex (12B) model in the paper".
https://crfm-models.stanford.edu/static/help.html
The next stronger Codex model is called code-davinci-001 which appears to be a fine-tuned version of the GPT-3 Davinci model which is known to have 175B parameters. The model naming is alphabetical in the order of the model size:
https://blog.eleuther.ai/gpt3-model-sizes/
See also A.2 here: https://arxiv.org/pdf/2204.00498.pdf#page=6
Comment · 3 years ago
Amazing if this is only a 12B model. If this already increases coding productivity by up to 50% (depending on kind of work), imagine what a 1T model will be capable of! I do wonder if some programmers at FAANG are already having access to a way more powerful coding assistants, and whether they code much at all at this point, or only make high level code specifications and then fix up the automatically generated code.
Comment · 3 years ago
Can you use a different Wi-Fi network at the same time for internet access or does Wi-Fi Direct block any other Wi-Fi access?