JackRumford

Not all other nations - just Palestine. Because Israel is in position of power.

yes but the size of the portfolio is the issue here

I solve exponentially harder problems.

When I’m im at the peak, I will start pursuing another field that is foreign to me and, again, grind to the top.

A cycle like this takes me about two years depending on the sub topic.

I have no interest in being the top 100 protein language model scientist in the world and stay there forever. I want to be a polymath.

Concerning /s

Almost everyone I know in the dev community agree that both generative art and code is fine. Most people don't care about copyright, to say the least. At least the people I know in real life and chat with online.

Only thing I've seen was artists on Reddit and Twitter getting angry about the AI art.

Abstract: The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that in order to develop a holistic understanding of these systems we need to consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. This approach - which we call the teleological approach - leads us to identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. We predict that LLMs will achieve higher accuracy when these probabilities are high than when they are low - even in deterministic settings where probability should not matter. To test our predictions, we evaluate two LLMs (GPT-3.5 and GPT-4) on eleven tasks, and we find robust evidence that LLMs are influenced by probability in the ways that we have hypothesized. In many cases, the experiments reveal surprising failure modes. For instance, GPT-4's accuracy at decoding a simple cipher is 51% when the output is a high-probability word sequence but only 13% when it is low-probability. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system - one that has been shaped by its own particular set of pressures.

Criticisms of Elon Musk are not solely based on ideological differences, but also on concerns regarding his behavior, management style, and the working conditions at his companies, which some believe warrant scrutiny irrespective of his personal beliefs. He just doesn't feel like a good person and isn't relatable. This is why I think people hate him.

"With FLASHATTENTION (Dao et al., 2022), there is negligible GPU memory overhead as we increase the sequence length and we observe around 17% speed loss when increasing the sequence length from 4,096 to 16,384 for the 70B model."

"For the 7B/13B models, we use learning rate 2e−5 and a cosine learning rate schedule with 2000 warm-up steps. For the larger 34B/70B models, we find it important to set a smaller learning rate (1e−5) to get monotonically decreasing validation losses."

"In the training curriculum ablation study, models trained with a fixed context window of 32k from scratch required 3.783 × 10^22 FLOPs and achieved performance metrics like 18.5 F1 on NarrativeQA, 28.6 F1 on Qasper, and 37.9 EM on Quality."

"Continual pretraining from short context models can easily save around 40% FLOPs while imposing almost no loss on performance."

"Through early experiments at the 7B scale, we identified a key limitation of LLAMA 2’s positional encoding (PE) that prevents the attention module from aggregating information of distant tokens. We adopt a minimal yet necessary modification on the RoPE positional encoding (Su et al., 2022) for long-context modeling – decreasing the rotation angle."

Pretty exciting stuff. Getting close to GPT-4 hopefully soon!