sacred_numbers

The whole paradigm changes, though, when you can do daily cancer scans. You don't get a biopsy when the scan shows a lump. You get a biopsy after a couple weeks of daily scans showing the lump growing. Plus, having all the data from the daily scans improves your testing accuracy so false positives and negatives are more rare.

GPT-4 is not the same product. I know it seems like it due to the way they position 3.5 and 4 on the same page, but they are really quite separate things. When I signed up for ChatGPT plus I didn't even bother using 3.5 because I knew it would be inferior. I still have only used it a handful of times. GPT-4 is just so much farther ahead that using 3.5 is just a waste of time.

Based on my research, GPT-3.5 is likely significantly smaller than 70B parameters, so it would make sense that it's cheaper to run. My guess is that OpenAI significantly overtrained GPT-3.5 to get as small a model as possible to optimize for inference. Also, Nvidia chips are way more efficient at inference than M1 Max. OpenAI also has the advantage of batching API calls which leads to better hardware utilization. I don't have definitive proof that they're not dumping, but economies of scale and optimization seem like better explanations to me.

The reason cement is a major contributor to CO2 emissions is because of how much cement we produce. I don't know the lifetime or effectiveness of this catalyst, but typically you only need a tiny amount of catalyst to start a reaction and the catalyst material can be used over and over for a long time.

Yes, with 4 bit quantization.

We could not do that inadvertently. To block even 1% of light using Starlink sized satellites (~30 m^2 with solar panels deployed) would require tens of billions of satellites. We could do it on purpose with huge rotating solar reflectors, and it should honestly be considered as a real option, but we couldn't and wouldn't do so just by launching communication satellites.

The quality difference is substantial. I don't care if it's wasteful to use something that has many uses for a supposedly narrow task (although I don't see translation as a particularly narrow task anymore than I see writing as a narrow task). I would gladly waste untold trillions of floating point operations for a 1% increase in translation quality. From my experiments, though, it's much higher than 1% increase in translation quality. And regardless of how wasteful the compute is, it's actually cheaper in terms of dollars. Using GPT-3.5 to translate Korean to English would cost about $11 per million words, based on the average characters per token of the small sample of text I gave it. DeepL (the best translation service I could find) costs $25 per million characters, or for my sample text, about $64 per million words. At $11 per million words I can have GPT-3.5 perform multiple translation passes and use it's own judgment to pick the best translation and STILL save money compared to DeepL.

It's worse on English and a lot of other common languages (see Appendix C of the paper). It does better on less common languages like Latvian or Tajik, though.

I would bet money against that. Replicating GPT-4 pre-training with current hardware would cost about 40-50m in compute. Compute will continue to decrease in cost and algorithmic improvements may allow for more efficient training, but probably not 3 orders of magnitude in a few years. I think there will be plenty of open source models that will claim GPT-4 quality, and some of them will be close, but they will be models that used millions of dollars (probably from some corporate benefactor but possibly from crowdsourcing) in compute to train. You will probably be able to fine-tune and run inference on fairly cheap hardware, but you can't cheat scale. It's going to take a major innovation to move away from the expensive base model paradigm.

It's unlikely that OSS LLMs will ever be able to compete with corporate LLMs. I can only think of a few scenarios where this could work:

1. Someone develops a procedure for training models with distributed computing resources, including consumer CPUs and GPUs. Even this is not really guaranteed to work, since corporations will probably just buy up all the consumer GPUs, since they are cheaper on a FLOPS/$ metric.

2. One or more governments provide a lot of funding for OSS models, including being willing to pay competitive salaries for the best talent (potentially millions of dollars per year). This is unlikely for a lot of reasons. The only thing that could speed up the process enough to compete with private organizations is a major war that required AI to win. In that case, though, open source would be the least of their concern.

3. Scaling laws stop working and Moore's law catches up. In that case adding more compute won't really help and eventually even organizations with small budgets can afford to train a SOTA LLM. We can't know until we find the limit, but we haven't hit the limit of scaling laws so far, despite scaling up massively over the last few years, so I doubt we will find the limit any time soon.

4. A bunch of corporations that have no hope of reaching first place decide to combine their resources to beat OpenAI and thus prevent a monopoly. I'm not sure if there is precedent, but even if there is, that would still require a lot of coordination and resources for little direct monetary gain.

We'll see what happens, but I'm not really confident about any of these possibilities.

I can think of a few ways: 1. The ChatGPT web search plugin becomes standard protocol for every prompt. If you ask a factual question ChatGPT will first look up an answer with a search engine, then use the results to craft an answer. This could also be implemented using a knowledge database created by scraping the web that is occasionally updated rather than actually performing a web search every time. This also allows OpenAI to shift the blame to whatever source it found rather than ChatGPT if it does find libelous content. 2. OpenAI will just add a disclaimer to every response (or at least every response that appears to be asking a factual question) saying that asking factual questions is unreliable. 3. If there is a blacklist of topics that require a disclaimer or cause the LLM to refuse to answer, it can generate an answer, check it against the blacklist using embeddings/semantic search and either re-generate an answer or generate a refusal before showing the answer to the user.

My best guess is that it will be a combination of 1 and 2. I have always maintained that LLMs are very unlikely to develop into AGI on their own, but are very likely to be a critical piece of an AGI. The most recent research into scaling laws (Chinchilla, Llama) finds significant improvements from scaling data size much farther than parameter size, so memorizing facts within the parameters will become less and less feasible. This is actually ideal, though, since you want your model parameters to encode language and reasoning patterns, not memorize facts. If it's memorizing facts you either need more data or better (i.e. deduplicated) data. I'm not an expert, though, and I'm too lazy for a research review, so please don't sue me for libel if my facts are out of date.

If you bought an 8xA100 machine for $140k you would have to run it continuously for over 10,000 hours (about 14 months) to train the 7B model. By that time the value of the A100s you bought would have depreciated substantially; especially because cloud companies will be renting/selling A100s at a discount as they bring H100s online. It might still be worth it, but it's not a home run.