ethn
Joined 10 years ago
Joined 10 years ago
However, in this article I contend that those limitations have posed little adversity in the field given the success of the latest models. As a result, it may be a bit premature to be concerned about it.
I wanted to avoid my experience but I worked at FAANG and helped create a multi-billion dollar corporation (from a handful of people to 1,500 people).
The self-employed is now able to rapidly develop their ideas while employed reducing their risk and cost.
Per VC, that's right and secondarily it's more difficult to justify earlier funding when one no longer needs to hire a substantial team. I expect the current major VC firms to get even bigger.
In this world the neurodivergent is empowered, they no longer are charged with persuasion of their ideas to a team or corporation. They can build their ideas themselves and form a limited partnership with someone with the talents they lack.
Though I make it a point to describe the new economic order without regard to its desirability. However, other authors on the subject (e.g. Lysander Spooner, Rothbard, etc) would be pleased by the development in terms of its social welfare.
> This is irrelevant for AI, because people throw more hardware at bigger problems
GAI is a fixed problem which is Solomonoff Induction. Further Amdahl's law is a limitation on neither software nor a super computer.
Both inference and training rely on parallelization, LLM inference has multiple serialization points per layer. Vegh et al 2019 quantifies how Amdahl's law limits success in neural networks[1]. He further states:
"A general misconception (introduced by successors of Amdahl) is to assume that Amdahl’s law is valid for software only". It would apply to a neural network as it does equally to the problem of self-driving cars.
> These two sentences contradict each other
There is no contradiction only a misunderstanding of what "eviscerates" means and even with that incorrect definition resulting in your threshold test, it still remains applicable.
1. https://pmc.ncbi.nlm.nih.gov/articles/PMC6458202/
Further reading on Amdahl's law w.r.t LLM:
2. https://medium.com/@TitanML/harmonizing-multi-gpus-efficient...
3. https://pages.cs.wisc.edu/~sinclair/papers/spati-iiswc23-tot...
Interested in: - epistemology - social/financial structures. - startups