acdc4life

> evolution selected for a language skill

Yes it did, and we do know a lot about general principles behind, from different disciplines. This is still an early science. NLP research still hasn’t considered many important aspects of language learning that we have discovered in such a short period of time.

> starting with an empirical result

What makes you think these benchmarks are empirical? They were hand constructed to fit some objective, assuming that being good at the said objective is required for NLP tasks. Where’s the empirical experiments to validate the notion that said objectives lead to language? Science hasn’t worked this way, datasets aren’t constructed, you do an experiment and MEASURE it. Then you make models, try to explain the phenomenon, and test new ideas and validate your models. Can your model extrapolate new information and suggest new experiments to validate? I used the word extrapolate over predict intentionally.

Not Google search, but many companies are exploiting cheap labor over seas for different industrial use cases because our algos suck. Not just in manufacturing, there are tech companies outsourcing labeled data for tasks like object detection (Hive.ai as anvexample). Whether it makes you depressed or not, humans are cheaper than algorithms, and I don’t see that changing unless we abandon the current paradigm for ai/ml.

I think this is false. My general opinion of computer scientists and engineers (especially in silicon valley) lack the scientific training that one gets in other disciplines like in physics or other hard sciences. Psychology, linguistics, cognitive science and neuroscience has generated a rich diverse experimental data in the past 50 years, and now is the prime time for a theory to emerge to connect everything. To make what I’m saying clearer, you needed Newtonian mechanics, Maxwell and Plank and Coulomb to develop and discover the science of electricity and atoms in order for us to engineer the silicon transistor. Without the original scientific knowledge, building such devices would have been impossible. This machine learning, ai, deep learning and this obsession with benchmarks are, in my opinion, a hinderance to AGI

> Comparing your pre-trained brain

You’re making sweeping assertions that require domain experts in several different disciplines. The scientific evidence points to the contrary. It is demonstrated that humans have a general ability to learn wide range of things, without being genetically programmed to. None of us evolved to drive cars. Yet nearly everyone in my grandparents generation were able to learn this totally new skill despite being a new invention, where you couldn’t possibly have had time for evolution to act. They weren’t genetically evolved to drive, it was learned within their lifetime and generation. There are tribal humans in different parts of the world that haven’t developed written language. Yet you can teach them written language. Where’s your “pre trained brain” theory there?

Well, that’s not entirely true. Alpha go self played 29 million games of Go, which is in feasible for humans. Assuming a game of Go is 5mins, it would take a human 286 years to achieve the same, assuming this person doesn’t sleep, eat or do anything else other than just play Go. CPU time is an order of magnitude faster than real time, especially on GPU clusters.

I see BERT as non sensical. You need to be scientific and have a mathematical theory on how humans learn language, which is a multi disciplinary task requiring physicists, mathematicians, neuroscientists, cognitive psychologists and linguists. Benchmarks are useless, theories, models, experiments and testable predictions is how science progresses. You’re making a comment on cognitive science, and trying to imply that language learning in humans isn’t learned, but pre baked. The psychological, linguistic, evolutionary biology and neuroscience evidence doesn’t seem to corroborate. The evidence points stronger to humans having general learning and problem solving abilities. For instance, there was no evolutionary pressure for humans to be good at math or programming. I was not born knowing english or calculus or probability theory, these were learned abilities. Evolution favoured brain mechanisms that lead to behaviour for success in a rapidly changing world. Had I been born in ancient Rome as a farmer, I would learn to speak Latin, and learn how to be a successful farmer, instead of the physics, math, probability, computer, driving, reading skills that I learned in my life time.

My brain does nlp better than any system out there. I’m also able to ride bicycles and do motor control better than Boston Dynamics. I can also construct and prove math, do physics and code all of this in Matlab and C. My brain can handle all these wide range of tasks almost seamlessly with just 15 watts of power, that Silicon Valley’s super computers can barely do 1% of.

Slippery slope, with all evidence pointing to the contrary. Apple will never have a monopoly, linux will always be an alternative.

> There should be a name for this kind of ridiculous hubris

Pseudoscience? To me this paper is no different from astrology.

The wiki post mentioned that people are right 45% of the time. How does dl and ml stack up? Also not mentioned, how long does it take for humans to start recognizing new races, vs ml and dl?

Why doesn’t the model trained on one race generalize to different races? That sounds inferior to human vision.

I prefer Googles snippet to visiting a shitty site. This is one of the few instances that I fully support Google. If websites want to stay relevant, they need to be more inventive.

"No matter what the function, there is guaranteed to be a neural network so that for every possible input, x , the value f(x) or some close approximation) is output from the network"

Okay, so what? You require more and more neurons (ie. parameters) to approximate your function better and better. You can do the same with piecewise constant (Riemann sums). You can do this with trig functions too (Fourier transform).

"This result tells us that neural networks have a kind of universality."

I don't know what this statement means. What mathematical properties do neural networks have that other functions don't? The ability to approximate continuous functions isn't special. Given 5 points, I can perfectly fit an elephant to your function. And it's not like you are fitting the function with as few parameters as possible.

There is no "ai", it's the same bull crap we had in the 80s. We haven't had true innovation in this space for a very long time, the current hype was caused by GPU, cloud, and abundance of data for training.