> Do you understand every line and possible state of all software you created, or any LLM which is far larger
Is your company's backend conscious? Is a Turing machine running busy beaver? Is SHA256? Emergent behavior isn't limited to brain-equivalent things.
> why does it matter
Genetic optimization, whether through actual evolution, in software, or on FPGAs creates extremely interdependent systems that prove difficult to untangle and trivial to break. If experts can't fully analyze an evolved tone discriminator, because of all the tiny emergent interactions beyond the lumped circuit model in a poorly characterized environment, why suddenly claim we can isolate what process in the brain is responsible for an ill-defined notion of consciousness other than the whole?
> Neural nets can be almost any function
Neural nets can _approximate_ any function with enough layers (see XOR problem), but so can an infinite look-up table with enough entries. They are the best models we've got, but they are not magical.
Look at the most recent and illustrative example with fly brain connectomes. These scans only capture the relative strength of connections between the fly's neurons, to actually turn it into neural activity we need some sort of model.
You could imagine a human connectome with an ANN-based excitation model trained from EEG scans, but that would still be hardly equivalent to a human brain. You can fit on a narrow range of behaviors, but without modeling the actual mechanisms behind them, you'll fail to predict a lot more. You'd be missing the influence of hormones, which doesn't show up on a short MRI scan, glia cells, and a whole lot more nuance only a real biologist can educate you and me about, making the digital twin, if we anthropomorphise it, at best a cripple.
And that's a system directly mimicking a functioning human brain, rather than a highly de-correlated objective like webtext. Just as an NN trained to run towards a finish line could learn bipedal locomotion but will more likely abuse a collision bug and launch itself forward, a machine trained to mimick text could spontaneously instantiate a compressed model of the human brain, or much more likely become whatever alien thing can generate appropriate outputs, bearing less similarity to human thinking than any living animal.
Is your company's backend conscious? Is a Turing machine running busy beaver? Is SHA256? Emergent behavior isn't limited to brain-equivalent things.
> why does it matter
Genetic optimization, whether through actual evolution, in software, or on FPGAs creates extremely interdependent systems that prove difficult to untangle and trivial to break. If experts can't fully analyze an evolved tone discriminator, because of all the tiny emergent interactions beyond the lumped circuit model in a poorly characterized environment, why suddenly claim we can isolate what process in the brain is responsible for an ill-defined notion of consciousness other than the whole?
> Neural nets can be almost any function
Neural nets can _approximate_ any function with enough layers (see XOR problem), but so can an infinite look-up table with enough entries. They are the best models we've got, but they are not magical.
Look at the most recent and illustrative example with fly brain connectomes. These scans only capture the relative strength of connections between the fly's neurons, to actually turn it into neural activity we need some sort of model.
You could imagine a human connectome with an ANN-based excitation model trained from EEG scans, but that would still be hardly equivalent to a human brain. You can fit on a narrow range of behaviors, but without modeling the actual mechanisms behind them, you'll fail to predict a lot more. You'd be missing the influence of hormones, which doesn't show up on a short MRI scan, glia cells, and a whole lot more nuance only a real biologist can educate you and me about, making the digital twin, if we anthropomorphise it, at best a cripple.
And that's a system directly mimicking a functioning human brain, rather than a highly de-correlated objective like webtext. Just as an NN trained to run towards a finish line could learn bipedal locomotion but will more likely abuse a collision bug and launch itself forward, a machine trained to mimick text could spontaneously instantiate a compressed model of the human brain, or much more likely become whatever alien thing can generate appropriate outputs, bearing less similarity to human thinking than any living animal.