Compression is a component of general intelligence. A few years ago I was very sceptical of machine learning ever leading to general intelligence. I've since changed my mind. There are a lot of parallels to this work and the concept of "embeddings" in machine learning.
Intelligence requires the ability to generalize. A prerequisite for generalization is the ability to take something high-dimensional and reduce it to a lower-dimensional representation to allow comparison and grouping of concepts.
We're doing this all the time. Take a pen for example: we're able to combine information from sight, touch, and sound. Through some mechanism, our brains reduce the multi-sensory information and create a consistent representation that is able to invoke past memories and knowledge about pens.
Our brains encode the embeddings in a very different way to deep learning neural networks, but the commonality is that both are able to compress data into a _useful_ representation. Note that as a result of this, the quality of the compression is important. Some forms of compression might be very efficient but they also tangle concepts together, resulting in loss of composability. The ideal compression (from an intelligence point of view) is both information efficient and maximally composable.
Intelligence requires the ability to generalize. A prerequisite for generalization is the ability to take something high-dimensional and reduce it to a lower-dimensional representation to allow comparison and grouping of concepts.
We're doing this all the time. Take a pen for example: we're able to combine information from sight, touch, and sound. Through some mechanism, our brains reduce the multi-sensory information and create a consistent representation that is able to invoke past memories and knowledge about pens.
Our brains encode the embeddings in a very different way to deep learning neural networks, but the commonality is that both are able to compress data into a _useful_ representation. Note that as a result of this, the quality of the compression is important. Some forms of compression might be very efficient but they also tangle concepts together, resulting in loss of composability. The ideal compression (from an intelligence point of view) is both information efficient and maximally composable.