The human brain, the authors argue, in fact uses multiple networks when interpreting and producing language. These include:
- the language network, which delivers formal linguistic competence
- the multiple demand network, which provides reasoning ability
- the default network, which tracks narratives above the clause level
- the theory of mind network, which infers the mental state of another entity
This leads to their argument that a modular structure would lead to enhanced ability for an LLM to be both formally and functionally competent. (While LLMs currently exhibit human-level formal linguistic competence, their functional competence--the ability to navigate the real world through language--has room for improvement.)
Transformer models, they note, have degree of emergent modularity through "allowing different attention heads to attend to different input features."
I was wondering, is it possible to characterize the degree of emergent modularity in current systems?
- the language network, which delivers formal linguistic competence - the multiple demand network, which provides reasoning ability - the default network, which tracks narratives above the clause level - the theory of mind network, which infers the mental state of another entity
This leads to their argument that a modular structure would lead to enhanced ability for an LLM to be both formally and functionally competent. (While LLMs currently exhibit human-level formal linguistic competence, their functional competence--the ability to navigate the real world through language--has room for improvement.)
Transformer models, they note, have degree of emergent modularity through "allowing different attention heads to attend to different input features."
I was wondering, is it possible to characterize the degree of emergent modularity in current systems?