Comment Comment by thegrim33 From the article:"We currently don't understand how to make sense of the neural activity within language models.""Unlike with most human creations, we don’t really understand the inner workings of neural networks.""The [..] networks are not well understood and cannot be easily decomposed into identifiable parts""[..] the neural activations inside a language model activate with unpredictable patterns, seemingly representing many concepts simultaneously""Learning a large number of sparse features is challenging, and past work has not been shown to scale well."etc., etc., etc.People say we don't (currently) know why they output what they output, because .. as the article clearly states, we don't. parent thegrim33 2 years ago
"We currently don't understand how to make sense of the neural activity within language models."
"Unlike with most human creations, we don’t really understand the inner workings of neural networks."
"The [..] networks are not well understood and cannot be easily decomposed into identifiable parts"
"[..] the neural activations inside a language model activate with unpredictable patterns, seemingly representing many concepts simultaneously"
"Learning a large number of sparse features is challenging, and past work has not been shown to scale well."
etc., etc., etc.
People say we don't (currently) know why they output what they output, because .. as the article clearly states, we don't.