You can never ask why a model did a certain thing, or what it was "thinking" when it said something - just like you can't ask a human which neurons were firing when they had a certain thought. The information just isn't available at that level.
You absolutely can have deep nuanced discussions with LLMs however, you just need to better understand their strengths and weaknesses.
You absolutely can have deep nuanced discussions with LLMs however, you just need to better understand their strengths and weaknesses.