Output can appear logical without the model actually applying formal logic. If its training has associated tools with wood, and sharp edges with splitting, then a prompt containing those ideas can make “use a sharp-edged tool to split wood” a likely continuation even if that exact combination never appeared in training.
By asking follow-up questions and establishing an in-context vocabulary, users can steer output toward a more complete description. For example, “Let’s call that tool and ‘axe’. How might I hold it?”; the model’s learned associations with holding sharp knives carefully can make “by the handle, with the blade facing away from you” a likely response.
I’m confident you could arrive at a near perfect description of an axe this way without any references or depictions of an axe in the training data. I’d imagine the same applies to many cybsecurity/cyber offense concepts (but probably not all).
By asking follow-up questions and establishing an in-context vocabulary, users can steer output toward a more complete description. For example, “Let’s call that tool and ‘axe’. How might I hold it?”; the model’s learned associations with holding sharp knives carefully can make “by the handle, with the blade facing away from you” a likely response.
I’m confident you could arrive at a near perfect description of an axe this way without any references or depictions of an axe in the training data. I’d imagine the same applies to many cybsecurity/cyber offense concepts (but probably not all).