Yes, you're right, understanding informs the constraints and the possible solution spaces. That should be added to the list.
Practically speaking, I don't think the "black box" metaphor is as broken as you're saying, though I like the point you made. There is still a ton of software (the majority, I think -- crud apps of all flavors) where an unthinking, untechnical PM wants something they can specify, and an LLM can give it to them, and it will more or less solve their problem. It won't be the best possible thing, but it will be good enough.
Practically speaking, I don't think the "black box" metaphor is as broken as you're saying, though I like the point you made. There is still a ton of software (the majority, I think -- crud apps of all flavors) where an unthinking, untechnical PM wants something they can specify, and an LLM can give it to them, and it will more or less solve their problem. It won't be the best possible thing, but it will be good enough.