You're absolutely right about the fungible teams / Scrum fallacy, and it's one of reasons I always ask the person who's doing the story to re-point it if they feel the point allocation was wrong. This also corrects for the peer-pressure people feel in planning to give stories fewer points [1].
This way at the end of the sprint you get a better idea for how much work was really done. There's an opportunity for abuse - if someone wants to slack off, they can make a 2 point story into an 8 pointer, but that's going to get noticed by everyone, and it hasn't been an issue in practice.
[1] "That's not 4 points, it's definitely 2, it's really not that much work ...", etc.
This way at the end of the sprint you get a better idea for how much work was really done. There's an opportunity for abuse - if someone wants to slack off, they can make a 2 point story into an 8 pointer, but that's going to get noticed by everyone, and it hasn't been an issue in practice.
[1] "That's not 4 points, it's definitely 2, it's really not that much work ...", etc.
EDIT: Fixed typo