But it does matter. ISPs simply don't have the means to provide capacity for every user to simultaneously max out their connection at once. That is not, and never has been how Internet infrastructure works. That'd be like building a 100 lane highway because on Christmas it gets congested.
You're paying for peak performance, not 24/7 performance.
You're paying for peak performance, not 24/7 performance.