That's not my understanding of the NP domain. Take Prime Factorization for example. Which isn't even NP-complete, or Sudoku which is NP-complete. Both of these problems can be made really hard by changing 1 small variable, size.
Edit: I understand that your argument is about the average case and not the worst case.
Edit: I understand that your argument is about the average case and not the worst case.