Comment by throwaway_bob

the fact that random noise can be classified as strongly belonging to some class, and the fact that classification results can be unstable, is simply a result of the fact the input space is very high dimensional, and the output space is very small (say a few isolated points). That is, if you are discriminatively training a mapping from images in R^(224 x 224 x 3) to 1000 points (class labels), there is going to be a tremendous amount of instability in the inverse direction.