Comment by o_____________o

You can turn this behavior off in .ssh/config with

  Host *
    PubkeyAuthentication no

Put that at the bottom of the file, then turn back on for each host:

  Host site.com
    PubkeyAuthentication yes