Comment by voltagex_

I wonder if you can turn off automatic use of public keys for "*", and then enable it in .ssh/config on an as-needed basis.