Comment by jkire

Yup, if you're using ssh-agent (as opposed to something like gnome-keyring) setting `AddKeysToAgent confirm` to your ssh config should cause a pop up to happen every time anything requests a key from the agent.