I believe that in 1Passwd you can define / preselect a key per host now. So you can pinpoint key -> host. Some hosts have firewall rules that will block after X attempts were X might be low.
However the agent still has access to all your keys, obviously.
However the agent still has access to all your keys, obviously.