Comment by JoachimSchipper

ssh-agent has prompting and you can set up a Yubikey with ssh.

The problem here was agent forwarding, which you should almost always replace with opening a new connection via ssh -J (or equivalent.)