Comment by zhfliz

have you considered using ykcs11?

ykcs11 allows you to use the native SSH agent (or even no agent at all for individual ssh invocations) with an ssh key on a yubikey using their pkcs11 provider.

https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PK...