are you extending this to the usage of yubikey-agent and ssh-tpm-agent as well?
both variants, whether it's using a PKCS11 provider using a standardized interface, or using a completely custom SSH agent, will need to deal with secret material.
although I'm no expert on the inner workings of SSH, I'd expect there to not be much difference between having the OpenSSH agent interface with ykcs11 (which is also open source and can be reviewed) and using an alternative agent with piv capabilities that was found on github.
both variants, whether it's using a PKCS11 provider using a standardized interface, or using a completely custom SSH agent, will need to deal with secret material.
although I'm no expert on the inner workings of SSH, I'd expect there to not be much difference between having the OpenSSH agent interface with ykcs11 (which is also open source and can be reviewed) and using an alternative agent with piv capabilities that was found on github.