Comment by davemtl

I've been doing this for a while using the Google Authenticator PAM plugin[1]. I use the same module to add two factor authentication to both SSH and OpenVPN connections.

[1] http://code.google.com/p/google-authenticator/