I think that this is seriously a great idea.
PAM should be a service that communicates over an UNIX socket. Upsides:
* You can completely isolate it from anything else. Lock it into a sandbox. Use SELinux to ensure it doesn't do anything weird.
* Much more secure -- you have a very narrow communication channel, and no exposure to stuff like LD_PRELOAD.
* You avoid the mess of the above for the applications using it. Apps using PAM execute PAM code, and that means that anything that uses PAM needs to be able to do any arbitrary thing a PAM module might want to do.
* Applications using PAM no longer need to have root access. Your screensaver/lock screen can be completely unprivileged.
* If it can be an UNIX socket, it probably can also be a TCP socket, so you can have it run remotely if such a thing makes sense.
* The protocol can be inspected
* Good opportunity for updating the API and fixing anything lacking.
PAM should be a service that communicates over an UNIX socket. Upsides:
* You can completely isolate it from anything else. Lock it into a sandbox. Use SELinux to ensure it doesn't do anything weird.
* Much more secure -- you have a very narrow communication channel, and no exposure to stuff like LD_PRELOAD.
* You avoid the mess of the above for the applications using it. Apps using PAM execute PAM code, and that means that anything that uses PAM needs to be able to do any arbitrary thing a PAM module might want to do.
* Applications using PAM no longer need to have root access. Your screensaver/lock screen can be completely unprivileged.
* If it can be an UNIX socket, it probably can also be a TCP socket, so you can have it run remotely if such a thing makes sense.
* The protocol can be inspected
* Good opportunity for updating the API and fixing anything lacking.