Comment by marwis

X11 is also just a protocol and you can implement it securely and expose access control through extensions. In fact it already exists (XACE) just nobody is using it.