Comment by welterde

X11 does have various ways to restrict access (one of which ssh does use for instance) and some more advanced security extensions. But as far as I can tell there has never been that much motivation to widely deploy any of it.