https://www.x.org/wiki/Releases/https://www.gtk.org/docs/installations/linux/
I mean it will probably not be painless and other applications u run might break* but xorg is relativly stable.
Liba are out there are free to get. Usually people are arguing that the conveniences isnt there not that its not possible.
* if u dont sandbox this a bit with custom lib paths