The Windows version is quite portable if you manually set the .textadept home to your USB stick. A while ago I wrote a Portable Apps launcher [1] that did that automatically for you, haven't used it in quite a while though. (I wrote this before GTK was included in the Windows package.)
[1] https://github.com/rgieseke/textadept-portable