Sounds interesting, and promising! I'm no expert in streaming videos, but your shell reminded me of mosh [1]. Mosh's auth approach is also interesting - it leverages ssh for authentication, something that you might consider too. Mosh unfortunately has shortcomings though - mainly, it lacks tunneling ability.
[1] https://mosh.org