Comment by Evidlo

Did you ever look at only transmitting the changed regions of the framebuffer? You can really cut down on data rate that way: https://github.com/pl-semiotics/mxc_epdc_fb_damage

Thats how the rM VNC project does it, but I prefer the UX of your app not needing any client side software.