With not-quite current hardware as supported by Pocketblue, performance is not that much of an issue, despite the OnePlus 6 being introduced in 2018. GNOME Shell mobile is quite smooth on it.
That said, if you want to start without the entire Linux desktop stack, you can, and there's even a project that already does something like that IIUC: https://sr.ht/~mil/framebufferphone/
That said, if you want to start without the entire Linux desktop stack, you can, and there's even a project that already does something like that IIUC: https://sr.ht/~mil/framebufferphone/