Shameless plug but I hit these exact same kinds of walls and I tried out alternatives like agent-deck or herdr, but ultimately didn't enjoy the TUI approach that herdr/Tmux solutions offer, so I built Muster [0] and I'm quite proud of it -- it's turning out really well and is my daily driver at home and at work. It uses libghostty so I get the same sort of Ghostty-feel as Ghostty by itself provides. It supports SSH / remote machines well too and allows agents from different harnesses to message and work together thru 'muster daemons'.
[0] https://github.com/amterp/muster