I found even these simple USB Switchers to be quite expensive, and sometimes they require drivers that don't work on Linux. I quickly sketched a simple circuit board with USB hubs and a switch and ordered them assembled (!) from one of those DIY-fabs (JLCPCB or PCBWay). They cost 25€ for 5 pcs, each with one 2-to-1 switch. I don't have any experience in circuit design and didn't have to solder anything, and they work perfectly.
These would go pretty well with this software, maybe I'll give it a try.
These would go pretty well with this software, maybe I'll give it a try.