Comment by fxtentacle

In general, that should work. Here's an example board with 4x CM4 modules and a RTL8367N switching chip: https://github.com/will127534/Miniature-CM4-Cluster