Comment by JepZ

If you run a federated server, not all contacts might use the same server. With end-to-end encryption that doesn't really matter.

EDIT: Moreover, trust is not binary. So while your family might trust you, maybe your dad doesn't want you to be able to read everything he writes your mom or so (you get the idea).