I was under the impression that the Signal server, if compromised could be utilised to potentially log metadata of communication between contacts. Sealed sender [1] is a feature of the Signal protocol to mitigate overt metadata retention but it may fail against certain correlation attacks.
[1]: https://signal.org/blog/sealed-sender/