Discovery is important and a hard problem, worth making some trade-offs for. Signal published a good blog post about this, including this passage:
> In order to turn the address book on a device into a social graph for Signal, the Signal client needs to be able to determine which of the device’s contacts are Signal users. [...] The obvious problem with this method is that the hash of a user identifier can almost always be inverted. [...] This method of contact discovery isn’t ideal because of these shortcomings, but at the very least the Signal service’s design does not depend on knowledge of a user’s social graph in order to function.
> In order to turn the address book on a device into a social graph for Signal, the Signal client needs to be able to determine which of the device’s contacts are Signal users. [...] The obvious problem with this method is that the hash of a user identifier can almost always be inverted. [...] This method of contact discovery isn’t ideal because of these shortcomings, but at the very least the Signal service’s design does not depend on knowledge of a user’s social graph in order to function.
Src.: https://signal.org/blog/private-contact-discovery/
edit: Also, Signal is not peer-to-peer. It is a centralized, client-server architecture.