Comment by tptacek

Allowing the Signal client to access your contact list is literally the premise of Signal; it's the core security UX trade it makes: no durable logs of who's talking to who on the servers, and contact lists stored exclusively on the client.