Comment by Summershard

Signal does not know who you correspond with. The only information they keep is the account creation timestamp, and the date that the account last connected to the Signal service.

You may have confused this information with WhatsApp which indeed keeps a lot of metadata on each user.