That's like saying if I send an email via Gmail, Google should feel free to do whatever it wants with the headers, because it's supposed to be publicly viewable so the mail server knows where it should go. (See also the NSA phone call metadata scandal.)
In reality, when you send a letter, you are placing it into the care of the postal service. Your letter is only handled by postal employees, and mailbox privacy is codified in federal law. There's clearly an expectation that the postal service is not releasing your information to third parties without your consent. The information on the outside of the envelope is not meant for, nor is it needed by, anyone other than the postal service
In reality, when you send a letter, you are placing it into the care of the postal service. Your letter is only handled by postal employees, and mailbox privacy is codified in federal law. There's clearly an expectation that the postal service is not releasing your information to third parties without your consent. The information on the outside of the envelope is not meant for, nor is it needed by, anyone other than the postal service