Comment by JohnTHaller

Could be. Or it could be a server on either end. Or a connection in between. Email is inherently insecure. So, even though you have an SSL connection to your server when you send your email and they have an SSL connection to their server when they receive it, your two servers make a plaintext connection to actually send the email from one to the other.