Comment by CCoffie

I believe you only lose the referer header when switching between http and https.