So, if traffic numbers are EXACTLY the same (as noted) but less people click on a single button, you've "lost those users"? Doesn't make any sense to me. You've lost interaction with a button from your unchanged user base. Like you said yourself, click conversion from your users changed. Users changing interaction ≠ losing users
If you had lost 50% of your users, you would have half of the overall site traffic that you had last time.
If you had lost 50% of your users, you would have half of the overall site traffic that you had last time.