You're not nuts - this has been a huge issue for a long time. It's what makes taking calls outside on a street so frustrating (or even turning on the faucet to wash a mug while on a call).
It's weird because this wasn't an issue with the initial versions of AirPods firmware. But somewhere along the line, they tuned the firmware to introduce the issue you're describing.
It's weird because this wasn't an issue with the initial versions of AirPods firmware. But somewhere along the line, they tuned the firmware to introduce the issue you're describing.