My partner and I have both an Alexa and a Google Home Assistant and we leave the mic off on both devices. The Alexa we got because we have a Blink doorbell and it connects to the Alexa in order to make a sound when someone rings the doorbell. The Google Home Assistant came with the Google Nest Wifi Mesh extender. We both care about privacy so that's why we turned them off (but not enough to not have the physical devices in our house). I sometimes wonder if those mic switches are hardware switches and can amazon or google override them.
edit: fixed typo, missing a "to" in "to not have"