I am guessing you live alone. Try telling a one year old to put on a snowsuit to eat soup in. Or your dog. When the weather is -40 degrees, a sweater won't save you. When it is 110 you will need a trunk with batteries behind you to power your personal AC 24/7. Give it a try and turn off the heat and AC for a season. You might make it, but will it be worth the $100-200/month?
Edit: don't forget that AC is much more efficient than heating. With AC you are looking to change the temperature by at most 35 degrees and you are moving heat, so you are getting a multiplier there. With heating, you are heating from freezing or below for most places.
Edit: don't forget that AC is much more efficient than heating. With AC you are looking to change the temperature by at most 35 degrees and you are moving heat, so you are getting a multiplier there. With heating, you are heating from freezing or below for most places.