I pack some backup clothes (undershirt, underwear, socks), they mostly stay in my bike bag because I rarely use them. If I do need them, I just go to the bathroom, then change back into the bike clothes for the ride home. If it's really hot, I might switch it up and put the office clothes in the bag from the start - meaning I know I'll need a change.
Really not that complicated.