I see this when my equipment roams back into my private network and the wireguard server is inside that LAN. It can be solved by NAT'ing packets arriving on your edge router's inside interface, destinated to your outside IP, back to the inside wireguard server IP.
Alternatively if your client is Linux, there is:
https://github.com/WireGuard/wireguard-tools/tree/master/con...