Back when domains were easy to spoof (I could setup a server and send mail as ycombinator.com easily enough) it made sense to track the IPs, but now that you have DKIM and SPF links to cross-check, you should be able to use the domain reliability as a strong indicator. Sure you would have to catch people buying a "good" domain that expired, but that shouldn't be an insane hurdle.
The real story is nobody cares.