Comment by stephenr

Yet again, there is already a mechanism for this.

Add your fingerprints to DNS (they can be obtained using ssh-keygen -r <hostname>)

Set the SSH option VerifyHostKeyDNS to "yes" (or "ask" if you want to confirm, with a notice about if the DNS entry matches).