After having my domain hijacked today I'd like to share an incident postmortem just in case you find yourself in the same situation (swearing at your terminal output at 10am on Friday morning when your website is down).
9:58am I can no longer deploy to production
10:15am Finished troubleshooting all services, no problems identified
10:16am nslookup resolves to some random IP address instead of my prod server (WTF!!!!!!)
10:20am Log into registrar and find out they replaced my custom DNS servers with their own and added records to serve a "Parked free courtesy of GoDaddy" page with ads and a button that says "Get This Domain"
10:30am Changed my domain on the registrar website back to my custom DNS servers
10:32am Changed my password on the registrar website
10:38am Got told by GoDaddy support they didn't have anything to do with this and it was my fault it happened (f-me, right?)
11:55am DNS records across the internet are still jacked
12:00pm Manually blow out the cache on cloudflare for my domain
Postmortem Suggestions:
* If your website goes down; don't blow 15+ minutes troubleshooting your app services before checking DNS
* Enable 2fa with your registrar (even though there was no alert for us)
* Set up an alert for when your domain resolves to a different IP address (make a script and host it elsewhere or pay for a service)
* Don't trust your registrar!!!!
* Take a screenshot of your registrar settings and DNS settings right now so you have a record when they disappear
* Get access to your registrar account ASAP after the attack and change your DNS records back using the screenshots you just took
* Manually purge the cache of major DNS providers (for your domain) to allow your DNS records to propagate: https://cloudflare-dns.com/purge-cache/
9:58am I can no longer deploy to production
10:15am Finished troubleshooting all services, no problems identified
10:16am nslookup resolves to some random IP address instead of my prod server (WTF!!!!!!)
10:20am Log into registrar and find out they replaced my custom DNS servers with their own and added records to serve a "Parked free courtesy of GoDaddy" page with ads and a button that says "Get This Domain"
10:30am Changed my domain on the registrar website back to my custom DNS servers
10:32am Changed my password on the registrar website
10:38am Got told by GoDaddy support they didn't have anything to do with this and it was my fault it happened (f-me, right?)
11:55am DNS records across the internet are still jacked
12:00pm Manually blow out the cache on cloudflare for my domain
Postmortem Suggestions:
* If your website goes down; don't blow 15+ minutes troubleshooting your app services before checking DNS
* Enable 2fa with your registrar (even though there was no alert for us)
* Set up an alert for when your domain resolves to a different IP address (make a script and host it elsewhere or pay for a service)
* Don't trust your registrar!!!!
* Take a screenshot of your registrar settings and DNS settings right now so you have a record when they disappear
* Get access to your registrar account ASAP after the attack and change your DNS records back using the screenshots you just took
* Manually purge the cache of major DNS providers (for your domain) to allow your DNS records to propagate: https://cloudflare-dns.com/purge-cache/