Ask HN: What should we do if someone is crawling our website?
A couple days ago I found out someone was crawling our website. she was sending low traffic (150 rpm) and it was not a real problem! (we have around 15k rpm on average).
I created an automated service to find crawler IPs and ban them, I did this for fun (parsing stream of requests, finding malicious behavior and blocking using firewall API was a challenging task).
Not only this service didn't stop her but she is trying harder and her request rate has tripled today (she is using more IPs. today, 1.5k of her IPs were banned).
What do you think I should do, let her crawl or chase this rabbit hole?
I created an automated service to find crawler IPs and ban them, I did this for fun (parsing stream of requests, finding malicious behavior and blocking using firewall API was a challenging task).
Not only this service didn't stop her but she is trying harder and her request rate has tripled today (she is using more IPs. today, 1.5k of her IPs were banned).
What do you think I should do, let her crawl or chase this rabbit hole?
Thanks