Comment by mmh0000

Something like this can be implemented in Linux's NetFilter quite easily using the recent module:

    iptables -A FORWARD -i eth0 -m recent --update --seconds 600 --hitcount 10 --name scan --mask 255.255.255.255 --rsource -j DROP
    iptables -A FORWARD -d 173.165.141.72/29 -i vmexfwbr -m recent --update --seconds 86400 --name instaban --mask 255.255.255.255 --rsource -j DROP
    …PUT STANDARD ACCEPT/REJECT RULES HERE…
    iptables -A FORWARD -d 173.165.141.72/29 -i vmexfwbr -p tcp -m multiport --dports 21,23,110,1433,3389,5060,8080,8088 -m recent --mask 255.255.255.0 --set --name instaban --rsource -m comment --comment "Instabans port scanning bots."
    iptables -A FORWARD -d $YOUR_IP_RANGE/29 -i eth0 -m recent --mask 255.255.255.0 --set --name scan --rsource
    iptables -A FORWARD -j REJECT --reject-with icmp-port-unreachable

The above would start dropping packets from anyone who hits 10 or more ports not previously accepted by an accept rule, and block the /24 of anyone who hits one of the more commonly scanned ports.