Comment by SHIT_TALKER

Doesn't that break admin-ajax.php access? It sits behind /wp-admin/. WordPress recommends allowing access when using IP whitelists.

Apache:

  # Allow access to wp-admin/admin-ajax.php
  <Files admin-ajax.php>
      Order allow,deny
      Allow from all
      Satisfy any
  </Files>

Nginx:

  location /wp-admin/admin-ajax.php {
    allow all;
  }