Firefishy

I am part of the team which runs the OpenStreetMap.org infrastructure, we also get a lot of weird and wonderful requests / emails. .mil, .gov, .edu and GeoTLD variants.

I should really do a write-up sometime.

Disclosure: I am part of the mostly volunteer run OpenStreetMap ops team.

Technically we able to block and restrict the scrapers after the initial request from an IP. We've seen 400,000 IPs in the last 24 hours. Each IP only does a few requests. Most are not very good at faking browsers, but they are getting better. (HTTP/1.1 vs HTTP/2, obviously faked headers etc)

The problem has been going on for over a year now. It isn't going away. We need journalists and others to help us push back.

Disclosure: I am part of the OpenStreetMap mostly-volunteer sysadmin team fighting this.

The scrapers try hard to make themselves look like valid browsers, sending requests via residential IP addresses (400,000+ IPs at last count).

I reached out to journalists because despite strong technical measures, the abuse will not go away on its own.

I help to run the OpenStreetMap infrastructure, and for many years the OSUOSL has generously hosted some of our servers.

Without OSUOSL OpenStreetMap would be more difficult to host and significantly slower to access from North America.

I hope OSUOSL can get the financial support they rightfully deserve.

OpenStreetMap is now back up and running :-)

The new stack has some unique feature like the vector tiles being minutely updated directly from OSM mapping changes.

There are still issues to fix as it is still only a technical preview.

We don't offer 2x raster tiles because we simply don't have the resources to do it while keeping the tiles minutely updated and open access. We serve around 60k req/sec on a tiny donation backed budget.

The raster tiles are primarily for our OSM mappers to see their map changes rendered quickly to improve the feedback loop.