MattBlissett
Joined 9 years ago
Karma
104
Submissions
48
Member since
Mar 1
Recent submissions
Comment · 2 days ago
15+ years of expectations around how a website can be run were upended by scrapers for LLM training.
They ignore robots.txt rules and seem to scrape as quickly and as broadly as possible. Unlike search engines, it doesn't matter to the LLM provider if the website is offline. Unlike search engines, they take steps to be anonymous and difficult to block, including using residential proxy services and other shared IP space.
I try and keep an open data repository online, but it was not designed for this. Cloudflare is hopefully a temporary fix while we redesign backend and frontend components.
The end result is going to be worse for users — slower querying, less flexibility as we primarily expose what we can precalculate, shifting CPU and bandwidth load to the user's device rather than our servers.
Comment · 15 days ago
Are any sites you'd expect to be on paid-for Cloudflare also blocked?
I put https://www.gbif.org/ behind Cloudflare on the business plan, and so far (it's been roughly a year) we've only had a couple of reports of blocking, and both seemed likely to be caused by other users on the same IP scraping the site.
Comment · 1 month ago
Some of the bird data used is available in https://www.GBIF.org (website + APIs + data dumps) but not all of it.
For a visualization like this, the place to start is probably the most recent monthly data snapshot which is available on AWS/Google/Azure's open data portals: https://www.gbif.org/occurrence-snapshots
e.g. start with a DuckDB query:
SELECT scientificName, individualCount, year, month, day, decimalLatitude, decimalLongitude, issue
FROM read_parquet('s3://gbif-open-data-eu-central-1/occurrence/2026-09-01/occurrence.parquet/*')
WHERE class = 'Aves';
Comment · 2 months ago
With the GROUPING SETS or CUBE features in DuckDB, I think.
https://duckdb.org/docs/current/sql/query_syntax/grouping_se...
Comment · 9 months ago
GBIF track the use of data we provide to scientists, where they later publish papers citing that data [1]. For iNaturalist, the list of known citations is at [2]. In most cases a download of data from GBIF will include data from more than one dataset (iNaturalist is one of over 110,000). To find how particular records in a download were used — or even if they were discarded — requires reading the paper.
As an example from the list, "Aedes albopictus Is Rapidly Invading Its Climatic Niche in France: Wider Implications for Biting Nuisance and Arbovirus Control in Western Europe" [3] cites 5348 iNaturalist records.
[1] https://www.gbif.org/literature-tracking
[2] https://www.gbif.org/resource/search?contentType=literature&...
[3] Paper https://doi.org/10.1111/gcb.70414 citing GBIF data download https://doi.org/10.15468/dl.gzdq3f
PS we are recruiting an engineer, but the deadline for this position is today: https://gbif.link/senior-data-engineer
Comment · 2 years ago
Global Biodiversity Information Facility (GBIF) | Software developer | Copenhagen, Denmark | ONSITE, VISA
Global Biodiversity Information Facility (GBIF) | Senior data engineer | Copenhagen, Denmark | REMOTE
GBIF is an international organization and open-data research infrastructure, whose purpose is to make scientific biodiversity data available to all on the Internet. Multiple research papers are published every day citing this data.
See https://www.gbif.org/ for our public biodiversity data platform, including its open-data API built using a Hadoop cluster and Elasticsearch. All our code is open source and on GitHub [https://github.com/gbif/].
We have two positions:
- a software developer (onsite) for the backend systems (Java, Elasticsearch) https://www.gbif.org/news/2b6mW6uaebea3cZUfXIxog/gbif-is-rec...
- a data engineer (remote) focused more on the Hadoop-based data pipeline and biodiversity data https://www.gbif.org/news/5oePaOpxHplJsbw57BIdK9/gbif-seeks-...
Comment · 3 years ago
uk.maven.org went offline years ago, although it's still listed in the mirror metadata at https://repo.maven.apache.org/maven2/.meta/repository-metada...
If anyone who can fix that is reading, there's an issue here: https://issues.apache.org/jira/browse/MNGSITE-471
Comment · 4 years ago
That's essentially correct.
You might be interested in the snapshots we make available in Azure, Amazon AWS and Google Cloud: https://www.gbif.org/occurrence-snapshots
Or the API: https://www.gbif.org/developer/summary
Comment · 4 years ago
Global Biodiversity Information Facility (GBIF) | Site Reliability Engineer / DevOps | Copenhagen, Denmark | €91k p.a. exempt from Danish income tax | ONSITE, VISA
GBIF is an international organization and open-data research infrastructure, whose purpose is to make scientific biodiversity data available to all on the Internet. Several research papers are published every day citing this data.
See https://www.gbif.org/ for our public biodiversity data platform, including its open-data API built using a Hadoop cluster and Elasticsearch. All our code is open source and on GitHub.
We're looking for an SRE/DevOps engineer who can upgrade and improve our data cluster and other infrastructure. The ideal candidate will also be comfortable working on the Java applications running on the cluster.
The salary is €91k per annum, and as an intergovernmental organization it's exempt from Danish income tax. Staff are recruited on the broadest possible geographical basis, and a visa is no problem at all.
https://www.gbif.org/news/GFYcHp4sJuCvnn6b8UM8e/gbif-secreta...
Comment · 4 years ago
Global Biodiversity Information Facility (GBIF) | Site Reliability Engineer / DevOps | Copenhagen, Denmark | ONSITE, VISA
GBIF is an international organization and open-data research infrastructure, whose purpose is to make scientific biodiversity data available to all on the Internet.
See https://www.gbif.org/ for our public biodiversity data platform, including its open-source and open-data API built using Hadoop cluster and Elasticsearch.
We're looking for an SRE/DevOps engineer who will update and improve our data cluster and other infrastructure.
The salary is competitive, and as an intergovernmental organization it's exempt from Danish income tax. Staff are recruited on the broadest possible geographical basis, and a visa is no problem at all.
https://www.gbif.org/news/7MZC6usRc9SNMLlN2znfLK/gbif-secret...
Comment · 5 years ago
This looks great!
I hacked around with OpenMapTiles to generate vector tiles in Arctic, Antarctic and WGS84 projections (the latter with 2×1 tiles at zoom 0) [1]. The time spent waiting for PostGIS to do reprojections etc made development very slow, and I've put off updating the tiles.
This tool is reprojects from WGS84 to Web Mercator while reading the tiles, so optimistically changing that would be much of what's needed.
[1] See the map at the bottom of gbif.org, there's a button to change the projection
Comment · 5 years ago
> [7] https://species.nbnatlas.org/species/NBNSYS0000021551
With data from GBIF you would have almost all of the British records published on the NBN Atlas (about 100 are not shared), plus 2,300 more from other sources.
Comment · 5 years ago
AsciiDoctor supports Unicode just fine.
We've integrated it with PO4A[0] and CrowdIn.com to support a translation workflow, and so far have one document in Chinese: [1].
Another (tiny) example is [2] with an emoji star, you can "Edit this page" and see the Unicode source.
[1] https://docs.gbif.org/collections-idea-paper/zh/
[2] https://ipt.gbif.org/manual/en/ipt/2.5/data-hosting-centres#...
Comment · 5 years ago
I use https://openmaptiles.org/ at work. For normal, Web Mercator maps we just use the free download of the whole world, available at [1]. Running the OpenMapTiles tools to generate a current tileset for a particular region is straightforward, but if you need all of Europe you will need some decent compute capacity. Unless everything can be done on a single computer, you'll also need to implement some missing pieces, e.g. a message system to distribute the rendering across your computers.
https://matt.blissett.me.uk/
https://github.com/MattBlissett
https://orcid.org/0000-0003-0623-6682
Software developer at https://www.gbif.org/ — mblissett@ that domain.