The state of data sharing seems to be still quite sad.
* Hosting problems. The first link I tried was already broken.
* Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita
And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?).
* Update problems. Many data-sets change over time (e.g. GDP). How can I subscribe to updates? "git pull" would be nice.
* Provenance problems. I want to know who put which record into the dataset, when and why? "git log" would be nice.
* Presentation problems. (This is OK sometimes) I necessarily want to download 5Gb file before I looked into it. The first few rows of the dataset should be presented on the page, with information about it.
Wrote down a few more thoughts a while ago here: https://github.com/HeinrichHartmann/data-sharing#in-the-idea...
Approaches I have seen so far in the wild:
* figshare.com -- Addresses Hosting and Presentation.
* https://quiltdata.com/ -- (!) looks great. Still exploring.
* github.com -- works fine for small datasets (<1GB)
* packaging (yum, pkg, pip) -- (?) Not sure if that works, but at least they solve: Hosting, Update, Provenance.
This seems to be a wide open problem to me.
* Hosting problems. The first link I tried was already broken.
* Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?).
* Update problems. Many data-sets change over time (e.g. GDP). How can I subscribe to updates? "git pull" would be nice.
* Provenance problems. I want to know who put which record into the dataset, when and why? "git log" would be nice.
* Presentation problems. (This is OK sometimes) I necessarily want to download 5Gb file before I looked into it. The first few rows of the dataset should be presented on the page, with information about it.
Wrote down a few more thoughts a while ago here: https://github.com/HeinrichHartmann/data-sharing#in-the-idea...
Approaches I have seen so far in the wild:
* figshare.com -- Addresses Hosting and Presentation.
* https://quiltdata.com/ -- (!) looks great. Still exploring.
* github.com -- works fine for small datasets (<1GB)
* packaging (yum, pkg, pip) -- (?) Not sure if that works, but at least they solve: Hosting, Update, Provenance.
This seems to be a wide open problem to me.