PolarizedPoutin
Joined 8 years ago
Karma
321
Submissions
64
Member since
Sep 5
Recent submissions
Comment · 9 months ago
Comment · 2 years ago
Thank you for reading and for the links!
I'm trying out Clickhouse for the next post. Definitely excited for sub 24 hour data loading!
I haven't heard of VictoriaMetrics but that's some impressive performance. Will check it out!
Comment · 2 years ago
The data is publicly available!
The data is freely available from the Climate Change Service [1] which has a nice API but download speeds can be a bit slow. You'll have to sign up for this.
NCAR's Research Data Archive [2] provides some of the data (as pre-generated NetCDF files) but at higher download speeds. No signup necessary.
It's not super well documented but I hosted the Python scripts I used to download the data on the accompanying GitHub repository [3].
[1]: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysi...
[2]: https://rda.ucar.edu/datasets/ds633-0/
[3]: https://github.com/ali-ramadhan/timescaledb-insert-benchmark...
Comment · 2 years ago
What do you mean by the heat distribution of energy? Do you mean like how much heat is received from the sun at a particular location and particular time? If so, then temperature is only the result of this plus other factors. Factors like cloud cover influence how much radiation is received at the surface.
Comment · 2 years ago
GraphCast was trained on this exact same data!
From https://deepmind.google/discover/blog/graphcast-ai-model-for...
> Crucially, GraphCast and traditional approaches go hand-in-hand: we trained GraphCast on four decades of weather reanalysis data, from the ECMWF’s ERA5 dataset.
Comment · 2 years ago
The full dataset is quite huge (~9 petabytes and growing) out of which I'm using just ~8 terabytes.
The data is freely available from the Climate Change Service [1] which has a nice API but download speeds can be a bit slow.
NCAR's Research Data Archive [2] provides some of the data (as pre-generated NetCDF files) but at higher download speeds.
It's not super well documented but I hosted the Python scripts I used to download the data on the accompanying GitHub repository [3].
[1]: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysi...
[2]: https://rda.ucar.edu/datasets/ds633-0/
[3]: https://github.com/ali-ramadhan/timescaledb-insert-benchmark...
Comment · 2 years ago
The full dataset is quite huge (~9 petabytes and growing) out of which I'm using just ~8 terabytes. Still quite big to upload.
The data is freely available from the Climate Change Service [1] which has a nice API but download speeds can be a bit slow.
NCAR's Research Data Archive [2] provides some of the data (as pre-generated NetCDF files) but at higher download speeds.
It's not super well documented but I hosted the Python scripts I used to download the data on the accompanying GitHub repository [3].
[1]: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysi...
[2]: https://rda.ucar.edu/datasets/ds633-0/
[3]: https://github.com/ali-ramadhan/timescaledb-insert-benchmark...
Comment · 2 years ago
Yeah I was thinking about this and hoped that Postgres had a `float2` data type but `int2` would have to work. I could scale the numbers to fit them into 2 bytes with minimal loss of precision, but decided I'd rather take the storage space hit since TimescaleDB promises good compression. Still haven't measured this though haha.
Comment · 2 years ago
It's a good question! It's true that the output is massive, I believe ~9 petabytes and growing. But running the model is super expensive. It runs on ECMWF's supercomputer. Not sure how many cores but I would guesstimate in the 10k - 100k core range just for one instance. So computing the data on the fly is super expensive, probably much more expensive than just storing it.
Comment · 2 years ago
I'm hoping to compare TimescaleDB and Clickhouse to see how big the difference is for different queries! My impression is that TimescaleDB gives you some columnar features, but maybe Clickhouse is a true columnar database.
Comment · 2 years ago
Had a read through parts 1 and 2, thank you for the engaging reads! Love how you've formatted your posts with the margin notes too. Thank you for providing the function to write numpy structured arrays to Postgres binary, I couldn't figure this out before.
Comment · 2 years ago
Thank you for reading through and for your feedback! Excited to try your settings to disable the WAL and other overhead and see if I get even faster inserts.
Also glad to hear an expert say that WAL data isn't really necessary for bulk loading, especially with chunks. I should get through the ~20 days it takes to load data without a power outage haha (no UPS yet :() but it sounds like even in the worst case I can just resume.
Comment · 2 years ago
Hoping to find out when I compare TimescaleDB vs. Clickhouse!
Comment · 2 years ago
Haha simplistic but probably faster and more space-efficient than a relational database. Sounds like rabernat and open-meteo who commented here do something similar to you and find it fast as well!
I recently redesigned my personal website to mimic Earth’s layers and also started blogging. Thanks for putting this together, I love browsing through unique personal websites!