I'm in the exploratory stages of starting a noncommercial project focused on hosting, processing and serving a vast amount of climate data for scientists and researchers. I've "seeded" this project with my own bare metal hardware, electricity and internet access. This includes 136 vCPUs, 512GB of RAM and 128TB of hard drive storage capacity (with room for another 456TB). As my own money permits, I plan to slowly add to this lab over time.
My initial consideration is to develop and host a sort of public Wolfram Alpha, but specialized for climate data queries and visualization. I'm open to other suggestions, as I'm currently in the process of architecting the system and reaching out to climate scientists for feedback on what they need that isn't well-served yet.
Once all the infrastructure work is done, I'd be happy to contribute to climate modeling more theoretically. But for now my prerogative is to develop a central repository for as much climate data as possible, normalize it as reasonable, and store it in a queryable format.
There are interesting technical problems here due to the scale of the data, the variety of its native (raw) formats, the frequency with which each source updates and the types of information involved. But speaking directly to the title of this post: I don't think anything I'm doing is strictly an unsolved technical problem, it's just a complex technical undertaking. There are plenty of data serving and processing pipelines for this that have already been proven capable.
My initial consideration is to develop and host a sort of public Wolfram Alpha, but specialized for climate data queries and visualization. I'm open to other suggestions, as I'm currently in the process of architecting the system and reaching out to climate scientists for feedback on what they need that isn't well-served yet.
Once all the infrastructure work is done, I'd be happy to contribute to climate modeling more theoretically. But for now my prerogative is to develop a central repository for as much climate data as possible, normalize it as reasonable, and store it in a queryable format.
There are interesting technical problems here due to the scale of the data, the variety of its native (raw) formats, the frequency with which each source updates and the types of information involved. But speaking directly to the title of this post: I don't think anything I'm doing is strictly an unsolved technical problem, it's just a complex technical undertaking. There are plenty of data serving and processing pipelines for this that have already been proven capable.