I run https://oikolab.com - it's a weather data service with global, hourly historical weather data from 1950 to 16-day forecast. It launched about a year ago and I think it's probably the largest data offering of its kind.
I don't have dev ops experience (I still don't know much) so a lot of what I did was to exhaust the limits of the setup before implementing any new infrastructure, sort of learning as I go along, with many 'oh, so that's why they do this' moments. I've read about K8 but still can't see when I would need that though.
A lot of what saved effort for me was to use off-the-shelf component whenever I can. I was quoted $10~20k to build a Django website for user & API management to connect to my back-end but it was much easier to use Azure API Management service (~$400/month), which also came with simple, reasonable looking web front-end that I was able to launch within a week.
Before moving things to cloud (Digital Ocean), I've also exhausted what I was able to do with simple NAS servers (overall I processed about 200TB of raw data with ~1TB/day on-going using what I call a 'poor man's HPC' setup) - luckily I'm based out of Hong Kong so had access to fiber-optic internet at home.
I don't have dev ops experience (I still don't know much) so a lot of what I did was to exhaust the limits of the setup before implementing any new infrastructure, sort of learning as I go along, with many 'oh, so that's why they do this' moments. I've read about K8 but still can't see when I would need that though.
A lot of what saved effort for me was to use off-the-shelf component whenever I can. I was quoted $10~20k to build a Django website for user & API management to connect to my back-end but it was much easier to use Azure API Management service (~$400/month), which also came with simple, reasonable looking web front-end that I was able to launch within a week.
Before moving things to cloud (Digital Ocean), I've also exhausted what I was able to do with simple NAS servers (overall I processed about 200TB of raw data with ~1TB/day on-going using what I call a 'poor man's HPC' setup) - luckily I'm based out of Hong Kong so had access to fiber-optic internet at home.