I learned early on, it's all about proactive monitoring, 'self healing' (e.g. restart on fail) services, and simplicity in everything.
I've been running between 300-500 Apache Solr search cores continuously for almost 5 years, starting with 1.4, migrating up to now 4.10.4 (5 and 6 are in the works), all with around 99.98% uptime (averaged on all the servers over time).
I have detailed logs and per-search-core stat tracking (queries, index size, query time) for the past 3 months and archives back further, and I have only had to remove noisy neighbors a few times (and did so quickly, by first isolating them on their own VM, then helping them move off to dedicated resources if needed).
Also, I explicitly state I offer no SLA in the support docs—some people are okay with that.
But yes, I'm always on call, technically. I've only had to fix 'emergency' scenarios about 3 times in the past 5 years though. Even security updates are automated via Ansible/Jenkins. I just need to log in and click a button and things are updated, or a new server is built.
I highly recommend Google's new book on SREs; whether a one man shop or a multi billion corp, the learnings are exactly the same!
I've been running between 300-500 Apache Solr search cores continuously for almost 5 years, starting with 1.4, migrating up to now 4.10.4 (5 and 6 are in the works), all with around 99.98% uptime (averaged on all the servers over time).
I have detailed logs and per-search-core stat tracking (queries, index size, query time) for the past 3 months and archives back further, and I have only had to remove noisy neighbors a few times (and did so quickly, by first isolating them on their own VM, then helping them move off to dedicated resources if needed).
Also, I explicitly state I offer no SLA in the support docs—some people are okay with that.
But yes, I'm always on call, technically. I've only had to fix 'emergency' scenarios about 3 times in the past 5 years though. Even security updates are automated via Ansible/Jenkins. I just need to log in and click a button and things are updated, or a new server is built.
I highly recommend Google's new book on SREs; whether a one man shop or a multi billion corp, the learnings are exactly the same!