Comment
Comment by tharkun__
No you can not do this in all cases and I described multiple cases in my comment already in which you can't reasonably do anything automatic. Let me explain.
But why would I do that for every `StatementTimeoutException` at every moment of the night and why do I need to bake that into my error handling? That isn't actually handling the error.
Please never ever do this to either your users, who may believe it or you "system administrators" who do value their sleep or have other more pressing matters to attend to.
I described a case in which automatically adding resources would actually be wrong in that it would completely mask the actual problem, which is that the developer did not think about the access patterns of the software they wrote enough and did not add the right index to the database. If you keep scaling your database automagically you'll pay AWS until you run out of money and have not solved anything. Believe me, a missing index can eat up a lot of resources before anything gets better. And until then your software will just keep failing and keep adding resources. And in this case it won't help at all because your new machine will not even be used by the query. It'll still only be one node handling your read and that is constrained by actually reading data from the disk and that's super slow because you forgot the index!
I've yet to see anything that the developer should do here based on an individual error in the software. One case where something should happen automatically on the database would be if the database was running out of space. You should have monitoring in place that takes care of that. And no it should not be your software doing that scaling because it received a `SQLError -> DatabaseOutOfDiskSpace` error. If it gets that far, all of your calls to the database will fail. Which of your error handlers should be the one handling it and why should we let things get that far in the first place? Have monitoring set up outside of your actual software that adds disk space automatically before you run out of space and then make it scream very loudly to your system administrators and developers about it, so that they can take a look at it and determine if this was a legitimate "well I guess we got too many more paying customers now, this was OK" or if it was the last update that went out having a bug that keeps filling up the database with BS data and you need to make an emergency bug fix or maybe you're deliberately being DDoS'd and it got past your DDoS protection and you better do something about that or the DDoS'er is gonna make your AWS bill go crazy.
Again, nothing and nobody cares about up-stack. I care about down-stack when handling such errors.
Nothing to do with agile at all. You should think about error handling every time you code anything. For example as we've seen above, think ahead about your database running out of disk space. But don't make every developer think about it for every single piece of code they ever write. That makes no sense to have them try to handle these situations. It'd actually make it worse and these people would never get any actual work done either.