I applaud the transparency of the GitLab team in their recent outage, but felt bad for the engineer who's typo was called out. Anyone who's done something similar will know the feeling immediately after realising your mistake...
To show that this sort of thing happens to the best of us, let's share some of our horror stories :)
A few months ago, I joined a new team and was still finding my way around the environments. I was tasked with performing manual deployments to a Dev, QA and Staging environment that weren't wired up to our automation system yet. We'd scheduled maintenance windows a week apart for the QA and Staging envs as we allow customers to test against these.
So the day of my QA deployment, I start by applying the database changes which all complete successfully.
Next, I upload the new .ear files and deploy the new build of our web app. Again, all looks good, so I tell the QA team they can start testing.
Then the alerts started...
I deployed the app to the Staging env by mistake (and unexpectedly restarted the app server). I didn't realise the naming scheme of the hostnames indicated the environment in this case :/
Our UI broke immediately due to the schema changes, so my mistake was _very_ visible. I was lucky I could roll back the change easily, but I don't think I'll forget that day any time soon.
To show that this sort of thing happens to the best of us, let's share some of our horror stories :)
A few months ago, I joined a new team and was still finding my way around the environments. I was tasked with performing manual deployments to a Dev, QA and Staging environment that weren't wired up to our automation system yet. We'd scheduled maintenance windows a week apart for the QA and Staging envs as we allow customers to test against these.
So the day of my QA deployment, I start by applying the database changes which all complete successfully. Next, I upload the new .ear files and deploy the new build of our web app. Again, all looks good, so I tell the QA team they can start testing.
Then the alerts started...
I deployed the app to the Staging env by mistake (and unexpectedly restarted the app server). I didn't realise the naming scheme of the hostnames indicated the environment in this case :/
Our UI broke immediately due to the schema changes, so my mistake was _very_ visible. I was lucky I could roll back the change easily, but I don't think I'll forget that day any time soon.