I was working support at a VPS provider for my first real-world tech job fresh out of college and a customer was having issues with their system not booting correctly. They were smart enough to use our integrated backups service so I told them that they could delete their current disks and restore from backup. So they did...
Or at least they tried to.
The backups system was incredibly wobbly at the time and would corrupt its archives pretty frequently which is exactly what happened. They lost everything on that server.
Did I mention that was their sole server and they had no other backups?
It turned out that they were a company providing services to a government entity and had some pretty strict record-keeping requirements which they relied on our service to fulfill.
I was freaking out thinking I was going to be fired after being there for less than a year but everything was resolved fairly well (somehow).
I learned to never trust backups and the rule of thumb "two is one, one is none" as it applies to them.
Or at least they tried to.
The backups system was incredibly wobbly at the time and would corrupt its archives pretty frequently which is exactly what happened. They lost everything on that server.
Did I mention that was their sole server and they had no other backups?
It turned out that they were a company providing services to a government entity and had some pretty strict record-keeping requirements which they relied on our service to fulfill.
I was freaking out thinking I was going to be fired after being there for less than a year but everything was resolved fairly well (somehow).
I learned to never trust backups and the rule of thumb "two is one, one is none" as it applies to them.