I once shut down an algorithmic trading server by hastily typing (in bash):
- Ctrl-r for reverse-search through history
- typing 'ps' to find the process status utility (of course)
- pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh)
I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the server and resume trading in half a minute or so.
That's $250 million to $400 million of orders on pause for half a minute. Not to mention my heartbeat.
Renamed stopserver.sh to stop_server.sh after that incident :|
P.S. typing speed is not merely overrated, but dangerous in some contexts. Haste makes waste.
- Ctrl-r for reverse-search through history
- typing 'ps' to find the process status utility (of course)
- pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh)
I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the server and resume trading in half a minute or so.
That's $250 million to $400 million of orders on pause for half a minute. Not to mention my heartbeat.
Renamed stopserver.sh to stop_server.sh after that incident :|
P.S. typing speed is not merely overrated, but dangerous in some contexts. Haste makes waste.