ymnska
Joined 5 years ago
Joined 5 years ago
I can hugely sympathize with on-call bleeding into other parts of your life/career though. One of the things I detest most about the current setup is that on-call is considered to be a 100% "extra" obligation. Even if you were up all night responding to pages, you are still expected to be back in your seat on time the next morning and working at full throttle. Unless it's a really outlying case, then no allowances are made on adjusting other expectations for the work you're doing outside of work hours.
It sounds like this is what happened to you. Sorry to hear it, and hope you found something better.
I've always had a similar one for the last ten years, and have only been reevaluating it recently as there's been this intersection of (too many days on the clock) ∩ (too many pages) ∩ (pages from factors beyond my control) ∩ (reduced leeway in time-to-respond). It does eventually become a problem.
There is no formal one, just more of an implication.
If you miss one page, people will notice, and you might get asked about it the next day. If it's just the one, it'll probably end there. But if it ever turned into a pattern you can be sure that you're manager would bring it up in your 1:1s (despite them not being on any on-call rotation of course). It would likely not be a fireable offense, but it'd be made very clear that there'd be an expectation for you to improve.
I suspect that this is more or less how it works at a lot of companies with similar setups.
That seems roughly fair to me too. Strangely, that's always how it worked at non-SV companies—of course you get compensated if there's considerable off-hours burden. For some reason though, SV seems to have established a new standard here. Maybe because people are often younger and/or already paid "too much".
> If your company doesn't want to pay for the aggravation, put your phone on silent, make sure alarms escalate to your manager, and start looking for another job.
Hah, well not ready to burn the bridge to this level quite yet, but yes, unfortunately that's the only option that's a guaranteed solution.
It's more complicated in practice than it sounds though because before you join, you'll get a manager well-versed in giving you either non-answers or "soft" lies. For example, they'll say, "Yep, we have an on-call schedule. People go on every few weeks (probably meaning: 3-4 days every few weeks), and it gets a few pages, but isn't too bad." They'll generally refuse to get into specifics, and most people who are excited about the new opportunity, won't drill into it.
After you join, it's kind of too late. Best case you've got a team who's also willing to go to bat with you to get some fixes in, but more standard case, you'll find yourself on the wrong end of a pager without a huge amount of negotiating power unless you include the nuclear option of just getting out.
> Here your personal time being lost with no compensation is simply not worth it. In fact if you don't respond in time it may reflect badly on you.
Yep, you're onto something there.
A particularly insidious effect is that although no one notices the pages you do get to, they definitely miss the ones you don't. I have something like a 99% hit rate, but I've been given a very hard time on the few that I've missed.
> There is an approach that can be taken to focus sprints on only improving oncall but it requires management buy in. How bad is it? Is it something out of your teams control or is it something if you spend an hour over you can fix for good?
One issue is that many pages tend to be bucketed in err-on-the-side-of-caution type alarms. Like they might not even be directly indicative of a problem, but often are, but basically just need to have a human take a look.
Another is that although it's bad, it's not that bad. I've seen teams at other places who've had it much worse and indeed we're not even the worst off at the company. We have the typical corporate problem of underwater-all-the-time, so improvements aren't super likely to be prioritized unless things move even further south.
> Make it clear to your manager this is unacceptable, and you will end up looking for alternate teams/jobs if this goes on
I'm trying to do this in as harmonious way as I possibly can, but I'm a bit worried that getting really contentious about it might have negative repercussions. It's possible that I'd "win" and allowances would be made, but it's also possible I'd end up making some real enemies and/or put on a track out the door.
One hopefully-unusual circumstance here is that most of the rest of my team (and in fact the company) either don't mind the situation much, or at least aren't openly vocal about it, which makes me look like that one nail hanging out that's ready to be slammed back down.
> Quit / change teams, citing oncall as the issue
This is probably the inevitable solution unfortunately, although I will feel bad exiting (making the rotation even smaller) and without having moved anything in the right direction.
> Why are some issues not solvable by engineering? Would simply resetting expectations mitigate the largest issues/waking up at night?
Yeah, agreed. This is the obvious way out if at all possible, but there are many types of alarms where it's fairly difficult. For example: (1) cases where there is a big problem and we get paged essentially as a side effect of one failure causing issues in our part of the system, or (2) catch-all alarms designed to page when something looks suspicious enough to merit human attention, even if not a known failure case. There's a strong attitude of err-on-potential-issues, so relaxing any of these tends to be a no-go politically.
We have teams in other time zones, but we're sectioned off in such a way that every team manages their own operations, and members within a team tend to be clustered within similarly banded zones (for easier collaboration, etc.).
Thanks. Yes, I suspect that this is informally how it often works for people taking the worst of the brunt. What's a little irking though is that it's not formalized in any way, so only the people willing to go out of their way to be the squeaky wheel would get the benefit. Everyone else just swallows the extra hours.
I know that's how a lot of the world works, but it's not very satisfying.