bdimcheff
Joined 17 years ago
Karma
281
Submissions
33
Member since
Nov 25
[ my public key: https://keybase.io/bdimcheff; my proof: https://keybase.io/bdimcheff/sigs/XkOC1XfG-gygc18wBCz2so47D88MWio5ymI1bThE-08 ]
View on Hacker NewsRecent submissions
Comment · 7 years ago
Our backend is 100% go, and we're happy to answer any questions about tech or insurance.
Comment · 7 years ago
Openly, Inc | Backend Engineer (mostly go) | Ann Arbor, Boston | Remote(US) or Onsite, Full Time | Salary, Benefits, Equity | https://openlyinsured.com
We’re a currently 7-person insure-tech startup with 3 engineers: me in Ann Arbor, the CTO (one of the cofounders) in Boston, and someone who works remotely from socal. We’re building an insurance company that uses technology from this century to make insurance agents’ lives easier. We're obviously a small team, so you'd be able to have a large impact quickly.
We intend on staying a remote-first team, but we're also trying to start a small Ann Arbor office if we can find the right person to join me here. Having a physical presence in a2 will allow us to hire interns/juniors and mentor them and such, which is more difficult to do remotely.
More details at https://www.welovegolang.com/jobs/software-engineer-57359861.... I know it says Ann Arbor only, but you can ignore that.
Comment · 8 years ago
RBAC itself is enabled by default on new gke clusters afaik, and has been enabled on ours for months. We're a small team so we haven't tried to do anything particularly complex with RBAC inside the cluster, but I can't say I've run into anything particularly troublesome.
Comment · 8 years ago
We use GKE at Olark for the last year and have been very happy with it.
Comment · 8 years ago
Yeah I think that there is still some potential in the terminationGracePeriod strategy, but we found this other way that worked reliably and stopped exploring that path. If I can repro the issue I'll let you know.
One extra thing I remember that was sort of problematic was that when a pod was Terminating it'd get removed from the Endpoints, so any tooling that was using the API info to keep an eye on connections was basically unusable at that point.
Comment · 8 years ago
One thing I didn't put in here that's also turned out to be useful: We can prerelease things relatively easily this way too. Each deployment has a git sha, and we can have a canary/beta/dogfood version that points at an entirely different sha.
Comment · 8 years ago
In our case it's not the websockets that are the problem, it's the XMPP connection that each websocket connection creates. Logging in thousands of users takes several minutes. While a user reconnects, any conversations that the users are having with their website visitors are disrupted.
Comment · 8 years ago
Yeah I think if restartPolicy were changeable at runtime, we could simply have the pods exit once their connections are drained enough. If we were to exit using the current strategy, they'd just be restarted by kube.
Comment · 8 years ago
yeah I don't know why terminationGracePeriodSeconds hacks didn't work. It could have been a different, unrelated factor that we didn't discover. It certainly could have been service-loadbalancer/haproxy's fault instead of the termination grace period itself. I'm certainly happy to be proven wrong there.
Comment · 8 years ago
We force reconnects eventually, there just aren't that many people affected at that point. There's a very long tail of people keeping their browsers open for days, but it's only a handful of people.
Comment · 8 years ago
We actually have this functionality as well: we can send a signal to the process which will cause it to display a message to the user asking them to reload to upgrade. There is a similar feature in slack and riot that I've seen as well.
Making this handoff automatic is definitely possible as well, though we do want people to reload occasionally to get new client-side code.
Comment · 8 years ago
The way this Rainbow Deploy works means that if I only deploy once a month, I only have a single "color" running for that whole month, plus couple days of overlap where there are 2. If I have blue/green, I have 2 colors running all month. If I have more fixed colors, even more. The sha thing is just a convenient way of creating "colors" dynamically when we need to do a new deploy, without having to use a meaningless representation like "blue", "green", "taupe", "chartreuse", etc.
Comment · 8 years ago
Yes, as lotyrin pointed out below, the backend is already effectively an XMPP <-> Websockets bridge. Every time a user logs in, one way or the other we need to establish a connection between the websockets backend and the XMPP backend. The backend does do other things besides simply proxy, and we certainly could separate out the part that needs to keep the state. This deployment strategy is effectively a way to avoid having to do that work, at least for now.