jules2689

The deploy queue usually gets to be a couple of "trains" deep which usually includes work from 4-10 devs. This represents a couple of hours. We have had issues with it taking too long, but this work I wrote about has improved that! We continue to try to improve it.

a ton of internal debug logs :) We log all state machine transitions, poll the state machine to make sure it stays running, and do a ton of checks in the process. The debug button is a timeline of all of that. Basically trace-level logs

Yes there was. The change would have been quite a lot to do at once, and we aren't ready to safely add that circular dependency yet. We aren't done on this path, and this certainly won't be the last iteration. I suspect that iteration will come at some point after we can figure out the circular dependency issue.

It also makes sense internally. Our actions team is much larger than the team that manages the project I wrote/lead, so it also makes logistical sense IMO

We also started before the CD product was ready on actions and we directly influenced it, so yea haha

It's actually longer than 5 minutes. There is the duration of the 2% canary deploy where we start to see pick up of traffic, a 5 minute wait, then a 20% "deploy", and a 5 minute wait. All in all this comes out to around 10-15ish minutes in canary. This is a stage where we can almost instantly shut off traffic to the canary deploy.

Could we reduce risk by lengthening the process? Maybe, but you also make deploys longer which means less stuff can get through in a day. This makes devs respond with larger PRs, for example, which increases the risk profile.

So we need to balance time and duration. Typically large problems will manifest quickly, or take a lot longer to detect (and thus are generally more minor problems) when you have our scale of a user base in my experience.

It's mainly the simplicity of the deployment system as it's inline and visible, coupled with habit. It all actuality that is just what _can_ trigger the deploy, the actual deploy is based on an internal deploy application and deploys can be triggered from there as well.

We monitor Datadog dashboards, exceptions, and other metrics mainly, as well as smoke testing the application

This is correct :)

There is some GitHub used, but as others stated we don't want to create a circular dependency on ourselves in case we deploy something that is broken.