kccqzy

Property based testing is great because it’s implemented as a library in many languages. You don’t have to rewrite your codebase in Lean.

Every incident having a root cause is required. If your organization doesn’t do this, it basically means they don’t value reliability at all. The best SREs quit. Well either that or the SREs have the ability to transfer oncall responsibility to SWEs. Whether or not that can be fixed at the origin is an entirely different matter. Bug-free code doesn’t really matter: the best incidents are the kind that reveals hitherto unknown bugs.

Google’s AI mode is roughly the same as using the free ChatGPT web app (both without logging in). Of course they use a lower-intelligence model for free users. They are certainly worse than real AI subscriptions. I do the same thing you did: only use it for basics and low-stake things, and it’s good enough for basics.

I stopped reading halfway because it doesn’t describe competent SREs in an organization where the culture values reliability.

> Alert noise reduction and intelligent grouping, runbook execution for known issues, log pattern detection and anomaly flagging, and basic root-cause suggestions from historical incident data, and auto-remediation for well-understood failure modes like restarting a crashed pod or scaling up a service that’s running hot, are all fairly automated today.

Let’s stop here and ask, why do we not fix the underlying issue? Alert noise? Design better alerts. Runbook execution? Make it a cronjob. Known issues? Why aren’t people fixing these known issues? Root cause suggestion from historical data? Sorry if root cause A is in the historical data you expect problem A is fully solved so it wouldn’t appear again in the next incident. Restarting things? Why doesn’t your health monitor restart things automatically?

SREs also need to climb up the abstraction ladder. Instead of using AI to assist with grunt work, use AI or human thinking to build more reliable systems with fewer incidents. Every incident should be novel.

You should try staying at small boutique hotels. They tend to just sign up for a regular business internet and you’ll get IPv6 that way. Large chain hotels want central management and their management isn’t modern enough to support v6.

How? Since when do banks care about IPO crashes? A lot of IPOs crashed after listing; I can’t remember a single time when a bailout was needed after an IPO.

Nope. The balance sheet of big tech companies are healthy enough not to need to bailout anyways. AI startups can of course fail.

Break glass systems are very good but they are also challenging to make. It requires at minimum a system to get approvals from n other people and a system to do logging. It means your authentication and authorization systems must both be working, at least a small amount of non-volatile storage is available, and this often implies the availability of more foundational services such as service discovery, name lookup, distributed consensus, and replicated storage.

The problems can be overcome, but management types may balk at the cost of a complete break glass system.

> when something breaks the app will tell me.

Then you have to require these permission controls to tell the app the user denied permission, and not just giving a valid but empty result. Some people on HN feel strongly that when you deny contacts permission to an app, the app should receive a list of dummy contacts. Without it a bad app can lock its UI and not do anything unless the permission is granted.

Non-sandboxed apps have this permission prompt too.

This is why I prefer to use C++ and its reversed iterators. The rbegin and rend accessors are there in the containers from the beginning. It’s less error prone than the C version using indices. (Of course some situations require me to use indices, so I do use the i-->0 trick, but it is much rarer.)

In America sometimes construction of mass transit requires so much money that it is less cost effective in terms of riders served on a per-unit-cost basis than subsidized taxi. A lot of that tax money goes to intermediate consultants who come up with extravagant station designs that seem to value grandeur more than utility. Another large chunk of that tax money goes to union workers when equivalent non-union workers are much cheaper to hire, and when fewer non-union workers are needed to do the same job than union workers. The sheer size of mass transit projects means they become even bigger. I don’t like this but it is what it is.