nlcs

I fear the current economic and political situation of “too big to fail” the most, and I think this will be the main reason why no real safeguards will be implemented. The only reason a proper safety concept may eventually be introduced is because it will be written in blood, and I fear that by then it will already be too late.

An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.

But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.

If safety isnt required, most companies wont implement it voluntarily. Once something becomes safety relevant, you need a safety concept, failure rate calculations, defined safety functions, verification, etc. Even a relatively simple safety subsystem in a consumer controller can suddenly mean thousands of pages of documentation and years of development to reach the required ASIL or PL.

A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.

Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.

OpenAI and other frontier labs won't ever introduce safety-level standards like those used for railways or nuclear plants until they are forced to do so by customers or by law. The reason is simple: safety is expensive, and if safety is introduced properly, development is no longer mainly about how to implement feature A. Instead, it becomes much more about how to design two or more redundant systems to implement feature A safely, while also documenting everything clearly and having it audited by an independent auditor.

So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.

Job market is currently really bad, it has never been worse. Two years ago, it was almost impossible to find an expert for a more specialized domain like computer vision or RTOS. Now, it’s impossible not to receive applications from multiple experts for a single role (and that’s only counting experts; senior and junior software developers or architects aren’t even included) that isn't even a sepcialized role and at best, "just a simple" senior role.