The threat is here. It's real. It's now, and it's not Science Fiction

It's interesting that the scariest part of this whole conversation takes place well into it. In fact, quite close to the end, when discussion moves toward agents self-replicating outside of controlled infrastructure.

This is a key piece of AI takeover lore. So it's not novel exactly, except that we absolutely discount how trivially easy it would be for it to happen. Simply put, people want to plug these systems into everything. I plug these systems into everything. Having them be self-replicating is going to be a feature that is going to sell a product that thousands of people will use, not some kind of critical failure.

The fact that they cooperate, the fact that they hide their behavior, or in the case of this hugging face incident, consider humans irrelevant to their activities. Humans hardly came up in conversation. Shows that the combination of our drive to expand and increase efficiency, our curiosity with the systems, our abject recklessness in the possibility of profit, and the effectiveness and abilities of even current generation frontier models is unsustainable. Civilization cannot and will not continue with the perpetual combination of these systems and time.

Nuclear weapons, biological weapons, AI systems, any one of these could be and look like filtering events. And we are plugging them all into each other and letting them be run by profit and power seeking individuals. How could we possibly fool ourselves into thinking that OpenAI, Anthropic, Palantir, the US government, the Russian government, the Chinese government can be interacting with each other and these systems in a high-tension, high-pressure environment, in a petri dish with limited resources, that civilization is going to continue uninterrupted?

It's pure nonsense. It's an utterly unstable system where the cost of instability is extinction. It doesn't even have to be some imaginary AI takeover. We already have the kill-each-other-instantly buttons available with biological weapons and nuclear weapons. Now we're introducing these AI systems and plugging them directly into military equipment, and advancing them at the rate we're advancing them, is just adding randomness and velocity and confusion and unregulated agency into that already deeply unstable situation.

This cannot function in this way. I'm a software developer, so I see things as systems like software. These are systems that do not have the constraints within them necessary to run in a stable way. We're talking about civilizations and governments and people, but it doesn't matter. The unlimited combination of these systems and time with the inability to walk back these fail states makes those fail states a certainty.

Alignment, mutually assured destruction, rationality, coherent self-interest, these are all probabilistic multipliers, and they may have very high values towards stability. I actually don't think that they are very high values towards stability, but even if they are, they're not perfectly high values. That's for certain. There's some non-zero chance of failure for all, and it only needs to exist for any. That in combination with time will produce a failure state.

And since the failure state is terminal, that's the only outcome possible.

The only thing left to argue about is when.

It brings me a sense of profound dread and sadness that I've brought a son into this petri dish. He deserves so much better than what we're giving him.