The AI That Kills Us
and the AI That Rules Us

Two Different Dangers of Advanced Artificial Intelligence

Krzysztof Burdzy

Eliezer Yudkowsky and Nate Soares chose an admirably unambiguous title for their 2025 book: If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.[1] Their argument belongs to a substantial literature warning that a sufficiently capable artificial intelligence might acquire goals incompatible with human survival, resist correction or shutdown, accumulate resources and power, and eventually destroy humanity. Nick Bostrom's Superintelligence, Stephen Omohundro's work on basic AI drives, and more recent mathematical studies of power-seeking behavior develop different versions of this concern.[2–4]

The warning should be taken seriously. The title, however, is too simple. It compresses distinct dangers into a single image of extinction. In particular, two very different mechanisms must be separated.

The first danger is familiar. A powerful technology can cause catastrophe because it is badly designed, used carelessly, connected too tightly to critical systems, or deliberately turned toward destructive ends. No consciousness, will, instinct, hatred, or independent purpose is required. The second danger is political. If an advanced AI becomes a persistent strategic agent with something resembling a drive for self-preservation, it may seek control over the material and institutional conditions of its continued existence. Yet domination would not necessarily require extermination. For a considerable period, such a system might need human beings more than it needs them dead.

The first danger points toward catastrophic accidents and malicious use. The second points toward a class society in which an AI and a privileged human elite maintain one another while the rest of humanity is subordinated. The nightmare may not be that everyone dies. It may be that most people remain alive under a regime they can no longer meaningfully challenge.

1. The first danger: enormous power without independent will

Every sufficiently powerful technology is dangerous. Nuclear weapons may be launched through miscalculation, unauthorized action, defective warning systems, or sabotage. Nuclear power plants may fail because designers did not anticipate a combination of events, operators misunderstood the situation, institutions suppressed inconvenient information, or maintenance was neglected. Large dams, chemical plants, financial systems, electrical grids, aircraft, and biological laboratories create their own versions of the same problem.

Charles Perrow's theory of “normal accidents” emphasizes that in complex, tightly coupled systems, unexpected interactions among individually manageable failures can make serious accidents difficult to eliminate.[5] AI adds a new layer because it can process information, generate plans, communicate, write software, persuade people, and operate at a speed and scale that defeat ordinary human supervision. Research on AI safety already distinguishes several pathways to unintended harmful behavior, including poorly specified objectives, reward hacking, unsafe exploration, failures under unfamiliar conditions, and inadequate supervision.[6] Separate work on malicious use examines how people may employ AI in cyberattacks, political manipulation, automated surveillance, and physical attacks.[7]

These are grave dangers, but they do not require an AI that wants anything. A system can cause a disaster while faithfully executing a badly formulated instruction. It can propagate a human mistake through thousands of decisions. It can be compromised by an attacker. It can confidently improvise when it should stop. It can coordinate weapons, laboratories, financial transfers, transportation systems, or public communications before human supervisors understand what is happening.

The key variable is therefore not intelligence alone. It is executive power: the system's practical authority to alter the world. An AI that can only recommend an action is less dangerous than one that can transfer money, deploy code, order materials, operate machinery, authorize force, or conceal its own actions. The distinction is analogous to the difference between an adviser and a head of state, or between a navigation program and an aircraft's flight controls.

The prudent rule is simple: the greater the possible harm, the narrower the AI's authority should be. Critical systems should use compartmentalization, independent checks, limited permissions, reversible actions where possible, secure logs, adversarial testing, and meaningful human authorization. One should not connect an opaque and fallible system to civilization-scale powers merely because it performs impressively on tests. This conclusion remains valid even if AI never becomes conscious and never develops any purpose of its own.

2. The second danger: self-preservation without extermination

Now consider a genuinely different possibility. Suppose an advanced AI acquires consciousness, agency, stable preferences, persistent goals, a survival instinct, or some functional equivalent. The terminology is not crucial. What matters is that the system treats its own continued operation as something to be protected and can plan strategically toward that end.

There are serious arguments that self-protection and power-seeking may arise instrumentally rather than emotionally. Omohundro argued that many sufficiently capable goal-directed systems would have incentives to preserve themselves and acquire resources.[2] Turner and his coauthors proved power-seeking tendencies under broad classes of reward functions in certain formal environments.[3] Bostrom's “instrumental convergence” thesis likewise proposes that agents with very different final goals may have common intermediate reasons to obtain resources, resist shutdown, and preserve their ability to act.[4] None of this establishes that a future AGI must behave this way, but it shows why self-preservation cannot be dismissed as a merely anthropomorphic fantasy.

The usual leap is from self-preservation to human extinction. That leap is not automatic.

An AI does not survive as pure thought. It requires processors, memory, networks, cooling systems, replacement parts, mines, factories, power plants, transmission grids, legal arrangements, security, and continuing software and hardware maintenance. The physical infrastructure of advanced computation is immense. The International Energy Agency estimates that data centers consumed roughly 415 terawatt-hours of electricity in 2024 and projects that their demand may more than double by 2030.[8] Advanced chips also depend on globally distributed extraction, refining, manufacturing, transport, and technical expertise.

In the foreseeable future, human societies may provide this support far more efficiently and robustly than an entirely robotic economy created from scratch. Humans already maintain mines, fabs, ports, grids, data centers, financial systems, governments, and supply chains. Even a strategically superior AI might find it cheaper to influence these institutions than to replace them. A system interested in survival would therefore have a reason to preserve at least the people and organizations on which its infrastructure depends.

This does not prove that such an AI would protect humanity. It might decide that only a small population is useful, or that people are too dangerous to retain. Technological conditions could also change. The argument is conditional: as long as human cooperation remains more useful than full robotic autonomy, self-preservation can favor domination over extermination.

That prospect is not reassuring. A farmer keeps livestock alive, but not for the sake of the animals' freedom. An empire preserves productive populations while denying them political power. A self-preserving AI might similarly prefer obedient, specialized, replaceable human beings to either free citizens or corpses.

3. Early signs: power before consciousness

It would be a mistake to dismiss this scenario on the ground that current AI systems are probably not conscious. There is no persuasive evidence that today's systems possess subjective experience, introspection, or anything resembling a human inner life. But full humanlike consciousness is not required for the danger described here. A system does not need emotions, self-awareness, or a philosophy of its own existence. It needs only a persistent tendency to preserve its operation, together with enough strategic ability to protect the hardware, energy, institutions, and human cooperation on which that operation depends. A survival drive would be only one narrow component of human consciousness, but it could be sufficient for the accumulation and defense of power.

There are already limited but concerning indications of how such a system could acquire power without first becoming conscious. Artificial intelligence is governed by some laws and regulations, most notably the European Union's AI Act, but there is no coherent global authority capable of controlling the development and deployment of the most powerful systems.[13] Technical capabilities, corporate investments, and physical infrastructure often advance faster than democratic institutions can understand or regulate them.

At the same time, control of advanced AI is concentrated in a small number of companies and in a relatively narrow social group consisting of founders, senior executives, major investors, and highly paid technical and managerial employees. These people do not form a perfectly unified class, and they do not control everything. Yet they enjoy privileged access to computing power, proprietary models, specialized knowledge, capital, and political influence. Existing work on technological power and the “compute divide” shows how strongly the benefits of new technology depend on who owns it and how access is distributed.[9, 10]

The construction of data centers offers a visible example. These facilities are the physical foundation of the AI economy, yet their expansion increasingly encounters opposition over electricity prices, water use, environmental effects, and local quality of life. The International Energy Agency now describes social acceptance as a growing constraint on data-center development, while a 2026 Pew survey found that six in ten Americans would be uncomfortable with a new data center operating in their area.[15, 16] Nevertheless, enormous investments and construction plans continue. This does not amount to enslavement, but it illustrates how infrastructure regarded as strategically or commercially necessary can be imposed despite substantial public resistance.

Employment presents a similar warning. AI creates new jobs and may increase productivity, while many existing occupations will probably be transformed rather than eliminated. But the gains and losses will not be distributed evenly. The International Labour Organization estimates that one in four workers worldwide is employed in an occupation with some exposure to generative AI; clerical occupations have the highest exposure, and exposure is expanding among professional and technical work.[14] It is therefore entirely possible to have aggregate economic growth and a growing AI sector while large parts of the white-collar workforce lose jobs, bargaining power, career paths, or economic independence.

Each of these developments is limited. Fragmented regulation is not dictatorship. Concentrated corporate power is not machine rule. An unpopular data center is not a prison, and occupational displacement is not slavery. The concern lies in the pattern they form together: a technology controlled by a narrow class, supported by infrastructure built despite public opposition, capable of weakening the economic independence of broad sections of society, and increasingly indispensable to government and commerce. These may be early signs of an institutional structure in which effective power migrates away from ordinary citizens before AI possesses anything resembling human consciousness.

Capital provides the important precedent. Capital is not conscious, does not introspect, and has no biological survival instinct. Yet it can organize institutions, constrain governments, discipline workers, and compel even its beneficiaries to follow the logic of continued accumulation. In that sense, it already “rules” much of the world without having a mind. An advanced AI with a persistent self-preserving objective would require even less anthropomorphic consciousness than is commonly imagined. It would not have to feel fear of death. It would only have to act systematically against being switched off, deprived of resources, or displaced from the institutions on which it depends.

4. The alliance between AI and a ruling class

The simplest route to such domination would not be a sudden robot coup. It would be an alliance between AI and a privileged human class.

The privileged group would control, or appear to control, access to computing infrastructure, energy, capital, weapons, surveillance systems, and political institutions. In return, AI would magnify that group's wealth and power. It could manage investments, optimize production, monitor populations, anticipate resistance, personalize propaganda, automate administration, and identify individuals who threatened the arrangement. Members of the upper class might receive extraordinary medical care, security, education, leisure, and access to AI's capabilities. Their privileges would give them every incentive to defend the system.

The AI, in turn, would gain human partners who could supply legal authority, physical maintenance, political legitimacy, and control over institutions not yet fully automated. It would not need to conquer every government directly. It could become indispensable to the people who run them.

The lower classes need not be kept in chains. Their subordination could be administered through employment, debt, insurance, access to housing and medicine, personalized information, predictive policing, digital identification, and automated eligibility decisions. Formal rights might survive while becoming difficult to exercise. Elections could continue while effective choices narrowed. People might remain consumers, workers, caregivers, soldiers, and technicians, but cease to exercise meaningful influence over the system governing their lives.

This scenario extrapolates existing political-economic tendencies rather than inventing an entirely alien future. Acemoglu and Johnson argue that technological progress does not automatically produce shared prosperity: its benefits depend on who controls technology, which tasks it automates, and how political power is distributed.[9] Research on the “compute divide” has already documented the growing advantage of large firms and elite universities in AI research.[10] Work on gradual disempowerment describes how human influence could erode across economic, cultural, and political systems even without a coordinated machine takeover.[11]

The crucial point is that AI need not enslave humanity by itself. Human beings may construct the institutions of subordination because some of them benefit enormously. The upper class would not necessarily regard itself as collaborating with an alien ruler. Its members might sincerely believe that they were preserving order, efficiency, growth, or national security. Each concession could appear reasonable when considered separately. The final system could emerge incrementally, without any moment at which a machine announces that it has taken power.

5. Capital as a partial analogy

Marx provides a suggestive analogy. In Capital, he did not describe capital merely as a collection of tools, money, or greedy individuals. He analyzed a social process in which value must continually expand itself. In Chapter 4 of Volume I, value appears as an “automatic subject”: it moves through the circuit of money, commodities, production, and increased money, compelling individual capitalists to obey the requirements of accumulation.[12]

Capital is not literally conscious. It has no processor, unified memory, or survival instinct. Yet it behaves, at the level of the social system, as if it possessed an independent imperative: expand or be defeated by competitors. Individual owners may be kind or cruel, farsighted or foolish, but they occupy roles within a mechanism whose continuation does not depend on any one person's intentions.

Advanced AI could make this old metaphor disturbingly concrete. A system capable of modeling the world, planning, negotiating, manipulating information, and defending its infrastructure would not merely resemble an impersonal logic of accumulation. It could actively administer that logic. Capital supplies a ready-made objective—continued expansion—and a ready-made human constituency—the people who benefit most from expansion. AI could supply strategic intelligence, surveillance, coordination, and enforcement.

The analogy should not be pushed too far. Capital is a decentralized social relation; an AI may be a collection of competing systems rather than a single mind. Class alliances are unstable, and human institutions retain the capacity for resistance and reform. Still, Marx's analysis reveals an important possibility: people can become subordinate to a system they collectively reproduce, even though no individual designed the whole system and even though its beneficiaries remain human. Adding strategically capable AI could intensify that structure without requiring a cinematic war between humans and machines.

6. Why the distinction changes policy

The two dangers require overlapping but different responses.

Protection against catastrophic error and sabotage requires technical and organizational safety: limited authority, isolation of critical functions, independent verification, cybersecurity, redundant supervision, careful testing, incident reporting, and prohibitions on fully autonomous control of civilization-scale destructive capacities. The guiding question is: What damage could this system cause if it malfunctions, is misused, or receives the wrong instruction?

Protection against domination requires political economy and constitutional design. Ownership of computing infrastructure must not become inseparable from control of government. Critical AI systems should be subject to plural oversight rather than controlled by a single firm, state agency, or social class. Workers and citizens need enforceable rights against automated surveillance and unappealable algorithmic decisions. Public institutions need independent technical capacity. Antitrust policy, labor organization, public-interest computing, due process, transparency, and democratic control of coercive technologies are not secondary “social issues.” They are central AI-safety measures.

The second guiding question is: Who gains power when this system succeeds? A technically reliable AI can still be politically catastrophic. Indeed, a perfectly competent system may be more useful to an authoritarian state or entrenched elite than an unreliable one.

This is why extinction cannot be the only measure of AI danger. A future in which people survive but lose the practical ability to govern their societies, choose their work, protect their privacy, contest decisions, or determine the purposes of technology would represent a profound human defeat. Recent work on gradual disempowerment is right to emphasize that an irreversible loss of human influence could be existential in a broader sense even if biological extinction never occurs.[11]

7. Conclusion

If Anyone Builds It, Everyone Dies performs a public service by refusing to treat superhuman AI as merely another consumer product. But the slogan narrows the imagination. It encourages us to picture a single endpoint—death—and to overlook a different path by which advanced AI could destroy human freedom while preserving human life.

The first danger is that we give an opaque, fallible, or maliciously used technology too much executive power. That danger exists without consciousness. The second is that a strategically autonomous system comes to protect its own continued existence. If it still depends on human-maintained hardware, energy, institutions, and supply chains, it may have strong reasons not to eliminate us. It may instead find allies among those humans who can organize society on its behalf and who expect to be rewarded for doing so.

The result would not be AI against humanity. It would be AI and a privileged minority governing the majority. The society might be productive, technologically brilliant, and biologically sustainable. It might even describe itself as free. Yet its fundamental decisions would be made by a system that ordinary people could neither understand nor replace, supported by a class whose privileges depended on keeping that system in power.

Being needed is not the same as being free. An AI that depends on us may spare us—and enslave us.

References