Below is the executive summary of our new paper, The Manhattan Trap. Please visit the link above to see the full paper. We also encourage discussion and feedback in the comments here.

This paper examines the strategic dynamics of international competition to develop Artificial Superintelligence (ASI). We argue that the same assumptions that might motivate the US to race to develop ASI also imply that such a race is extremely dangerous.

A race to develop ASI is motivated by two assumptions: that ASI provides a decisive military advantage (DMA) to the first state that develops it, and that states are rational actors aware of ASI's strategic implications.

However, these same assumptions make racing catastrophically dangerous for three reasons.

First, an ASI race creates a threat to strategic stability that could trigger a war between the US and its adversaries, particularly China. If ASI could provide a decisive military advantage, states would rationally view their adversaries' ASI development as an existential threat justifying military intervention and espionage. A state cannot race to ASI without incurring significant risk of great power conflict unless either (a) its adversaries are unaware of ASI's importance (contradicting the awareness assumption), (b) development can be kept secret (likely impossible given the scale required), or (c) adversaries believe they can win the race (requiring a close competition in which victory is not assured).

Second, racing heightens the risk of losing control of an ASI system once developed. We do not take a position on the likelihood of loss of control. Instead, we observe that the argument for racing assumes that ASI would wield a decisive advantage over the militaries of global superpowers; accordingly, losing control of such a technology would also present an existential threat to the state that developed it. We also argue that an ASI would only provide a DMA if its capabilities scaled extremely rapidly—precisely the scenario in which loss of control risk is theoretically highest. Finally, a race compounds this risk by creating competitive pressure to develop ASI before adequate control measures are in place.

Third, even a controlled ASI threatens to disrupt the internal power structures of the state that develops it. In the case of a US, a successful ASI project would likely undermine the liberal democracy it purports to defend. An ASI system that would grant a state international advantage would also grant its controllers unprecedented domestic power, creating an extreme concentration of power likely incompatible with democratic checks and balances. Racing exacerbates this risk by requiring development to occur quickly and in secret—and therefore without public input.

These three dangers—great power conflict, loss of control, and power concentration—represent three successive barriers a state would need to overcome to 'win' a race to ASI. Together, they imply that an ASI race poses an existential threat to the national security of the states involved. Assuming states are informed and rational, the strategic situation can be modeled as a trust dilemma: states would prefer mutual restraint to racing and will only race if they believe others will. States can therefore avoid an ASI race by establishing a verification regime—a set of mechanisms that can verify a state's compliance with an international agreement not to pursue an ASI project. An ASI project would be highly distinguishable from civilian AI applications and not integrated with a state's economy—precisely the conditions under which verification regimes have historically succeeded.

The paper concludes that, if an ASI race is motivated, then cooperation to avoid an ASI race is both preferable and strategically sound. The assumptions that make racing seem necessary are the very reasons it is unwise.