That’s one issue.

Another is that even if you want to do so, it’s a staggeringly difficult enforcement problem.

What they’re calling for is basically an arms control treaty.

For those to work, you have to have monitoring and enforcement.

We have had serious problems even with major arms control treaties in the past.

en.wikipedia.org/…/Chemical_Weapons_Convention

The Chemical Weapons Convention (CWC), officially the Convention on the Prohibition of the Development, Production, Stockpiling and Use of Chemical Weapons and on their Destruction, is an arms control treaty administered by the Organisation for the Prohibition of Chemical Weapons (OPCW), an intergovernmental organization based in The Hague, Netherlands. The treaty entered into force on 29 April 1997. It prohibits the use of chemical weapons, and the large-scale development, production, stockpiling, or transfer of chemical weapons or their precursors, except for very limited purposes (research, medical, pharmaceutical or protective). The main obligation of member states under the convention is to effect this prohibition, as well as the destruction of all current chemical weapons. All destruction activities must take place under OPCW verification.

And then Russia started Novichoking people with the chemical weapons that they theoretically didn’t have.

Or the Washington Naval Treaty:

en.wikipedia.org/wiki/Washington_Naval_Treaty

That had plenty of violations.

And it’s very, very difficult to hide construction of warships, which can only be done by large specialized organizations in specific, geographically-constrained, highly-visible locations.

But to develop superintelligence, probably all you need is some computer science researchers and some fairly ordinary computers. How can you monitor those?

You can maybe tamp down on the deployment in datacenters to some degree, especially specialized ones designed to handle high-power parallel compute. But the long pole here is the R&D time. Develop the software, and it’s just a matter of deploying it scale, and that can be done very quickly, with little time to respond.