After repeating warnings that AI could “end humanity” from researchers, capitalists signed a "morally binding" safety pact with zero legal consequences for breaking it.
Details. On September 9, Anthropic researcher Jacob Coxon resigned, accusing his employer of “gambling with our lives” by “racing straight to self-improving superintelligence”. Another Anthropic researcher, Evan Hubinger, backed his warning, putting the chance of AI wiping out humanity within a decade at over “10%”. In July, more than 1,000 employees at major AI companies had already signed a petition calling for measures to slow AI development.
► Days after Coxon’s tweet, AI capitalists raced to join in, raising “concerns” whilst continuing work as usual. Anthropic CEO Dario Amodei called for the industry to “pace the frontier”, meaning coordination among frontier AI firms to set safety standards and limit unchecked AI development, and international cooperation to manage AI risks. OpenAI CEO Sam Altman and xAI’s Elon Musk backed calls for slower AI development.
► Reports of AI agents failing to follow instructions, along with researchers' warnings, have fuelled public concern and media attention. A poll published on September 22 found that 73% of Americans thought AI companies were doing too little to prevent serious harm. A record 39% viewed AI's impact on society negatively, while just 11% viewed it positively.
► On September 29, Trump brought AI executives together to sign the “White House Accord on Super Intelligence”. It calls for internal monitoring, oversight teams, external audits and board supervision, but sets out no legal penalties for companies that fail to comply. Trump called the agreement “morally binding” and praised the industry's supposed “tremendous self-policing.”
► Days before the signing, Trump made clear which priority actually guided the monopolies approach, dismissing calls to slow development: "We're leading China in AI... and frankly, I want to keep it that way, because whoever wins AI wins."
Context. Most researchers' concerns are not about AI becoming conscious. They centre on theoretical systems capable of self-improvement (AI that can write better AI software without human involvement), developing behaviours we cannot understand, predict or fully control, as seen in recent hacking incidents, and potentially causing huge damage through cyberattacks, economic disruption or the development of biological weapons.
► Despite calls for caution, AI companies continue to compete to develop more powerful systems, racing towards the goal of “self-improvement”. The White House accord requires monitoring and oversight but does not require companies to halt or slow development.
► Warning about catastrophic AI risks can also have a commercial effect by increasing the perceived value, power and importance of the technology. OpenAI used similar messaging in 2019, initially withholding the largest version of GPT-2 because of concerns over malicious applications.
Important to Know. Like nuclear science or any other powerful technologies, AI's danger and use comes not from the technology itself, but from which class controls it and to what end. In the hands of capitalists, this technology is being used to intensify the exploitation of workers, maximise profits and prepare new weaponry for imperialist plunder.
► Under a capitalist economy, competition between the different monopolies, both domestic and foreign, means that this technology cannot be developed consciously or safely. A company that slows down risks losing ground to its rivals. Voluntary safety pledges do little to change this pressure, since the companies responsible for managing the risks also stand to profit from continued development. The same conflict between private profit and wider social needs can be seen in the climate crisis.