Matrice Jacobine
Posts 12
Comments31
If there's no humans left after AGI, then that's also true for "weak general AI". Transformative AI is also a far better target for what we're talking about than "weak general AI".
The "AI Dystopia" scenario is significantly different from what PauseAI rhetoric is centered about.
The PauseAI rhetoric is also very much centered on just scaling LLMs, not acknowledging other ingredients of AGI.
Metaculus puts (being significantly more bullish than actual AI/ML experts and populated with rationalists/EAs) <25% chance on transformative AI happening by the end of the decade and <8% chance of this leading to the traditional AI-go-foom scenario, so <2% p(doom) by the end of the decade. I can't find a Metaculus poll on this but I would halve that to <1% for whether such transformative AI would be reached by simply scaling LLMs.
To be clear, my point is that 1/ even inside the environmental movement calling for an immediate pause on all industry from the same argument you're using is extremely fringe, 2/ the reputation costs in 99% of worlds will themselves increase existential risk in the (far more likely) case that AGI happens when (or after) most experts think it will happen.
Industry regulations tend to be based on statistical averages (i.e., from a global perspective, on certainties), not multiplications of subjective-Bayesian guesses. I don't think the general public accepting any industry regulations commit them to Pascal-mugging-adjacent views. After all, 1% of existential risk (or at least global catastrophic risk) due to climate change, biodiversity collapse, or zoonotic pandemics seem plausible too. If you have any realistic amount of risk aversion, whether the remaining 99% of the futures (even from a strictly strong-longtermist perspective) are improved upon by pausing (worse, by flippant militant advocacy for pausing on alarmist slogans that will carry extreme reputation costs in the 99% of worlds where no x-risk from LLMs happen) is important!
Crucially, p(doom)=1% isn't the claim PauseAI protesters are making. Discussed outcomes should be fairly distributed over probable futures, if only to make sure your preferred policy is an improvement on most or all of those (this is where I would weakly agree with @Matthew_Barnett's comment).
Most surveys of AI/ML researchers (with significant selection effects and very high variance) indicate p(doom)s of ~10% (among a variety of different kinds of global risks beyond the traditional AI-go-foom), and (like Ajeya Cotra's report on AI timelines) a predicted AGI date in the mid-century according to one definition, in next century by another.
Pausing scaling LLMs above a given magnitude will do ~nothing for non-x-risk AI worries. Pausing any subcategory below that (e.g. AI art generators, open-source AI) will do ~nothing (and indeed probably be a net negative) for x-risk AI worries.
Here are Thorstad's earlier posts on the institutional critique:
https://reflectivealtruism.com/2023/06/09/the-good-it-promises-the-harm-it-does-part-4-lori-gruen/
https://reflectivealtruism.com/2023/07/15/the-good-it-promises-part-5-de-coriolis-et-al/
https://reflectivealtruism.com/2023/09/08/the-good-it-promises-part-7-crary-continued/
In addition to what @gw said on the public being in favor of slowing down AI, I'm mostly basing this on reactions to news about PauseAI protests on generic social media websites. The idea that LLMs scaling without further technological breakthrough will for sure lead to superintelligence in the coming decade is controversial by EA standards, fringe by general AI community standard, and resoundly mocked by the general public.
If other stakeholders agree with the existential risk perspective then that is of course great and should be encouraged. To develop further on what I meant (though see also the linked post), I am extremely skeptical that allying with copyright lobbyists is good by any EA/longtermist metric, when ~nobody think art generators pose any existential risk and big AI companies are already negotiating deals with copyright giants (or even the latter creating their own AI divisions as with Adobe Firefly or Disney's new AI division), while independent EA-aligned research groups like EleutherAI are heavily dependent on the existence of open-source datasets.
I don't think rationalism is that small a subculture in the Bay at this point, and the Bay Area rate of cult creation has historically been pretty high since the 1960s at least. Watching Slimepriestess' interview (linked in the LW thread comments), my impression is that the Zizians' beliefs and actions stemmed from a fusion of rationalist/EA beliefs, far-left anarchist politics, and general Bay Area looniness.