I take your claim in the post not to be "the fact that an offer is +EV is one strong reason to be in favor of it," but rather "you ought to take the cosmic coin flip, risking the universe, just because it is +EV." (Because being +EV definitionally means the good scenario is super unbelievably good, much better than most people considering the thought experiment are probably imagining.)

But even within the thought experiment, abstracting away all empirical uncertainties, I have enough philosophical uncertainty about EV maximization that I don't want to take the bet.

Again, just giving my impressions from interacting with AI safety people: it doesn't seem to me like I get this impression by drawing a larger circle -- I don't recall hearing the types of arguments you allude to even from people I consider "core" to AI safety. I think it would help me understand if you were able to provide some examples? (Although like I said, I found examples either way hard to search for, so I understand if you don't have any available.)

I still disagree about the Dial post: at the end Zvi says

Seeing highly intelligent thinkers who are otherwise natural partners and allies making a variety of obvious nonsense arguments, in ways that seem immune to correction, in ways that seem designed to prevent humanity from taking action to prevent its own extinction, is extremely frustrating. Even more frustrating is not knowing why it is happening, and responding in unproductive ways.

So my read is that he wants to explain and understand the position as well as possible, so that he can cooperate as effectively as possible with people who take the Dial position. He also agrees on lots of object-level points with the people he's arguing against. But ultimately actually using the Dial as an argument is "obvious nonsense," for the same reason the Technology Bucket Error is an error.

I agree that there's been a phenomenon of people suddenly all agreeing that all of SBF's opinions were wrong post-FTX collapse. So I appreciate the effort to make the case for taking the deal, and to portray the choice as not completely obvious.

To the extent that you're hoping to save "maximizing utility via maximizing expected value," I think it's still an uphill battle. I like Beckstead and Thomas's "A paradox for tiny probabilities and enormous values" on this, which runs essentially the same thought experiment as "flip the coin many times," except with the coin weighted to 99.9% heads (and only your own life in play, not the universe). They point out that both positions, "timidity" and "recklessness", have implausible conclusions.

I'm ultimately quite philosophically troubled by this "concentrating all the value into narrow regions of probability space" feature of EV maximization as a result (but I don't have a better alternative on hand!). This makes me, in particular, not confident enough in EV-maximization to wager the universe on it. So while I'm more sympathetic than most to the position that the coin flip might be justifiable, I'm still pretty far from wanting to bite that bullet.

I agree with the main ideas of this post. But I want to flag that, as someone who's engaged with the AI safety community (outside of the Bay Area) for several years, I don't recognize this depiction. In my experience, it's very common to say "even a [1, 5, 10]% chance of AI x-risk justifies taking this very seriously."

I don't have time to track down many examples, but just to illustrate:

• Neel Nanda's "Simplify EA Pitches to 'Holy Shit, X-Risk'": "If you believe the key claims of "there is a >=1% chance of AI causing x-risk and >=0.1% chance of bio causing x-risk in my lifetime" this is enough to justify the core action relevant points of EA."
• Toby Ord estimated AI x-risk at 10% in The Precipice.
• Scott Alexander is sympathetic to libertarianism and put AI x-risk at 20-25% in 2023. 

To address another claim: "The Dial of Progress" by Zvi, a core LessWrong contributor, makes the case that technology is not always good (similar to the "Technology Bucket Error" post) and the comments overwhelmingly seem to agree.

I'm sure someone has said that 10-25% x-risk would not be worth addressing due to libertarianism -- but I don't believe I've heard this argument, and wasn't able to find someone making it after a few minutes of searching. (But it's a hard thing to search for.)

I don't doubt that Holly is accurately reporting her experiences, and she's almost certainly engaged more widely than I have with people in AI safety. I wouldn't be surprised if there are people saying these things in the Bay Area. But I don't have the impression that they represent the mainstream of the AI safety community in any forum I'm familiar with (Twitter, LW, and definitely EA).

While I agree with many points in this post, I think it would be stronger if it engaged more with the existing discussion within EA on mental health, on the Forum and elsewhere. 

For example:

• The "Self-care and wellbeing in the EA community" tag contains over 270 posts, including some of the most highly-upvoted posts on the site.
• 80,000 Hours has dedicated significant attention here:
  • Their most popular podcast episode (iirc) focuses on mental health in EA
  • They have an episode with a mental health practitioner about perfectionism and related challenges
  • The first three sections of the career guide’s "How to be successful in any job" chapter emphasize self-care, with the second section stating that dealing with poor mental health should be the top priority
• Several programs have popped up over the years offering to provide or connect EAs with mental health services. Currently Rethink Wellbeing is active and provides CBT and IFS-based peer-facilitated programs. (There may be others I’m forgetting.)

A few of these seem to me like the sort of thing the suggestions were asking for, e.g. "a few podcast episodes that could succinctly demonstrate the way therapy might explore common blind spots held by EAs, providing a rapid update with less resistance than other methods."

I've personally experienced mental health challenges due to EA, so I'm certainly not saying the problems are all solved, or that the resources above cover everything. Publishing one podcast doesn't solve a community-wide problem. But parts of this post read to me as suggesting these resources and discussions don't exist, so I want to provide an alternate perspective.

What makes the agriculture transition stand out as the "point of no return"? Agriculture was independently invented several times, so I'd expect the hunter-gatherer -> agriculture transition to be more convergent than agriculture -> AGI.

I didn't say universal or 50% support.

Sorry you're right, you didn't say this -- I misread that part of your comment. 

I still think your framing misses something important: the logic "50% of people are women so I think women’s suffrage had a pretty strong support base" applies at all points in time, so it doesn't explain why suffrage was so unpopular for so long. Or to put it another way, for some reason the popularity and political influence of the suffrage movement increased dramatically without the percentage of women increasing, so I'm not sure the percentage of people who are women is relevant in the way you're implying.

The idea that you can go regulating without considering public support/resistance is silly

On the other hand I didn't say this! The degree of public support is certainly relevant. But I'm not sure what your practical takeaway or recommendation is in the case of an unpopular movement.

For example you point out abolition as an example where resistance caused massive additional costs (including the Civil War in the US). I could see points 1, 3, 7, and possibly 8 all being part of a "Ways I see the Quaker shift to abolitionism backfiring" post. They could indeed be fair points that Quakers / other abolitionists should have considered, in some way -- but I'm not sure what that post would have actually wanted abolitionists to do differently, and I'm not sure what your post wants EAs to do differently.

Maybe you just intend to be pointing out possible problems, without concluding one way or another whether the GH -> AW shift is overall good or bad. But I get a strong sense from reading it that you think it's overall bad, and if that's the case I don't know what the practical upshots are.

I'm not super knowledgeable about women's suffrage, but

• It was not universally supported by women (See e.g. here; I couldn't quickly find stats but I'd be interested).
• Surely the relevant support base in this case is those who had political power, and the whole point is that women didn't. So "50% support" seems misleading in that sense.

I could similarly say ">99.999% of animals are nonhumans, so nonhuman animal welfare has an extremely large support base." But that's not the relevant support base for the discussion at hand.

if I assume that mosquitoes fall somewhere between black soldier flies and silkworms in their welfare range then killing 100-1000 mosquitoes a year (assuming this causes suffering) could be the moral equivalent to killing a human.

I don't think this is a correct reading of the welfare range estimates. If I understand correctly, these numbers would mean that a mosquito can have hedonic states 0.1% - 1% as intense as humans. So 100-1000 days of mosquito suffering might be on par with one day of human suffering. (And of course this number is a wild guess based on other insects, whose numbers are already very uncertain.)

The harm of death is a different question that RP's numbers don't straightforwardly address. Even a purely hedonic account has to factor in lifespan (mosquitos live for about six weeks). And killing a human is bad for a whole host of additional reasons unrelated to preventing future happiness.

So while I think the welfare range estimates suggest huge moral updates, they're not as huge as you say. It's good to be able to take bold conclusions seriously, but it's also worth taking seriously that there might be a good reason for a result to be extremely counterintuitive.

If you missed this (I did), it looks like the recording will be released on August 9.