Community building has recently had a surge in energy and resources. It scales well, it’s high leverage, and you can get involved in it even if six months ago you’d never heard of Effective Altruism. Having seen how hard some people are rowing this boat, I’d like to see if I can’t steer it a bit.

The tl;dr is:

• Some current approaches to community building especially in student groups are driving away great people
• These approaches involve optimising for engaging a lot of new people in a way that undermines good epistemics, and trades off against other goals which are harder to measure but equally important (cf Goodhart’s Law)
• I argue that this is a top priority because the people that these approaches drive away are in many cases the people EA needs the most.

I’ll start by describing why I and several friends of mine did not become EAs.

Then I’ll lay out my sense of what EA needs and what university community building is trying to achieve. I’ll also discuss things that I have encountered in community building that I’ve found troubling.

After that I’ll give my model for how these approaches to community building might be causing serious problems

Finally I’ll explain why I think that this issue in particular needs to be prioritised, and what I think could be done about it.

Part 1 - Reasons I and others did not become an EA

1

I was talking to a friend a little while ago who went to an EA intro talk and is now doing one of 80,000 Hours' recommended career paths, with a top score for direct impact. She’s also one of the most charismatic people I know, and she cares deeply about doing good, with a healthy practical streak.

She’s not an EA, and she’s not going to be. She told me that she likes the concept and the framing, and that since the intro talk she’s often found that when faced with big ethical questions it’s useful to ask “what would an EA do”. But she’s not an EA. Off the back of the intro talk and the general reputation as she perceived it, she got the sense that EA was a bit totalising, like she couldn’t really half-join, so she didn’t. Still, she enjoys discussing the concept of it with me, and she’s curious to hear more about AI.

Certainly there are some professions, like AI safety, where one person going all in is strikingly better than a lot of people who are only partly engaged, but in her area I don’t think this applies. I’ll build on this later.

2

A friend of mine at a different university attended the EA intro fellowship and found it lacking. He tells me that in the first session, foundational arguments were laid out, and he was encouraged to offer criticism. So he did. According to him, the organisers were grateful for the criticism, but didn’t really give him any satisfying replies. They then proceeded to build on the claims about which he remained unconvinced, without ever returning to it or making an effort to find an answer themselves.

He recently described something to me as ‘too EA’. When I pushed him to elaborate, what he meant was something like ‘has the appearance of inviting you to make your own choice but is not-so-subtly trying to push you in a specific direction’.

3

Another friend of mine is currently working on Bayesian statistical inference, but has an offer to work as a quantitative trader. He hopes to donate some of his income to charity. He does not want to donate to EA causes, or follow EA recommendations, and in fact he will pretty freely describe EA as a cult. He has not, as far as I know, attended any EA events. He has already made his mind up.

As far as I can tell, this is the folk wisdom among mathematicians in my university: I’ve heard the rough sentiment expressed several times, usually in response to people saying things like “so what do you guys make of EA?”

4

I have a friend who has just started a career in an EA cause area. She knows about EA because I have told her about it, and because I once gave her a copy of The Precipice. But there’s never really been a way for her to get engaged. Her area of interest is distinctly neartermist, and even though she lives in one of the most densely EA cities in the world, she’s never become aware of any EA events in her area.

Me
When I came to university I had already read a lot of the Sequences and I’d known about effective altruism for years, and even read some of the advice on 80,000 Hours. But upon investigating my local group I was immediately massively put off. The group advertised that I could easily book a time to go on a walk with a committee member who would talk to me about effective altruism and give me career advice, and to me this felt off. Every student society was trying to project warm inviting friendliness, but EA specifically seemed to be trying too hard, and it pattern-matched to things like student religious groups.

I asked around, and quickly stumbled upon some people who confidently told me that EA was an organisation that wanted to trick me into signing away my future income to them in exchange for being part of their gang. The fact that anyone would confidently claim this was enough to completely dissuade me from ever engaging.

Nonetheless, I retained a general interest in the area, and indeed my interest in rationality led me to get to know various older engaged EAs. They never tried to convince me to adopt their values, but they were pretty exemplary in their epistemics, and this made them very interesting to talk to. The groups I floated in were a mix of EAs and non-EAs, but eventually it rubbed off on me. And I’m pretty sure that if I hadn’t encountered EA in university it would have rubbed off a lot sooner.

Part 2 - My concerns with current community building approaches

I have a tentative model for how EA community building could be improved, which I’ve arrived at from a synthesis of two things. The first is my received sense of where EA is currently facing difficulty; the second is things that I have personally found concerning. I’ll lay these out, then present my best guess for what is going wrong in the next section.

Where is EA facing difficulty

As far as I can tell, the most basic account is that EA is talent-constrained: there aren’t enough good people ready to go out there and do things. This yields the most basic account of what EA should be doing: producing more Highly Engaged EAs (HEAs)

But the picture is slightly more complex than that, because in fact there are only some kinds of talent on which EA is constrained. Indeed, openings for EA jobs tend to be massively oversubscribed. So what specific kinds of talented people does EA need more of? Well, the most obvious place to look is the most recent Leader Forum, which gives the following talent gaps (in order):

• Government and policy experts
• Management
• The ability to really figure out what matters most and set the right priorities
• Skills related to entrepreneurship / founding new organizations
• One-on-one social skills and emotional intelligence
• Machine learning / AI technical expertise

As you can see, there are in fact five categories which rank above AI technical expertise. So the question is, if many EA jobs are flooded with applicants, why are we still having trouble with these? What I will go on to claim is that current community building may be selecting against people with some of these talents.

What I have found disconcerting

The most concrete thing is community builders acting in manners which I would consider to be too overtly geared at conversion. For instance, introducing people to EA by reading prepared scripts, and keeping track of students in CRMs. I find this very aversive, and I would guess that a lot of likely candidates for EA entrepreneurs, governmental officials, and creative types would feel similarly.

This point bears repeating because as far as I can tell a lot of community builders just don’t think this is weird. They do not have any intuitive sense that somebody might be less likely to listen to the message of a speech if they know that it’s being read from a script designed to maximise conversion. They are surprised that somebody interested in EA might be unhappy to discover that the committee members have been recording the details of their conversation in a CRM without asking.

But I can personally confirm that I and several other people find this really aversive. One of my friends said he would “run far” if, in almost any context, someone tried to persuade him to join a group by giving a verbatim speech from a script written by someone else. Even if the group seemed to have totally innocuous beliefs, he thought it would smack of deception and manipulation.

Another red flag is the general attitude of persuading rather than explaining. Instead of focusing on creating a space for truth-seeking - learning useful tools and asking important questions - it seems like many community builders see their main job as persuading people of certain important truths and coaxing them into entering certain careers. One admitted to me that, if there were a series of moves they could play to convert a new undergrad into an AI safety researcher or someone working on another job that seems important, they would ‘kind of want to’ play those moves. This is a very different approach from giving exceptional people the ‘EA toolkit’ and helping them along their journey to figuring out how to have the biggest impact they can.

EA may not in fact be a form of Pascal’s Mugging or fanaticism, but if you take certain presentations of longtermism and X-risk seriously, the demands are sufficiently large that it certainly pattern-matches pretty well to these.

And more generally, I find it odd to know that people are being put in charge of student groups who have only known about effective altruism for single digit months. Even if they’re not being directly hired by CEA/OpenPhil, they’re still often being given significant resources and tasked with growing their groups. This is an obvious environment for misalignment to creep in, not through any malice but just through a desire to act quickly without a real grip on what to do.

Part 3 - My model of what is going wrong

My central and most important worry is that activities doing something close to optimising for the number of new HEAs will disproportionately filter out many of the people it’s most valuable to engage. I’ll reiterate the list of things we need more than technical AI expertise:

• Government and policy experts
• Management
• The ability to really figure out what matters most and set the right priorities
• Skills related to entrepreneurship / founding new organizations
• One-on-one social skills and emotional intelligence

My impression is that there are some people who will, when presented with the arguments for Effective Altruism, pretty quickly accept them and adopt something approximating the EA mindset and worldview. I think that the people who excel in some of the areas I’ve listed above are significantly less likely to also be the kinds of people who get engaged quickly. I’ll lay my thoughts out in detail, but first let me give an easy example: “The ability to really figure out what matters most and set the right priorities”

People who care a lot about what matters most are likely to be the kinds of people who don’t just go along with arguments. They’ll be the kind that push back, pick holes, and resist attempts to be persuaded. I think it would be tempting to assume that the best of these people will already have intuited the importance of scope sensitivity and existential risk, and that they’ll therefore know to give EA a chance, but that’s not how it works. The community needs to contain people who won’t take the importance of existential risk seriously until they’ve had some time to think hard about it, and it will take more effort to get such people engaged.

If you don’t intentionally encourage the kinds of people who instinctively pick holes in arguments while you’re presenting EA to them for the first time, your student group is not going to produce people who are fantastic at coming up with thoughtful and interesting criticisms. I can point to specific people who I believe have useful criticisms of EA, but who have no interest in getting hired to write them up even if it can be funded, because they just don’t care that much about EA, because when they tried to present criticism early on they were ignored.

I’m going to address the following points in this order:

1. Noticing the problem is itself hard, but too much focus on creating HEAs will sometimes cause you to miss the most impactful people
2. A speculative model of things going wrong
3. If these problems are real, they’re systemic
4. Scaling makes them worse
5. The faster your community is growing, the less experienced the majority of members will be.

After that, I will at least try to offer some recommendations.

Noticing the problem is itself hard, but too much focus on creating HEAs will sometimes cause you to miss the most impactful people

I think the basic problem is that firstly we might be failing to consider hard-to-measure factors, and secondly, we might be overweighing easy-to-measure factors. These are of course intimately connected.

On the first point: Zealous community building might sometimes cause big downsides that are really hard to measure. If somebody comes to an intro talk, leaves, and never comes back, you don’t usually find out why. Even if you ask them, you probably can’t put much weight on their answer: they don’t owe you anything and they might quite reasonably be more interested in giving you an answer that makes you leave them alone, even if it’s vague or incomplete. You should expect there to be whole types of reason (like ‘you guys seem way more zealous than I’m comfortable with’) which you’ll be notably less likely to hear about relative to how much people think it, especially if you’re not prioritising getting this kind of feedback.

Even worse, if something about EA switches them off before they even come to the intro talk, you won’t even realise. If something you say in your talk is so bad that it causes someone to go away and start telling all their most promising and altruistic friends that EA is a thinly-veiled cult, you will almost never find out - at least not without prioritising feedback from people who are no longer engaged.

Second, despite some pushback, current EA community building doctrine seems to focus heavily on producing ‘Highly Engaged EAs’ (HEAs). It is relatively easy to tell if someone is a HEA. The less engaged someone is, the harder it is to tell. Unfortunately, sometimes there will be people who will take longer to become HEAs (or perhaps forever), but who will have a higher impact than the median HEA even in proportion to however long it takes them to become however engaged they might become.

I think the model of prioritising HEAs does broadly make sense for something like AI safety: one person actually working on AI safety is worth more than a hundred ML researchers who think AI safety sounds pretty important but not important enough to merit a career change. But elsewhere it’s less clear. Is one EA in government policy worth more than a hundred civil servants who, though not card-carrying EAs, have seriously considered the ideas and are in touch with engaged EAs who can call them up if need be? What about great managers and entrepreneurs?

I don’t actually know the answer here, but what I do know is that the first option - one HEA in a given field - is much easier to measure and point to as evidence of success.

To be really clear, I’m not advocating for an absolute shift in strategy away from HEAs to broader and shallower appeal. What I’m saying is that I don’t think it’s clear-cut, but a focus on measurably increasing the number of HEAs is likely to miss less legible opportunities for impact.

Why can’t people appreciate the deep and subtle mysteries of community building? Well, this is where Goodhart’s Law crops up: a measure that becomes a target to be optimised ceases to be a good measure.

The main way Goodhart’s Law kicks in is that the people setting strategy have a much more nuanced vision than the people executing it. The reason everyone’s pushing for community building, I believe, is that people right in the heart of EA thought about what a more effective and higher-impact EA would look like, and what they pictured was an EA which was much larger and contained many more highly-engaged people. Implicit in that picture were a bunch of other features - strong capacity for coordination, good epistemics, healthy memes, and so on. 

But when that gets distilled down to “community building” and relayed to people who have only been in university for a year or so, quite understandably they don’t spontaneously fill in all the extra details. What they get is “take your enthusiasm for EA, and make other people enthusiastic, and we’ll know you’re doing well if at the end of the year there are more HEAs”.

But often the best way to make more HEAs is not the best way to grow the community!

A speculative model of things going wrong

This is a bit more speculative but I’d like to sketch out a model for how this plays out in a bit more detail. I’d like to conjure up two hypothetical students, Alice and Bob, at their first EA intro fellowship session.

Alice

Alice has a lot of experience with strange ideas. She’s talked to communists, alt-rights, crypto bros, all kinds of people. She’s very used to people coming along with an entirely new perspective on what’s important, and when they set the parameters, she expects them to have arguments she can’t reply to, because she’s an undergrad and they’re cribbing their notes from professors, and sometimes literally reciting arguments off a script. Of course she actually quite likes sitting down and thinking through the problems - she enjoys the intellectual challenge. She knows the world is full of Pascal’s Muggers. She doesn’t know if EAs are muggers, but she knows they like getting people to promise to give away 10% of their income (which sounds to her like a church tithe), and she’s heard they sweep people away on weekend retreats. Still, she can appreciate that if they are right, what they’re doing is important, so she suspends her judgement. 

At the opening session she disputes some of the assumptions, and the facilitators thank her for raising the concerns, but don’t really address them. They then plough on, building on those assumptions. She is unimpressed.

Bob

Bob came to university feeling a bit aimless. He’s not really sure what he wants to do with his life, or how he should even decide. Secretly he’d kind of like it if someone could just tell him what he was meant to do, because sometimes it feels like the world’s in a bad state and he doesn’t really get why or how to fix it. So when he hears the arguments in the opening session he’s blown away. He feels like he’s been handed a golden opportunity: if they’re right, he can be a good person, who does important work, with a close group of friends who all share his goals and values.

Are they right? He’s not sure. He’s never really considered these arguments but they seem very persuasive. And the organisers keep talking about epistemics, and top researchers. If they’re wrong, he’s not even sure how he’d tell, but if they’re right then it’s pretty important that he starts helping out right away. And he kind of wants them to be right. 

If these problems are real, they’re systemic

We should expect that new EAs doing community building will misunderstand high-level goals in systematic ways.

What this means is, it’s not just that some random cluster of promising people will get missed, it’s that certain kinds of promising people will get missed, consistently, and EA as a whole will shift its composition away from those kinds of people. To be clear, this isn't absolute: it’s not that everyone capable of criticism is filtered out, it’s that every group that prioritises producing HEAs will be slightly filtering against it and the effects will compound across the entire community.

If you’ve been told that CEA has hired you as a community builder you because they think that counterfactually this will lead to ten more HEAs, and indeed, you think that it’s really very important to get more HEAs so that there are more people working on the biggest problems, and you meet an Alice and a Bob, well, maybe you’d rather talk to Bob about how to get into community building instead of talking to Alice about alternative foundations to the Rescue Principle.

And maybe this really is the right choice in individual cases. The problem is if it gradually accumulates. Eventually EA as a whole becomes more Bob than Alice, not just in terms of how many people with really fantastic epistemics there are, but also in terms of the epistemic rigour of the median HEA.

Personally the thing I’m most worried about is that this effect starts to wreck EA group epistemics and agency. I’ve seen little traces here and there which have given me concerns, although I don’t yet feel I can confidently claim that this is happening. But I think it’s really really important that we notice if it is, so that we can stop it. And this phenomenon is hard to notice.

Ironically, we should expect community building to tend towards homogeneity because community builders will beget other community builders who find their strategies compelling. And we should expect this to tend towards strategies that are easy to quickly adopt.

There has been some emphasis lately on getting community builders to develop their own ‘inside views’ on important topics like AI safety, partly so that they can then relay these positions with higher fidelity. I welcome this, but I don’t think it’s sufficient to solve the problem of selecting against traits we value. Understanding AI safety better doesn’t stop you from putting people off for any reason other than your understanding of AI safety.

A little while after I first drafted this post, there was a popular forum post entitled “What psychological traits predict interest in effective altruism?” I commend the impulse to research this area but I can very easily picture how it goes wrong, because while it may be true that there are certain characteristics which predict that people are more likely to become HEAs, it does not follow that a larger EA community made up of such people would automatically be better than this one.

Scaling makes them worse

It might now occur to you that not everyone joins EA through student groups. Some people come from LessWrong, some people see a TED Talk, some people just stumble across the forum. It’s true, and these will all be filtering in different kinds of people with different interests and values.

As the community changes, the way it grows will change. The thing to avoid is a feedback loop that sends you spiralling somewhere weird. Unfortunately this is exactly what you encourage when you try to scale things up. The easier something is to scale, well, the faster you’ll scale it. 

If you have a way of community building which produces ten HEAs in a year, two of which will be able to follow in your footsteps, you will very quickly be responsible for the majority of EA growth. The closer a student organiser is to creating the maximum number of HEAs possible, the more likely they are to be Goodharting: trading away something else of value for more HEAs.

And bear in mind: the faster you’re growing, the newer the median member of the community will be. If EA doubled in size every year then half of EAs would only have been EAs for a year. And if any portion of EA managed to crack a way of doubling in size every year, it would very quickly make up the majority of the community.

The faster your community is growing, the less experienced the majority of members will be.

Concretely, I worry that university groups risk instantiating this pattern. The turnover is quick, and the potential for rapid growth and scaling is a big selling point.

I imagine that older EA groups will have had both time to grow and time to consider downside risks. They’ll have more experienced members who can be more careful, and also less of a pressure to expand. On the other hand, newer groups will be saddled with both less experience and more desire and opportunity to scale up quickly.

It’s also generally hard, as someone with experience, to remember what it was like being inexperienced, and what was or wasn’t obvious to you. It’s easy to assume that people understand all the subtext and implications of your claims far more than they actually do. We need to actively resist this when dealing with newer, more inexperienced community builders.

Part 4 - Why to prioritise this problem, and what to do about it

You might think that, while this problem exists, it’s not worth focusing resources on it because it’s not as high-priority as problems like AI safety research. If better epistemics trades off against getting more alignment researchers, maybe you think it’s not worth doing. However, it’s not clear at all that this is the case.

First, AI Safety researcher impact is long-tailed, and I claim that the people on the long tail all have really unusually good epistemics, such that trading against good epistemics in favour of more AI safety researchers risks trading the best for the worst.

Second, most groups in history have been wrong about some significant things, including groups that really wanted to find the truth, like scientists in various fields. So, our strong outside view should be that, either at the level of cause prioritisation or within causes, we are wrong about some significant things. If it’s also sufficiently likely that some people could figure this out and put us on a better path, then it seems really bad that we might be putting off those very people.

Third, imagine a world in which EA student groups are indeed significantly selecting against traits we value. Ask yourself if, in this world, things might look roughly as they do now. I think they might. It’s easy to let motivated reasoning slip in when one wants to avoid acknowledging a tradeoff - for example, I’ve often told myself I have time to do everything I want to do, when in fact I don’t. This problem could be happening, and you might think this problem isn’t happening even if it is! Until we spend some resources getting information, our uncertainty about whether / how badly the problem is happening should push us towards prioritising the problem. (It might be really bad, and we don’t yet know if it is!) If we later found out that it wasn’t a big deal, we could deprioritise it again.

For all the same reasons that doing community building is important, it is really important to do it right.

So what do you do about all this?

It’s probably not enough just to acknowledge that it might be a problem if you don’t prioritise it. It’s also not enough (though it may be useful) to select for virtuous traits when choosing, for example, your intro fellows. Even if you do this, you will still miss out on anyone who is put off after the selection process, or who doesn’t even apply because they’ve heard that EA is a weird cult.

Honestly, it’s hard. I have to admit the limits of my own knowledge here: I don’t know what constraints community builders are acting under, or what the right balance between these factors is. Moreover, the issue I’m pointing to is, in the most general terms, that there are lots of hard-to-measure things which people might not be properly measuring. It’d be very easy, I suspect, to read this post and think “Look at all these other factors I hadn’t considered! Well, I’d better start considering them,” and move on, when in fact what you need to do is one meta-level up: start looking for illegible issues, and factors that nobody else has even considered yet.

So, now that you know that I don’t have all the answers, and that literally following my advice as written will only sort of help, here is my advice. 
 

• Don’t actually think in terms of producing more HEAs. Yes, good community building will lead to more HEAs, but producing more HEAs is not enough to make what you’re doing good community building.
  • If you’re high-status within EA, think carefully about how you react to community builders who seem to create lots of HEAs. Don’t just praise them, but also coach them and monitor them. The more HEAs created, the more you should be suspicious of Goodharting (despite the best of intentions), so work together to avoid it.
• Consider the downside risks from activities you’re running.
  • A useful framework is to seriously consider what types of people might be put off / selected against by an activity, and a good list of types to start with is the ones the Leaders Forum says EA needs.
  • Adopt the rule of thumb: ‘If many people would find it creepy if they knew we were doing x, don’t do x.’
    • Notice that many EA community builders seem to have different norms from other students in this regard. Especially if you didn’t think things like reading pre-scripted persuasive speeches or recording details from 1-on-1s in a CRM without asking seemed sinister, default to asking a handful of (non-EA) friends what they think before introducing a new initiative.
  • It might be helpful for there to be a community-building Red Team organisation, which could scrutinise both central strategies (e.g. from CEA or OpenPhil) and the activities of individual student groups.

• Assume that people find you more authoritative, important, and hard-to-criticise than you think you are. It’s usually not enough to be open to criticism - you have to actually seek it out or visibly reward it in front of other potential critics.
  • Maybe try things like giving pizza to intro fellows who left in exchange for feedback.
  • You want good feedback from everyone, not just those who you thought would be highly impactful, since it’s easy for someone to be put off EA based on the message they hear for any other person, whether or not that person has potential for high impact.
    • No-one seems sure how much low-fidelity / misleading messages about EA are being spread. It would be great (and at least partly tractable) to research this.

• Be open to changing your mind. I know this is kind of overplayed, but there’s a whole sequence on it, and it’s pretty good. Remember that the marginal value of another HEA is way lower than the marginal value of an actual legitimate criticism of EA nobody else has considered yet.
  • Seriously consider adding more events geared towards presentation of ideas than persuasion.
     
• Don’t offer people things they want in exchange for self-identification and value adoption. Free pizza for showing up to a discussion group is fine, but if people feel like they’ll get respect and a friendship group only if they go around saying “AI safety seems like a big deal”, then that will be why some of them go around saying “AI safety seems like a big deal”.

• Message me. I’ll try to reply to comments and messages. It’s hard for me to predict in advance what parts of this will or won’t be clear, so I invite you to tell me what doesn’t make sense.

• Read these articles, if you feel so inclined (ranked from most to least useful in my opinion):

https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporative-cooling-of-group-beliefs

https://www.lesswrong.com/posts/L32LHWzy9FzSDazEg/motivated-stopping-and-motivated-continuation

https://www.lesswrong.com/posts/4PPE6D635iBcGPGRy/rationality-common-interest-of-many-causes

https://meaningness.com/metablog/upgrade-your-cargo-cult

https://meaningness.com/geeks-mops-sociopaths

http://benjaminrosshoffman.com/construction-beacons/