Anders Sandberg has written a “final report” released simultaneously with the announcement of FHI’s closure. The abstract and an excerpt follow.
Normally manifestos are written first, and then hopefully stimulate actors to implement their vision. This document is the reverse: an epitaph summarizing what the Future of Humanity Institute was, what we did and why, what we learned, and what we think comes next. It can be seen as an oral history of FHI from some of its members. It will not be unbiased, nor complete, but hopefully a useful historical source. I have received input from other people who worked at FHI, but it is my perspective and others would no doubt place somewhat different emphasis on the various strands of FHI work.
What we did well
One of the most important insights from the successes of FHI is to have a long-term perspective on one’s research. While working on currently fashionable and fundable topics may provide success in academia, aiming for building up fields that are needed, writing papers about topics before they become cool, and staying in the game allows for creating a solid body of work that is likely to have actual meaning and real-world effect.
The challenge is obviously to create enough stability to allow such long-term research. This suggests that long-term funding and less topically restricted funding is more valuable than big funding.
Many academic organizations are turned towards other academic organizations and recognized research topics. However, pre-paradigmatic topics are often valuable, and relevant research can occur in non-university organizations or even in emerging networks that only later become organized. Having the courage to defy academic fashion and “investing” wisely in such pre-paradigmatic or neglected domains (and networks) can reap good rewards.
Having a diverse team, both in terms of backgrounds but also in disciplines, proved valuable. But this was not always easy to achieve within the rigid administrative structure that we operated in. Especially senior hires with a home discipline in a faculty other than philosophy were nearly impossible to arrange. Conversely, by making it impossible to hire anyone not from a conventional academic background (i.e., elite university postdocs) adversely affects minorities, and resulted in instances where FHI was practically blocked from hiring individuals from under-represented groups. Hence, try to avoid credentialist constraints.
In order to do interdisciplinary work, it is necessary to also be curious about what other disciplines are doing and why, as well as to be open to working on topics one never considered before. It also opens the surface to the rest of the world. Unusually for a research group based in a philosophy department, FHI members found themselves giving tech support to the pharmacology department; participating in demography workshops, insurance conferences, VC investor events, geopolitics gatherings, hosting artists and civil servant delegations studying how to set up high-performing research institutions in their own home country, etc. - often with interesting results.
It is not enough to have great operations people; they need to understand what the overall aim is even as the mission grows more complex. We were lucky to have had many amazing and mission-oriented people make the Institute function. Often there was an overlap between being operations and a researcher: most of the really successful ops people participated in our discussions and paper-writing. Try to hire people who are curious.
Where we failed
Any organization embedded in a larger organization or community needs to invest to a certain degree in establishing the right kind of social relationships to maintain this embeddedness. Incentives must be aligned, and both parties must also recognize this alignment. We did not invest enough in university politics and sociality to form a long-term stable relationship with our faculty.
There also needs to be an understanding of how to communicate across organizational communities. When epistemic and communicative practices diverge too much, misunderstandings proliferate. Several times we made serious missteps in our communications with other parts of the university because we misunderstood how the message would be received. Finding friendly local translators and bridgebuilders is important.
Another important lesson (which is well known in business and management everywhere outside academia) is that as an organization scales up it needs to organize itself differently. The early informal structure cannot be maintained beyond a certain size, and must be gradually replaced with an internal structure. Doing this gracefully, without causing administrative sclerosis or lack of delegation, is tricky and in my opinion we somewhat failed.
So, you want to start another FHI?
Did FHI become humanity's best effort at understanding and evaluating its own long-term prospects? We leave that to the future to evaluate properly, but we certainly think we did unexpectedly well for a “three-year project”.
FHI is ending and we are sad to see it go. We think it could have achieved far more than it did, but circumstances made it impossible to continue. On the plus side, we know FHI did not live past its time. There is a real risk that organizations lose their mission and become self-perpetuating users of resources that could better be used for other things, preventing the flowering of the new.
In fact, as mentioned above, FHI has seeded a number of new organizations, fields and topics. In a biological or memetic sense, it would count as having had great fitness in propagating successors, although many are not FHI-like or have different goals.
What would it take to replicate FHI, and would it be a good idea? Here are some considerations for why it became what it was:
- Concrete object-level intellectual activity in core areas and finding and enabling top people were always the focus. Structure, process, plans, and hierarchy were given minimal weight (which sometimes backfired - flexible structure is better than little structure, but as organization size increases more structure is needed).
- Tolerance for eccentrics. Creating a protective bubble to shield them from larger University bureaucracy as much as possible (but do not ignore institutional politics!).
- Short-term renewable contracts. Since firing people is basically impossible within the University, only by offering short-term contracts (two or three years) was it possible to get rid of people who turned out not to be great fit. It was important to be able to take a chance on people who might not work out. Maybe about 30% of people given a job at FHI were offered to have their contracts extended after their initial contract ran out. A side-effect was to filter for individuals who truly loved the intellectual work we were doing, as opposed to careerists.
- Valued: insights, good ideas, intellectual honesty, focusing on what’s important, interest in other disciplines, having interesting perspectives and thoughts to contribute on a range of relevant topics.
- Deemphasized: the normal academic game, credentials, mainstream acceptance, staying in one’s lane, organizational politics.
- Very few organizational or planning meetings. Most meetings were only to discuss ideas or present research, often informally.
A comment from a member:
“I think there’s no cookie-cutter template for replicating FHI because it depends critically on having the right (rare) people and a particular intellectual culture. The secret sauce was not an organizational structure or some kind of management process. But with the right people and culture, then shielding from other constraints can become enabling.
“To the extent that it can be replicated, I think it is because (a) it was an existence proof of organization, template ideas and research results, and legitimization, and (b) the intellectual culture has spread (e.g. in the wider rationalist and EA networks). But it could probably not be replicated by having some random administrator or manager trying to reproduce the same organizational structure - that would be like the cargo cult.”
So, the conclusion may be that while the above considerations give a recipe to aim for, the key question for any replication should be: “What are the important topics this organization should aim at?” Pursuing those topics must always be at the center of what is being done (both in research and administration), even when new knowledge and developments change them and their priorities.
I think it is worth appreciating the number and depth of insights that FHI can claim significant credit for. In no particular order:
Note especially how much of the literal terminology was coined on (one imagines) a whiteboard in FHI. “Existential risk” isn't a neologism, but I understand it was Nick who first suggested it be used in a principled way to point to the “loss of potential” thing. “Existential hope”, “vulnerable world”, “unilateralist's curse”, “information hazard”, all (as fa... (read more)
I'm awestruck, that is an incredible track record. Thanks for taking the time to write this out.
These are concepts and ideas I regularly use throughout my week and which have significantly shaped my thinking. A deep thanks to everyone who has contributed to FHI, your work certainly had an influence on me.
I want to take this opportunity to thank the people who kept FHI alive for so many years against such hurricane-force headwinds. But I also want to express some concerns, warnings, and--honestly--mixed feelings about what that entailed.
Today, a huge amount of FHI's work is being carried forward by dozens of excellent organizations and literally thousands of brilliant individuals. FHI's mission has replicated and spread and diversified. It is safe now. However, there was a time when FHI was mostly alone and the ember might have died from the shockingly harsh winds of Oxford before it could light these thousands of other fires.
I have mixed feelings about encouraging the veneration of FHI ops people because they made sacrifices that later had terrible consequences for their physical and mental health, family lives, and sometimes careers--and I want to discourage others from making these trade-offs in the future. At the same time, their willingness to sacrifice so much, quietly and in the background, because of their sincere belief in FHI's mission--and this sacrifice paying off with keeping FHI alive long enough for its work to spread--is something for which I am incr... (read more)
Strong +1 re: 'hero' work culture. especially for ops staff. This was one of the things that bothered me while there and contributed to my moving on - an (admittedly very nice) attitude of praising (especially admin/management) people who were working stupidly hard/long, rather than actually investing in fixing a clearly dysfunctional situation. And while it might not have been possible to fix later on due to embedded animosity/frustration on both sides => hiring freeze etc, it certainly was early on when I was there.
The admin load issue was not just about the faculty. And the breakdown of relationship with the faculty was really was not one-sided, at least when I was there (and I think I succeeded in semi-rescuing some of the key relationships (oxford martin school, faculty of philosophy) while I was there, at least temporarily).
This is really sad news. I hope everyone working there has alternative employment opportunities (far from a given in academia!).
I was shocked to hear that the philosophy department imposed a freeze on fundraising in 2020. That sounds extremely unusual, and I hope we eventually learn more about the reasons behind this extraordinary institutional hostility. (Did the university shoot itself in the financial foot for reasons of "academic politics"?)
A minor note on the forward-looking advice: "short-term renewable contracts" can have their place, especially for trying out untested junior researchers. But you should be aware that it also filters out mid-career academics (especially those with family obligations) who could potentially bring a lot to a research institution, but would never leave a tenured position for short-term one. Not everyone who is unwilling to gamble away their academic career is thereby a "careerist" in the derogatory sense.
On your second point, FHI had at least ~£10m sitting in the bank in 2020 (see below, from the report). So the fundraising freeze, while unusual, wasn't terminal. A rephrasing of your question is "What adminstrative and organisational problems at FHI could possibly have prompted the Faculty to take the unusual step of a hiring and fundraising freeze in 2020, and why could it not be resolved over the next two to three years?"
There is now a dedicated FHI website with lists of selected publications and resources about the institute. (Thanks to @Stefan_Schubert for bringing this to my attention.)
See also Anders’s more personal reflections:
... (read more)I made a perma.cc copy of the Final Report here: https://perma.cc/3KP9-ZSFB
I'd be curious about a list of topics they would like others to investigate/continue investigating, or a list of the most important open questions.
There is a list by Sandberg here. (The other items in that post may also be of interest.)
That's sad. For anyone interested in why they shut down (I'd thought they had an indefinitely sustainable endowment!), the archived version of their website gives some info:
I still don't understand why the University of Oxford was not cooperative with the institute, and why later it decided to freeze it completely. What was that about?
I'm also confused by this. Did Oxford think it was a reputation risk? Were the other philosophers jealous of the attention and funding FHI got? Was a beaurocratic parasitic egregore putting up roadblocks to siphon off money to itself? Garden variety incompetence?
Having worked there and interfaced with the Faculty for 4 years, yes, I would expect garden variety incompetence on Bostrom's part in terms of managing the relationship was a big part; I would predict the single biggest contributer to the eventual outcome.
Why was relationship management even necessary? Wasn't FHI bringing prestige and funding to the university? Aren't the incentives pretty well aligned?
Why are people pressing the "disagree" button? Do they disagree with the idea that FHI brought prestige? Do they disagree with the framing? Is it because I have a silly username?
Clearly there's some politics going on here, but I have no idea who the factions are or why.
Someone help me out?
I’m very confused why you think that FHI brought prestige to Oxford University rather than the other way around
You’re answering a somewhat different question to the one I’m bringing up
Influence =/= prestige
The point I’m trying to make is that there are many ways you can be influential (including towards people that matter) and only some of them increase prestige. People can talk about your ideas without ever mentioning or knowing your name, you can be a polarising figure who a lot of influential people like but who it’s taboo to mention, and so on.
I also do think you originally meant (or conveyed) a broader meaning of influential - as you mention economic output and the dustbins of history, which I would consider to be about broad influence.
Andrew Tate is very influential, but entirely lacking in prestige.
I might think of FHI as having borrowed prestige from Oxford. I think it benefited significantly from that prestige. But in the longer run it gets paid back (with interest!).
That metaphor doesn't really work, because it's not that FHI loses prestige when it pays it back -- but I think the basic dynamic of it being a trade of prestige at different points in time is roughly accurate.
Thanks Habryka. My reason for commenting is that a one-sided story is being told here about the administrative/faculty relationship stuff, both by FHI and in the discussion here, and I feel it to be misleading in its incompleteness. It appears Carrick and I disagree and I respect his views, but I think many people who worked at FHI felt it to be severely administratively mismanaged for a long time. I felt presenting that perspective was important for trying to draw the right lessons.
I agree with the general point that maintaining independence under this kind of pressure is extremely hard, that there are difficult tradeoffs to make. I believe Nick made many of the right decisions in maintaining integrity and independence, and sometimes incurred costly penalties to do so that likely contributed to the administrative/bureaucratic tensions with the faculty. However, I think part of what is happening here is that some quite different things from working-inside-fhi-perspective are being conflated under broad 'heading' (intellectual integrity/independence) which sometimes overlapped, but often relatively minimally, and can be usefully disaggregated - intellectual vision and integrity; fol... (read more)
And I guess I should just say directly. I do wish it were possible to raise (specific) critical points on matter like faculty relations where I have some direct insight and discuss these, without immediate escalation to counterclaims that my career’s work has been bad for the world, that I am not to be trusted, and and that my influence is somehow responsible for attacks on people’s intellectual integrity. It’s very stressful and upsetting.
I suffer from (mild) social anxiety. That is not uncommon. This kind of very forceful interaction is valuable for some people but is difficult and costly for others to engage with. I am going to engage less with EA forum/LW as a result of this and a few similar interactions, and I am especially going to be more hesitant to be critical of EA/LW sacred cows. I imagine, given what you have said about my takes, that this will be positive from your perspective. So be it. But you might also consider the effect it will have on others who might be psychologically similar, and whose takes you might consider more valuable.
This makes me sad as I enjoy reading your comments and find them insightful. That said, I understand and support your reasoning. I feel as though some amount of "mistake mindset" has disappeared a little in the two years I've been reading the forum.
Thanks Rían, I appreciate it. And to be fair, this is from my perspective as much a me thing as it is an Oli thing. Like, I don't think the global optimal solution is an EA forum that's a cuddly little safe space for me. But we all have to make the tradeoffs that make most sense for us individually, and this kind of thing is costly for me.
I agree with this, but also think the forum "not being cuddly for Sean" and "not driving contributors away" aren't mutually exclusive. Maybe I am not seeing all the tradeoffs though.
I feel your pain. I hope the amount of upvotes and hearts you're getting helps you feel better, but I know brains don't always work that way (mine doesn't).
Thanks Sean. I think this is a good comment and I think makes me understand your perspective better.
I do think we obviously have large worldview differences here, that seem maybe worth exploring at some point, but this comment (as well as some private conversations sparked by these comments with others at FHI) made me feel more sympathetic to the perspective of "there is some history-rewriting happening that seems scary, where the university gets portrayed as this kind of boogeyman, and while it does seem the university did some unreasonable-seeming things, I think a lack of empathy and practicality in relation to that university had a lot of bad effects on both the university and FHI, and we should be very wary of remembering the story of FHI as a purely one-sided one".
Deleting this because on re-reading I think I'm just repeating myself, but in a more annoyed way. Thanks for checking with other people, I'll leave it at that.
Thank you. I’m grateful you checked with other people. Yes, I do think there is some history rewriting and mythologising going on here compared to my own memory of how things were, and this bothers me because I think the truth does matter.There is a very real sense in which Nick had a pretty sweet setup at Oxford, in terms of having the power and influence to do an unusual thing. And there were a bunch of people around him working insanely hard to help make that happen. I also do think there is a degree to which, yes, Nick blew it. I don’t really want to dwell on this because it feels a bit like bad-mouthing FHI at its funeral. And it’s not the whole story. But it is a source of some frustration to me, because I did not have that position of power and influence in trying to do somewhat similar things, and have spent years banging my head against various walls, and I would have liked to see the FHI story go well. That is not to say i would have made all the right decisions had I had that power and influence (I'm sure I would n... (read more)Yeah I made a similar point here.
Sorry Oli, but what is up with this (and your following) comment?
From what I've read from you[1] seem to value what you call "integrity" almost as a deontological good above all others. And this has gained you many admirers. But to my mind high integrity actors don't make the claims you've made in both of these comments without bringing examples or evidence. Maybe you're reacting to Sean's use of 'garden variety incompetence' which you think is unfair to Bostrom's attempts to tow the fine line between independence and managing university politics but still, I feel you could have done better here.
To make my case:
- When you talk about "other organizations... become a hollow shell of political correctness and vapid ideas" you have to be referring to CSER & Leverhulme here right, like it's the only context that makes sense.
- If not, I feel like that's very misleadingly phrased.
- But if it is, then calling those organisations 'hollow shells' of 'vapid ideas' is like really rude, and if you're going to go there at least have the proof to back it up?
- Now that just might be you having very different politics from CSER & Leverhulme people. But then you say "he [Bostrom] didn't comprom
... (read more)Fwiw I think part of the issue that I had[1] with your comment is that the comment came across much more aggressively and personally, rather than as a critique of an organization. I do think the bar for critiquing individuals ought to be moderately higher than the bar for critiquing organizations. Particularly when the critique comes from a different place/capacity[2] than strictly necessary for the conversation[3].
I expect some other people like JWS had a similar reaction to me, and stronger in magnitude. I did think your comment was on net useful for the conversation (not including the more global effects/externalities).
- ^
- ^
... (read more)Before your comment blew up, I upvoted/agreevoted it because I do think there's a true and important point to be made about FHI being much more successful at doing future-of-humanityish research than "peer" organizations that are more successful at looking like a normal/respectable organization. But I did wince (and didn't strong upvote). I also lacked the information necessary to judge whether your perceived causal models were correct.
Put ano
Sure, social aggression is a rather subjective call. I do think decoupling/locality norms are relevant here. "Garden variety incompetence" may not have been the best choice of words on Sean's part,[1] but it did seem like a) a locally scoped comment specifically answering a question that people on the forum understandably had, b) much of it empirically checkable (other people formerly at FHI, particularly ops staff, could present their perspectives re: relationship management), and c) Bostom's capacity as director is very much relevant to the discussion of the organization's operations or why the organization shut down.
Your comment first presents what I consider to be a core observation that is true and important, namely, FHI did a lot of good work, and this type of magic might not be easy to replicate if you do everything with apparent garden-variety competence. But afterwards, it also brought in a bunch of what I consider to be extraneous details on Sean's competency, judgment, and integrity. The points you raise are also more murkily defined and harder to check. So overall I think of your comment as more escalatory.
- ^
... (read more)or maybe it was under the circumstances. I don't kno
It wasn't carefully chosen. It was the term used by the commenter I was replying to. I was a little frustrated, because it was another example of a truth-seeking enquiry by Milena getting pushed down the track of only-considering-answers-in-which-all-the-agency/wrongness-is-on-the-university side (including some pretty unpleasant options relating to people I'd worked with ('parasitic egregore/siphon money').
>Did Oxford think it was a reputation risk? Were the other philosophers jealous of the attention and funding FHI got? Was a beaurocratic parasitic egregore putting up roadblocks to siphon off money to itself? Garden variety incompetence?
So I just did copy and paste on the most relevant phrase, but flipped it. Bit blunter and more smart-arse than I would normally be (as you've presumably seen from my writing, I normally caveat to a probably-tedious degree), but I was finding it hard to challenge the simplistic fhi-good-uni-bad narrative. It was one line, I didn't think much about it.
I remain of the view that the claim is true/a reasonable interpretation, but de novo / in a different context I would have phrased differently.
One other observation that might explain some of the different perceptions on 'blame' here.
I don't think Oxford's bureaucracy/administration is good, and I think it did behave very badly at points*. But overall, I don't think Oxford's bureaucracy/behaviour was a long way outside what you would expect for the reference class of thousand-year-old-institutions with >10,000 employees. And Nick knew that was what it was, chose to be situated there, and did benefit (particularly in the early days) from the reputation boost. I think there is some reasonable expectation that having made that choice, he would put some effort into either figuring out how to operate effectively within its constraints, or take it somewhere else.
(*it did at point have the feeling of grinding inevitability of a failing marriage, where beyond a certain point everything one side did was perceived in the worst light and with maximal irritation by the other side, going in both directions, which contributed to bad behaviour I think).
For what it's worth, I'm (at least partly) sympathetic to Oli's position here. If nothing else, from my end I'm not confident that the combined time usage of:
[Oli producing book-length critique of CSER/Leverhulme, or me personally, depending] +
[me producing presumably book-length response] +
[further back and forth] +
[a whole lot of forum readers trying to unpick the disagreements]
is overall worth it, particularly given (a) it seems likely to me there are some worldview/cultural differences that would take time to unpick and (b) I will be limited in what I can say on certain matters by professional constraints/norms.
I think this might be one of the LTFF writeups Oli mentions (apologies if wrong), and seems like a good place to start:
https://forum.effectivealtruism.org/posts/an9GrNXrdMwBJpHeC/long-term-future-fund-august-2019-grant-recommendations-1#Addendum__Thoughts_on_a_Strategy_Article_by_the_Leadership_of_Leverhulme_CFI_and_CSER
And as to the claim "I also wouldn't be surprised if Sean's takes were ultimately responsible for a good chunk of associated pressure and attacks on people's intellectual integrity" it seems like some of this is based on my online comments/wri... (read more)
In my experience people update less from positive comments and more from negative comments intuitively to correct for this asymmetry (that it's more socially acceptable to give unsupported praise than unsupported criticism). Your preferred approach to correcting the asymmetry, while I agree is in the abstract better, doesn't work in the context of these existing corrections.
I don't understand your lack of understanding. My point is that you're acting like a right arse.
When people make claims, we expect there to be some justification proportional to the claims made. You made hostile claims that weren't following on from prior discussion,[1] and in my view nasty and personal insinuations as well, and didn't have anything to back it up.
I don't understand how you wouldn't think that Sean would be hurt by it.[2] So to me, you behaved like arse, knowing that you'd hurt someone, didn't justify it, got called out, and are now complaining.
So I don't really have much interest in continuing this discussion for now, or much opinion at the moment of your behaviour or your 'integrity'
Like nobody was discussing CSER/CFI or Sean directly until you came in with it
Even if you did think it was justified
Sean is one of the under-sung heroes who helped build FHI and kept it alive. He did this by--among other things--careful and difficult relationship management with the faculty. I had to engage in this work too and it was less like being between a rock and a hard place and more like being between a belt grinder and another bigger belt grinder.
One can disagree about apportioning the blame for this relationship--and in my mind, I divide it differently than Sean--but after his four years of first-hand experience, my response to Sean is to take his view seriously, listen, and consider it. (And to give it weight even against my 3.5 years of first-hand experience!)
As a tangent, respectfully listening to people's views and expressing gratitude--and avoiding unnecessary blame--was a core part of what allowed ops and admin staff to keep FHI alive for so long against hostile social dynamics. As per Anders' comment posted by Pablo here, it might be useful for extending EA's productive legacy as well.
Sean thank you so much for all you did for FHI.
Sad to hear this happened, but it seems the situation was irrecoverable, and the organisation was already dead for a bit before it officially shuttered.
Glad for this post and all the comments.
From Bostrom's website, an updated "My Work" section reads:
A very interesting summary, thanks.
However I'd like to echo Richard Chappell's unease at the praising of the use of short-term contracts in the report. These likely cause a lot of mental health problems and will dissuade people who might have a lot to contribute but can't cope with worrying about whether they will need to find a new job or even career in a couple of years' time. It could be read as a way of avoiding dealing with university processes for firing people - but then the lesson for future organisations may be to set up outside a university structure, and have a sensible degree of job security.
FHI almost singlehandedly made salient so many obscure yet important research topics. To everyone that contributed over the years, thank you!
Nick Bostrom's website now lists him as "Principal Researcher, Macrostrategy Research Initiative."
Doesn't seem like they have a website yet.
The Macrostrategy Research Initiative was registered as a company in August 2023. Its Director is Toby Newberry ("Occupation: Chief of Staff"), who's been at FHI/GPI for a few years.
https://find-and-update.company-information.service.gov.uk/company/15054502
For anyone wondering about the definition of macrostrategy, the EA forum defines it as follows:
Executive summary: The Future of Humanity Institute (FHI) achieved notable successes in its mission from 2005-2024 through long-term research perspectives, interdisciplinary work, and adaptable operations, though challenges included university politics, communication gaps, and scaling issues.
Key points:
- Long-term research perspectives and pre-paradigmatic topics were key to FHI's impact, enabled by stable funding.
- An interdisciplinary and diverse team was valuable for tackling neglected research areas.
- Operations staff needed to understand the mission as it g
... (read more)