Introduction
This post seeks to estimate how much we should expect a highly cost-effective charity to spend on reducing existential risk by a certain amount. By setting a threshold for cost-effectiveness, we can be selective about which longtermist charities to recommend to donors.
We appreciate feedback. We would like for this post to be the first in a sequence about cost-effectiveness thresholds for giving, and your feedback will help us write better posts.
How many beings does extinction destroy?
This chart gives six estimates for the size of the moral universe that would be lost in an extinction event on Earth this century. There is a truly incredible range in the possible size of the moral universe, and the value you see in the future depends on the moral weights you put on different types of moral patients.
(Click this link to explore the value of the future. You can change the moral weights on humans, vertebrates, invertebrates, and beings of the future; the number of years you expect humans and animals to live without inevitable extinction; and the number of animals and humans you expect to exist at one time, from conservative to liberal estimates. The value at the bottom, T, shows you the total value you put on the future. You can then see how changing your moral weights radically influences the value you put on the future – and you can recalculate the cost-effectiveness thresholds below based on that.)
If the universe could have between 8 x 10^9 and 5 x 10^55 morally valuable beings, then a 0.01% absolute reduction in cumulative x-risk is “equivalent” (given several assumptions) to saving 8 x 10^5 to 5 x 10^51 lives. This is a reduction in risk of one basis point, or bp. Note that this is a much greater accomplishment than reducing per-century x-risk by one basis point, which we will discuss later.
How much should we pay to prevent this destruction?
Using near-termist thresholds as a starting point
Using the SoGive Gold Standard for cost per life saved in a near-termist framework (£5,000), this would mean a “good cost” for a 0.01% absolute reduction in the cumulative risk of extinction would be between £4 billion and £2.5 * 10^55. For context, all the money in the world is probably between £10^13 and £10^14.
Are either of these the thresholds we should be using? The cost-effectiveness of longtermist interventions is different from near-termist interventions:
- The interventions we are analyzing are of a different character.
- The evidence of their cost-effectiveness is more speculative.
- It might be much cheaper, or much more expensive, to prevent the loss of future life than to save a person now. It would not be useful to set a threshold that ~all highly effective longtermist charities would be above or below, when the purpose of a threshold is to help us select the top tier of existing cost-effective interventions within a cause area.
Furthermore, we might never have enough evidence to say whether an intervention has reduced cumulative x-risk by a certain amount. It might be more manageable to set a threshold based on reduction in per-century x-risk.
Per-century risk
Epistemic status: Uncertain. Please leave feedback and help us improve the math here.
Briefly looking at this, let’s assume that humans will last some time into the future, and that those humans are our moral patients. (If we didn’t put value on future humans, then cumulative risk would be equal to the risk of extinction for present beings. This would give us a lower bound on a good cost for a 0.01% absolute reduction in per-century risk of £4 billion.)
Let’s also assume for now that we are not in a time of perils, so that we have a simpler relationship between cumulative risk and per-century risk. This will give us a lower bound for what we should spend on a reduction in per-century risk under our assumption above. (Given the chance we are in a time of perils, a “good cost” for a reduction in per-century risk this century would only be higher. If we are in an extreme time of perils where a reduction in per-century risk is about equivalent to a reduction in cumulative risk, then we already have our upper bound on a good cost for a 0.01% reduction in per-century risk – £2.5 * 10^55.)
According to David Thorstad, assuming a future of 1 billion years, an absolute reduction in cumulative x-risk of only 1/100,000,000 requires that we reduce per-century x-risk down from its current figure (estimated by Toby Ord at 1/6) at least down to below 16/10,000,000.[1]
Let’s try to use the same math. On the low end of the size of the future, suppose humans could exist for a maximum of 800,000 more years before inevitable extinction. Then, an absolute reduction in cumulative risk of 1/10,000 (1 bp) would require us to drive per-century risk at least down to below 11/10,000.
Bringing per-century risk from ⅙ to 11/10,000 is an absolute reduction of 1656 basis points. We might say that if the future is 800,000 years and there are 10^14 moral patients (low end) that can exist in that time, then we should pay 10^14 * 0.01% * £5000 / 1656 = £30.19 billion[2] to reduce per-century x-risk by 1 bp.
On the high end of the size of the future, suppose humans could exist for a maximum of 100 billion years before inevitable extinction. Then, an absolute reduction in cumulative risk of 1/10,000 (1 bp) would require us to drive per-century risk at least down to below 92/10,000,000,000.
Bringing per-century risk from ⅙ to 92/10,000,000,000 is an absolute reduction of 1667 basis points. We might say that if the future is 100 billion years and there are 5 x 10^55 moral patients (high end) that can exist in that time, then we should pay 5 * 10^55 * 0.01% * £5000 / 1667 = £1.50 × 10^52 to reduce per-century x-risk by 1 bp.
Clearly, this is a very difficult task in either situation. What does this mean for our threshold? Do we think that reducing cumulative x-risk by one basis point is worth a different amount, when we know that it means reducing per-century x-risk so extremely?
- Should we tolerate a higher cost threshold for a more difficult task?
- Should we reduce our tolerance for the cost, since the task may be less tractable?
Overall, our range for what we should spend on a reduction in a 0.01% absolute reduction in x-risk for this century is £4 billion, if you only put moral value on present humans; £30.19 billion, if you believe the future is small, the class of moral patients is small, and we are not in a time of perils; £1.50 × 10^52, if you predict a very large future and a large class of moral patients, and believe we are not in a time of perils; and £2.5 * 10^55, if you predict a very large future and a large class of moral patients, and believe we are in an extreme time of perils.
Using benchmarks for cost-effectiveness from current longtermist charities
We are currently seeking estimates of how much longtermist charities are reducing x-risk. These estimates can give us a benchmark for cost-effectiveness. Benchmarking is another method we can use to set a threshold. However, estimates we get from charities for their own cost-effectiveness are likely to be biased.
Using the estimates of others
Now let’s look at cost-effectiveness thresholds from others, taken from this post by Linch and this post by Vasco and their comments sections, which use USD. The table below is adapted and expanded from Vasco’s post.
Most commenters seem to have been thinking of an absolute reduction of one basis point this century (it’s unclear) – but one commenter, NunoSempere, gave a range for an absolute reduction by one basis point in the yearly x-risk for one century, and another range for an absolute reduction by one basis point in the x-risk this century. The commenters used a range of methods to answer the question, but their answers clustered around $1 billion.
Please note Linch’s disclaimer, which likely also applies to the estimates of others:
“EDIT 2022/09/21: The 100M-1B estimates are relatively off-the-cuff and very not robust, I think there are good arguments to go higher or lower. I think the numbers aren't crazy, partially because others independently come to similar numbers (but some people I respect have different numbers). I don't think it's crazy to make decisions/defer roughly based on these numbers given limited time and attention. However, I'm worried about having too much secondary literature/large decisions based on my numbers, since it will likely result in information cascades. My current tentative guess as of 2022/09/21 is that there are more reasons to go higher (think averting x-risk is more expensive) than lower. However, overspending on marginal interventions is more -EV than underspending, which pushes us to bias towards conservatism.”
| Estimator | Highly cost-effective | Middling (default column if not specified) | Upper bound |
| Linch | $100 million per bp | $300 million per bp to $1 billion per bp | $10 billion per bp |
| Ajeya Cotra | “AI risk is something that we think has a currently higher cost effectiveness” | “$200 trillion per world saved” Or $20 billion per bp | |
| Oliver Habryka | “I think we are currently funding projects that are definitely more cost-effective than that” | “My very rough gut estimate says something like” $1 billion per bp | “Probably more on the margin” |
| NunoSempere | $99 billion per bp, lower bound assuming a “one-off 0.01% existential risk reduction over a century” | $16 trillion per bp, lower bound per yearly reduction of 1bp … to $330T per bp, upper bound assuming a “one-off 0.01% existential risk reduction over a century” | $3.8 x 10^15 per bp, upper bound per yearly reduction of 1bp |
| Anonymous person from Vasco’s post | $1 trillion per existential catastrophe averted, or $100 million per bp | $100 trillion per existential catastrophe averted, or $10 billion per bp | |
| Simon Skade | “Note that I do think that there are even more effective ways to reduce x-risk, and in fact I suspect most things longtermist EA is currently funding have a higher expected x-risk reduction than 0.01% per 154M$.” | $154 million per bp | “I just don't think that it is likely that the 50 billionth 2021 dollar EA spends has a much higher effectiveness than 0.01% per 154M$, so I think we should grant everything that has a higher expected effectiveness.” |
| William Kiely | $34 million per bp | “I thought it would be interesting to answer this using a wrong method (writing the bottom line first).” ~ $100 million per bp | $340 million per bp |
| Zach Stein-Perlman | $25 million per bp | $50 million per bp | “Unlike Linch, I would be quite sad about trading $100M for a single measly basis point — $100M (granted reasonably well) would make a bigger difference, I think.” $100 million per bp |
| Median | $100 million per bp | $1 billion per bp | $5.17 billion per bp |
This suggests that a reasonable range for the cost of reducing absolute x-risk by 0.01% this century could be between USD $100 million and $5.17 billion, or £75 million and £3.88 billion.[3]
Earlier, we put a “good cost” of a 1 bp absolute reduction in per-century x-risk, based on SoGive’s near-termist Gold Standard, between £4 billion and £2.5 * 10^55, depending on how much you value the future what parts of the future you value, and your forecast for the future. Note that the estimates given in this table are all below that range.
Why is that?
- Were these commenters expecting it to be much cheaper to save a life by preventing the loss of potential in an extinction, than to save a life using near-termist interventions?
- Were they envisioning a relatively small future?
- Were they discounting the future?
Probably, instead of these possibilities, the commenters were giving an implicit haircut to their estimates based on uncertainty.
What does this say for our threshold?
Conclusion
For near-termist interventions, when SoGive has high confidence that the intervention is effective, we use a Gold Standard. A tentative threshold for a “SoGive Longtermist Gold Standard” could be £750 million / bp. This is the median estimate given by the commenters above.
Having a threshold for cost-effectiveness can help us decide whether to give now or give later. If all the available funding opportunities cost more than our threshold, then we can hold onto our donations until a more cost-effective opportunity comes along. We may come back to the nuanced relationship between thresholds and give now/give later in a future post.
There remains a practical problem with using a cost-effectiveness threshold for deciding where to give within longtermism. Most longtermist projects do not publish cost-effectiveness estimates – and if they did, they could be as flawed as misleading cost-effectiveness estimates from near-termist charities. For third-party evaluators, there is no clearly reliable method for estimating how much x-risk has been reduced by one project.
Thus, a threshold for longtermist cost-effectiveness could be better applied to the average cost-effectiveness of categories, e.g., “AI risk-reducing projects in Europe,” rather than to individual projects. (It is not clear that cost-effectiveness for projects varies less within geographic regions than across regions – this is just an example of a category of funding opportunities.) Analyzing cost-effectiveness over a larger category of funding opportunities, instead of per-intervention or per-project, could help us avoid false precision.
These estimates are not robust enough to make the most important decisions we face. We recommend conducting a survey of funders, charities, and experts to get a stronger picture of what the standard should be and the cost-effectiveness of different types of work.
- ^
This represents a reduction from a cumulative risk of 1 to a cumulative risk of 0.9999. If we are starting with a cumulative risk below 1, then we need to reduce per-century risk somewhat more, but likely not orders of magnitude more.
- ^
(number of moral patients) * (% saved in expectation given 0.01% reduction in cumulative risk) * (“good cost” to save a life) / (conversion from 0.01% reduction in cumulative risk to 0.01% reduction in per-century risk)
- ^
Converting at an average exchange rate of 0.75.
This was a difficult post, and my first post for SoGive as the Lead Researcher & Philanthropy Advisor! I hope it can be useful to our discussions on cost-effectiveness.
I hope my uncertainty comes though. I haven't been thinking about the size of the future for a very long time, but I learned a lot from writing this. As I mentioned at the beginning, please leave feedback on my assumptions, math, and methods, so I can write better posts about thresholds in the future.
It might be a while, but I'd like to do some writing about cost-effectiveness thresholds for animal advocacy and multipliers as well. Feel free to leave your thoughts about those as a reply to this comment as well.
Interesting read, and a tricky topic! A few thoughts:
Thanks Matt!
Re 2 - ah yeah, I was assuming that at least one alien civilisation would aim to 'technologize the Local Supercluster' if humans didn't. If they all just decided to stick to their own solar system or not spread sentience/digital minds, then of course that would be a loss of experiences.
Thanks for clarifying 1 and 3!
Great that SoGive is setting these benchmarks. I'm of the opinion that we are very much in a time of perils, and indeed the entire future may well hinge on the next few years (or less), with regard to AI x-risk. I also think that work aimed at slowing down AI is woefully neglected. Right now, I think such work (examples) could easily be as cost effective as $1M per bp of x-risk reduction (this century at least).
Note also that if timelines are short, then the cost effectiveness of saving lives via traditional neartermist interventions decreases. IIRC, GiveWell equates a "life saved" with ~40 years extra life (or 40 QALYs). So if we only have 5 years left before extinction, then the cost effectiveness estimates worsen by a factor of 8 (40/5). So instead of £5,000 to save a life, neartermist interventions would need to clear a bar of £625 to save a life. Although I think at this point we might be better off just thinking in terms of £/QALY. And interventions aimed at averting severe depression ("£200 to avert one year of severe depression") or chicken suffering ("£5 to avert the suffering of one chicken who is living in very poor conditions") seem more promising.
Interesting point about how any extinction timelines less than the length of a human life change the thresholds we should be using for neartermism as well! Thank you, Greg. I'll read what you linked.
Also, for this reason I think that x-risk reduction should very much not be lumped in with longtermism. It is, in fact, very near term, now.
A couple of nitpicks on the calculations for the size of the moral universe:
This is not to say that we won't go extinct from artificial environmental and evolutionary causes of our own making. Just that bounding to a number (1M years) that originates from natural (nonanthropogenic) processes seems highly arbitrary.
Directionally, I agree with your points. On the last one, I’ll note that counting person-years (or animal-years) falls naturally out of empty individualism as well as open individualism, and so the point goes through under the (substantively) weaker claim of “either open or empty individualism is true”.[1]
(You may be interested in David Pearce’s take on closed, empty, and open individualism.)
For the casual reader: The three candidate theories of personal identity are empty, open, and closed individualism. Closed is the common sense view, but most people who have thought seriously about personal identity—e.g., Parfit—have concluded that it must be false (tl;dr: because nothing, not memory in particular, can “carry” identity in the way that's needed for closed individualism to make sense). Of the remaining two candidates, open appears to be the fringe view—supporters include Kolak, Johnson, Vinding, and Gomez-Emilsson (although Kolak's response to Cornwall makes it unclear to what extent he is indeed a supporter). Proponents of (what we now call) empty individualism include Parfit, Nozick, Shoemaker, and Hume.
Agree. I find Empty Individualism pretty depressing to think about though. And Open Individualism seems more natural, from (my) subjective experience.
If we're looking at upper bounds, even the Stelliferous Era is highly conservative. The Black Hole era could last up to 10<sup>100</sup> years https://en.wikipedia.org/wiki/The_Five_Ages_of_the_Universe and it's at least conceivable under known physics that we could farm their rotational energy or still more speculatively their Hawking radiation.
Usual Pascalian reasoning applies in that this would allow such a ridiculously large number of person-years that even with an implausibly low credence in its possibility the expectation dwarfs the whole stellar era.
The problem with this position is that the Black Hole Era—at least, the way the “Five Ages of the Universe” article you link to defines it—only starts after proton decay has run to (effective) completion,[1] which means that all matter will be in black holes, which means that conscious beings will not exist to farm black holes for their energy. (If do, however, agree that life is in theory not dependent on luminous stars, and so life could continue beyond the Stelliferous Era and into the Degenerate Era, which adds many years.)
Whether proton decay will actually happen is still a major open question in physics. See, for example, Hadhazy (2021) or Siegel (2020).
(Additionally, if proton decay does happen, there's then the question of “could a technologically mature civilization stop proton decay?”. My money would be on “no”, but of course our current understanding of particle decay physics could be incorrect, or an advanced civilization might find an ingenious workaround.)
As you say, whether proton decay will happen seems to be an open question. If you're feeling highly confident you could knock off another couple of zeroes to represent that credence and still end up with a number that eclipses everything else.
Hi Spencer,
You may be interested in Brian Tomasik's analysis on How the Simulation Argument Dampens Future Fanaticism. I think its essence is well captured in this short comment by Pablo Stafforini:
If I recall correctly, the argument goes more or less as follows. The larger the future, the greater the likelihood of us being in a short-lived simulation, thus having negligible influence in the far future. The 2 effects cancel out, and therefore the ratio between far future value and near future value does not depend on the size of the future. The ratio is roughly inversely proportional to the fraction of resources going towards simulations, i.e. 2 times as much resources going to simulations means the far future is half as valuable relative to the near term future.
The possibility of us living in a short-lived simulation isn't enough to count much against longtermism, because it's also possible we could live in a long-lived simulation or a long-lived world, and those possibilities will be much higher stakes, so still dominate expected value calculations unless we assign them tiny probability together.
I think the argument crucially depends on the assumption that simulations will be disproportionately short-lived, and we have acausal influence over agents in other simulations. If for each long-running world (simulated or otherwise) with moral agents and moral patients, there are N short-lived worlds with (moral) agents and moral patients, and our actions are correlated with those of agents across worlds, then we get to decide for more agents in in short-lived worlds than long-lived ones. Basically, acausal influence will boost the expected value of all interventions, but if moral patients are disproportionately in short-lived simulations with agents whose decisions we're correlated with relative to long-run simulations with agents whose decisions we're correlated with (or more skewed towards the short-lived than it seems for our own world), acausal influence will disproportionately boost the expected value of neartermist interventions relative to longtermist ones.
Also, ~all of the expected value will be acausal if we fully count the value of acausal influence, based on the evidentialist's wager and similar, given the possibility of very large or even infinite numbers of agents with whom we're correlated.
Thanks for clarifying, Michael!
Yes, the argument depends on Brian's parameter F not being super small. F is "fraction of all computational sent-years spent non-solipsishly simulating almost-space-colonizing ancestral planets (both the most intelligent and also less intelligent creatures on those planets)". "A non-solipsish simulation is one in which most or all of the people and animals who seem to exist on Earth are actually being simulated to a non-trivial level of detail". Brian guessed F = 10^-6, but it feels like it should be much smaller to me. If the value of the future is e.g. 10^30 times the value of this century, it is maybe reasonable to assume that the vast vast majority of computational sent-years are also simulations of the far future, as opposed to simulations of almost-space-colonizing ancestral planets.
I find this argument unconvincing. The vast majority of 'simulations' humans run are very unlike our actual history. The modal simulated entity to date is probably an NPC from World of Warcraft, a zergling from Starcraft or similar. This makes it incredibly speculative to imagine what our supposed simulators might be like, what resources they might have available and what their motivations might be.
Also the vast majority of 'simulations' focus on 'exciting' moments - pitched Team Fortress battles, epic RPG narratives, or at least active interaction with the simulators. If you and your workmates are just tapping away in your office on your keyboard doing theoretical existential risk research, the probability that someone like us has spent their precious resources to (re)create you seem radically lowered than if you're (say) fighting a pitched battle.
Thanks for commenting!
Agreed, but could you explain why that would be an objection to Brian's argument?
I do not know, because I agree with your 1st paragraph about it being quite hard to predict future simulated entities based on past history.
I mainly had in mind Pablo's summary. It's been a long time since I read Brian's essay, and I don't have bandwidth to review it now, so if he says something substantially different there, my argument might not apply. But basically every argument I remember hearing about how the simulation argument implies we should modify our behaviour presupposes that we have some level of inferential knowledge of our simulators (this presupposition being hidden in the assumption that simulations would be primarily ancestor simulations). This presupposition seems basically false to me, because, for example:
a. A zergling would struggle to gain much inferential knowledge of its simulators' motivations.
b. A zergling looking around at the scope and complexity of its universe would typically observe that it itself is 2-dimensional (albeit with some quasi-3D properties), and is made from approx 38x94 'atoms'. Perhaps more advanced simulations would both be more numerous (and hence a higher proportion of simulationspace) and more complex, but it still seems hard to imagine they'll average to anything like the same level of complexity as we see in our universe, or have a consistent difference from it.
c. If the simulation argument is correct for a single layer of reality, it seems (to the degree permitted by a and b) far more likely that it's correct for multiple, perhaps vast numbers of layers of reality (insert 'spawn more Overlords' joke here). Thus the people whose decisions and motivations a zergling is trying to ultimately guess at is not ours, but someone whose distance from us is approx n(|human - zergling|), where n is the number of layers. It's hard to imagine the zergling - or us - could make any intelligible assumptions at all about them at that level of removal.
To show this in Pablo's argument:
For this to be 'plausible' is to assert that we know our simulators' motivations well enough to know that whatever they hoped to gain by running us will 'plausibly' be motivating enough for them to do it a second time in much the same form, and that their simulators will at least permit it, and so on.
Another version of the anti-x-risk argument from simulation I've heard (and which I confess with hindsight I was conflating Pablo's with - maybe it's part of Brian's argument?) is that the simulators will likely switch off our universe if it expands beyond a certain size due to resource constraints. Again, this argument implies IMO vastly too high confidence in both their motivation and resource limits.
Thanks for explaining that!
Brian concludes that L/S = T*D/F, where:
Brian guesses T = 10^4, D = 10^-3, and F = 10^-6, thus concluding L/S = 10^7. I guess you are saying with your comment just above that F should be much lower than 10^-6? For reference, here is Brian's motivation for F = 10^-6:
The informality of that equation makes it hard for me to know how to reason about it. For eg,
If you feel like you understand it, maybe you could give me a concrete example of how to apply this reasoning?
For what it's worth, I have much more prosaic reasons for doubting the value of explicitly longtermist work both in practice (the stuff I've discussed with you before that makes me feel like it's misprioritised) and in principle (my instinct is that in situations that reduce to a kind of Pascalian mugging, xP(x) where x is a counterfactual payoff increase and P(x) is the probability of that payoff increase, approaches 0 as x tends to infinity).
I agree.
I think F = "sent-years respecting the simulations of the beings in almost-space-colonizing ancestral planets"/"all sent-years of the universe". Brian defines sent-years as follows:
I said Brian concluded that L/S = T*D/F, but this was after simplifying L/S = T*D/(E/N + F), where:
Then Brian says:
The simulation argument dampening future fanaticism comes from Brian assuming that E/N << F, in which case L/S = T*D/F, and therefore prioritising the future no longer depends on its size. However, for the reasons you mentioned (we are not simulating our ancestors much), I feel like we should a priori expect E/N and F to be similar, and correlated, in which case L/S will still be huge unless it is countered by a very small D (i.e. if the typical low tractability argument against longtermism goes through).
I think L/S is just supposed to be a heuristic for how much to prioritise longtermist actions relative to neartermist ones. Brian's inputs lead to 10^7, but they were mainly illustrative:
However, it seems to me that, even if one thinks that both E/N and F are super small, L/S could still be smaller than 1 due to super small D. This relates to your point that:
I share your instinct. I think David Thorstad calls that rapid diminution.
I think Brian's reasoning works more or less as follows. Neglecting the simulation argument, if I save one life, I am only saving one life. However, if F = 10^-16[1] of sentience-years are spent simulating situation like my own, and the future contains N = 10^30 sentience-years, then me saving a life will imply saving F*N = 10^14 copies of the person I saved. I do not think the argument goes through because I would expect F to be super small in this case, such that F*N is similar to 1.
Brian's F = 10^-6 divided by the human population of 10^10.
Appreciate the patient breakdown :)
This is more of a sidenote, but given all the empirical and model uncertainty in any far-future oriented work, it doesn't seem like adding a highly speculative counterargument with its own radical uncertainties should meaningfully shift anyone's priors. It seems like a strong longtermist could accept Brian's views at face value and say 'but the possibility of L/S being vastly bigger than 1 means we should just accept the Pascalian reasoning and plow ahead regardless', while a sceptic could point to rapid diminution and say no simulationy weirdness is necessary to reject these views.
(Sidesidenote: I wonder whether anyone has investigated the maths of this in any detail? I can imagine there being some possible proof by contradiction of RD, along the lines of 'if there were some minimum amount that it was rational for the muggee to accept, a dishonest mugger could learn that and raise the offer beyond it whereas an honest mugger might not be able to, and therefore, when the mugger's epistemics are taken into account, you should not be willing to accept that amount. Though I can also imagine this might just end up as an awkward integral that you have to choose your values for somewhat arbitrarily)
For the record, this kind of thing is why I love Brian (aside from him being a wonderful human) - I disagree with him vigorously on almost every point of detail on reflection, but he always come up with some weird take. I had either forgotten or never saw this version of the argument, and was imagining the version closer to Pablo's that talks about the limited value of the far future rather than the increased near-term value.
That said, I still think I can basically C&P my objection. It's maybe less that I think F is likely to be super small, and more that, given our inability to make any intelligible statements about our purported simulators' nature or intentions it feels basically undefined (or, if you like, any statement whatsoever about its value is ultimately going to be predicated on arbitrary assumptions), making the equation just not parse (or not output any value that could guide our behaviour).
Interesting. But how soon is "soon"? And even if we are a simulation, to all intents and purposes it is real to us. It doesn't seem like much of a consolation that the simulators might restart the simulation after we go extinct (any more than the Many Worlds interpretation of Quantum Mechanics gives solace over many universes still existing nearby in probability space in the multiverse).
Maybe the simulators will stage an intervention over us reaching the Singularity. I don't think we can rely on this though (indeed, this is part of the exotic scenarios that make up the ~10% chance that I think we aren't doomed from AGI by default).
Thanks for engaging, Greg!
I seem to remember a comment from Carl Shulman saying the risk of simulation shut-down should not be assumed to be less than 1 in 1 M per year (or maybe it was per century). This suggests there is still a long way before it happens. On the other hand, I would intuitively think the risk to be higher if the time we are in really is special. I do not remember whether the comment was taking that into account.
Yes, it is not a consolation. It is an argument for focussing more on interventions which have nearterm benefits, like corporate campaigns for chicken welfare, instead of ones whose benefits may not be realised due to simulation shut-down.
I still don't think this goes through either. I'm saying we should care about our world going extinct just as much as if it were the only world (given we can't causally influence the others).
Agreed, but if the lifespan of the only world is much shorter due to risk of simulation shut-down, the loss of value due to extinction is smaller. In any case, this argument should be weighted together with many others. I personally still direct 100 % of my donations to the Long-Term Future Fund, which is essentially funding AI safety work. Thanks for your work in this space!
Thanks for your donations to the LTFF. I think they need to start funding stuff aimed at slowing AI down (/pushing for a global moratorium on AGI development). There's not enough time for AI Safety work to bear fruit otherwise.
Thank you Vasco! This seems hard to model, but worthwhile. I'll think on it.
Cool! One point from a quick skim - the number of animals wouldn't be lost in many kinds of human extinction events or existential risks. Only a subset would erase the entire biosphere - e.g. a resource-maximising rogue AI, vacuum decay, etc. Presumably with extinction of just humans the animal density of reclaimed land would be higher than current, so the number of animals would rise (assuming it outweighs the end of factory farming).
The implications of human existential risks for animals is interesting, and I can see some points either way depending on the moral theory (e.g. end of factory farming in human extinction, but rise of wild animal suffering; total number and quality of animal lives in a beyond-Earth humanity; potential of a completely re-wilded Earth in a beyond-Earth humanity; risks of astronomical suffering if a beyond-Earth humanity retains the equivalent of factory farming...)
Thanks Ben! I totally agree. The math in this post was trying to get at upper and lower bounds and a median -- but for setting one's personal thresholds, the nuance you mention is incredibly important. I hope this post, and the Desmos tool I linked, can help people play with these numbers and set their own thresholds!
Hi Spencer,
You mention the following methods to estimate thresholds:
I think the 2nd is the most promising. It is the one employed to establish thresholds for neartermist charities, which are usually benchmarked against GiveWell's top charities. Open Philanthropy is arguably the largest funder of longtermist projects, so I think it would be valuable to:
In addition, I believe it is worth estimating the cost-effectiveness of longtermist charities in terms of a neartermist metric like DALYs averted per $ without accounting for future generations, and then see how they compare with GiveWell's top charities.
Nice post, Spencer!
Nitpick, an absolute reduction of 10^-4 is often indicated as 0.01 pp (0.01 percentage points).
I would go a little further, and say that we might never have enough evidence to say whether non-extinction x-risk this century was reduced, as I think the evidence base for non-extinction value lock-in is quite poor (related links here). So I believe it is better to focus on extinction risk, or probability of a given population loss, as forecasted in the Existential-Risk Persuasion Tournament.
“$200 trillion per world saved”
Or $20 billion per bp
To clarify, the above estimate is a conservative cost-effectiveness bar for Open Philanthropy's longtermist grants. In this section of episode 90 of The 80,000 Hours Podcast, Ajeya says it concerns "meta R&D to make responses to new pathogens faster", and "[Open Philanthropy] were aiming for this to be conservative".
I guess so.
I would say better quantitative models would also be needed to more reliably estimate the cost-effectiveness of interventions aiming to decrease extinction risk.
Thanks Vasco! This helps my understanding.
I think that commenters are looking at the cost-effectiveness they could reach with current budget constraints. If we had way more money for longtermism, we could go to a higher cost per basis point. That is different than the value of reducing a basis point, which very well could be astronomical, given GiveWell costs for saving a life (though to be consistent, one should try to estimate the long-term impacts of a GiveWell intervention as well).