This is the second part of "Mistakes in the moral mathematics of existential risk", a series of blog posts by David Thorstad that aims to identify ways in which estimates of the value of reducing existential risk have been inflated. I've made this linkpost part of a sequence.

In the decades to come, advanced bioweapons could threaten human existence. Although the probability of human extinction from bioweapons may be low, the expected value of reducing the risk could still be large, since such risks jeopardize the existence of all future generations. We provide an overview of biotechnological extinction risk, make some rough initial estimates for how severe the risks might be, and compare the cost-effectiveness of reducing these extinction-level risks with existing biosecurity work. We find that reducing human extinction risk can be more cost-effective than reducing smaller-scale risks, even when using conservative estimates. This suggests that the risks are not low enough to ignore and that more ought to be done to prevent the worst-case scenarios.
Millett and Snyder-Beattie, “Existential risk and cost-effective biosecurity”

1. Introduction

This is Part 2 of a series based on my paper “Mistakes in the moral mathematics of existential risk”.

Part 1 introduced the series and discussed the first mistake: focusing on cumulative rather than per-unit risk. We saw how bringing the focus back to per-unit rather than cumulative risk was enough to change a claimed `small’ risk reduction of one millionth of one percent into an astronomically large reduction that would drive risk to almost one in a million per century.

Today, I want to focus on a second mistake: ignoring background risk. The importance of modeling background risk is one way to interpret the main lesson of my paper and blog series “Existential risk pessimism and the time of perils.” Indeed, in Part 7 of that series, I suggested just this interpretation.

It turns out that blogging is sometimes a good way to write a paper. Today, I want to expand my discussion in Part 7 of the existential risk pessimism series to clarify the second mistake (ignoring background risk) and to show how it interacts with the first mistake (focusing on cumulative risk) in a leading discussion of cost-effective biosecurity.

Some elements of this discussion are lifted verbatim from my earlier post. In my defense, I remind my readers that...

Mistakes in the moral mathematics of existential risk - Reflective altruism