EA - Mistakes in the moral mathematics of existential risk (Part 1: Introduction and cumulative risk) - Reflective altruism by BrownHairedEevee

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mistakes in the moral mathematics of existential risk (Part 1: Introduction and cumulative risk) - Reflective altruism, published by BrownHairedEevee on July 3, 2023 on The Effective Altruism Forum.This is the first part of "Mistakes in the moral mathematics of existential risk", a series of blog posts by David Thorstad that aims to identify ways in which estimates of the value of reducing existential risk have been inflated. I've made this linkpost part of a sequence.Even if we use . conservative estimates, which entirely ignor[e] the possibility of space colonization and software minds, we find that the expected loss of an existential catastrophe is greater than the value of 1016 human lives. This implies that the expected value of reducing existential risk by a mere one millionth of one percentage point is at least a hundred times the value of a million human lives.Nick Bostrom, “Existential risk prevention as global priority”1. IntroductionThis is Part 1 of a series based on my paper, “Mistakes in the moral mathematics of existential risk.”(Almost) everyone agrees that human extinction would be a bad thing, and that actions which reduce the chance of human extinction have positive value. But some authors assign quite high value to extinction mitigation efforts. For example:Nick Bostrom argues that even on the most conservative assumptions, reducing existential risk by just one millionth of one percentage point would be as valuable as saving a hundred million lives today.Hilary Greaves and Will MacAskill estimate that early asteroid-detection efforts saved lives at an expected cost of fourteen cents per life.These numbers are a bit on the high side. If they are correct, then on many philosophical views the truth of longtermism will be (nearly) a foregone conclusion.I think that these, and other similar estimates, are inflated by many orders of magnitude. My paper and blog series “Existential risk pessimism and the time of perils” brought out one way in which these numbers may be too high: they will be overestimates unless the Time of Perils Hypothesis is true.My aim in this paper is to bring out three novel ways in which many leading estimates of the value of existential risk mitigation have been inflated. (The paper should be online as a working paper within a month.)I’ll introduce the mistakes in detail throughout the series, but it might be helpful to list them now.Mistake 1: Focusing on cumulative risk rather than per-unit risk.Mistake 2: Ignoring background risk.Mistake 3: Neglecting population dynamics.I show how many leading estimates make one, or often more than one of these mistakes.Correcting these mistakes in the moral mathematics of existential risk has two important implications.First, many debates have been mislocated, insofar as factors such as background risk and population dynamics are highly relevant to the value of existential risk mitigation, but these factors have rarely figured in recent debates.Second, many authors have overestimated the value of existential risk mitigation, often by many orders of magnitude.In this series, I review each mistake in turn. Then I consider implications of this discussion for current and future debates. Today, I look at the first mistake, focusing on cumulative rather than per-unit risk.2. Bostrom’s conservative scenarioNick Bostrom (2013) considers what he terms a conservative scenario in which humanity survives for a billion years on the planet Earth, at a stable population of one billion humans.We will see throughout this series that is far from a conservative scenario. Modeling background risk (correcting the second mistake) will put pressure on the likelihood of humanity surviving for a billion years. And modeling population dynamics (correcting the third mistake) will raise the possi...