EA - AI strategy nearcasting by Holden Karnofsky

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI strategy nearcasting, published by Holden Karnofsky on August 26, 2022 on The Effective Altruism Forum. Cross-posted from the Alignment Forum / Less Wrong This is the first in a series of pieces taking a stab at dealing with a conundrum: I believe this could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future. But when it comes to what actions we can take to help such a development go well instead of poorly, it’s hard to say much (with a few exceptions). This is because many actions that would be helpful under one theory of how things will play out would be harmful under another (for example, see my discussion of the “caution” frame vs. the “competition” frame). It seems to me that in order to more productively take actions (including making more grants), we need to get more clarity on some crucial questions such as “How serious is the threat of a world run by misaligned AI?” But it’s hard to answer questions like this, when we’re talking about a development (transformative AI) that may take place some indeterminate number of decades from now. This piece introduces one possible framework for dealing with this conundrum. The framework is AI strategy nearcasting: trying to answer key strategic questions about transformative AI, under the assumption that key events (e.g., the development of transformative AI) will happen in a world that is otherwise relatively similar to today's. One (but not the only) version of this assumption would be “Transformative AI will be developed soon, using methods like what AI labs focus on today.” The term is inspired by nowcasting. For example, the FiveThirtyEight Now-Cast projects "who would win the election if it were held today,” which is easier than projecting who will win the election when it is actually held. I think imagining transformative AI being developed today is a bit much, but “in a world otherwise relatively similar to today’s” seems worth grappling with. Some potential benefits of nearcasting, and reservations A few benefits of nearcasting (all of which are speculative): As with nowcasting, nearcasting can serve as a jumping-off point. If we have an idea of what the best actions to take would be if transformative AI were developed in a world otherwise similar to today’s, we can then start asking “Are there particular ways in which we expect the future to be different from the nearer term, that should change our picture of which actions would be most helpful?” Nearcasting can also focus our attention on the scenarios that are not only easiest to imagine concretely, but also arguably “highest-stakes.” Worlds in which transformative AI is developed especially soon are worlds in which the “nearcasting” assumptions are especially likely to hold - and these are also worlds in which we will have especially little time to react to crucial developments as they unfold. They are thus worlds in which it will be especially valuable to have thought matters through in advance. (They are also likely to be worlds in which transformative AI most “takes the world by surprise,” such that the efforts of people paying attention today are most likely to be disproportionately helpful.) Nearcasting might give us a sort of feedback loop for learning: if we do nearcasting now and in the future, we can see how the conclusions (in terms of which actions would be most helpful today) change over time, and perhaps learn something from this. I could easily imagine that we will learn more from repeated attempts to ask “What should we do if transformative AI is just around the corner?”, whose conclusions will change as the world changes, than from a persistent eff...