EA - Katja Grace on Slowing Down AI, AI Expert Surveys And Estimating AI Risk by Michaël Trazzi

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Katja Grace on Slowing Down AI, AI Expert Surveys And Estimating AI Risk, published by Michaël Trazzi on September 16, 2022 on The Effective Altruism Forum. Katja runs AI Impacts and recently published What do ML researchers think about AI in 2022, a new survey of AI Experts. We start this episode by discussing what Katja is currently thinking about, namely an answer to Scott Alexander on slowing down AI Progress, and then go on discussing her survey and other considerations for estimating AI risk. Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript. On Slowing Down AI The Astral Codex Ten Post Katja Is Replying To "[Scott] wrote a blogpost about whether we shouldn't or why we shouldn't try and slow down artificial intelligence progress. He's noticing that lately, various people have been saying that maybe we should slow down AI progress if AI progress is a threat to the existence of humanity. He outlines some reasons you might not want to slow down AI progress." Are we actually in an arms race? "The first question is, are you actually in an arms race? It seems not obvious to me that the AI situation at present is an arms race or likely to be an arms race. In the classic arms race scenario, it's not the case that when you win the arms race, you get destroyed. In the very simplified version of this arms race, it's quite unlike a usual arms race. The usual arms race is a prisoner's dilemma, basically. You always have an incentive to build more arms regardless of what the other player does. That's not true if you get killed in the end. Then you can build more complicated models, and sometimes you should race perhaps depending on the details of it, but often, not." Surveying AI Experts in 2022 Forecasting High Level Machine Intelligence "[High Level Machine Intelligence is] when unaided machines can accomplish every task better and more cheaply than human workers, ignoring things where it's intrinsically advantageous to be a human, for instance, being on a jury [...] We gave [AI Capabilities Researchers] three different probabilities, 10%, 50% and 90%, and we asked them in what years that amount of probability would've been reached basically. And then, to a different set of people, we gave them years and we asked them what the probability was that we would have high level machine intelligence by those years. And then, a different set of people again, we asked about not high level machine intelligence but full automation of labor, which is when AI can do all of the occupations. Which, to my mind is quite similar to when will AI be able to do all of the tasks. But it turned out to be much later." ML Researchers do consider extremely bad outcomes "They were all authors at NeurIPS or ICML. We just wrote to 50% of the authors at those conferences [...] I think in the end [the response rate] was maybe roughly 15%. [...] A lot of people who work on AI capabilities do think there are serious concerns with safety. So I think it's very unclear what different people want here, and probably there are a lot of different views. It seems like, among people working on AI capabilities, I guess in this recent survey I worked on, a lot of them said they thought there was more than a 10% chance of an extremely bad outcome." AI Alignment is increasingly becoming a concern [...] We describe something the alignment problem and ask, "Do you think this is an important problem? Is it a hard problem? Is it a valuable problem to work on at the moment?" And I think, for all of those answers, the distribution shifted toward it being important and valuable and hard[...] They had five options for how important it is, say. For importance, the top eva...