EA - What can superintelligent ANI tell us about superintelligent AGI? by Ted Sanders

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What can superintelligent ANI tell us about superintelligent AGI?, published by Ted Sanders on June 12, 2023 on The Effective Altruism Forum.To what extent can humans forecast the impacts of superintelligent AGI?From one point of view, trying to understand superintelligence seems utterly intractable. Just as a dog or chimpanzee has little hope of comprehending the motivations and powers of humans, why should humans have any hope of comprehending the motivations and powers of superintelligence?But from another point of view, forecasting the impacts of superintelligence may yet be possible. The laws of reality that constrain us will similarly constrain any superintelligence. Even if a superintelligence achieves a more refined understanding of physics than us humans, it very likely won’t overturn laws already known. Thus, any inventions optimized against those physical laws, even if superior to our own, may end up looking familiar rather than alien.No matter how intelligent an AGI is, it will still be bound by physics. No matter how smart you are, you still must obey the law of conservation of energy. Just like us, an AGI wishing to affect the world will require an energy industry full of equipment to extract energy from natural sources. Just like us, its energy will have to come from somewhere, whether it’s the sun (solar, wind, biofuels, fossil fuels, hydro), the Earth (geothermal, nuclear), or the Moon (tidal, nuclear). Just like us, any heat engines will be limited by Carnot efficiency. Just like us, energy will need to be transported from where it is collected to where it is consumed, likely by electromagnetic fields in the presence of bound electrons (e.g., chemical fuels) or unbound electrons (e.g., electricity) or neither (e.g., lasers).If there are economies of scale, as there likely will be, that transportation will take place across networks with fractal network topologies, similar to our electric grids, roads, and pipelines. The physics of energy production are so constrained and so well understood that no matter what a superintelligence might build (even fusion electricity, or superconducting power lines, or wireless power), I suspect it will be something that humans had at least considered, even if our attempts were not as successful.One way to preview superintelligent AGI is to consider the superintelligent narrow AIs humanity has attempted to develop, such as chess AI.Lessons from chess AI: superintelligence is not omnipotenceIn 2017, DeepMind revealed AlphaZero. In less than 24 hours of (highly parallelized) training, it was able crush Stockfish, the reigning AI world chess champion. AlphaZero was trained entirely de novo, with no learning from human games and no human tuning of chess-specific parameters.AlphaZero is superhuman at chess. AlphaZero is so good at chess that it could defeat all of us combined with ease. Though the experiment has never been done, were we to assemble all the world’s chess grandmasters and give them the collective task of coming up with a single move a day to play against AlphaZero, I’d bet my life savings that AlphaZero would win 100 games before the humans won 1.From this point of view, AlphaZero is godlike.Its margin of strength over us is so great that even if the entire world teamed up, it could defeat all of us combined with easeIt plays moves so subtle and counterintuitive that they are beyond the comprehension of the world’s smartest humans (or at least beyond the tautological comprehension of ‘I guess it wins because the computer says it wins’)....but on the other hand, pay attention to all the things that didn’t happen:AlphaZero’s play mostly aligned with human theory—it didn’t discover any secret winning shortcuts or counterintuitive openings.AlphaZero rediscovered openings commonly played by humans ...