EA - Comments on Manheim's "What's in a Pause?" by RobBensinger

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comments on Manheim's "What's in a Pause?", published by RobBensinger on September 18, 2023 on The Effective Altruism Forum.This post is part of AI Pause Debate Week. Please see this sequence for other posts in the debate.I agree with David Manheim's post at a high level. I especially agree that a pause on large training runs is needed, that "We absolutely cannot delay responding", and that we should be focusing on a pause mediated by "a multilateral agreement centered on countries and international corporations". I also agree that if we can't respond to the fire today, we should at least be moving fast to get a "sprinkler system".The basic reason we need a (long) pause, from my perspective, is that we are radically unprepared on a technical level for smarter-than-human AI. We have little notion of how to make such systems reliable or safe, and we'll predictably have very little time to figure this out once smarter-than-human AI is here, before the technology proliferates and causes human extinction.We need far, far more time to begin building up an alignment field and to develop less opaque approaches to AI, if we're to have a realistic chance of surviving the transition to smarter-than-human AI systems.My take on AI risk is similar to Eliezer Yudkowsky's, as expressed in his piece in TIME and in the policy agenda he outlined. I think we should be placing more focus on the human extinction and disempowerment risks posed by AGI, and should be putting a heavy focus on the arguments for that position and the reasonably widespread extinction fears among ML professionals.I have disagreements with some of the specific statements in the post, though in many cases I'm unsure of exactly what Manheim's view is, so the disagreement might turn out to be non-substantive. In the interest of checking my understanding and laying out a few more of my views for discussion, I'll respond to these below.[1]So the question of whether to stop and how to do so depends on the details of the proposal - but these seem absent from most of the discussion.This is not apparent to me. I think it would take a pretty unusual proposal in order for me to prefer the status quo over it, assuming the proposal actually pauses progress toward smarter-than-human AI.It's important to get this right, and the details matter. But if a proposal would actually work then I'm not picky about the additional implementation details, because there's an awful lot at stake, and "actually working" is already an extremely high bar.An immediate, temporary pause isn't currently possible to monitor, much less enforce, even if it were likely that some or most parties would agree.A voluntary and temporary moratorium still seems like an obviously good idea to me; it just doesn't go far enough, on its own, to macroscopically increase our odds of surviving AGI. But small movements in the right direction are still worthwhile.Similarly, a single company, or country announcing a unilateral halt to building advanced models is not credible without assurances,"Not credible" sounds too strong here, though maybe I'm misunderstanding your claim. Scientists have voluntarily imposed restrictions on their own research in the past (e.g., Asilomar), and I don't think this led to widespread deception. Countries have banned dangerous-but-profitable inventions without pursuing those inventions in secret.I don't think it would be that hard for many companies or countries to convince me that they're not building advanced models. It might be hard for me to (for example) get to 95% confidence that DeepMind has suspended frontier AI development, merely on DeepMind's say-so; but 75% confidence seems fairly easy to me, if their say-so is concrete and detailed enough.(Obviously some people will pursue such research in secret, somewhere in t...