EA - Guesstimate Algorithm for Medical Research by Elizabeth

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Guesstimate Algorithm for Medical Research, published by Elizabeth on September 22, 2022 on The Effective Altruism Forum. This document is aimed at subcontractors doing medical research for me. I am sharing it in the hope it is more broadly useful, but have made no attempts to make it more widely accessible. Intro Guesstimate is a tool I have found quite useful in my work, especially in making medical estimates in environments of high uncertainty. It’s not just that it makes it easy to do calculations incorporating many sources of data; guesstimate renders your thinking much more legible to readers, who can then more productively argue with you about your conclusions. The basis of guesstimate is breaking down a question you want an answer to (such as “what is the chance of long covid?”) into subquestions that can be tackled independently. Questions can have numerical answers in the form of a single number, a range, or a formula that references other questions. This allows you to highlight areas of relative certainty and relative uncertainty, to experiment with the importance of different assumptions, and for readers to play with your model and identify differences of opinion while incorporating the parts of your work they agree with. Basics If you’re not already familiar with guesstimate, please watch this video, which references this model. The video goes over two toy questions to help you familiarize yourself with the interface. The Algorithm The following is my basic algorithm for medical questions: Formalize the question you want an answer to. e.g. what is the risk to me of long covid? Break that question down into subquestions. The appropriate subquestion varies based on what data is available, and your idea of the correct subquestions is likely to change as you work. When I was studying long covid last year, I broke it into the following subquestions What is the risk with baseline covid? What is the vaccine risk modifier? What is the strain risk modifier? What’s the risk modifier for a given individual? In guesstimate, wire the questions together. For example, if you wanted to know your risk of hospitalization when newly vaccinated in May 2021, you might multiply the former hospitalization rate times a vaccine modifier. If you don’t know how to do that in guesstimate, watch the video above, it demonstrates it in a lot of detail. Use literature to fill in answers to subquestions as best you can. Unless the data is very good, these probably include giving ranges and making your best guess as to the shape of the distribution of values. Provide citations for where you got those numbers. This can be done in the guesstimate commenting interface, but that’s quite clunky. Sometimes it’s better to have a separate document where you lay out your reasoning. The reader should be able to go from a particular node in the guesstimate to your reasoning for that node with as little effort as possible. Guesstimate will use log-normal distribution by default, but you can change it to uniform or normal if you believe that represents reality better. Sometimes there are questions literature literally can’t answer, or aren’t worth your time to research rigorously. Make your best guess, and call it out as a separate variable so people can identify it and apply their own best guess. This includes value judgments, like the value of a day in lockdown relative to a normal day, or how much one hates being sick. Or the 5-year recovery rate from long covid- no one can literally measure it, and while you could guess from other diseases, the additional precision isn’t necessarily worth the effort. Final product is both the guesstimate model and a document writing up your sources and reasoning. Example: Trading off air quality and covid. The final model is available here. Every year California ...