EA - Getting Actual Value from “Info Value”: Example from a Failed Experiment by Nikola

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting Actual Value from “Info Value”: Example from a Failed Experiment, published by Nikola on January 26, 2023 on The Effective Altruism Forum.TL;DR: Experiments that generate info value should be shared between EA groups. Publication bias makes it more likely that groups repeat each other’s mistakes. Doing things “for the info value” produces info value only insofar as 1) you actually try to do them well and 2) you process and share the information that they generate. As an example, we present a failed experiment we ran in the Spring of 2022 where we changed the format of our Precipice Reading Group, which had very low attendance due (we think) to a structure that felt impersonal and unaccountable. This, among other experiences, has taught us that personalization and accountability is very important for the success of a program. We invite other group organizers (or anyone else who does things “for the info value”) to share potentially helpful results of failed experiments in the comments.How “Info Value” Experiments Lose Their Potential ValueEA group organizers often consciously change a program’s setup, or run an unusual program, “for the info value” – that is, they might not have high expectations that it will succeed, but “at least we’ll know if something like this works.” It is very valuable to try new things; EA groups have explored very little of the space of field-building programs, and good discoveries can be scaled up!But we want to raise two caveats:Running things “for the info value” is often paired with “80-20’ing,” where organizers run low-overhead versions of programs and don’t sweat the details. But if you actually expect a program to be impactful, you might not want to “80-20 it” – you might want to make the operations smooth, pay lots of attention to participants, and so on. This means that even while piloting a program, you get much worse info value if the program is understaffed or otherwise badly run and understaffed. If the program works anyway, this can indeed signal to organizers that an even better version could be great, but often the program goes poorly and fails to generate the info value of whether a well-executed version of the program would have worked. (We think our Precipice group described below mostly executed everything except the format reasonably well and does not fit in this category, but have witnessed this a few times, including in our own programs.)People seem much more likely to post on the forum about their groups’ successful experiments than their failed experiments, which might be resulting in lots of wasted effort. It’s understandable not to want to write a whole post about a failed experiment, though, so we’d like to invite group organizers to post about other things that didn’t work but generated useful information value in the comments!Example: Precipice Reading Group Format ChangeHarvard organizers have been doing Precipice reading groups every semester since the Fall semester of 2021. Their format was similar to an introductory fellowship: specific people were assigned to cohorts with an assigned discussion leader, and they were to meet at specific times and locations. There wasn’t an extensive application form, just a sign up form, and people were automatically admitted to the program.Dropoff rates were pretty high (~60% of signups came to the first meeting, and as we got more into the semester, only around 15% of signups would actually finish the reading group). This dropoff seems to widely apply to reading groups/book clubs. For example, a remote summer book club hosted by the Harvard Data Science Initiative went from around 15-20 participants to around 5-7 regulars. For comparison, in the Fall 2022 Arete Fellowship (with a more formal syllabus and application process), ~90% of admitted students atten...