EA - Learning as much Deep Learning math as I could in 24 hours by Phosphorous

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learning as much Deep Learning math as I could in 24 hours, published by Phosphorous on January 8, 2023 on The Effective Altruism Forum.TL:DR I designed an experiment where I committed to spend two 12 hour days trying to learn as much deep-learning math as possible, basically from scratch.Table of ContentsOrigins and MotivationsResultsTakeawaysExperiment set-upThe CurriculumDocumentation on hoursOrigins and MotivationsFor a long time, I’ve felt intimidated by the technical aspects of alignment research. I had never taken classes on linear algebra or multivariable calculus or deep learning, and when I cracked open many AI papers, I was terrified by symbols and words I didn’t understand.7 months ago I wrote up a short doc about how I was going to remedy my lack of technical knowledge: I collected some textbooks and some online courses, and I decided to hire a tutor to meet a few hours a week. I had the first two weeks of meetings, it was awesome, then regular meetings got disrupted by travel, and I never came back to it.When I thought about my accumulating debt of technical knowledge, my cached answer was “Oh, that might take six months to get up to speed. I don’t have the time.”Then, watching my productivity on other projects over the intervening months, I noticed two things:There appeared to be massive variance in my productivity. Sometimes, in a single day, I would get more done than I had accomplished in previous weeks.I seemed to both enjoy and get more done by “sprinting” through certain projects, eg. by spending 10 hours on it in a single day, rather than spreading that same work out over 2 hours a week for 5 weeks. It was, for some reason, way more motivating and seemingly more efficient to sprint.Also, when I asked myself what I thought the main bottlenecks were for addressing my technical debt problem, I identified two categories:Time (I felt busy all the time, and was afraid of committing too much to one project)A combination of lacking Motivation, Accountability and FunThen, as my mind wandered, I started to put 2 and 2 together: Perhaps these new things I had noticed about my productivity, could be used to address the bottlenecks in my technical debt? I decided to embark on an experiment: how much technical background on deep learning could I learn in a single weekend? My understanding of the benefits of this experiment were as follows:Committing “a weekend” felt like a much smaller time cost than committing “a few months”, even if they were the same number of hours.No Distraction: I could design my environment to minimize distractions for two days, something it would be intractable to do to the same degree for several months.“Trying to learn as much as possible” felt like a challenge. I was, to be honest, pretty scared. I didn’t know what I was doing, it felt extreme, but that also made it exciting and fun.I had some historical data that I might be good at this kind of sprinting, and framing this as an experiment to see what I could learn about my productivity added another layer of discovery-driven motivation and fun. What if I learned more about how to be productive and get hard things done via this experiment?As far as I knew, nobody else among my peers had done this - but I suspected that more people than me had the same problems, and that if I conducted this experiment, I might learn things that would be helpful to others, which added yet another layer of discovery-driven motivation and fun.Accountability: Once I told somebody about this, it was hard to back out. It’s way easier for them to monitor me for a weekend than for a few months.ResultsI’d consider the experiment a success: I finished the whole curriculum in ~18 hours, and I got a lot of neat take-aways I’ll go over below.At the end of day 1, after 12 hours of cram...