Casey Muratori is one of the top programming educators on YouTube. He won’t put a probability on the risk of AI doom — he doesn’t have P(Doom) — but he believes the AI extinction threat is worth taking seriously. I probe how he arrived at that position.
00:00:00 — Cold Open
00:00:31 — Introducing Casey Muratori
00:02:07 — Recap of Crypto Series: Is Crypto Dead?
00:11:46 — On Programming & Why Coding Agents Don’t Faze Casey
00:18:51 — Coding Is a Craft: Drum Machine Example
00:36:24 — Pivoting to AI: Can We Forecast Its Capabilities?
00:43:36 — Debating if AI Can Acquire Taste
00:54:59 — What’s Your P(Doom)™?
01:04:11 — Even 0.1% X-Risk Is Unacceptable
01:07:34 — Modeling Uncertainty: Bayesianism vs “Tie Goes to the Catastrophe”
01:09:34 — Is This Pascal’s Wager?
01:17:12 — Casey’s (Cynical) Mainline Scenario
01:24:22 — Would You Un-Invent the Internet?
01:27:18 — Let’s Ride the Doom Train 🚂
01:27:51 — Can AI Become Ridiculously Powerful?
01:29:11 — Will AI Preserve Humanity (Orthogonality Thesis)?
01:32:26 — Will AI Seek Power (Instrumental Convergence)?
01:36:06 — Wrap-Up
Transcript, links, and show notes:
source

40 Comments
liran so handsome
my guy on the left you don't even remotely understand decision theory and you spread bayes around like a wet cloth you can use to extrapolate over the massive cracks in your knowledge
If you see yourself as able to risk the lives of everybody… That tels me a lot about you…
Lol… Feeling a probability is knowing???
my man actually tried for over a half hour to get casey to give him a random probability that the human race will end in the next few years off the top of his head even though he kept saying its an impossible thing to come up with due to his inexperience in the field, just hilarious
Wait, is he the granny.dll guy ?
Well met! Windows XP talked a lot about you!
Casey does what AI don't.
What! Hope to achieve a permanent/accepted dynamic AI psychosis? Business is faster because you are ignoring the validation. Don't care if is true or not, it gets you close enough to an accepted answer. Yikes.
1:24:10 "It's always a cynical take to assume that any new technology will go about as well as the most recent one" ( -> 1:18:43 "…AIs will get roughly good enough for the people who make them to make a lot of money, and then they will stagnate") – No, that's just being a realist socialized in a capitalist society…
47:07 "…we have never built anything that attempts to simulate how humans determin that something that's never been seen before and never been done before is pleasing to them"
Aperture Science Psychoacoustic Laboratories has you covered! 😁
Casey is trying to explain that you can't put a number on "What's percentage change would you give me pulling a red ball out of this bag of balls?" where you don't know what colors there are, what size the bag is, what size the balls are and how many balls there could be. Casey correctly states that, knowing the bag and the balls exist, all you can know is that it's a non-zero chance. The interviewer feels incorrectly justified to just say "5 percent!"
Ai is just a tool today and the foreseeable future. don't see it (AGI) happen in my lifetime. LLMs advances are still happening that will improve but it's really still a small lense. World models like Jepa aren't even main stream. Maybe a new discovery has to happen (should ask the Neural Link folks). To get to AGI (human level) you will need a feelings and intent bio/chips. I agree with Casey, A machine couldn't go from zero to learning (paraphrasing) or discovering a music genre like Rock if it's never heard of it.
In the 100 years (10-20 is just improving LLMs frontier models not world models like JEPA and others). Ai is still in the very infancy where even if you capture all the human senses and experience it doesn't have intuition or intent or good morals. If you teach it good morals it would not "want to" kill us all.
I think the doom sayers assume we as a human race couldn't figure a way to stop the eminent doom to humanity. If accounting for a percentage there are also other percentages of (extrapolations pulling something out of a hat) that resist the .01 % of pDoom. Humans, new discoveries that will improve our lives, new forms of energy, medical discoveries, improve the brain. Ai will have to advance on its own and have some moral compass. Just don't see it (pDoom). In order for pDoom like in the terminator movie Ai would have to be sentient and able to truly self improve (unlimited context and energy, evil intent for what ever reason). Once we are dead and gone then will Ai continue?
The "What if it cures cancer thing" also felt kind of weird to me as AI is very likely to harm people with disabilities and diseases more if it goes into Casey's cynical view. Outside of white collar jobs, there aren't that many job opportunities for people in these camps.
There are so many misconceptions; it's painful to watch. First, we have had AI for decades. We have used AI in research for decades. So the argument about having or not having AI is nonsensical in the first place. We have been using AI in cancer research for decades as well. Second, we don't have any clue whether superintelligence is even possible. Literally zero. Debating is like debating faster-than-light travel. Exactly the same amount of evidence. Zero. Just for context, I have a Ph.D. in AI.
lol this guy is on deep ai psychosis
that p(doom) AI jingle made me want to blow my brains out. so yes, the doom is here.
Re: the music portion of this video. Prong 1: LLMs can already compose music that sounds good if you're using the right harness. Can they invent rock music or an equivalently popular genre? (Or one which will become equally popular given enough time?). This question is flawed because no one human can invent Rock alone by themselves either. It requires the collective history of all music prior to rock. It requires the interaction of the humans who were iteratibg on new styles and genres before rock was rock. And, crucially, it required the feedback from other humans as essential to the meme which is considered rock today. There was no "human preference genie" human that developed rock in their basement. That is an oversimplification of a complex process. For this and other reasons, LLMs today can be creative and as inventive, if not moreso, than humans.
For prong 2: yes, we do have such ai system simulating user preferences via RL post-training on implicit and explicit like/dislike signals. It's called Suno. Check it out, it's been around for a while now. There's a few of these now and they're producing music that people really enjoy listening to for months/years now
He kind of sounds like John Carmack to me. And I don't mean Casey lol.
This guy very clearly contradicted himself at the end about hating the internet 😂
No he doesn't believe what he's saying — he's just a grifter making a buck on this AI hysteria. Listen — I just got of an hour of wasted time with Gemini trying to get it to figure how to alphabetize a list in Apple's "Notes" app. Complete and utter failure. If that's the deep understanding of general AI you might as well pack it in. It took several iterations before it could even conceive of the fact there was a difference between alphabetizing folders from alphabetizing a list within a folder. When I finally asked it why it didn't just throw the list into Word and use the alphabetize feature there, it gave up and said that was a more elegant solution. Stupidity on stilts kids. It ain't planning on any high level shenanigans anytime soon. By then, Liron should have a nice nest egg to retire on.
54:00 I really like Casey but this is a terrible argument. No training data? What do humans have? Humans have training data. The difference is the processing that humans do. Humans didn't just create rock and roll. They created many genres of music way before which inspired rock and roll. For find what pleases people, you don't need to assess a humans brain to do that. You categorize the things that make people like things and cross reference across similar areas and the data speaks for itself by pulling out features. This is ML 101 that existed years ago.
So many cuts.
20:30 That is – sorry – so dumb. If you want to "express it" for the machine, sooner or later it must be represented as code. Yes, you can express yourself much better in englisch than in C, but that's (a) a you-problem and (b) also the problem. You can try for months and wont be able to write a 100 page document with ambiguous statements i can misinterpret or an AI can misinterpret. Most specs are far from implementable reality, they ignore existing logic, rules, aren ot logically coherent and so on and so on. Sure, code might have bugs, but specs are full of them – the only thing saving most Business Analysts from losing their job are people able ignore 30% of their garbage and fix it on the fly. But all day every day is code a better source for information than any spec vomited by a BA.
and another hard truth: 34:55 if you are "a programmer", you don't consider memory leaks and slow responsiveness "not worth my time". Doing this stuff is not time consuming, if you are a professional. My junior devs are always complaining that they don't have time to do it right – but thats BS, they are just not good – not good at all. This is like not mounting tires correctly because there was no time to do it right. People with so little knowledge and such low standards for themselves get the same advice from me: find a different profession. Nobody needs them just to type a prompt into ChatGPT and they are obviously not interested in the actual work.
I think sometimes talking with Casey can be so frustrating because he refuses to see programming as a means to a product. Like when Liran said the thing about the dashboard and Casey deflected with "I would just rewrite the dashboard engine". I think this kind of purism is very similar to Stallman's: extremely important to pull the focus to overlooked, important things (FOSS, performance), but actually living what's being preached requires a capital (both social and financial) that most people simply don't have.
Liron, please for the love of god, listen to the guest.
1:25:40 No, because nearly every company is trying to get “AI” to do that.
funny how you said it's lower then three ago but not two years ago. one year ago would have been to obvious right?
at least the guest gets it
38:13 finally some one mentions what I learned in my statistics class in 2013, extrapolation is to be engaged very carefully…
Maybe management should use it to illustrate the feature/spec request but let the engineers engineer. That could be a win win although it shifts the whip to management itself since they can now technically communicate “perfectly “ and even noticed some obvious pitfalls early.
This whole bit where you're trying to get Casey to pin down a probability below which the risk of AI resulting in human extinction is an acceptable risk is peak sophistry. Even if you're just using Bayesian reasoning to make your decisions, assuming you value the continuation of the human species above any amount of increase in economic productivity (which you should), the the value of an extinction event is negative infinity, so the expected value of just letting things play their course is necessarily negative if the probably of extinction is greater than 0. This isn't a hard concept.
That doesn't mean we should just stop doing AI, but it does mean we should take the risks far more seriously than we currently are.
I find Casey's argument so contradictory. On one hand he says modern frameworks/libraries/languages are so badly designed because if you want to actually do something like create a 7 days summary for certain data in some color on UI you can't just describe it simply enough to invalidate the utility of an LLM, on the other hand he constantly compares coding directly in any other environment than apparently performance critical and relatively low level applications as basically just writing a spec file and "not programming" but doing what a manger does, while complaining about how modern programming has moved towards explicit higher-level syntax that is not as easily attributable to machine commands
Such a 1% conversation, why do "true devs" do this? Perfect belly button lint every time
I played Pokémon emerald battle factory.
Then a lapras with quick claw (10%) and sheer cold (30%) one shot my team in the first 3 turns. I did not get to use a move.
That day i learned that 0.0027% is NOT 0%.
I dont have a p.doom i have an "ai as a force multiplier for incompetence" worry.
Too many ai debates seem to overlook the difference between discovery and innovation.
Ai can be useful for discovery, but it can't innovate.
Why wouldn't a human be able to come up with a new genre of music, and by extension why wouldn't an AI be able to come up with a new genre of music? Humans already invented all of the existing genres, so it follows that a human, and then an AI could invent a new genre. If you accept that they could do that, and you accept that both a human and an AI could come up with new music (of any genre) that other humans find appealing within that genre, then the human or the AI could come up with a new genre of music that people find pleasing.
Casey made my argument for climate change. If there's a 0.1% chance that burning gasoline and breeding copious flatulent bovines could turn Earth into Venus, why don't we cut down. Sadly in both cases, reducing what we're doing isn't in the interest of the investor class (pun intended).
It takes a lot of power to turn a spec into code. If specs replaced code we would need a Dyson sphere to power the LLMs.
1:22:00 That's some necron thinking right there lol.. They definitely cured the cancer… Lol
It's a good thing that Casey had that money value universal Star Trek Communist future debate a while back lol.. Right off the cuff with the human labor value isms… Actually I really liked that little back and forth, it's a good reference for anybody that's willing to actually bother (sadly) with regards to cutting through the fat when discussing money time value etc in a society. Y'all followed that thread pretty far, far enough that most people haven't thought through at least half of it, but they probably would get there eventually.