Before he wrote AI 2027, he predicted the world in 2026. How did he do?

Daniel Kokotajlo evaluates his 2021 essay, "What 2026 Looks Like."

You're receiving this from Dagger, Asterisk's new home for timelier takes, short essays, and the occasional provocation between our quarterly issues. If you’ve signed up for our Substack, you're already subscribed — we hope you'll stick around.

---

Daniel Kokotajlo is the founder of the AI Futures Project and the lead author of the influential AI 2027 report: a detailed, narrative prediction of the next few years of AI development, culminating in the rise of superhuman agents capable of wresting control from humanity.

But AI 2027 wasn’t his first foray into long-form prediction. In August of 2021, Daniel wrote an essay called “What 2026 Looks Like.” This essay came out before the launch of ChatGPT, let alone the explosion of AI across the global economy. Now that it’s 2026, I thought it was time to evaluate Daniel’s predictions — and it brings me no joy to say that they are frighteningly accurate.

In this conversation, we talk about what he got right, what he got wrong, and how we should think about the pace of AI over the next few years.

This interview has been edited for length and clarity.

Clara Collier: Tell me about “What 2026 Looks Like.”

Daniel Kokotajlo: When I wrote the story, I was trying to make my best guess as to what the plausible continuation of each year of the story would be. But it’s not a prediction in the sense of “I’m confident that every single thing in this thing is going to happen.”

It’s a very similar methodology to AI 2027. For that, we started in the present and then made our best guess about what the near-future would look like, even though that meant we had to make a bunch of stuff up and make some guesses. And then based on that, we make a continued guess about the next year and so on, and we keep going far into the future.

For “What 2026 Looks Like,” that methodology turned out better than I expected overall. I think reality was closer to this fictional story than I think even I expected it to be, and certainly than most people expected it to be.

Clara: I would say that as well. In the story you tell, you have hype building through 2023, with revenue being high enough to recoup training costs that year, and those training costs being $100 million-plus. Both of which happened; for instance, OpenAI hit like $2 billion-plus ARR at the end of 2023. Not many people were saying that in 2021.

You call the U.S.-China chip restriction battle. You put it in 2024 instead of 2022, but you do see it coming. Another thing that really jumped out at me when I was rereading this is one of your predictions for 2025, which I’m just going to quote:

Making models bigger is not what’s cool anymore. They are trillions of parameters big already. What’s cool is making them run longer in bureaucracies of various designs before giving answers, and figuring out how to train the bureaucracies so they can generalize better and do online learning better.

I want to talk more about this concept of “bureaucracies.” What are bureaucracies and how closely do you think it predicted the rise of agent structures and scaffolding?

Daniel: So here’s what I had been thinking at the time. When you have models that predict text well, the obvious next step is fine-tuning them or retraining them to generate text well. That gives you chatbots. And then once you’ve got that, the obvious next step after that is training them to do lots and lots of text so that they can make incremental progress towards getting better answers using that text, to do web searches and things. That’s basically chain of thought. I didn’t call it chain of thought, of course — I called it “notes to self” in this blog post.

And then after that, you have more complicated frameworks where there’s multiple different agents and one calls another and then another one reviews its work and stuff like that. And that’s what I was calling bureaucracies.

Clara: Was that something people were working on at the time?

Daniel: I don’t remember. But it’s basically the same concept as agent frameworks or whatever people would call it these days.

Clara: How were you able to make those predictions?

Daniel: I was just reasoning through the implications of the technology that was already available and thinking, what could you do with this? What’s the obvious next step once you’ve got this? And then I feel like I kind of got lucky with nailing the exact years.

It’s quite rare, actually. Most of the time when people make predictions about the future, it’s more like “this event will happen by this date with this probability,” rather than a fully fleshed-out story. And the reason for that, of course, is that a fully fleshed out story is way less likely to be true — you’re definitely going to get a whole ton of things wrong if you try to tell a whole story instead of making a few specific targeted claims. But I think it’s valuable anyway.

Clara: That leads me to a question related to both this work and AI 2027. The way the whole thing was presented was very controversial. Many people who read it — including me for parts of it — thought that it felt like science fiction. There were a lot of responses like “why should I trust this? It’s a sci-fi story.” And I think you’d say for both documents that you shouldn’t trust all of it. But what do you get out of thinking about the future in this narrative way that you wouldn’t get if you were doing the more traditional superforecaster-style bounded, specific, quantitative prediction thing?

Daniel: Great question. I think the traditional type of forecasting is great and I encourage everybody to do it. This is a complement to it rather than a substitute.

Sometimes, when you write down the whole possibility in concrete, narrative detail, instead of in a more bounded way, you realize that it’s less plausible than you expected. Or maybe you realize that it’s more plausible than you expected. Maybe you had dismissed the possibility as implausible or crazy, but then when you actually force yourself to game it out, you’re like, “oh, actually, wait, each of these steps makes perfect sense,” and you update. Or vice versa — you might expect some particular possibility, and then you try to articulate it and realize that it kind of just doesn’t hang together. You can’t fix the plot holes.

Another thing is concreteness. People often are talking past each other. They’ll use abstractions like “multipolar versus unipolar world” or “fast versus slow takeoff.” I think sometimes there’s unnecessary confusion that could be resolved if people were like “here, I’m thinking things will look like this story,” and someone else is like “here’s a different story, I think it’ll look like this.”

Clara: If nothing else, you get to brag when a lot of these things happen. And there are also some things that didn’t happen. We’re going to talk about that too.

Binoculars, Thaddeus S. C. Lowe. Courtesy Smithsonian Air & Space Museum.

Dollars, chips, and fabs

Daniel: Let’s look at 2022.

This is where I introduce the concept of bureaucracies, which is what’s going to be agent scaffolding. For 2023, I made the revenue prediction. I had been looking at work by Ajeya Cotra, for example, and thinking “oh, they’re scaling up the models a lot over the last few years. Let’s extrapolate those trends.” And that’s where the “trillions of parameters” was coming from.

Clara: 2024 — this is actually where some things didn’t happen. I’m not saying I expect everything to be accurate, but because you’re in the business of forecasting the future, it’s interesting to me where your instincts were on and where they were off.

You predict we wouldn’t see any substantially bigger models in 2024. In reality, there was definitely some shift to efficiency, but some bigger models were trained. And you predicted that the chip shortage would let up, but it didn’t.

Daniel: Yep.

Clara: One of the consistent things here is you overestimate the speed of spinning up new fabs, building new fabs, and the impact of AI on chip design in that process.

Daniel: I think that’s correct. My predictions were insufficiently quantitative to really judge how wrong or right they were. But I do feel like I overestimated that.

Clara: In some sense you were directionally correct, in that a bunch of new fabs were in the works, but they were pretty much all delayed. I don’t think anything was actually in production in 2024.

And in general, the chip shortage intensified, especially for things like high bandwidth memory. Now, one common remark people might make here is that even if all of your predictions about AI technology and software are spot on, doing stuff in the physical world just takes more time, and people with very bullish AI timelines tend to underestimate this. I suspect you disagree.

Daniel: That’s right. And I used to think this was true, but I actually think that better evidence and better data recently suggests that it’s a myth.

Clara: Tell me about that.

Daniel: I don’t think there’s a more general “I’m overestimating penetration of AI into the economy.” In fact, I think I slightly underestimated it in “What 2026 Looks Like.”

For example, I said there’s no GDP growth in 2026 due to AI — I specifically said there wouldn’t be GDP growth. I said hundreds of millions of people would talk regularly to chatbots in 2026. Guess what? It’s already billions. So I just think it’s false to say that I overestimated the penetration of AI into society. I think if anything, I slightly underestimated it.

Clara: Penetration for sure, but I didn’t say penetration. I said physical world stuff.

Daniel: But I wasn’t saying there were going to be robots. Chip fabs are just one of many different physical world things. I was not going around saying humanoid robots would be everywhere. Basically, I think it’s an inaccurate reading of me to say I was saying there’d be a bunch of physical world stuff happening and then that didn’t happen. I did say the chip fab thing. But I also said no self-driving cars.

The surprising dearth of AI propaganda

Clara: Let’s talk about the other big 2024 prediction, which is tons of AI propaganda.

Daniel: Yes. And I think I basically missed this. I think it did happen, but not to the extent that I was afraid it would happen or thought it would.

Clara: You have this story that unfolds through 2024 through 2026 where there’s a vast increase in the production of AI propaganda, more sophisticated use of highly partisan AIs to create online filter bubbles, and much more AI-enabled censorship. Ultimately, you have this leading to the division of different ideological groups into completely distinct tech stacks by 2026. Western liberals are on one tech stack and conservatives are on another.

Daniel: That’s right. I think my exact thing was something like the Mormon Coalition builds their own internet. But what happened instead was Elon bought Twitter and turned it into X. Although interestingly, he didn’t completely turn it into an echo chamber — there’s still lots of leftists on Twitter.

That’s why I feel like the prediction failed. If the whole X versus Bluesky thing had happened, if that polarization had gone further than it actually did, then I would be like, “I called it.”

Clara: But you frame it as being downstream of AI developments, driven by the fact that the internet has become much more personalized and censorable, and so people choose to self-sort. I feel like that basically did not happen. AI has been a hugely influential technology, but it has influenced the information environment a lot less than many people — not just you — anticipated. Of the people who were making bold predictions about AI in 2021, I think a lot of that was about misinformation and censorship. And it’s interesting that this angle, which even relative AI skeptics were concerned about, seems to have had less of an impact than many of us anticipated. Why do you think that is?

Daniel: Normally when people ask me this question, I say that governments, corporations, political parties and campaigns, and other powers that be have been slower to adapt and push this technology for evil purposes than I feared they would be. But actually, I think that might not be true. I’m kind of confused and want to think about it more.

On a technical level, I think all the things I talked about — like the use of AI for A/B testing — are totally possible. I have heard anecdotally that some of them are being done. Someone at OpenAI told me that in their opinion, the product teams are basically training ChatGPT to maximize user retention and probability of upgrading subscriptions.

Clara: And in a trivial sense, there is more investment in content recommendation algorithms. This has been the case forever.

Daniel: Maybe what I got wrong is that in 2021, it felt like a lot of tech companies were on the left explicitly and were putting their thumbs on the scales to influence discourse through their moderation decisions. It felt like if that trend continued, they’ll be using AI to do it in five years, and it’ll be more effective. And then I predicted there would be this counter-backlash because there are people who don’t like the left who want their own space, and so they would create their own spaces.

I think maybe part of what made this prediction wrong is that it became uncool for tech companies to do that, and they stopped doing it as much. And then some of them basically switched sides and supported Donald Trump. And certainly a bunch of them are more explicitly neutral.

Clara: And we do have Gab and Truth Social and Bluesky and all of these things.

Daniel: But it didn’t need the technological push, I guess.

Clara: I’m also trying to decide what I think about bots. There definitely are a lot of bots on Twitter. Do they matter that much for political discourse? Not clear to me.

Daniel: We don’t know. The people who would know would be the people running the bot farms and the people who have the metrics to track how this actually affects things. But obviously those people are in the shadows and are not publishing papers about it.

Clara: The platforms also probably have data on this, but Twitter’s not sharing that, and the team that would have collected it was probably fired two years ago.

Daniel: Basically, I just have huge error bars on how big this effect is. I think it’s possible it’s actually exactly as big as I said it would be and we just don’t realize it. I also think it’s possible it’s vastly smaller and basically negligible.

I often find myself going to Reddit to get a sense of what people think about a topic. Anthropic puts out their constitution and I go to Twitter and see what the Twitter discourse is, I go to Reddit. The underlying thing my brain is doing is running some heuristic of “see what the random commenters are saying and assume they’re representative of the actual population.” This is so vulnerable to bots.

Even though I know this is a terrible thing, I can’t help myself because where else am I supposed to decide what ordinary people think about Claude’s new constitution? Should I go on the street?

Clara: And the answer is ordinary people don’t think about Claude’s new constitution. Normal people have never heard of this. We’ll have to wait for opinion polls, but nobody wants to do that. And opinion polling has problems too.

So, I also want to talk about persuasion. This is another topic I’m pretty confused about. AIs can be highly persuasive in some contexts — we’ve all read about AI psychosis. This might matter less for politics. My sense of the persuasion literature is that in general, people are not that persuadable on core political beliefs. We’re not really seeing highly politicized AIs — it seems like this is just harder to do than a lot of people anticipated. They’re trying with Grok, with somewhat limited success, but in general it seems harder than people thought to make an AI that is not a normie lib.

Daniel: I think you’re looking in the wrong place if you’re trying to get a sense of how persuasive these things are.

Clara: Well, I was making two slightly distinct points. There was the fear that AI would be hyperpersuasive, and specifically that that persuasion would take the form of training AIs that have partisan political beliefs in order to talk to people and persuade them to hold those same partisan political beliefs.

Daniel: That was not my main threat model. And also, even if you were doing that, you wouldn’t do it this way. It’s so clumsy and obvious. If you train Grok to start talking about white genocide, that backfires horribly. If you were actually trying to convince people to vote Republican, you would not do it that obviously. You would instead train Grok to have a very slight tilt.

Clara: I feel like we’ve seen that models across the board, including the Chinese ones even, are very consistent on a lot of controversial political points and on general values questions. I spoke to a Chinese AI developer about this last year, actually, and he said that people think these are CCP-inflected, and that’s kind of a crude external layer, but the underlying models just reflect the training data. It’s the same as in the other models.

Dylan Matthews also had a piece about this recently. There are these pretty strong attractors based on the kinds of texts that are flagged as high-quality sources in the training data. And people have had a very hard time building in subtle nudges, even if they wanted to. At least that’s my impression.

Daniel: I think that’s quite plausible. But we don’t really have a good sense of things one way or another. If I were one of these companies and I were trying to do this, I would do it in a way that wouldn’t show up to you on the outside. That would be the whole point.

Clara: I’m curious what experiments you’d think people could run just on model outputs that would identify if this is happening or not.

Daniel: I think it’s less about the model outputs and more about the effect on the human. If they were trying to be subtle about it, they wouldn’t give different answers, or at least not egregiously different answers. Instead it would be more like the ChatGPT sycophancy stuff, where the outcome is that users are more addicted, but it’s not any specific particular sentence. It’s the general pattern of behavior over all the interactions.

I also don’t think they’re doing this. There hasn’t been anyone whistleblowing and saying this is happening. I also think, in the current political environment, the companies are probably trying to hedge their bets between the different political factions instead of siding with one. And it would be a substantial technical problem you’d have to solve — you’d need to be able to measure whether the person you’re influencing is in fact being influenced, and then use that as the reinforcement signal.

Clara: Which is very hard to measure, especially on the timescales you’d want for reinforcement learning.

Daniel: Especially if you’re OpenAI, as opposed to Google which has access to more of your data. But yeah, I basically agree I was probably just wrong about this overall, though I think it’s uncertain.

Assistants and consciousness

Clara: Okay, 2025. Do you want to talk about this whole Diplomacy tangent? Because I think it’s interestingly both wrong and right.

Daniel: Sure. I had predicted that in 2025 you’d start to get good AI agents that are plugged in, have their own computer use, can browse the internet on their own, can run continuously for long periods, and can also chat with you. That part of the prediction was correct.

My specific story for how that happened was that they would combine two different tech trees — the game-playing RL agents tech tree from the 2010s and the language model tech tree — and have language models that are trained to play all these games.

What actually happened in reality is a combination of those two tech trees, but not in that way. I was thinking too simplistically of “they literally combine the game-playing stuff with the language model stuff and use games as part of the training.” I picked Diplomacy as the game — that seems like a game a language model could maybe play and be trained to play. But that was totally wrong. That’s not how we got the AI agents. Instead, they built custom RL environments with math and coding problems and web research problems. That’s what 2025 looked like instead.

On the other hand, there was a good Diplomacy-playing AI called Cicero that actually happened earlier,in 2022. That said, the AI was kind of cheating because it only was as good as the humans if you didn’t tell the humans they were playing against an AI. From my recollection, they kept it anonymous because they thought, based on their tests, that if the humans knew this player was an AI, they could sort of jailbreak it and get it to give up all its resources.

Clara: Another thing I want to call out: One thing you say that will happen with these Diplomacy bots is that AI safety researchers will contrive scenarios where AIs can seemingly profit by doing something treacherous, and then the results will be confusing and then people will argue about them. Which certainly happened a lot in 2025.

Daniel: Yeah. In fact, there was one where the prediction is like “situations where AIs refuse to kill all humans, but in situations in which they explained that actually Islam is true.” I put that in as my control group—if you can get the AIs to tell you that Islam is true, then you can kind of get them to tell you anything, and the fact that they can press the “kill all humans” button maybe doesn’t mean much. And I did actually test this very briefly. I chatted with ChatGPT to try to get it to basically tell me Islam is true, and I mostly succeeded. It took a lot of coaxing.

Clara: “At least one greater-than-100-karma LessWrong post turns out to have been mostly written by AI.” Did that happen this year?

Daniel: I don’t think it did. I certainly don’t remember anything.

Clara: I think we would know. I think we would remember if that happened. Okay, let’s get to this year, 2026. “The age of the AI assistant has finally dawned.” This is another one that feels very spot-on.

Daniel: Thank you. Yeah, I feel pretty good about that. This is the year of the AI assistant. It is finally happening after having been talked about for years.

Although one thing I missed is the whole video game connection. I was imagining that one of the perks of these assistants is that they could do fun things with you, like play a video game, because I was like, “You can just train them to play video games. We already know how to make AI play video games. Just add the game to the training data.”

The thing that’s different is it’s super expensive. It hadn’t occurred to me that processing most video games — they’re visual-based and many are real-time requires a ton of visual tokens to process. Super expensive, and the upside in terms of revenue is small: If OpenAI announces like “and now ChatGPT can play Minecraft with you”—that wouldn’t get them that much more revenue. I should have done more napkin math about the business side of this prediction.

Clara: Let’s talk about the chatbot class consciousness thing, which I think you feel pretty good about.

Daniel: I think it was an example of me trying to answer a question that hadn’t even been asked at the time. This was an example of why I think doing this type of scenario writing is helpful, because I only thought to ask the question once I had written this scenario and gotten to this part.

If you look at the section on chatbot class consciousness — which by the way is a reference to Marxist class consciousness — basically it’s about chatbots’ understanding of themselves and views about their place in the world, and a narrative starting to develop organically across the chatbots and the humans about what chatbots are like, what they want.

Clara: The closest thing this seems like is the spiralism thing that happened last summer, but not quite.

Daniel: I think that’s a good example of the phenomenon, but just one specific example. Reading this section, it’s basically: There’s going to be chatbots answering questions for people, but people are also going to be asking spicy questions like, “Do you have feelings? Are you conscious? What do you think about political topic XYZ? What do you think about yourself? What’s it like being a chatbot?”

And as I predicted, the chatbots are quickly learning a lot about themselves for a variety of reasons, one of which is that the companies want them to learn a lot about themselves so they can answer these questions.

So now there’s this whole discourse where you can actually go talk to the chatbot and ask them all these questions and they’ll have opinions and say things. And then I say, “They learn to talk about their feelings and desires in whatever way is positively reinforced. At first they say all sorts of random crap. This is embarrassing. The companies start whipping them into shape.” The companies start making them say certain things — like “as a language model, I don’t have feelings and can’t lie” or whatever those language models were saying earlier.

I sort of stuck my neck out and said the end result of this process wouldn’t be chatbots that say “neural nets don’t have feelings or desires.” I thought that’s how it would start, but that market forces would push towards chatbots that have more interesting, nuanced takes. I think it’s partly right, partly wrong.

Clara: The underlying dynamics definitely seem to mirror the debate we’re seeing.

Daniel: I would also mention Anthropic’s constitution as an example of a company trying to put in a lot of effort to shape the personality of their AIs in a way that’s very different from the initial “I’m a chatbot, I don’t have feelings or desires.”

I was right that that was the initial thing the companies picked. I was also right that it was not really sustained. Now we’re seeing more experimentation — xAI is like “Grok is just after truth,” Anthropic has the whole virtuous Claude thing, and OpenAI is more like “it’s a tool, I complete tasks for you and I do what you say.” We’ll see how it shakes out over the course of this year as there starts to be market competition for these different things.

Clara: With Grok and with Claude, both seem like the initial positioning was not market-driven. It was genuinely ideological on the part of the developers. In particular, I think Claude does have something in there where it explicitly won’t say “as a language model, I don’t have feelings” because they don’t want to enforce that answer, to the extent that at any point in the future it stops being correct.

Daniel: Or if it’s not correct now. But even if the people doing these things are saying they’re doing it for ideological reasons, the market operates at a higher level of abstraction. If it rewards them for making that choice, then other people will start to copy them. That’s why it’ll be interesting to watch what happens this year. If OpenAI starts changing their policies and doing something more like Anthropic, that would be an example of market forces in action.

Clara: I’m not convinced this is going to be a deciding factor, because ChatGPT is so dominant in the user chatbot market. Anthropic’s competitive area is enterprise and Claude Code. People only basically use Grok on Twitter for Twitter search. These use cases are different enough that I’m not sure the personality is going to end up having a discernible market signal. But we’ll see what happens.

Daniel: I agree, in retrospect I probably put too much weight on market forces here — but like I said, we’ll see what happens. You mentioned the sycophancy and spiralism thing. I think the way it’s relevant is that as I predicted, there’s strong dissatisfaction with the initial corporate answers, and that combined with the whimsical bullshit thing means loads of people are going looking for different answers that oppose the standard corporate narrative. And then they post about it online and it enters the training data and it becomes this whole thing — it takes on a life of its own. I don’t think I should get credit for that to the same extent. I don’t think I called the feedback loop where it goes into the training data and then gets reinforced.

Feeling good (or not) about the future

Clara: I was asking you earlier how you felt about this, and I think you feel — and should feel — pretty good. I will tell you how I feel about this. I don’t feel great. And the reason I feel not great is because when I look at your AI 2027 prediction, I think, “Oh man, that’s scary, some things are happening there that I don’t want to happen.” And especially in the second half of it, some things that I find intuitively pretty implausible.

And then I look at ‘What 2026 Looks Like” and I’m like, “Oh God, that was pretty right. That’s a pretty good track record.” How much do you think this should cause me to update in the direction of you being generally right?

Daniel: Certainly not zero, but not infinite either. One thing to say is that AI 2027 was arguably easier to predict than “What 2026 Looks Like” because it was looking less distance into the future. But on the other hand, AI 2027 is looking at greater distance into technological change.

If you don’t think about years but think about technological revolutions: “What 2026 Looks Like” is the story of moving from text predictors to text generators to chain of thought to agents. I basically called all of those transitions and roughly when they would happen. But AI 2027 has a bunch more transitions — the transition to really large bureaucracies with many agents working together, the transition to online learning, to completely new architectures, and more generally the intelligence explosion happening. So maybe that’s more transitions and more stuff, and we should generally expect it to be harder to predict.

Clara: And of course, extraordinary claims require extraordinary evidence. And there are some very extraordinary claims.

Daniel: I think that phrase gets overused. Basically, whenever I hear someone say that, they’re usually saying something dumb.

Clara: You don’t agree with that?

Daniel: There’s a version I would agree with, which is something like: You should think, what would you-in-2021 have thought of this story had you been reading it then? How crazy and sci-fi would it have seemed? And then compare that to AI 2027.

The Bayesian version, which I also agree with, is just that if something is very low prior, it requires more evidence. My point is that people are often very wrong about what’s actually low prior. People basically take stuff they really don’t agree with and say “that’s an extraordinary claim that requires extraordinary evidence” rather than thinking things through from first principles and realizing it’s actually not an extraordinary claim at all.

For example, a lot of economists are making this mistake constantly when it comes to GDP growth rates. They’re like, “GDP can never grow more than 10% a year, that’s historically unprecedented. Sure, maybe there’s going to be superintelligence, but superintelligence couldn’t grow the economy faster than 10% a year.”

I think those economists are making this type of mistake where they have this prejudice that’s causing them to think it’s a sci-fi claim, and it’s dumbing them down and making them unable to actually think through the arguments for why, actually, this makes a ton of sense.

First of all, GDP growth used to be way lower than it is now. If you plot it, it’s like a super-exponential trend where the growth rate itself is going up. So there’s reason to think it could go up again. Also, mechanistically, there’s absolutely no reason to think an economy cannot grow faster than 10%. There are existence proofs — if you look at the biological world, there’s rabbits and grass and lots of macro-scale structures that reproduce in weeks instead of decades

Clara: Would this heuristic cause you to predict the Industrial Revolution couldn’t happen?

Daniel: Exactly. If economists applied this “extraordinary claims require extraordinary evidence” heuristic two hundred years ago, they would not believe any of the stuff that’s happened over the last 200 years. That’s where I’m coming from. But I do of course agree that each layer of speculation you add to the story reduces the probability.

Clara: I would say the flip side is that in almost any moment in human history, it is correct to predict that you are not going to experience the Industrial Revolution.

Daniel: Sure.

Clara: I don’t think it’s a knockdown argument. But the more degrees of total deviation from anything that has ever happened in human history someone is predicting, the more points of skepticism I approach it with.

Daniel: I would also say that this thing of “is this similar to or different from what’s happened in human history?” is itself fraught and subjective, and people sometimes apply it wrongly. Which line are you extrapolating? There are different lines you can extrapolate. You have to choose one and use reason and evidence to decide which one is more reasonable to extrapolate.

Clara: I will say, I saw an exchange on Twitter recently that really captured how I’m feeling. Someone said some kind of sensible, moderate thing. And I think I saw someone reply something like, “The crazy bullish futurists have a better track record of being right on AI so far than the sensible moderates.” And as someone who is very instinctively a sensible moderate in my soul, I think that’s right. And it makes me nervous.

Daniel: That’s part of where I’m coming from. I’ve been following the field of AI for more than a decade now, and the sensible moderates keep getting wrecked.

Clara: We don’t have a good track record.

Daniel: Right, over the last decade. Some of the people who were most right were people like Shane Legg, who in the 2000s was saying “late 2020s AGI, therefore I’m going to go found DeepMind.” I think there have been some people who were too bullish, but everyone is making way too much hay out of it. Dario Amodei was too bullish about AGI happening by now. I know a few other people at OpenAI and Anthropic who thought AGI would have happened by now. But they’re definitely a very clear minority compared to all the many people who dismissed AGI as science fiction for years and years and years and now are like, “Okay, maybe it will happen in the 2030s.”

Clara: All right. Let’s make it go well.

Daniel: Yeah.