How long until AI doesn’t need humans?
She thinks it's nearly imminent. He doesn't.
This article originally appeared in Issue 14: Risk. Subscribe to the print magazine by June 30th to receive our next issue, Work.
How long until AI systems can sustain their own existence — such that, if every human died, they could keep growing their own population?
METR’s Ajeya Cotra and Understanding AI’s Timothy B. Lee discuss the path toward this “self-sufficient AI.” She thinks it’s nearly imminent; he believes it might never happen. The two talk through the skills and shortcomings of today’s humanoid robots, profit incentives, tacit knowledge and how these affect the timeline, non-robot paths to self-sufficiency, and the benchmarks to watch for in the next few years.
This interview has been edited for length and clarity.
Clara Collier: This conversation is happening because you two got into a debate on Twitter about when you expect us to have fully autonomous AI systems. Ajeya, this is your concept. Do you want to talk about what you mean by that?
Ajeya Cotra: I wrote a blog post about self-sufficient AI, which is AI systems integrated with physical infrastructure — factories, mines, fabs, robots to operate all of those — such that they don’t need any cognitive or physical inputs from human labor to keep growing their own population.
If all humans died of the plague one day, the AI systems would be able to maintain themselves, repair things that might be breaking down in their physical environment, and keep up the power plants that run them. They’d also have to expand themselves, which would require eventually consuming more physical resources — going out and mining the quartz and turning it into silicon sheets and then etching those into chips and so on.
This is an interesting forecasting point in my mind because, one, it’s relatively concrete and easy to imagine, and two, because a number of people are concerned about the possibility that AI systems might literally drive humanity extinct because they’re pursuing goals at odds with humanity’s goals. Self-sufficiency seems like a requirement for carrying out full extinction of humanity on the part of the AIs.
I do want to flag that I think this would be a pretty late milestone, and you could degrade this milestone in interesting ways and forecast weaker endpoints as well. For example, how many physical humans are needed to sustain a certain population of AIs? To what extent do they need to have specialized expertise versus “the AI can just direct them around because the AI has all the knowledge”? I ultimately think one or more of those weaker milestones will be more directly relevant for policymaking, but you can generate them starting from the concept of self-sufficient AI.
Clara: So when do you think we’re going to have this self-sufficient AI?
Ajeya: More likely than not within 10 years. And it very well could happen sooner.
Clara: Tim?
Timothy B. Lee: I think that timeline is pretty unlikely. Hard to put a number on it, but 20 years is the earliest it sounds plausible to me. I’d say less than 10% chance that it happens within 20 years. I’d say there’s a 10 or 20% chance it’s never, and my median would be 50 years.
It’s hard to reason about what will happen in the future, but I have a strong intuition based on writing about robotics, particularly self-driving cars and sidewalk robots, that these things always take longer than you think they will. Practical barriers mean stuff in the physical world takes a lot longer than software stuff. It’s capital-intensive. There’s just a lot of stuff you have to do.
Let me give you a specific example. Six years ago, I went out to George Mason University and looked at Starship, which has these sidewalk delivery robots. The robot is simple — it’s a box on wheels, drives four miles an hour, and delivers lunch to people. I wrote an article saying it seems like we’re on the verge of these things being everywhere because they seem to work great and seem useful. Six years later, that company is far bigger than before, but they’re not everywhere
I couldn’t tell you exactly why they haven’t grown faster. I don’t think there were any major technological breakthroughs needed, at least on the hardware side. This was something that worked totally fine. I think it’s some combination of engineering work to make the robots more reliable and easy to repair and some amount of human labor needed to maintain them, and the margins aren’t that high, so it actually wasn’t that profitable a business.
And I think you could tell a similar story about self-driving cars, which is more of a software progress story. Waymo basically now has working self-driving technology, and it’s still going to take them years to just scale that up from the scale they’re at now to where most taxis are driverless. It takes time to build factories, it takes time to get regulatory approval, and so on.
To have a replacement for humans, you’re going to need humanoid robots, which I don’t think are anywhere close to what’s needed. Once we have a working prototype that has all the characteristics you’d expect from a human, it seems like a 10- to 20-year process to go from initial prototype to 100 million or a billion cost-effective, easy-to-repair robots all over the place.
Two challenges for humanoid robots
Ajeya: That’s helpful. One thing I wanted to flag is that you said we don’t have humanoid robots now. I don’t think that’s true. My understanding is that we’re producing thousands or tens of thousands of units of humanoid robots.
Timothy: Right.
Ajeya: What is true is they’re not very useful right now. One question that feels very important is: Why are humanoid robots not useful right now? Is it because they have bad bodies, or is it because they have bad brains?
I lean toward thinking that cognitive capabilities are the bigger bottleneck, as I think they were for self-driving cars too.
You said in your tweet that you wouldn’t be surprised if the cognitive capabilities necessary for self-sufficient AI were there within 10 years, but that they wouldn’t have enough physical bodies to operate. If we condition on that world — where the brains are good enough to operate the bodies that exist today — then AI systems may actually have many bodies to operate, at least if we’re currently manufacturing 10,000 per year and production is growing. It becomes a question of how many robots are needed.
It could also be that the bodies are insufficient — the thumbs don’t have enough dexterity, there isn’t enough haptic feedback. That’s a debate within the robotics community.
Timothy: Let me be more precise. We definitely do have robots with arms and legs that can walk around, can pick objects up, have eyes and ears and so forth. But I think there are two things we don’t currently have that we will need for self-sufficient AI.
One is that I do not think the humanoid robots shipping today are anywhere near as sophisticated as human beings in terms of their ability to do physical manipulation of objects. The human hand in particular is a miraculous thing. It has a ton of touch sensors, it’s able to do very precise movements, it has a lot of degrees of freedom.
My understanding is that things that are hard for people, like dancing and certain gross motor skills, are actually pretty easy for robots because they’re just physics. But interacting with the physical world — for example, picking something up without crushing it— is actually quite difficult because you have to apply exactly the right amount of force.
Clara: I assume both of you have read the Rodney Brooks post about this.
Ajeya: I have skimmed that post.
Timothy: I did read it a while ago. Is that the one that talks about how the way the robots balance means that if they screw up, they can fall over and hurt people?
Clara: That, but also just how complicated and sophisticated human touch is as a sense. His argument, I think, is that we don’t even really have the sensors that would give you training data for the kinds of signals that inform human manual dexterity right now.
But I’ve also seen people argue in response that “yes, but we have different kinds of sensors — like with self-driving cars, where a camera can’t do everything the human eye does, but we can supplement it with lidar and radar. Maybe for touch you can’t do what a human hand does, but you can supplement a pressure sensor with infrared and other things.” I think that’s the debate, at least as I understand it.
Timothy: Yeah, I may be just poorly summarizing Rodney Brooks because I think I did read that essay and some of these ideas probably come from that. I think that particularly in terms of balance and physical manipulation of objects, the current hardware is not on the same level as the human body.
But the second really huge thing is energy efficiency, reliability, and repair cost. This is why I think the Starship sidewalk robot example is instructive. There’s a big gap between “I have a prototype that maybe cost a million dollars, can go for an hour or two before it runs out of battery, and needs to be repaired after a few days of use” versus a human body, which, with the right inputs, can work for 10, 20, 30 years in a wide range of situations.
It definitely does not seem to me like we are close to the point of developing humanoid robots that have most human capabilities and the low cost, reliability, and durability of a human body.
The problem of scale
Ajeya: Would you agree that if we had bodies everyone agreed were as sophisticated as human bodies, and we could see production doubling every year — 1,000 this year, 2,000 next year, 4,000 the year after — then the right thing to do is extrapolate that trend to the number we’d need for self-sufficient AI?
In other words, we’re having this argument because we don’t agree on whether existing humanoids count, so we don’t agree that a trend has started. But if there were a clear trend, like we have for solar panels or batteries, should we extrapolate it?
Timothy: There are two problems. One is scaling up the manufacturing and getting the cost down. It’s true that once that has started, that’s a very predictable thing. You can calculate a learning curve and say “every time you double production you get this much reduction in cost.”
But, again, the other problem is repairability. If you build a robot that only works for a week before you have to send it back to the factory because a particular motor in the elbow grinds down and stops working, that’s not going to be economically viable.
And I think one of the things you see in the industry right now is that for most tasks it doesn’t make sense to use a humanoid robot because there’s already a simpler robot that is cheaper, easier to repair, and more durable that can do 90% of the task, and all you need is a human to take the box of parts and set it next to the robot so the robot doesn’t have to walk around. So for most tasks it’s likely that humans and robots will remain complements for a long time.
That timing is hard to predict. It’s similar to the Waymo problem where 10 years ago Waymo’s software was handling 99% of cases correctly and we were saying, “Well, we can extrapolate how long until it’s 99.99%.” But it turned out to be really hard to predict how long it would take to get to that level of accuracy.
Ajeya: Does this mean your timeline is much earlier for a weaker milestone — say, an AI population sustained by a thousand humans doing unskilled physical labor, with cameras on their heads and earpieces so the AI can direct them, rather than needing specialized semiconductor fab workers? Do you think that happens many years before full self-sufficiency?
Timothy: I don’t know if I’d say a thousand, but yeah, definitely. I think we’ll reach a point where if you had willing human bodies, you could have a society where some AI model ran the whole society, made all the high-level decisions, and you had a group of humans wearing cameras and earpieces that tell them exactly what to do.
I don’t think the humans in that scenario would put up with that — I hope they wouldn’t — but yes, that would be a theoretical possibility a decade or longer before we get to the point where robots can do everything humans can do in a cost-effective way.
I do think it would take a lot more than a thousand humans to sustain that kind of society. I think we’re talking tens of millions. In a world where only a thousand humanoid bodies were necessary, the AI could get by with humanoid robots that break down once a week and just constantly repair or replace them.
Ajeya: When I say thousands of humans, I mean millions of robots doing physical tasks that are much more automated than now, but still with stuff around the edges that humans have to do, like in almost fully automated factories today where someone picks things up and puts them down. You would need millions of bodies to sustain an AI civilization of reasonable size, but do you think it’s plausible that only thousands of those could be humans?
Timothy: No, I don’t. I’d have to think about what the exact number is, but I think there are a lot of tasks across the economy that require humans. Think about mining, or logging, or climbing up telephone poles to replace fiber optic cables, or moving through crawl spaces to connect fibers. There are a lot of tasks like that, and they’re distributed in such a way that you couldn’t have one area populated by all the people necessary to do them. For many tasks you can have the robot do 99% of the work, but you need a person there for the 1% it can’t do.
If a resource is needed from a mine in Chile, there’s got to be someone in Chile at that particular mine ready to do that job. If you tally those kinds of examples, more than thousands of humans are needed. I don’t know if it’s millions or tens or hundreds of millions, but a modern civilization has a lot of different products with a lot of different inputs, and when you think about the volume of the world and all the things humans are doing, it would add up to quite a lot of people.
Stranger paths
Timothy: It’s also possible that there are unique characteristics of human biology that mean you would actually need a radically different set of technologies to build robots that have some of the characteristics of humans.
Think about flying. We have airplanes that in some respects are much more sophisticated than birds, but if we wanted to build a literal bird with the energy efficiency and the ability to land on a branch — we’re nowhere close to knowing how to do that. It’s possible there’s a similar issue with the human body, that we don’t have any technological equivalent for certain kinds of things we can do, like working for hours or having your finger heal after it’s cut.
The human body has some cells that are self-repairing. I don’t think we know how to make anything like that via traditional manufacturing. The human hand might always have an inherent advantage performing certain tasks because it’s based on a different kind of technology stack than steel and plastic. I’m not sure about that.
Ajeya: We’ve been talking about achieving self-sufficient AI by scaling up robots, but it’s plausible that self-sufficient AI actually comes first by leapfrogging that stack — developing a more energy-efficient, biologically-inspired technology.
I don’t want to anchor too much on robots being the only path. I can imagine superintelligent systems in a data center directing humans and robots to do wet lab research, resulting in small, self-replicating, semi-biological —
Clara: Classic nanobots.
Ajeya: There’s a range. You could get microbots, you could get nanobots.
Timothy: Yeah, I don’t want to put forth a strong thesis that it’s never. I think probably if you had enough smart AI, they would figure something out. But I think it’s plausible it would require a different paradigm that would take decades to develop.
Tacit knowledge
Timothy: Can I mention one other factor I think is relevant? There’s a question about whether these hypothetical AIs would have all the knowledge they needed.
Imagine if all the employees in the entire semiconductor industry disappeared — the machines and textbooks remain, but none of the people. How long would it take for the rest of humanity to restart the fabs? It’s quite possible that would take decades. Because even though you might have the textbooks, there’s a lot of tacit knowledge inside these machines. They’re very complicated. There are thousands of people in Taiwan who handle very specific things — this specific machine needs these settings or else this problem occurs.
There are analogous situations throughout the economy. Now, maybe we’ll be in a situation where we’ve already automated a lot of the routine stuff and there’s some AI out there that has all the information the other AIs need. But I think there’s a question about how much information the AIs have.
Ajeya: I do want to flag that my 10-year 50% forecast comes from expecting parallel automation of the physical stack and the software stack — which people are very much trying now — and from expecting big increases in overall AI intelligence.
There are two counters to the tacit knowledge hypothetical.One is that we’d have trained AI systems with reinforcement learning on that tacit knowledge because it’s profitable to automate what the Taiwanese worker was doing. The other is that AIs might get really generally intelligent in the sense of quickly figuring out new things by trying them, reading textbooks, and experimenting efficiently.
Clara: Tim, you mentioned earlier that one of the things holding back the sidewalk robots was that the profit isn’t clear. The financial incentives of automating workers at a TSMC fab seem much stronger and clearer to me. If you could get these fabs to run autonomously, wouldn’t that be huge?
Timothy: I’m not sure. Are the people the bottleneck? Obviously you’d save the salaries of thousands of workers, so that’s certainly some money. Would the fabs run dramatically faster if you replaced the people? If not, why would it be hugely profitable?
Clara: If we’re imagining Ajeya’s scenario, we’re imagining vastly increasing demand for these chips. And also, every time there’s discussion of American reshoring, the issue that comes up is that we don’t have enough people and that the people we do have don’t have the right training. This seems like a real bottleneck if we want to massively expand chip production.
Ajeya: Yeah, it’s not mainly that a single fab would print chips faster, though I expect some effect like that too. It’s that AI companies are already straining TSMC, and TSMC isn’t ramping up production fast enough to meet the demand I expect in a year or two for AI chips. Companies have vertically integrated chip design to varying extents — Google designs its own Tensor Processing Units — but TSMC is still the bottleneck for actual fabrication. There’s going to be big economic pressure in the next few years to scale up fab production.
Timothy: I completely agree that if you can make more fabs, that would be profitable. I guess the question I have is: Is that about physical things people do, or is it that the Taiwanese experts have expertise that’s hard to write down? Maybe the difficulty of transferring tacit knowledge to America is actually the main bottleneck.
I don’t know enough about the semiconductor industry to have a strong view, but that would be why I’d be a little skeptical that they’d have big gains from automation. Clearly, if there are routine tasks people are doing that you could automate, that’s going to speed you up. But I would think most routine tasks inside fabs would already be automated — you would still need people because sometimes the machine needs to be fixed. To do that, you need somebody to work the night shift. And Taiwanese people are maybe more willing to work those night shifts than Americans. And you need somebody with a very specific set of knowledge that’s hard to get in America.
Ajeya: Stepping aside from onshoring, the question is whether within 10 years there will be AI systems plus robots that have either been trained on the long tail of physical and cognitive tasks involved in AI production, or are generally competent enough, cognitively and physically, that when something unexpected comes up they can say, “We’ll comb through the textbooks, do it badly at first, then figure it out by trial and error” such that a new task they haven’t done before doesn’t spell doom.
I think both will be much greater in 10 years, from improvements in general sample-efficient learning, and from having actually been trained on the high-value physical tasks bottlenecking chip production. The second is what Clara was pointing out — if AI could do these tasks, that would be quite profitable because it would let us ramp up production and onshore. So people will be pushing on that.
Timothy: I think that’s true.
Clara: I had a question for you, Ajeya. What do we mean by fully automating the stack? Because it’s easy to think of the tech stack as making chips, but then someone has to mine the ores, and figure out logistics and more. It seems very hard for me to separate out the chips and energy that directly support the AI from the rest of the economy.
How do you think about the full automation of all the many thousands of component parts that would have to run for this whole process to work? For the chips, I understand doing reinforcement learning on this hyperspecific process, but there are many upstream components where there’s maybe less incentive to do so.
Ajeya: My guess is this gets done through a heavy dose of general-purpose robotics. It’s similar to the intellectual side, where the easier tasks — arithmetic where we just wrote down the algorithm, chess, certain kinds of search — were automated in specialized ways. Then the harder tasks get automated more broadly, even if it’s not the full breadth of human skills. We automated coding, conversation, essay writing, and answering history questions all in a mush around the same time by training on a very broad distribution. And I think we’re going to broaden that distribution further.
I think physical tasks will follow a similar pattern. Some we automated in the 1700s or 1800s or 1950s. Self-driving cars are more general than factory robots from the previous century, but still pretty specialized — they see a wide range of things, but driving is the only thing they can do, and they often have special-purpose components like route planning. By the time you’re imagining AI doing the mining, the power construction, the maintenance and manufacturing of robots, as well as any new and surprising tasks that come up — you need general-purpose robotics.
The reason I think that’s plausible goes back to where humanoid robots are bottlenecked. Again, I think it’s more likely to be cognitive capacities than physical capabilities. So it’s a big deal that people are starting to do imitation learning and RL in physical settings.
Prediction mode
Clara: Let’s switch to forecasting mode. I want to know, for both of you: What would you observe in the next two to three years that would make it seem like the 10-year forecast is on track or not on track?
Ajeya: It’s tough because there are two very different branches to self-sufficient AI. We’ve been talking about the humanoid robot branch. There’s also the superintelligence-to-nanobots branch. I think it’s hard to bound the probability of that second branch to a very low number, and Tim and I disagree on this too — we disagree on how quickly systems far more intelligent than top humans could discover radically new technology.
If we focus on the humanoid robot path and ignore the other one, I’d want to learn more about the haptics and physical limitations. If our robots don’t have good hands, why not? Could we build a great hand for $100 million? Is it that we don’t know how, or just that it’s not profitable and there’s some experience curve to ride? I’d want a line on a graph showing improvement of robotic hands, and another line showing the rate at which we’re manufacturing humanoid robots.
The cognitive side is similar to other AI forecasting — there are benchmarks to watch. Google DeepMind last year had an advance in general-purpose robotics, with videos of robots handling slight perturbations in their environment. I’d expect that technology to meaningfully improve in the next couple years. If it’s totally stagnant, that would push my timelines out for the robot path.
Timothy: I think we mostly agree about that. I’m going to want to watch how the humanoid robots develop: the number of robots, their capabilities, and particularly their cost and repairability. If I see those advancing more quickly than I expected, that’s going to shift me towards thinking this is possible earlier.
There’s some interplay between these. If you imagine a future where there are a lot of very reliable, very cheap humanoid robots with crappy hands that don’t work as well as human hands — I think you could still accomplish most tasks. Maybe you have to redesign all the machines in the world to have bigger knobs or something, but you could get a lot of value out of those cheap, reliable robots. So there’s some substitutability between the quality of the robots, the number of robots, and their cost. If we see a rapid takeoff in the number of humanoid robots, even if they’re not amazing, that’ll be a sign that I’m wrong and this is going to be possible relatively soon.
Ajeya: There’s also the actual integration of automation into key parts of the stack. There’s a research project someone could do: How physically automated is TSMC right now? How automated are the power production steps? Map out the whole stack — what do humans do and what do robots do? If my timelines are right, we should see more tasks subsumed by robotic processes over the next few years, whether specialized or general-purpose.
Timothy: I actually think this might be a place where we disagree. I think humanoid robots specifically are a canary in the coal mine. I expect robots to get more sophisticated, but I think it’s going to be cheaper and more efficient to have specialized robots — for example, robots that are bolted to a factory floor, or a self-driving car — paired with a human that carries the parts over to the robot, or repairs the robot when it breaks, or unloads the self-driving truck.
So the path is towards automating 95% to 99% of the human’s job, but hiring the human to do the 1%, and then scaling up production with the whole system. If we see rapid progress in robotics but very few humanoid robots being produced, that will seem like a sign that humans are still going to have an essential role that would be hard for AIs to replace. Whereas if we have a future where, increasingly, there are just humanoid robots that you can drop in for a larger and larger share of the workforce, then I could see that going up to 100% after a decade or so.
Ajeya: I see your point on humanoids being special, but if we don’t see automation eating more of the stack one way or another, I’ll feel like things are generally going slower. What I’m craving is a detailed mapping of the AI production stack — how many bodies, machines, power plants are currently necessary to sustain an AI population of millions? How many humans could drop dead today without affecting the AIs? I’d want to track that number over time.
For threat modeling, it really matters whether you need a million humans, 100,000, 10,000, or 1,000. Once you’re down to thousands or hundreds, a small number of coerced people or willing defectors becomes very salient for AI takeover scenarios. I don’t know what the number is today — my guess is tens of millions of humans are needed right now to sustain the current AI population.
Timothy: Yeah.
Ajeya: I’d want to map that out and track how it shrinks. A lot of the shrinking might happen through specialized robotics — though you might be right that you’d need fully humanoid robots or something more exotic to cross the last mile.
Ajeya Cotra is a researcher at METR working on assessing the risk of loss-of-control from advanced AI. Previously, she spent a decade at Coefficient Giving as a researcher and program officer.
Timothy B. Lee is a journalist who writes the newsletter Understanding AI. Previously he was on staff at the Washington Post, Vox.com, and Ars Technica. He has a master’s degree in computer science from Princeton.
This article originally appeared in Issue 14: Risk. Subscribe to the print magazine by June 30th to receive our next issue, Work.