The past decade has seen huge leaps in the fields of robotics and artificial intelligence. Quadruped robots are autonomously roaming industrial sites, collecting data and conducting inspections; humanoid robots are performing complex parkour routines and venturing into manipulation tasks; and large language models have become advanced and ubiquitous across fields.

The pace of progress has been breathtaking, astonishing, and even, to some, a bit unnerving.

It’s not enough for Marc Raibert.

“If you look at all the humanoids that are currently coming on deck, they are all kind of…just very carefully walking around,” says Raibert, executive director of The AI Institute and founder of Boston Dynamics. “You don’t see them handling things and almost no dexterity yet. About ten years ago at Boston Dynamics, we started programming humanoid robots to perform tasks – manipulation and motions like jumping and handstands – but it took a roomful of very skilled programmers, and every new thing you wanted them to do took an incredibly long time. Today’s robots are too hard to program, and it’s too hard to get them to do new things.”

While the rest of the world looks on in awe as robots assemble cars, fold laundry, star in dance videos, and load trucks in warehouses, Raibert can’t quite decide whether to call the current robots “as dumb as a doorknob” or “as dumb as a toaster.”

In 2022, Raibert founded The AI Institute, with the goal of solving the most important and fundamental challenges in robotics and AI. But first, the Institute had to define those challenges and work to create tangible milestones for teams to unlock on the path to their end goals.

The Institute hopes to answer the question: What does it mean to make progress in a field that is rapidly evolving, but in many ways is still an unexplored frontier?

Defining the Challenge

“If the only thing we wanted was a robot to change a tire on a car, we could build that robot right now,” says Jessica Hodgins, vice president of strategic projects and research for the Institute. “But that isn’t what we want. We want a robot that can do all repair jobs on all cars—or do any kind of work with more generality—and do it under all circumstances.”

In other words, the AI Institute isn’t interested in creating a bunch of robots that can each do just one thing well; rather, the goal is to lay the groundwork for robots that can do everything well—or, at least, robots that can do many tasks as well as humans do them. To do that, the Institute will need to develop robots that, like humans, can enter a jobsite, receive some simple instructions, and then get to work.

“What we want to do is figure out what the hard parts of the problem are,” Hodgins says.

This is a deceptively difficult question. Throughout the history of robotics, researchers have repeatedly been surprised at which tasks AI has been able to do successfully, and which the technology continues to struggle with. For instance, robots are generally still quite bad at dexterously manipulating small objects, something people find easy, while chess and go, games that most people struggle to master, are solved.

For people outside the industry, it can be difficult to even conceptualize just how challenging certain tasks remain for robots, says Jennifer Barry, a research lead at the Institute. “Humans are great at perception—at being able to look around a room and say, ‘That’s a desk, that’s a cubicle, that’s a monitor,’” Barry says. “We can do that without any brainpower or stress, but that’s still so hard for robots. I’ve had a lot of people ask me why we don’t just put AI on a robot. And I tell them: ‘Yes, that’s what we’re trying to do.’”

But not everyone inside the industry, Barry says, aligns with the Institute’s school of thought regarding generalized robots. Some, she notes, think that progress lies in perfecting niche robots for specific use cases, rather than in striving for robots that can clean your kitchen one day and rebuild your bicycle the next.

“This is where the Institute becomes very valuable,” Barry says. “If you’re trying to sell a robot right now, you’re not going to have the resources to solve the most fundamental challenges in the field. It’s way too hard, and it’s way too expensive.” The Institute sits in a unique position where the team can develop generalized robot skills without getting distracted by the rigors of productizing.

Charting the Path

Raibert believes that the secret to pushing AI and robotics forward lies in the combination of “cognitive intelligence” (defined as a robot’s ability to generalize, see relationships between things, use common sense, and plan) and “athletic intelligence” (meaning the ability to perceive, maneuver, balance, and grasp in the ways needed to complete these tasks).

Many of the trickiest, most important capabilities are seemingly simple things that people take for granted. Today, there are robots that can perform a backflip or hold a conversation. But there isn’t a robot in the world that can walk into a kitchen, read a recipe, and start chopping an onion. That’s the sort of progress that the Institute is after.

Raibert says that an organization like the Institute is necessary to bridge the gap between the research approaches of universities and private companies. In higher education, he notes, there are obstacles standing in the way of the sort of cross-disciplinary, collaborative research that tends to lead to breakthroughs in robotics. And in industry, he says, economic pressures lead companies to focus on technologies that they can bring to market within the next 18 to 24 months (compared to the five-to-eight-year time horizon that the Institute has targeted).

“On the academic side, it’s hard to do projects at scale,” Raibert says. “And companies need to have projects that pay off over the short term.”

Researchers at the Institute have devised projects aimed at pushing forward specific areas of robotics and AI. In one, dubbed “Watch, Understand, Do,” researchers are trying to develop robot bike mechanics that can learn how to perform basic repair and maintenance jobs after watching a human complete the same tasks. In another project, a team is working to design robots capable of performing BMX-style tricks, with the goal of improving the balance, perception, and planning capabilities of robots.

But the Institute also has researchers looking into broader research areas. The Foundation Models team, headed up by research leads Laura Herlant and David Watkins, is studying “models that can be used to generalize downstream tasks,” Watkins says. These include large language models, but also models that include data from video, audio, and movement data from the robots themselves.

“We’re taking the theoretical results in the field and seeing how they actually perform on a wide variety of robots,” Herlant adds.

Over the next half decade, Watkins says, this research might support robots that perform a variety of work in factories, clean up homes, help the disabled, and beyond. “Before the advent of foundation models, generalized tasks were challenges that felt absolutely insurmountable,” he says.

Achieving Real Progress

Al Rizzi, chief technology officer for the Institute, says the biggest surprise of his career in robotics has been the fact that meaningful advances rarely come from one person—or even one team—having an a-ha! moment. “Rather, the real progress comes from diverse groups getting together and making all of their different pieces come together,” he says.

This is because robots are, essentially, only as capable as their weakest link, and they are made up of many links. Incredible new hardware won’t do much good if battery power is lagging, for example, just as amazing new software can’t make much happen if the hardware lets you down.

Because of this complexity, the Institute is following what Raibert calls a “steppingstones to moonshots” model. “The moonshot is the long-term, hard problem,” Raibert says. “If you just try and work on that, and you don’t do anything until you’re there, you won’t have enough information to make progress. So instead, you take a stab at something smaller and more tangible, while keeping your eyes on the longer-term prize. Rather than waiting until there are game-changing breakthroughs, the research teams will aim to make steady, manageable advances, learn from intermediate results, and ultimately achieve transformational change.