Robotic manipulation is evolving far beyond static, single-purpose automation. At the RAI Institute, we develop robots that manipulate objects dynamically, using their whole body and reacting to their surroundings in real time. In earlier work, we showed Spot uprighting and rolling a car tire, with a controller that combined reinforcement learning and sampling-based trajectory optimization to coordinate the robot’s legs, body, and arm. The 15 kg tire is heavier than Spot’s arm can lift on its own (11 kg peak, 5 kg continuous), so the robot used its legs and body together with the arm to move the tire. The earlier system worked only in our lab, because it relied on three technologies that were not on-board the robot: motion-capture cameras in the ceiling that located the tire, a desktop workstation that ran the controllers, and a script that read the motion-capture state and decided when to switch from one skill to the next.

The new system handles all of this itself. Beyond moving to fully onboard execution, the robot now finds the tire with its own depth cameras instead of a motion-capture system, which lets it leave the laboratory and go outside. Most of this post is about how the policies learned to work with data from those cameras. With these changes, the robot can now function in previously unseen environments and perform autonomous tasks such as tire rolling, uprighting, dragging, and recovering from unexpected perturbations.

This video demonstrates how seamlessly our new policies work with the onboard cameras; now, a person simply gives the robot a goal location and the robot performs the entire sequence while being robust enough to stand the tire up when it tips over on the way.

Every skill is a neural network

The control stack has two layers: a locomotion policy trained with reinforcement learning that keeps the robot balanced and walking, and a skill policy that decides where the body should go and what the arm should do. What changed is the skill layer. The previous version sampled dozens of possible actions and simulated their futures across multiple CPU threads to upright, drag, and stack the tire, while reinforcement learning handled rolling. This strategy was effective, but needed a powerful desktop computer in the loop and an engineered script to properly mix these techniques. Now, every skill is a reinforcement learning policy trained in simulation with the tire’s exact pose and then transferred to use the robot’s cameras. Running these skills on the robot is as easy as a single neural network pass, greatly simplifying the system.

A skill policy outputs ten numbers at 50 Hz: three adjustments to the base velocity and seven to the arm’s joint targets. The adjustments accumulate, so an output of zero holds the current target, and a sustained push builds sustained force against the tire. All of the skills share this same action space and reward template, which rewards progress toward the goal, getting the gripper onto the tire, and positioning the body to successfully complete the task, and penalizes jerky motion of the tire. Some skills have a few specialized reward terms, such as keeping the tire upright while rolling or laying it flat while stacking. Tire mass, friction, and size are randomized during training so that the policy cannot rely on the exact tire it was trained with. With anywhere from 16,000 to 64,000 simulated robots training in parallel on a given scene, a single skill trains in a few hours on a single desktop GPU.

Simulated robots training the tire-rolling skill in parallel. Each skill takes a few hours on a single desktop GPU.

Teaching the policy to see

A policy trained with state information always knows exactly where the tire is because the simulator tells it. In the earlier system, motion capture played that role. The usual way to replace state-based inputs with cameras is to freeze the trained policy to act as a teacher and train a second, camera-based student to imitate it. While that allowed us to transfer capabilities to some of the tasks like upright and drag, more dynamic motions like tire re-orientation during rolling worked poorly. We therefore designed a pipeline that keeps reinforcement learning running for the entire training process. Over the course of training, the policy’s privileged inputs are gradually replaced with more realistic perceptive observations.

Our perception pipeline is composed of three stages that each train an encoder. Every successive encoder learns to output the same feature vector as the previous one, but from progressively less privileged information. The stages are as follows: 

  • Encoding object position from exact poses: This stage takes in the exact tire pose as input and outputs the tire pose’s representation in a latent vector space, which is used as a ground-truth comparison for the next stage.
  • Learning to handle intermittent visibility: This stage receives exact tire poses as input, but only when the tire is seen by the camera; otherwise, its input is zeroed out. This approach allows the encoder to learn robust behavior even when it does not see the tire. At this stage the network still relies on privileged information like exact object position and velocities.
  • Translating from poses to images: This stage receives depth images as input. Because  the previous stage already learned how to handle intermittent tire sightings, this stage adapts the neural network to only use information available in the vision-based sensors.

As each perception stage converges, the skill learning pipeline learns to rely less on the previous stage and more on the new stage until the new encoder fully replaces it. This process allows us to bootstrap the learning of various parts of the pipeline as we approach the realistic information used during real-world deployment. We call this method Crossfade RL because the policy continuously learns using reinforcement learning while using distillation to enhance newly introduced architecture components.  

The resulting vision-based policy behaves very differently from the previous state-based policy, even though the reward was not changed. The state-based policy rolls the tire aggressively and spins it toward the goal in one continuous motion, because of the availability of privileged information. On the other hand, the depth-based policy is risk-averse. It learned to walk around the tire, make contact from an angle that keeps the tire in front of it, and steer with smaller corrections so that the tire rarely leaves camera view or becomes unstable. While cautious with the rotation of the tire, the policy is still very fast and dynamic with rolling. Nothing in the reward design encourages either behavior. Each learns based on the perception constraints: the privileged policy that always knows the tire’s location takes risks, and the policy that cannot always see the tire learns not to. Furthermore, real depth is much noisier than simulated depth, so the simulated images were heavily corrupted during training with distance-dependent noise, dropped pixels, and blank strips where the arm blocks a camera. The noise, corruption, and introduction of foreign objects like walls were necessary for the sim-to-real transfer.

Deciding what to do next

Moving a tire to a goal takes a sequence of skills. The robot has to walk to the tire, stand it up if it is lying flat, get behind it, roll it, drag it the last stretch onto the target, and stand it up again whenever it tips over. In our earlier work, a scripted state machine stepped through that sequence using the tire's pose from motion capture. While using onboard sensors, such information is no longer available, so the robot does it with a small network we call the gating network. The gating network reads the same depth images as the skills, along with the robot's proprioception and the goal, and outputs which skill to execute at the given timestep. We trained the gating network with supervision from a scripted state machine. In simulation, hand-written rules with access to the exact tire state decide when to switch; the gating network learns to make the same decisions from what the cameras see. Each skill stays in a separate network that can be trained and checked on its own, and the gating network chooses among them. All of the networks run at once on the computer on the robot's back, and the gating network decides whose output reaches the robot. 

We first tried the more obvious design, distilling all of the skills into one network, and found the gating network solution to be more robust. In a clean run, the gating network switches exactly as many times as the sequence has stages, and it goes back to an earlier stage only when something has gone wrong. We tested this on the basketball court, deliberately knocking the tire over with a stick in the middle of a roll. The gating network seamlessly switched to uprighting, the robot stood the tire up, and the roll continued.

The deployed system has no external parts. Perception comes from the five depth cameras built into Spot's body: the two front cameras are read at 20 Hz, while the side and rear cameras are queried less frequently, at 7 Hz.  There is no object detector or tracker; depth images go straight into the networks. The goal is a position a few meters away, tracked in a global reference frame via visual odometry. An onboard Mac Mini computer runs the gating network, skills, and locomotion policy, and sends joint targets to the robot at 50 Hz. Where the sampling-based controller needed a desktop with 32 CPU threads, the new stack is now a few forward neural-network passes that take about 1 ms to query.

Outside the lab

We ran the system in four uninstrumented settings outside the lab: a skatepark, a basketball court, a garage, and the floor of our building. We used the same 15 kg car tire from previous experiments. The sequence was the same in each: the tire starts on its side, and the robot walks to it, stands it up, rolls it, and drags it onto the target, with the operator setting one goal position.

The tire drops out of the back of a van and lands on its side. The robot stands up, walks over, uprights the tire, rolls it toward the target, and drags it the last stretch. All of the switches between different skills are orchestrated by the gating network.
The same sequence at the skatepark
The same sequence on the basketball court.
A person knocks the tire over with a hockey stick in the middle of a roll. The gating network switches to uprighting, the robot stands the tire up, and the roll continues.

The van tire offload and the pallet stacking tasks are individual policies that are executed on their own.

Pulling the tire from the van. The robot hooks it with the gripper, walks backward, and drags it over the edge to the ground.
Stacking. The robot tips the standing tire so that it lands flat on the pallet.

What still breaks and what comes next

The policies were trained with a single object and walls in the depth view, not for obstacle avoidance, but to help disambiguate static environmental features from other entities in the scene, which worked reasonably well in the garage sequence. However, one class of failure comes from external objects in the scene. Anything that shows up as a blob in a depth image can be taken for the tire, whether a person walking past or a stray object on the floor. In a few runs the robot left the tire and went after something else. Training never required the policy to tell the tire from other objects, so it learned to treat any blob as the tire. Adding distractor objects during training, so that the policy has to learn what a tire looks like, is the obvious next step.

The robot gets stuck while pulling the tire off the van. It hooks the tire but cannot drag it over the edge, as it expects the tire to slide off more easily.

Two of the skills, the van pull and the pallet stack, work on their own but not yet under the gating network, and combining them with the rest is the next step. Each skill is its own network with its own tuned reward weights, and a new skill means tuning a new reward. A single training run takes hours, and finding the right reward specification takes many runs.

We have tested one tire on flat ground, and the policies treat anything blob-shaped as that tire. Multiple objects, rougher ground, and obstacles between the robot and the tire are the next challenges. The locomotion policy is the reusable foundation, and it is the part that has been repurposed throughout multiple efforts. Revisiting the interface and capabilities of the low-level controller could help us further improve the capabilities of the system.

Authorship

This work builds on our earlier system for dynamic whole-body manipulation. Contributors: Leonor Fermoselle*, Maks Sorokin*, Duy Ta, Dmitry Yershov, Farzad Niroui, Brandon Hung, Jan Brüdigam, Emmanuel Panov, Simon Le Cléac'h (* equal contributions)