In Behavior Cloning (BC), policies are trained on human demonstrations, allowing the robot to be taught to perform complex tasks simply by showing it how. BC is an effective way to teach robots new manipulation skills. However, while this approach is powerful, it lacks a mechanism for self-improvement beyond the demonstration data.  

The ability for self-improvement is critical for tasks and environments with changing conditions. The performance of a policy can quickly degrade if the task changes, object positions and/or lighting shifts, or if the robot encounters a situation that was not well represented in the demonstrations.

How can the ability for robust self-improvement be added to Behavior Cloning?

Moving Beyond Demonstrations

When a BC policy underperforms, one approach to improving its performance is reinforcement learning (RL), which enables the robot to learn from its own experience. However, moving from offline training (BC) to online learning (RL) is notoriously difficult, because of “catastrophic forgetting”. The RL training process can overwrite the “good” behaviors the robot learned by the policy, essentially unlearning what it already knows. As a solution to this problem, some works propose “warm-starting” RL training by learning a Q-function offline, but this approach requires large, diverse datasets.

Presented at RSS 2026, Q2RL (Q-Estimation and Q-Gating from Behavior Cloning for Reinforcement Learning) is a method that avoids catastrophic forgetting when transitioning from offline behavior cloning to online reinforcement learning, without requiring diverse offline datasets. 

The algorithm functions in two phases: Q-Estimation and Q-Gating. In the Q-Estimation phase, the pre-existing Behavior Cloning policies are used to estimate a Q-function. By analyzing the BC policy’s action probabilities, specifically its likelihoods and entropy, higher Q-values to actions are assigned to the policy it is already confident in. This approach provides a reliable starting point for online learning (RL) without needing the original, large training datasets.


Following Q-Estimation, policy performance is improved via online reinforcement learning, incorporating a gating procedure called “Q-Gating”. Q-Gating maintains two Q-functions, one frozen to represent the BC policy and one learnable to represent the new RL policy, to compare the estimated Q-values at every step. It then executes the action predicted to have higher value, ensuring the robot learns through RL, without discarding the high-quality behaviors initially learned from human demonstrations.

Q2RL Performance Results

Q2RL was evaluated across simulation benchmarks from D4RL and robomimic. Across these tasks, Q2RL generally improved over the initial BC policy and outperformed several offline-to-online learning baselines in success rate and convergence speed.

Results on D4RL (Kitchen, Pen, Door). Q2RL starts with strong initial performance from BC and continuously improves, outperforming baselines.
Results on robomimic (Lift, Can, Square) — both with and without access to offline training data. Q2RL is the only method that reliably learns without offline data, while remaining competitive with data access.

The real-world experiments focused on three manipulation tasks: peg insertion, pipe assembly, and kitting. These tasks require the robot to reason through contact, alignment, grasping, multi-step execution, and task distribution shift. 

The experiments used a 7-DOF Franka Panda robot equipped with a Robotiq 2F-85 gripper. Standard RGB images from workspace and wrist-mounted cameras were used to feed the policy. By leveraging an asynchronous framework that separates actor and learner processes, the training was both sample-efficient and stable enough for real-world execution.

On hardware, Q2RL improved performance on every task, within 1-2 hours of online interaction:

  • Achieved 100% success on a benchmark peg insertion task.
  • Raised success in pipe assembly, a long horizon task involving grasping then inserting a PVC pipe, from 20% success with the BC policy to 75%.
  • Improved a kitting task under task distribution shift, where the BC policy was trained under one task distribution and shifted to another. Q2RL raised performance from 35% to 70%.

In terms of qualitative results, Q2RL also helps behavior cloning policies “get unstuck.” A BC policy may know how to begin a task correctly, but fail during contact-rich alignment, insertion, or recovery when the scene differs slightly from the demonstrations. Instead of repeatedly executing the same unsuccessful behavior, Q2RL can switch to reinforcement learning actions to continue making progress toward the goal.

Q2RL often uses the BC policy for the early stages of the task, such as grasping and coarse alignment, then switches to RL actions for the final contact-rich insertion. In the kitting task, the BC policy was trained with one object in each bin, but the test setting contained multiple objects in each bin. Q2RL learned to use BC actions for familiar movements between bins, while using RL actions for the parts of the task affected by the distribution shift. This behavior is one of the main advantages of Q2RL: rather than replacing the BC policy outright, the robot learns when to trust it and when to improve upon it.

The real world experiments also highlighted the importance of safe exploration. One of the baseline methods we compared against produced excessive force against the board and faulted the robot. Q2RL avoided such safety violations, thanks to the algorithm’s effective use of the BC policy to bootstrap and initialize the RL policy.

What Comes Next

Q2RL demonstrates that you do not need to choose between the reliability of imitation learning and the adaptability of reinforcement learning. By extracting value estimates directly from BC policies, the learning process can be bootstrapped, making online reinforcement learning a viable, safer, and faster post-training strategy for robots.

The potential applications for this work include environments where the ability to improve deployed performance is paramount. For complex assembly tasks, for example, Q2RL excels because it intelligently switches to RL for high-precision, contact-rich segments of the task while using BC for general movement. The method also demonstrates strong adaptability to distribution shifts. In testing scenarios where the environment changed, such as encountering two parts in a bin instead of the single part used in initial training, Q2RL allowed the robot to effectively utilize BC-learned behavior for the general task while leveraging RL to adapt to grasping objects in new, previously unseen locations.

One limitation of Q2RL is that it requires the BC policy to provide specific action likelihoods and entropy. Future work will focus on extending the algorithm to broader policy classes that do not inherently provide these metrics. 

For more technical details, code, and videos of our results, visit the project page at https://q2rl.rai-inst.com/.