In This Story
From September 27th to October 1st, Pittsburgh, PA will become the epicenter of the robotics world as it hosts the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). As one of the premier global robotics conferences, IROS 2026 brings together researchers and industry leaders to present groundbreaking work, showcase robotic innovations and discuss the future of intelligent systems through technical sessions, workshops, and demos.
Research teams at the RAI Institute had several papers accepted, covering topics from closed- loop visuomotor policies and frameworks for self-supervised learning on outdoor LiDAR point clouds to whole-body model predictive control for quadruped table tennis.
RAI will also have a booth in the exhibition hall (#1102) where you can check out some of our custom built robotic research platforms including the Ultra Mobility Vehicle (UMV), Roadrunner, and our Koala grippers for robotic data collection.
Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition
Learning from Demonstration (LfD) enables robots to learn complex behaviors from expert examples, yet existing approaches often fail to generalize to new compositions of known skills without retraining. Modern generative policies model distributions over action trajectories alone, thus are unable to reason about the symbolic outcomes required for robust composition.
We propose that skills should jointly model action trajectories and the symbolic outcomes they induce. To address this gap, we introduce Predicate-Action Skills (PACTS), a class of closed- loop visuomotor policies that model skills as a joint generative process over action and predicate belief trajectories, producing coherent action–outcome rollouts within a single model. Jointly generating actions and predicates enables PACTS to learn internal representations that improve both action generation and predicate classification. Furthermore, we demonstrate zero-shot composition of learned skills via planning by leveraging online predicate predictions from PACTS as a symbolic interface for sequencing and monitoring execution.
Vernata: Self-Supervised Learning of LiDAR Point Representations
LiDAR serves as a primary sensing modality for robots operating in outdoor environments. However, the performance of deep learning models in this domain is severely limited by the scarcity of labeled data, a direct result of the high cost of 3D annotation. Self-supervised learning addresses this scarcity by learning general-purpose features from unlabeled data. In this work, we present a multi-modal, multi-teacher distillation framework for self-supervised learning on outdoor LiDAR point clouds.
Building upon the Sonata architecture, we introduce Vernata, consisting of three extensions: sparse view augmentation to improve robustness against varying point densities, a memory bank mechanism to stabilize resource-constrained training, and cross-modal distillation utilizing dense, high-resolution 2D image features to enable fine-grained semantic guidance. We evaluate our method on the GrandTour, TartanGround, and Waymo datasets, as well as data collected from our own robotic platforms.
Our experiments demonstrate a significant performance improvement over Sonata baselines, yielding mIoU scores of 54.7 on TartanGround (+5.9 points, +12.1%) and 57.1 on Waymo (+7.3 points, +14.7%). Finally, we show that the self-supervised approach maintains strong performance even in reduced-modality settings (lacking color or normals), achieving competitive mIoU scores of 49.4 and 50.2 on the respective datasets.
Whole-Body Model Predictive Control for Spin-Aware Quadrupedal Table Tennis
Developing table tennis robots that mirror human speed, accuracy, and ability to predict and respond to the full range of ball spins remains a significant challenge for legged robots. To demonstrate these capabilities, we present a novel continuous-time model predictive controller (MPC) for agile full-body control of a quadrupedal robot equipped with an arm.
This formulation enables the flexibility to place time-critical constraints, such as those required to strike a ball, anywhere along the trajectory. We further develop a hybrid model-based and learned spin estimator that can accurately predict ball spin from its observed trajectory and an aiming planner that dictates how the ball must be struck. Notably, a continuous set of stroke strategies emerge automatically from different ball return objectives after combining the aiming planner and whole-body MPC. We demonstrate the system on hardware with a Spot quadruped, evaluate the accuracy of each system component, and exhibit coordination through the ability to aim and return balls with varying spin types. As a further demonstration, the system is able to rally with human players.
EMPM: Embodied MPM for Modeling and Simulation ofDeformable Objects
Modeling deformable objects – especially continuum materials – in a way that is physically plausible, generalizable,
and data-efficient remains challenging across 3D vision, graphics, and robotic manipulation. Many existing methods
oversimplify the rich dynamics of deformable objects or require large training sets, which often limits generalization.
We introduce embodied MPM (EMPM), a deformable object modeling and simulation framework built on a
differentiable Material Point Method (MPM) simulator that captures the dynamics of challenging materials. From
multi-view RGB-D videos, our approach reconstructs geometry and appearance, then uses an MPM physics engine
to simulate object behavior by minimizing the mismatch between predicted and observed visual data. We further
optimize MPM parameters online using sensory feedback, enabling adaptive, robust, and physics-aware object
representations that open new possibilities for robotic manipulation of complex deformables. Experiments show
that EMPM outperforms spring-mass baseline models.
ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim
We present ZeroBot, a real2sim framework for learning a robot manipulation task from scratch in minutes under challenging conditions: zero human demonstrations, zero policy pre-training, and zero known object models. Given only a single view of an object and a goal pose for that object, ZeroBot uses image-to-3D generative models to obtain a complete object mesh, which is used in simulation for large-scale parallel reinforcement learning. To accelerate training, we introduce an action space which leverages the generated geometry and learned value function to sample states involving robot-object contact. When evaluated on real-world tasks including grasping, pushing, articulated object interaction, and multi-stage manipulation, ZeroBot achieves an 87% success rate with an average training time of 119 seconds. These results show the value of using image-to-3D models in a real2sim framework for rapid, autonomous robot learning.
Test-Time Spatial Reasoning for Robot Manipulation Using Generative Real-to-Sim
Spatial reasoning is fundamental to general robot intelligence, as it enables robots to complete long-horizon tasks involving multi-object interaction. We introduce Simify, a training-free, test-time framework that performs explicit spatial reasoning via massively parallel physics simulation. From a single RGB-D image of a scene, Simify reconstructs simulation-ready assets leveraging 3D generative models and vision-language models. Then given a task specified by a reward function (e.g., build the tallest tower), Simify launches thousands of parallel rollouts in simulation and performs an evolutionary search to optimize object arrangements, typically converging within seconds. We conduct quantitative experiments on real-robot hardware to demonstrate the ability of our framework to execute complex object rearrangement tasks end-to-end with previously unseen objects. Results show that our framework outperforms prior work on foundation models for spatial reasoning by effectively exploiting large-scale parallel simulation during inference, and also highlight the importance of complete and accurate geometry for successful sim-to-real transfer.
