A Real2Sim2Real Transformer Policy System for Bimanual R2R2H Tasks

We present a Real2Sim2Real system with a pair of UR3 arms equipped with Allegro 4-finger hands for bimanual, long-horizon R2R2H (Robot-to-Robot-to-Human) interaction tasks. Our system is composed of three main modules: (1) Real2Sim module to extract observation data and collect expert demonstrations from the real hardware setup; (2) Temporal-Context Transformer (TCT) policy module that train the policy in simulation using expert demonstrations to preserve human motion trajectories; (3) Sim2Real module to transfer the trained policy from simulation to hardware by leveraging domain randomization and real-time object detection. We have evaluated our system with two R2R2H tasks: (1) Task #1: an R2R2H tube handover and (2) Task #2: an R2R cup flipping and placing. Our presented system has completed the tasks, achieving success rates of 82.86% in simulation and 77.14% in hardware for Task #1, and 57.14% and 51.43% for Task #2. Demonstrating the feasibility of our system in real hardware.