# Imitation Learning

RDT-1B is a state-of-the-art imitation learning diffusion transformer with 1 billion parameters (currently the largest). It has been pre-trained on over 1 million multimodal robot scenarios. Given language instructions and RGB images from up to three views, RDT can predict the next 64 robot actions. RDT is compatible with nearly all modern mobile manipulators, including single to dual-arm systems, joint to end-effector configurations, position to velocity control, and even wheeled motions. The model has been fine-tuned on over 6,000 self-collected bimanual scenarios and deployed on the ALOHA dual-arm robot, achieving leading performance in dexterity, zero-shot generalization, and few-shot learning.

The Surgical Robot Transformer is a model that executes surgical operation tasks on the Da Vinci robot using imitation learning. This model overcomes the inaccuracies in forward kinematics of the Da Vinci system through relative motion formulas, enabling the successful training and deployment of policies. One significant advantage of this approach is the ability to leverage a large amount of clinical data containing approximate kinematics for robot learning without additional calibration. The model has demonstrated success in executing three fundamental surgical tasks, including tissue manipulation, needle handling, and suturing.

HumanPlus is a research project aimed at training humanoid robots to learn skills autonomously by imitating human actions. The project simulates reinforcement learning to train low-level policies, applying these policies to the real world for real-time tracking of human body and hand movements. Through shadow imitation techniques, operators can remotely control the robot to collect full-body data for learning different tasks. Additionally, behavior cloning techniques enable the robot to imitate human skills and perform various tasks.

LeRobot is an open-source project aimed at lowering the barriers to entry into the field of robotics, enabling everyone to contribute and benefit from shared datasets and pre-trained models. It includes the most advanced methods verified in the real world, with a special focus on imitation learning and reinforcement learning. LeRobot provides a set of pre-trained models, datasets with human-collected demonstration videos, and simulation environments, allowing users to begin without assembling a robot. In the coming weeks, we plan to add support for the most affordable and capable real-world robots.

DexCap is a portable hand motion capture system that combines holography and electromagnetic field technology to provide accurate and occlusion-resistant wrist and finger motion tracking. It collects data by 3D observing the environment. The DexIL algorithm leverages inverse kinematics and point cloud-based imitation learning to directly train dexterous robotic hand skills from human hand movement data. The system supports optional human-in-the-loop calibration mechanisms, enabling the robotic hand to replicate human movements and further improve its performance based on human hand actions.