

Maniwav
Overview :
ManiWAV is a research project aimed at learning robot manipulation skills from wild audio and visual data. It collects synchronized audio and visual feedback from human demonstrations and directly learns robot manipulation policies from these demonstrations through corresponding policy interfaces. The model demonstrates its system's capabilities through four contact-rich manipulation tasks, which require the robot to passively perceive contact events and patterns or actively perceive the material and state of object surfaces. Additionally, the system can generalize to unseen wild environments by learning from diverse wild human demonstrations.
Target Users :
ManiWAV is primarily targeted toward researchers and developers in the robotics and artificial intelligence fields. They can leverage ManiWAV to enhance robot manipulation skills in complex environments, particularly when visual information is unclear or incomplete, by augmenting the robot's perception capabilities through auditory information.
Use Cases
Using the ManiWAV system to wipe graphics off a whiteboard
Utilizing ManiWAV to flip pancakes in a frying pan
Using ManiWAV to transfer dice from one cup to another
Using ManiWAV to secure wires with Velcro tape
Features
Provides rich interaction and object attribute information through audio signals
Uses 'ear-in-hand' data collection devices to synchronously collect audio and visual feedback
Directly learns robot manipulation policies from human demonstrations
Demonstrates system capabilities in four contact-rich manipulation tasks
Learns through diverse wild human demonstrations to achieve generalization to unseen environments
How to Use
1. Install and configure ManiWAV data collection devices, ensuring synchronized audio and visual feedback.
2. Collect human demonstration data, including audio signals of contact events and object surface materials.
3. Train robot manipulation policies using the collected data.
4. Test and validate the effectiveness of the policies in diverse wild environments.
5. Adjust the policies based on testing results to improve the robot's generalization capability.
6. Apply the trained policies to actual robot manipulation tasks.
Featured AI Tools

RT Trajectory
RT-Trajectory is a robot control strategy based on rough trajectory sketches, enabling effective generalization to new tasks through trajectory sketches. It can generate trajectory sketches through manual drawing, video demonstrations, or image generation models. RT-Trajectory has been widely evaluated in various real-world robotics tasks and demonstrates broader task execution capabilities compared to language- and goal-based strategies.
AI machine
58.0K

Maniwav
ManiWAV is a research project aimed at learning robot manipulation skills from wild audio and visual data. It collects synchronized audio and visual feedback from human demonstrations and directly learns robot manipulation policies from these demonstrations through corresponding policy interfaces. The model demonstrates its system's capabilities through four contact-rich manipulation tasks, which require the robot to passively perceive contact events and patterns or actively perceive the material and state of object surfaces. Additionally, the system can generalize to unseen wild environments by learning from diverse wild human demonstrations.
AI machine
45.3K