Skip to content

Latest commit

 

History

History
75 lines (50 loc) · 2.23 KB

manipulation.md

File metadata and controls

75 lines (50 loc) · 2.23 KB
Raw

Manipulation tasks with natural language

This demo showcases the capabilities of RAI (Robotec AI) in performing manipulation tasks using natural language commands. The demo utilizes a robot arm (Franka Emika Panda) in a simulated environment, demonstrating how RAI can interpret complex instructions and execute them using advanced vision and manipulation techniques.

Manipulation Demo

Note

This readme is a work in progress.

Setup

  1. Follow the RAI setup instructions in the main README.
  2. Download additional dependencies:
poetry install --with openset

  1. Clone the manipulation demo repository:

    git clone https://github.com/RobotecAI/rai-manipulation-demo.git src/examples/rai-manipulation-demo

  2. Download the latest binary release for your ROS 2 distribution:

  3. Unpack the binary:

    For Humble:

    unzip RAIManipulationDemo_1.0.0_jammyhumble.zip

    For Jazzy:

    unzip RAIManipulationDemo_1.0.0_noblejazzy.zip

  4. Build the ROS 2 workspace:

    colcon build --symlink-install

Running the Demo

Note: Ensure that every command is run in a sourced shell using source setup_shell.sh

  1. Start the demo 
    ros2 launch examples/manipulation-demo.launch.py
  2. Interact with the robot arm using natural language commands. For example: 
    Enter a prompt: Pick up the red cube and drop it on other cube

How it works

The manipulation demo utilizes several components:

  1. Vision processing using Grounded SAM 2 and Grounding DINO for object detection and segmentation.
  2. RAI agent to process the request and plan the manipulation sequence.
  3. Robot arm control for executing the planned movements.

The main logic of the demo is implemented in the ManipulationDemo class, which can be found in:

examples/manipulation-demo.py