FoundationPose node failed to run

[Yeager-101]

I have successfully installed foundationPose according to the ISSAC_ROS documentation, and have downloaded and converted the three required model files: refine_trt_engine.plan, score_trt_engine.plan, and sdetr_grasp.plan, and I have modified the foundationPose configuration file for realsense camera startup as follows:

import os

from ament_index_python.packages import get_package_share_directory
import launch
from launch.actions import DeclareLaunchArgument
from launch.conditions import IfCondition
from launch.substitutions import LaunchConfiguration
from launch_ros.actions import ComposableNodeContainer, Node
from launch_ros.descriptions import ComposableNode

# Number of Realsense messages to be dropped in 1 second
HAWK_EXPECT_FREQ = 28
# Expected number of Realsense messages in 1 second
INPUT_IMAGES_DROP_FREQ = 30

RT_DETR_MODEL_INPUT_SIZE = 640  # RT-DETR models expect 640x640 encoded image size
RT_DETR_MODEL_NUM_CHANNELS = 3  # RT-DETR models expect 3 image channels

REALSENSE_IMAGE_WIDTH =  1280
REALSENSE_IMAGE_HEIGHT = 720

VISUALIZATION_DOWNSCALING_FACTOR = 10

REALSENSE_TO_RT_DETR_RATIO = REALSENSE_IMAGE_WIDTH / RT_DETR_MODEL_INPUT_SIZE

REFINE_MODEL_PATH = '/tmp/refine_model.onnx'
REFINE_ENGINE_PATH = '/home/nvidia/Pose/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan'
SCORE_MODEL_PATH = '/tmp/score_model.onnx'
SCORE_ENGINE_PATH = '/home/nvidia/Pose/isaac_ros_assets/models/foundationpose/score_trt_engine.plan'


def generate_launch_description():
    """Generate launch description for testing relevant nodes."""
    rviz_config_path = os.path.join(
        get_package_share_directory('isaac_ros_foundationpose'),
        'rviz', 'foundationpose_realsense.rviz')

    launch_args = [
        DeclareLaunchArgument(
            'hawk_expect_freq',
            default_value=str(HAWK_EXPECT_FREQ),
            description='Number of Realsense messages to be dropped in 1 second'),

        DeclareLaunchArgument(
            'input_images_drop_freq',
            default_value=str(INPUT_IMAGES_DROP_FREQ),
            description='Expected number of Realsense messages in 1 second'),

        DeclareLaunchArgument(
            'mesh_file_path',
            default_value='/home/nvidia/Pose/box/box.obj',
            description='The absolute file path to the mesh file'),

        DeclareLaunchArgument(
            'texture_path',
            default_value='/home/nvidia/Pose/box/baked_mesh_tex0.png',
            description='The absolute file path to the texture map'),

        DeclareLaunchArgument(
            'refine_model_file_path',
            default_value=REFINE_MODEL_PATH,
            description='The absolute file path to the refine model'),

        DeclareLaunchArgument(
            'refine_engine_file_path',
            default_value=REFINE_ENGINE_PATH,
            description='The absolute file path to the refine trt engine'),

        DeclareLaunchArgument(
            'score_model_file_path',
            default_value=SCORE_MODEL_PATH,
            description='The absolute file path to the score model'),

        DeclareLaunchArgument(
            'score_engine_file_path',
            default_value=SCORE_ENGINE_PATH,
            description='The absolute file path to the score trt engine'),

        DeclareLaunchArgument(
            'rt_detr_model_file_path',
            default_value='',
            description='The absolute file path to the RT-DETR ONNX file'),

        DeclareLaunchArgument(
            'rt_detr_engine_file_path',
            # default_value='',
            default_value='/home/nvidia/Pose/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan',
            description='The absolute file path to the RT-DETR TensorRT engine file'),

        DeclareLaunchArgument(
            'launch_rviz',
            default_value='False',
            description='Flag to enable Rviz2 launch'),

        DeclareLaunchArgument(
            'container_name',
            default_value='foundationpose',
            # default_value='foundationpose_container',
            description='Name for ComposableNodeContainer'),
    ]

Finally, I started the foundationPose node, ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose_realsense.launch.py, first I made sure I disabled X11 forwarding, my hardware environment is Jetson AGX Orin 32GB, ubuntu22.04 with CUDA12 JetPack6.0, the following error appeared after the node started:

[component_container_mt-1] [INFO] [1725524396.095299005] [drop_node]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/resize_left_viz' in container '/foundationpose'
[component_container_mt-1] [INFO] [1725524396.841828199] [foundationpose_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1725524396.841999883] [foundationpose_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1725524396.842041355] [foundationpose_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1725524396.842072204] [foundationpose_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/output", data_format="nitros_detection3_d_array"
[component_container_mt-1] [INFO] [1725524396.842099981] [foundationpose_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1725524396.842120493] [foundationpose_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/pose_estimation/pose_matrix_output", data_format="nitros_tensor_list_nchw"
[component_container_mt-1] [INFO] [1725524396.842146094] [foundationpose_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1725524396.842178414] [foundationpose_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1725524396.842199759] [foundationpose_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/rgb/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1725524396.842223503] [foundationpose_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_mono8"
[component_container_mt-1] [INFO] [1725524396.842243856] [foundationpose_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_32FC1"
[component_container_mt-1] [INFO] [1725524396.842652536] [foundationpose_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1725524397.096062001] [resize_left_viz]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1725524397.096230965] [resize_left_viz]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1725524397.096270166] [resize_left_viz]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1725524397.096303126] [resize_left_viz]: [NitrosPublisher] Use only the compatible publisher: topic_name="/rgb/image_rect_color_viz", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1725524397.096329175] [resize_left_viz]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1725524397.096348535] [resize_left_viz]: [NitrosPublisher] Use only the compatible publisher: topic_name="/rgb/camera_info_viz", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1725524397.096373304] [resize_left_viz]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1725524397.096397560] [resize_left_viz]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1725524397.096417721] [resize_left_viz]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/rgb/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1725524397.096836258] [resize_left_viz]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1725524397.121946293] [resize_left_viz]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/GGZLWRRQKN/GGZLWRRQKN.yaml"
[component_container_mt-1] [INFO] [1725524397.122149562] [resize_left_viz]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1725524397.136081473] [resize_left_viz]: [ResizeNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1725524397.136513706] [resize_left_viz]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1725524397.152245047] [resize_left_viz]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1725524399.044914598] [camera]: Stopping Sensor: Stereo Module
[component_container_mt-1] [INFO] [1725524399.344173027] [camera]: Starting Sensor: Stereo Module
[component_container_mt-1] [INFO] [1725524399.464196847] [camera]: Open profile: stream_type: Depth(0), Format: Z16, Width: 848, Height: 480, FPS: 30
[component_container_mt-1] [INFO] [1725524399.464502870] [camera]: Stopping Sensor: RGB Camera
[component_container_mt-1]  05/09 01:19:59,465 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,476 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,487 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,498 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,509 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,520 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,530 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,541 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,552 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,563 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,574 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1] [INFO] [1725524399.587672420] [camera]: Starting Sensor: RGB Camera
[component_container_mt-1] [INFO] [1725524399.615530386] [camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 1280, Height: 720, FPS: 30
[component_container_mt-1] [INFO] [1725524399.615742038] [camera]: Stopping Sensor: Motion Module
[component_container_mt-1] [INFO] [1725524399.623044497] [camera]: Starting Sensor: Motion Module
[component_container_mt-1] [INFO] [1725524399.633783412] [camera]: Open profile: stream_type: Accel(0)Format: MOTION_XYZ32F, FPS: 100
[component_container_mt-1] [INFO] [1725524399.633919095] [camera]: Open profile: stream_type: Gyro(0)Format: MOTION_XYZ32F, FPS: 200
[component_container_mt-1] [WARN] [1725524399.636094469] [camera]: 
[component_container_mt-1] [INFO] [1725524399.636563247] [camera]: RealSense Node Is Up!
[component_container_mt-1]  05/09 01:19:59,890 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,900 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,912 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,922 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,933 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/09 01:19:59,944 WARNING [281472187754720] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1] [WARN] [1725524400.346289752] [camera]: XXX Hardware Notification:Motion Module force pause,1.72552e+12,Error,Hardware Error
[component_container_mt-1] [WARN] [1725524400.346465724] [camera]: Hardware Notification:Motion Module force pause,1.72552e+12,Error,Hardware Error
[component_container_mt-1] [INFO] [1725524401.815843009] [foundationpose_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/EBEFIVHXIA/EBEFIVHXIA.yaml"
[component_container_mt-1] [INFO] [1725524401.815990660] [foundationpose_node]: [NitrosNode] Loading application
[component_container_mt-1] 2024-09-05 01:20:01.832 WARN  gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dummy_rx' in component ''.
[component_container_mt-1] 2024-09-05 01:20:01.832 WARN  gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dummy_rx' in component ''.
[component_container_mt-1] 2024-09-05 01:20:01.834 WARN  gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dev_id' in component 'stream'.
[component_container_mt-1] [INFO] [1725524401.837107395] [foundationpose_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] 2024-09-05 01:20:01.854 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] 2024-09-05 01:20:01.854 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] 2024-09-05 01:20:01.854 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] 2024-09-05 01:20:01.854 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] 2024-09-05 01:20:01.856 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] [INFO] [1725524401.867930479] [foundationpose_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1725524401.881938967] [detection2_d_to_mask]: No detection found with non-zero confidence
[component_container_mt-1] 2024-09-05 01:20:02.026 WARN  ./gxf/extensions/tensor_rt/tensor_rt_inference.cpp@155: TRT WARNING: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[component_container_mt-1] [INFO] [1725524402.567335280] [detection2_d_to_mask]: No detection found with non-zero confidence
[component_container_mt-1] 2024-09-05 01:20:02.583 WARN  ./gxf/extensions/tensor_rt/tensor_rt_inference.cpp@155: TRT WARNING: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[component_container_mt-1] [INFO] [1725524403.589012199] [detection2_d_to_mask]: No detection found with non-zero confidence
[component_container_mt-1] [INFO] [1725524404.709579357] [detection2_d_to_mask]: No detection found with non-zero confidence
[component_container_mt-1] [INFO] [1725524405.816409672] [detection2_d_to_mask]: No detection found with non-zero confidence
[component_container_mt-1] [INFO] [1725524406.854557534] [detection2_d_to_mask]: No detection found with non-zero confidence
[component_container_mt-1] [INFO] [1725524407.979675575] [detection2_d_to_mask]: No detection found with non-zero confidence
[component_container_mt-1] [INFO] [1725524409.090029516] [detection2_d_to_mask]: No detection found with non-zero confidence
[component_container_mt-1] [INFO] [1725524410.214247516] [detection2_d_to_mask]: No detection found with non-zero confidence
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[component_container_mt-1] [INFO] [1725524411.165306050] [rclcpp]: signal_handler(signum=2)
[ERROR] [component_container_mt-1]: process[component_container_mt-1] failed to terminate '5' seconds after receiving 'SIGINT', escalating to 'SIGTERM'
[INFO] [component_container_mt-1]: sending signal 'SIGTERM' to process[component_container_mt-1]
[component_container_mt-1] [INFO] [1725524417.122701960] [rclcpp]: signal_handler(signum=15)
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT) again, ignoring...
[component_container_mt-1] [INFO] [1725524418.262883998] [rclcpp]: signal_handler(signum=2)
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT) again, ignoring...
[component_container_mt-1] [INFO] [1725524418.470428965] [rclcpp]: signal_handler(signum=2)
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT) again, ignoring...
[component_container_mt-1] [INFO] [1725524418.638392462] [rclcpp]: signal_handler(signum=2)
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT) again, ignoring...
[component_container_mt-1] [INFO] [1725524418.782556002] [rclcpp]: signal_handler(signum=2)
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT) again, ignoring...
[component_container_mt-1] [INFO] [1725524418.926563315] [rclcpp]: signal_handler(signum=2)
[ERROR] [component_container_mt-1]: process[component_container_mt-1] failed to terminate '10.0' seconds after receiving 'SIGTERM', escalating to 'SIGKILL'
[INFO] [component_container_mt-1]: sending signal 'SIGKILL' to process[component_container_mt-1]
[ERROR] [component_container_mt-1]: process has died [pid 12888, exit code -9, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r __node:=foundationpose -r __ns:=/'].

It keeps looping [component_container_mt-1] [INFO] [1725524410.214247516] [detection2_d_to_mask]: No detection found with non-zero confidence, and I enabled launch_rviz to True in the launch file, but rviz displays image data but no pose information.
Who can help me analyze this problem, I have tried many methods but still failed to solve it, maybe I overlooked some details.Thanks!

[PPZPPZ]

Hello, when I recently ran this script, I replaced my own obj file and encountered the same issue as you. Could you please tell me how you resolved it?

[Yeager-101]

Hello, when I recently ran this script, I replaced my own obj file and encountered the same issue as you. Could you please tell me how you resolved it?

I think you can debug it as I suggested here before, but I will say that although I seemed to have completed the test in the end, the results showed that it was very slow and resource intensive.