RTABMAP shows black screen in 3D map and the visualizer frames are very laggy.

[Shivam7Sharma]

Hi,

I am using a custom stereo setup and driver for two OAK FFC OV9282 M12 cameras with an OAK FFC 3P. I have calibrated the camera using pinhole functions of OpenCV, and I have received reprojection error of 0.17. The problem I am facing is that I get black screen in the 3D map of RTABMAP visualizer with the following messages in the terminal:

[rtabmap-2] [WARN] [1721257512.432158353] [rtabmap.rtabmap]: rtabmap: Did not receive data since 5 seconds! Make sure the input topics are published ("$ ros2 topic hz my_topic") and the timestamps in their header are set. If topics are coming from different computers, make sure the clocks of the computers are synchronized ("ntpdate"). If topics are not published at the same rate, you could increase "sync_queue_size" and/or "topic_queue_size" parameters (current=10 and 1 respectively).
[rtabmap-2] rtabmap subscribed to (approx sync):
[rtabmap-2]    /rtabmap/odom \
[rtabmap-2]    /synced/left/image_rect \
[rtabmap-2]    /synced/right/image_rect \
[rtabmap-2]    /left/camera_info \
[rtabmap-2]    /right/camera_info \
[rtabmap-2]    /rtabmap/odom_info
[stereo_odometry-1] [INFO] [1721257512.570976086] [rtabmap.stereo_odometry]: Odom: quality=380, std dev=0.001542m|0.008712rad, update time=0.271491s
[stereo_odometry-1] [WARN] [1721257512.954000463] [rtabmap.stereo_odometry]: The time difference between left and right frames is high (diff=0.063504s, left=1721257512.375110s, right=1721257512.438614s). If your left and right cameras are hardware synchronized, use approx_sync:=false. Otherwise, you may want to set approx_sync_max_interval lower than 0.01s to reject spurious bad synchronizations.
[stereo_odometry-1] [INFO] [1721257513.272862162] [rtabmap.stereo_odometry]: Odom: quality=351, std dev=0.001901m|0.008712rad, update time=0.316917s

My launch command is :

ros2 launch rtabmap_launch rtabmap.launch.py \
    rtabmap_args:="--delete_db_on_start --Grid/RayTracing true --Vis/EstimateOdom true --Grid/3D false --subscribe_odom_info false --queue_size 30 --approx_sync_max_interval 0.02" \
    stereo:=true \
    frame_id:=base_link \
    wait_imu_to_init:=false \
    left_image_topic:=/synced/left/image_rect \
    right_image_topic:=/synced/right/image_rect \
    left_camera_info_topic:=/left/camera_info \
    right_camera_info_topic:=/right/camera_info \
    approx_sync:=true

Why is RTABMAP not mapping with the stereo setup I have? I will upload a link to the ROS bag in a few hours so that someone can help me debug.

Edit 1:
Link to ROS bag folder

[matlabbe]

Testing visual odometry against a white wall is not the best. In this case VO is loosing tracking and get lost, causing a red screen: https://github.com/introlab/rtabmap/wiki/Kinect-mapping#lost-odometry-red-screens

The major problem is that it seems the left and right images are not synchronized or stereo calibration is poor.

Here is an example where there is a vertical disparity between left and right images:

Here is an example where during motion between two consecutive stereo frames the horizontal disparity varies (meaning the left and right cameras are not synchronized):

Are the two OAK FFC OV9282 M12 cameras trigger a picture with same hardware signal?

[Shivam7Sharma]

Testing visual odometry against a white wall is not the best. In this case VO is loosing tracking and get lost, causing a red screen: https://github.com/introlab/rtabmap/wiki/Kinect-mapping#lost-odometry-red-screens

The major problem is that it seems the left and right images are not synchronized or stereo calibration is poor.

Here is an example where there is a vertical disparity between left and right images:

Here is an example where during motion between two consecutive stereo frames the horizontal disparity varies (meaning the left and right cameras are not synchronized):

Are the two OAK FFC OV9282 M12 cameras trigger a picture with same hardware signal?

Thank you for your reply!

I saw that you were getting some points, but I was not getting any points instead a black screen in 3D map. I am trying to synchronize the left and right frames using software. Is frame dropping also an issue with Rtabmap? I think my calibration is good because the disparity and depth was good. Once I synchronize left and right I will give an update on performance.

[Shivam7Sharma]

Testing visual odometry against a white wall is not the best. In this case VO is loosing tracking and get lost, causing a red screen: https://github.com/introlab/rtabmap/wiki/Kinect-mapping#lost-odometry-red-screens

The major problem is that it seems the left and right images are not synchronized or stereo calibration is poor.

Here is an example where there is a vertical disparity between left and right images:

Here is an example where during motion between two consecutive stereo frames the horizontal disparity varies (meaning the left and right cameras are not synchronized):

Are the two OAK FFC OV9282 M12 cameras trigger a picture with same hardware signal?

Hi,

I added synchronization logic to my Ros 2 driver and edited the QoS settings since that was causing some synchronization issues. I have attached the terminal output and a new robag link with this reply. Can you please help me understand why you were getting points on the first rosbag I sent and I was getting a black screen? Also, I have set the maximum time difference to 0.040 seconds, but still, RTABMAP complained about the 0.031 second time difference. Is this time difference not good enough?
I still get a black screen. The terminal output and link to ROSbag are attached below.

My command to run RTABMAP was :

ros2 launch rtabmap_launch rtabmap.launch.py rtabmap_args:="--delete_db_on_start --Vis/EstimateOdom true --subscribe_odom_info false --queue_size 30" stereo:=true frame_id:=base_link wait_imu_to_init:=false left_image_topic:=/left/image_rect right_image_topic:=/right/image_rect left_camera_info_topic:=/left/camera_info right_camera_info_topic:=/right/camera_info approx_sync_max_interval:=0.04 approx_sync:=true

rtabmap_debug7.txt

Link to google drive ROS Bag

Edit 1:

In this rosbag, I didn't move the camera, but when I moved the camera for real-time testing, I did see a red screen and some scene overlap, which still looks like a synchronization issue. I have tried many things to synchronize. Is there any particular setting in RTABMAP that can help with synchronization?

Current QoS setting is :

rclcpp::QoS qosSetting = rclcpp::QoS(rclcpp::KeepLast(1))
                                 .transient_local() // Durability policy
                                 .reliable(); 

I publish images with the same timestamp in a while loop. Do you think this synchronization issue is with the ROS 2 driver or the RTABMAP setting?

Edit 2:

I changed the header time stamp in frames to the depthai time and now the time difference is constant between left and right. 0.033 seconds. But I keep getting the following messge in the terminal:

[rtabmap_viz-3] [WARN] [1721858171.663666526] [rtabmap.rtabmap_viz]: rtabmap_viz: Did not receive data since 5 seconds! Make sure the input topics are published ("$ ros2 topic hz my_topic") and the timestamps in their header are set. If topics are coming from different computers, make sure the clocks of the computers are synchronized ("ntpdate"). If topics are not published at the same rate, you could increase "sync_queue_size" and/or "topic_queue_size" parameters (current=10 and 1 respectively).

I don't understand how it can not receive a single frame in 5 seconds when the fps is around 25.

[matlabbe]

It seems to work:

[stereo_odometry-1] [INFO] [1722114647.571042491] [rtabmap.stereo_odometry]: Odom: quality=326, std dev=0.001560m|0.000173rad, update time=0.034837s
[stereo_odometry-1] [INFO] [1722114647.612182747] [rtabmap.stereo_odometry]: Odom: quality=293, std dev=0.001731m|0.008712rad, update time=0.040051s
[stereo_odometry-1] [INFO] [1722114647.649321451] [rtabmap.stereo_odometry]: Odom: quality=316, std dev=0.001202m|0.000173rad, update time=0.034762s
[stereo_odometry-1] [INFO] [1722114647.684650695] [rtabmap.stereo_odometry]: Odom: quality=322, std dev=0.001380m|0.000173rad, update time=0.034368s
[stereo_odometry-1] [INFO] [1722114647.723724790] [rtabmap.stereo_odometry]: Odom: quality=334, std dev=0.001277m|0.008712rad, update time=0.037868s
[stereo_odometry-1] [INFO] [1722114647.760536512] [rtabmap.stereo_odometry]: Odom: quality=303, std dev=0.001547m|0.008712rad, update time=0.035568s
[stereo_odometry-1] [INFO] [1722114647.798730269] [rtabmap.stereo_odometry]: Odom: quality=324, std dev=0.001457m|0.008712rad, update time=0.036987s
[rtabmap-2] [INFO] [1722114647.820579185] [rtabmap.rtabmap]: rtabmap (6): Rate=1.00s, Limit=0.000s, Conversion=0.0007s, RTAB-Map=0.0574s, Maps update=0.0001s pub=0.0011s (local map=1, WM=1)
[stereo_odometry-1] [INFO] [1722114647.838038385] [rtabmap.stereo_odometry]: Odom: quality=303, std dev=0.001343m|0.000173rad, update time=0.038168s
[stereo_odometry-1] [INFO] [1722114647.874417770] [rtabmap.stereo_odometry]: Odom: quality=321, std dev=0.001341m|0.000173rad, update time=0.035408s
[stereo_odometry-1] [INFO] [1722114647.911916739] [rtabmap.stereo_odometry]: Odom: quality=309, std dev=0.001532m|0.000173rad, update time=0.034522s
[stereo_odometry-1] [INFO] [1722114647.951691461] [rtabmap.stereo_odometry]: Odom: quality=316, std dev=0.001110m|0.000173rad, update time=0.035869s
[stereo_odometry-1] [INFO] [1722114647.986655126] [rtabmap.stereo_odometry]: Odom: quality=317, std dev=0.001538m|0.010360rad, update time=0.034051s
[stereo_odometry-1] [INFO] [1722114648.024644190] [rtabmap.stereo_odometry]: Odom: quality=320, std dev=0.001351m|0.008712rad, update time=0.034484s
[stereo_odometry-1] [INFO] [1722114648.061335622] [rtabmap.stereo_odometry]: Odom: quality=308, std dev=0.001658m|0.011465rad, update time=0.035638s
[stereo_odometry-1] [INFO] [1722114648.097434390] [rtabmap.stereo_odometry]: Odom: quality=314, std dev=0.001364m|0.000173rad, update time=0.035141s
[stereo_odometry-1] [INFO] [1722114648.133750464] [rtabmap.stereo_odometry]: Odom: quality=333, std dev=0.001238m|0.000173rad, update time=0.035355s
[stereo_odometry-1] [INFO] [1722114648.175154550] [rtabmap.stereo_odometry]: Odom: quality=313, std dev=0.001399m|0.000173rad, update time=0.040162s
[stereo_odometry-1] [INFO] [1722114648.215593757] [rtabmap.stereo_odometry]: Odom: quality=306, std dev=0.001802m|0.000173rad, update time=0.034935s

Without moving, the stereo calibration looks fine. If you didn't change anything from the previous bag, then the time sync is causing a problem when moving. You cannot use rtabmap in stereo mode if the left and right camera shutters are not triggered at the same time (same hardware signal). Normally approx_sync should be false with stereo data, but we accept it when the camera driver cannot stamp with exact same time both images as long as they were still taken at the same time by hardware (e.g., depthai driver). Assuming the conditions above are true, normally the delay between left and right should not be more than 1 ms. This is what we do with depthai example here: approx_sync_max_interval:=0.001.

For the "Did not receive data since 5 seconds!" warning, is it when replaying the bag? You should launch rtabmap.launch.py with use_sim_time:=true and play the bag with --clock option. It is sure if you have sync issue and rtabmap node didn't publish for 5 sec, rtabmap_viz will then complain that it didn't receive data for past 5 sec.

cheers,
Mathieu

[Shivam7Sharma]

It seems to work:

[stereo_odometry-1] [INFO] [1722114647.571042491] [rtabmap.stereo_odometry]: Odom: quality=326, std dev=0.001560m|0.000173rad, update time=0.034837s
[stereo_odometry-1] [INFO] [1722114647.612182747] [rtabmap.stereo_odometry]: Odom: quality=293, std dev=0.001731m|0.008712rad, update time=0.040051s
[stereo_odometry-1] [INFO] [1722114647.649321451] [rtabmap.stereo_odometry]: Odom: quality=316, std dev=0.001202m|0.000173rad, update time=0.034762s
[stereo_odometry-1] [INFO] [1722114647.684650695] [rtabmap.stereo_odometry]: Odom: quality=322, std dev=0.001380m|0.000173rad, update time=0.034368s
[stereo_odometry-1] [INFO] [1722114647.723724790] [rtabmap.stereo_odometry]: Odom: quality=334, std dev=0.001277m|0.008712rad, update time=0.037868s
[stereo_odometry-1] [INFO] [1722114647.760536512] [rtabmap.stereo_odometry]: Odom: quality=303, std dev=0.001547m|0.008712rad, update time=0.035568s
[stereo_odometry-1] [INFO] [1722114647.798730269] [rtabmap.stereo_odometry]: Odom: quality=324, std dev=0.001457m|0.008712rad, update time=0.036987s
[rtabmap-2] [INFO] [1722114647.820579185] [rtabmap.rtabmap]: rtabmap (6): Rate=1.00s, Limit=0.000s, Conversion=0.0007s, RTAB-Map=0.0574s, Maps update=0.0001s pub=0.0011s (local map=1, WM=1)
[stereo_odometry-1] [INFO] [1722114647.838038385] [rtabmap.stereo_odometry]: Odom: quality=303, std dev=0.001343m|0.000173rad, update time=0.038168s
[stereo_odometry-1] [INFO] [1722114647.874417770] [rtabmap.stereo_odometry]: Odom: quality=321, std dev=0.001341m|0.000173rad, update time=0.035408s
[stereo_odometry-1] [INFO] [1722114647.911916739] [rtabmap.stereo_odometry]: Odom: quality=309, std dev=0.001532m|0.000173rad, update time=0.034522s
[stereo_odometry-1] [INFO] [1722114647.951691461] [rtabmap.stereo_odometry]: Odom: quality=316, std dev=0.001110m|0.000173rad, update time=0.035869s
[stereo_odometry-1] [INFO] [1722114647.986655126] [rtabmap.stereo_odometry]: Odom: quality=317, std dev=0.001538m|0.010360rad, update time=0.034051s
[stereo_odometry-1] [INFO] [1722114648.024644190] [rtabmap.stereo_odometry]: Odom: quality=320, std dev=0.001351m|0.008712rad, update time=0.034484s
[stereo_odometry-1] [INFO] [1722114648.061335622] [rtabmap.stereo_odometry]: Odom: quality=308, std dev=0.001658m|0.011465rad, update time=0.035638s
[stereo_odometry-1] [INFO] [1722114648.097434390] [rtabmap.stereo_odometry]: Odom: quality=314, std dev=0.001364m|0.000173rad, update time=0.035141s
[stereo_odometry-1] [INFO] [1722114648.133750464] [rtabmap.stereo_odometry]: Odom: quality=333, std dev=0.001238m|0.000173rad, update time=0.035355s
[stereo_odometry-1] [INFO] [1722114648.175154550] [rtabmap.stereo_odometry]: Odom: quality=313, std dev=0.001399m|0.000173rad, update time=0.040162s
[stereo_odometry-1] [INFO] [1722114648.215593757] [rtabmap.stereo_odometry]: Odom: quality=306, std dev=0.001802m|0.000173rad, update time=0.034935s

Without moving, the stereo calibration looks fine. If you didn't change anything from the previous bag, then the time sync is causing a problem when moving. You cannot use rtabmap in stereo mode if the left and right camera shutters are not triggered at the same time (same hardware signal). Normally approx_sync should be false with stereo data, but we accept it when the camera driver cannot stamp with exact same time both images as long as they were still taken at the same time by hardware (e.g., depthai driver). Assuming the conditions above are true, normally the delay between left and right should not be more than 1 ms. This is what we do with depthai example here: approx_sync_max_interval:=0.001.

For the "Did not receive data since 5 seconds!" warning, is it when replaying the bag? You should launch rtabmap.launch.py with use_sim_time:=true and play the bag with --clock option. It is sure if you have sync issue and rtabmap node didn't publish for 5 sec, rtabmap_viz will then complain that it didn't receive data for past 5 sec.

cheers, Mathieu

Thank you for your reply and for supporting this software. Getting this work done is part of my full-time job, so I really appreciate your help.

I have tested it in real time with the custom stereo setup I have. I am using my own ROS 2 driver using the depthai library for the mono oak cameras. So, it is a steep learning curve on how to synchronize the left and right images with software. I will check the rosbags with the flags you suggested and edit my reply. I am using Jetson Orin Nano; does that affect the behaviour of RTABMAP? In real time also, I get a black screen in the 3D visualization in RTABMAP, and in RVIZ, only the local map and last frame point clouds are being published. I am also changing some things in my ROS 2 driver to get better sync. I will update on Monday.

To answer your question, yes, I got the 5-second warning when I ran the first rosbag with RTABMAP.

@matlabbe Update:

I checked the second rosbag on my laptop, Ubuntu 22.04, and it was working, but on the Jetson Orin Nano with Jetpack 6 and ROS 2 humble, it is having some issues, i.e., a black screen and no 3D map. I also checked the Docker container in Jetson, but still had the same issue. What might the cause and solution be here? I installed the RTABMAP ROS 2 binaries in Jetson Orin nano. I will try to install it from source and see if the same issue persists.

Edit 1: When playing the same rosbag with the same command and with the use_sim_time and clock options, I get no images and a black screen in the visualizer in Jetson Orin. Also, rviz is only showing two point clouds: the local map and the last frame point cloud.

Edit 2: I installed RTABMAP from source, but it still didn't work on the same rosbag file. Same issue as before. How do I solve this problem?

I have installed Opencv 4.10 from source, but I don't know if rtabmap is getting linked to ROS 2 4.5 Opencv or the 4.10 version.

Edit 3: Same problem in the RTABMAP ROS 2 humble Docker container when playing Rosbag; no output in 3d map.

Edit 4: So in Docker, I see some point clouds in Rviz but not all. I see a map and odometry in rviz. The rtabmap visualization is not working in docker either. How do I get dense point clouds like in the rtabmap visualizer in Rviz? I think this has been a known issue since 2019, but I didn't know this was an issue with the latest jetpacks.

Edit 5: I am not getting dense point cloud on cloud_map even after setting the following parameters:
Grid/3D:=true Grid/CellSize:=0.05 Grid/VoxelSize:=0.01 Mem/DepthAsMask:=false Rtabmap/DetectionRate:=1 Rtabmap/CreateOccupancyGrid:=true

Edit 6: I am using sim time = true and clock option when I play Rosbag, but I get the following messages on RVIZ; let me know if these messages are affecting the map visualization or not. I don't know how to get rid of them.:

[INFO] [1722374780.211379479] [rviz]: Message Filter dropping message: frame 'odom' at time 1722372529.712 for reason 'discarding message because the queue is full'
[INFO] [1722374780.351191112] [rviz]: Message Filter dropping message: frame 'odom' at time 1722372529.745 for reason 'discarding message because the queue is full'

[INFO] [1722374791.391020810] [rviz]: Message Filter dropping message: frame 'odom' at time 1722372532.879 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[INFO] [1722374791.391152303] [rviz]: Message Filter dropping message: frame 'odom' at time 1722372532.879 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[INFO] [1722374791.400077773] [rviz]: Message Filter dropping message: frame 'base_link' at time 1722372530.612 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[INFO] [1722374791.400286612] [rviz]: Message Filter dropping message: frame 'base_link' at time 1722372530.612 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[INFO] [1722374791.760949219] [rviz2]: Trying to create a map of size 70 x 54 using 1 swatches
[INFO] [1722374792.314906130] [rviz]: Message Filter dropping message: frame 'base_link' at time 1722372531.679 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[INFO] [1722374792.315202172] [rviz]: Message Filter dropping message: frame 'base_link' at time 1722372531.679 for reason 'the timestamp on the message is earlier than all the data in the transform cache'

Edit 7: I recorded a new rosbag with motion and tested it on Jetson Orin, and the following is the performance:

Untitled.video.-.Made.with.Clipchamp.1.mp4

Edit 8:

Is it common to see the following messages sometimes?:

[rtabmap-2] [ WARN] (2024-07-31 00:54:59.805) Features2d.cpp:898::generateKeypoints3D() A large number (497/875) of stereo correspondences are rejected! Optical flow may have failed because images are not calibrated, the background is too far (no disparity between the images), maximum disparity may be too small (128.000000) or that exposure between left and right images is too different.
Does this mean the calibration is bad?

Edit 9:

So I got a denser point cloud in Rviz on Jetson without Docker by using rviz:= true. I am still fixing the synchronization, so I am not expecting a great result without that.

Edit 10:
I saw one video on YouTube where rtabmap was adding walls to the point cloud, even though the walls were featureless. Is this possible with stereo? In the video I posted above, it ignores the featureless walls. Will this improve with synchronization?