Empty pointcloud data when accelerate_gpu_with_glsl = True #3206
Comments
|
Hi @edgarcamilocamacho In rare cases on some computers, a pointcloud will disappear when GLSL is enabled. There is no way to predict in advance which computers will be unable to use GLSL, unfortunately. If you are using a Jetson then the usual way to accelerate a pointcloud will be to enable the librealsense SDK's CUDA support. This can be done by installing the Jetson version of the librealsense packages on the installation_jetson.md instruction page or by building from source code with the flag -DBUILD_WITH_CUDA=ON included in the CMake build instruction. |
|
So if I install
I should get both of them running with GPU, right? There is a way to identify that it's actually using the GPU? |
|
Following that idea I this in my Jetson container (the sumarized steps from the links I sent before): # apt-key adv --keyserver keyserver.ubuntu.com --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key # adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE
# add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo $(lsb_release -cs) main" -u
# apt-get install -y librealsense2-utils librealsense2-dev ros-humble-realsense2-*After that I get these versions installed: I got I could install |
|
Wait... seeing it closely, it installed
|
|
I am not involved in the compilation and publishing of packages, so do not have advice to provide about their contents. As a general rule, assume that if installation is not being performed with the Jetson packages on the |
|
Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please Thanks! |
|
These are the steps I'm following in my dockerfile (in the Jetson). I took the steps from the scripts you told me. # Install librealsense
RUN cd /tmp && \
wget https://github.com/IntelRealSense/librealsense/archive/refs/tags/v2.56.1.tar.gz && \
tar -xzf v2.56.1.tar.gz && \
cd librealsense-2.56.1 && \
mkdir build && \
cd build && \
cmake .. \
-DBUILD_PYTHON_BINDINGS:bool=true \
-DPYTHON_EXECUTABLE=/usr/bin/python3 \
-DBUILD_EXAMPLES=true \
-DCMAKE_BUILD_TYPE=release \
-DFORCE_LIBUVC=true \
-DFORCE_RSUSB_BACKEND=false \
-DBUILD_WITH_CUDA:bool=true \
&& \
make -j 10 && \
make install && \
cd ../.. && \
rm -r v2.56.1.tar.gz librealsense-2.56.1/
# Install realsense-ros
RUN source /setup_env.bash && \
mkdir -p /robot/additional_ros_packages/src && \
cd /robot/additional_ros_packages/src && \
git clone -b 4.56.1 https://github.com/IntelRealSense/realsense-ros.git && \
cd /robot/additional_ros_packages && \
colcon build --merge-install --cmake-args '-DBUILD_ACCELERATE_GPU_WITH_GLSL=ON' && \
rm -rf src build logAnd I still have the same problem. I attach the complete dockerfile. In addition this is the docker-compose configuration: cameras:
# common
network_mode: host
pid: host
privileged: true
runtime: nvidia
restart: unless-stopped
# specific
image: gary.core.cameras.cameras:jetsonorin.4.3.beta
environment:
# display
- XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR}
- DISPLAY=:0
env_file:
# common
- ./config/env/env
volumes:
# common
- ./config:/robot/config
- ./generic_persistent_data:/robot/generic_persistent_data
- ./component_persistent_data/cameras:/robot/component_persistent_data
# devices
- /dev:/dev
- /run/udev:/run/udev:ro
# display
- ${XDG_RUNTIME_DIR}/gdm/Xauthority:/root/.Xauthority
- /tmp/.X11-unix:/tmp/.X11-unix:rw |
|
My knowledge of Docker is admittedly limited, so there is not much that I can add to the previous advice, unfortunately. I would recommend removing the -DBUILD_ACCELERATE_GPU_WITH_GLSL=ON pointcloud acceleration parameter, since the pointcloud will already be being accelerated by -DBUILD_WITH_CUDA:bool=true when this Dockerfile is used on Jetson. Havibg said that, there have been a small number of past cases where problems disappear when not setting -DBUILD_WITH_CUDA to true. So you could try it either with -DBUILD_ACCELERATE_GPU_WITH_GLSL=ON or with DBUILD_WITH_CUDA:bool=true but not have both set to true at the same time. |
|
Hi @edgarcamilocamacho Do you require further assistance with this case, please? Thanks! |
|
Hello @MartyG-RealSense , thanks for the fast response and sorry for the slow testing, we have several things on the table, but for sure next week we'll build the container again and test. |
|
It's no problem at all. I look forward to your next report. Good luck! |
|
Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please? Thanks! |
|
Hi @MartyG-RealSense,
I've the following findings (Using Setup 1 hardware):
Long story: Scenario1:
sudo apt-get install ros-humble-realsense2-camera ros-humble-realsense2-camera-msgs ros-humble-realsense2-description
ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info
lsof -p $(pgrep -f /opt/ros/humble/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.so
realsense 76448 zzzzz mem REG 259,2 28264 807521 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so realsense 76448 zzzzz mem REG 259,2 27392 807519 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so realsense 76448 zzzzz mem REG 259,2 6581504 807412 /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 realsense 76448 zzzzz mem REG 259,2 1151296 813087 /opt/ros/humble/lib/librealsense2_camera.so realsense 76448 zzzzz mem REG 259,2 14344 807517 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so Scenario2:
sudo apt-get install librealsense2-utils librealsense2-dev librealsense2-gl-dev Note2: I've not patched the kernel
git clone https://github.com/IntelRealSense/realsense-ros.git cd realsense-ros colcon build --cmake-args -DBUILD_ACCELERATE_GPU_WITH_GLSL=1 #overlay this build on top of humble release source $(pwd)/install/setup.bash
ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info
lsof -p $(pgrep -f $(pwd)/install/realsense2_camera/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.s Output: realsense 80059 zzzzz mem REG 259,2 79448 138474 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so realsense 80059 zzzzz mem REG 259,2 60784 138463 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so realsense 80059 zzzzz mem REG 259,2 6581504 807412 /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 realsense 80059 zzzzz mem REG 259,2 1450648 3030330 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1 realsense 80059 zzzzz mem REG 259,2 2318512 138826 /home/zzzzz/realsense-ros/install/realsense2_camera/lib/librealsense2_camera.so realsense 80059 zzzzz mem REG 259,2 24904 138475 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so Note the following: /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1 When I run the node with accelerate_gpu_with_glsl:=True point cloud is not generated. Otherwise it is generated without problems because ros version of librealsense2 is running as expected. Scenario 3:
sudo apt-get remove ros-humble-realsense2-camera ros-humble-realsense2-camera-msgs ros-humble-realsense2-description At this point the following should return empty results: ls /opt/ros/humble/lib/aarch64-linux-gnu/librealsense* ls /opt/ros/humble/lib/librealsense*
ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info IMPORTANT: Now we don't get point cloud data in all cases. (So there must be a problem with the libraries built by librealsense, ros version is working, lets deep dive in Scenario4)
lsof -p $(pgrep -f $(pwd)/install/realsense2_camera/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.s Output: realsense 80962 zzzzz mem REG 259,2 79448 138474 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so realsense 80962 zzzzz mem REG 259,2 60784 138463 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so realsense 80962 zzzzz mem REG 259,2 7855232 3015185 /usr/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 realsense 80962 zzzzz mem REG 259,2 1450648 3030330 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1 realsense 80962 zzzzz mem REG 259,2 2318512 138826 /home/zzzzz/realsense-ros/install/realsense2_camera/lib/librealsense2_camera.so realsense 80962 zzzzz mem REG 259,2 24904 138475 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so As expected, this time, both libraries are linked correctly and against the ones built by librealsense /usr/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1 Scenario4:
sudo apt-get remove librealsense2-dev librealsense2-gl-dev librealsense2 librealsense2-gl librealsense2-utils librealsense2-udev-rules #Dependencies are already installed, skipping... cd ~ git clone https://github.com/IntelRealSense/librealsense.git cd librealsense #Switch to v2.55.1 tag git checkout v2.55.1 #Install USB udev rules sudo ./scripts/setup_udev_rules.sh #Skipping kernel patch we'll be using FORCE_RSUSB_BACKEND mkdir build cd build cmake ../ -DBUILD_EXAMPLES=false -DBUILD_GRAPHICAL_EXAMPLES=false -DBUILD_GLSL_EXTENSIONS=ON -DFORCE_RSUSB_BACKEND=true -DCMAKE_BUILD_TYPE=Release sudo make uninstall make clean make sudo make install
cd ~/realsense-ros colcon clean workspace -y colcon build #overlay this build on top of humble release source $(pwd)/install/setup.bash
ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info Point cloud generation works, however on the ros logs I see the following warning: (ros/realsense builds do not print this warning) WARNING (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0 [INFO] [1728932468.838272790] [camera_d435_front]: RealSense Node Is Up! WARNING (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0 |
|
Update: cmake ../ -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true -DBUILD_GLSL_EXTENSIONS=ON -DFORCE_RSUSB_BACKEND=true -DCMAKE_BUILD_TYPE=Release -DBUILD_TOOLS=true We're back to square 1. Point cloud can now be generated with CPU but not with GPU. |
|
control_transfer returned warnings can be safely disregarded if the pointcloud appears to be performing normally. |
…t provided. See: IntelRealSense/realsense-ros#3206 When called through realsense-ros package, depth mapping texture is not provided, thus point cloud is not generated. A new shader is generated to handle the case.
|
I've found the reason and implemented a solution, I'll test and let you know about the results. The problem occurs in "librealsense" library if point cloud generation is opengl accelerated and there's no depth_texture provided. depth_to_points function returns empty point data as this operation is deferred to get_texture_map function. However, since there's no depth texture provided, get_texture_map is never called and we get empty point cloud. I've created another shader which just does point cloud extraction but not depth texture mapping and executed it within depth_to_points function if texture map is not available. (See also resdocta/librealsense@81b78a1 for details) |
|
Thanks so much for sharing the details of your solution. I look forward to hearing about your test results. :) |
…t provided. See: IntelRealSense/realsense-ros#3206 When called through realsense-ros package, depth mapping texture is not provided, thus point cloud is not generated. A new shader is generated to handle the case.
|
I've tested it successfully on OrangePI 5, GPU utilization is around 9% and CPU utilization is around 7% with depth camera running at 848x480@90FPS. I should note that CPU utilization increases and frame rate drops to around 20FPS when subscribed to the ros2 topic (possibly due to message copying). But this is out of scope of this conversation. Pull request to the development branch: |
|
Thanks again for creating the PR! :) |
…t provided. See: IntelRealSense/realsense-ros#3206 When called through realsense-ros package, depth mapping texture is not provided, thus point cloud is not generated. A new shader is generated to handle the case.
Hello
Issue Description
When I set
accelerate_gpu_with_glsl = True(at the begining of the execution, not dinamically), I get empty data pointcloud like:But with
accelerate_gpu_with_glsl = Falseit works perfectly.Maybe I'm setting something wrong, I attach the list of ros params:
rs-params.txt
The text was updated successfully, but these errors were encountered: