Point cloud density of Livox lidar Horizon as a function of integration time, the pattern does not repeat itself.
We propose CamVox by adapting Livox lidars into visual SLAM (ORB-SLAM2) by exploring the lidars’ unique features. Based on the non-repeating nature of Livox lidars, we propose an automatic lidar-camera calibration method that will work in uncontrolled scenes.
The long depth detection range also benefit a more efficient mapping. Comparison of CamVox with visual SLAM (VINS-mono) and lidar SLAM (livox_horizon_loam) are evaluated on the same dataset to demonstrate the performance.
Developer: Yuewen Zhu, Chunran Zheng, Chongjian Yuan, Xu Huang.
Our related video: our related videos are now available on [YouTube Video] [bilibili Video] [OwnCloud]
Paper: our related paper has been posted on arXiv, and final ICRA2021 accepted version can be available CamVox.pdf.
Lidar-Camera Automatic calibration follow-up work: "Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments".
Ubuntu 64-bit 16.04 or 18.04.
ROS Kinetic or Melodic. Follow ROS Installation.
(Recommended version Ubunutu 16.04 LTS kernel version 4.15.0-140-generic)
We use OpenCV to manipulate images and features. Follow Opencv Installation. Required at leat 2.4.3. Tested with OpenCV 2.4.11 and OpenCV 3.4.1.
(1) *** Install dependencies: ***
sudo apt-get install build-essential libgtk2.0-dev libavcodec-dev libavformat-dev libjpeg.dev libtiff4.dev libswscale-dev libjasper-dev
(2) *** Install opencv-3.4.1: ***
cd opencv-3.4.1
mkdir build && cd build
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j
sudo make install
(4) *** Add opencv libraries to path: ***
sudo gedit /etc/ld.so.conf.d/opencv.conf
*** add in the file's end: ***
/usr/local/lib
sudo ldconfig
(5) *** bash configurtion: ***
sudo gedit /etc/bash.bashrc
*** add in the end: ***
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig
export PKG_CONFIG_PATH
source configurtion:
source /etc/bash.bashrc
sudo updatedb(Recommended version OpenCV-3.4.1.zip)
We use PCL to deal with point cloud features and Lidar-camera extrinsic parameters calibration. PCL Installation version recommended as follows.
(1) *** update your host: ***
sudo apt-get update
(2) *** install VTK ***
cd VTK-8.2.0
mkdir build && cd build
cmake ..
make -j
sudo make install
(3) *** install pcl: ***
sudo apt-get install libpcl-dev pcl-tools
sudo apt-get install freeglut3-dev(Recommended VTK version VTK-8.2.0.tar.gz)
We use Pangolin for visualization and user interface. Follow Pangolin Installation.
(1) *** install dependencies: ***
sudo apt-get install libglew-dev libpython2.7-dev libboost-dev libboost-thread-dev libboost-filesystem-dev -y
(3) *** install Pangolin: ***
cd Pangolin
mkdir build && cd build
cmake ..
make -j
sudo make install(Recommended version Pangolin.zip)
Follow Ceres Installation.
(1) *** install dependencies: ***
sudo apt-get install liblapack-dev
sudo apt-get install libsuitesparse-dev
sudo apt-get install libcxsparse3.1.4
sudo apt-get install libgflags-dev
sudo apt-get install libgoogle-glog-dev libgtest-dev
(2) *** install ceres: ***
cd ceres-solver-1.14.0
mkdir build && cd build
cmake ..
make -j
sudo make install(Recommended version Ceres-solver-1.14.0.tar.gz)
Follow Eigen Installation. Required at least 3.1.0.
(1) *** install eigen: ***
cd eigen
mkdir build && cd build
cmake ..
make -j
sudo make install(Recommended version Eigen-3.2.10.tar.gz)
Follow Livox-SDK Installation. (Recommended version Livox-SDK-master.zip)
Install the HIKROBOT camera SDK as follows.
tar zxvf MVS-2.0.0_x86_64_20191126.tar.gz
cd ./MVS-2.0.0_x86_64_20191126
chmod +x setup.sh
sudo ./setup.shIn addition, we supply a (software trigger Hikvisions' compatible Ros drivers), you can run it directly just use USB to connect the camera.
roslaunch mvs_camera mvs_camera.launch(Recommended version MVS-2.0.0.tar.gz)
Clone the repository and catkin_make:
cd ~/catkin_ws/src
git clone https://github.com/ISEE-Technology/CamVox.git
cd CamVox/isee-camvox && chmod a+x build.sh && chmod a+x build_ros.sh
./build.sh
./build_ros.sh
source ~/catkin_ws/devel/setup.bash
cd CamVox/isee-camvox/Vocabulary
tar zxvf ORBvoc.txt.tar.gz| Platform | Item | Pics | Shopping Link |
|---|---|---|---|
 |
Livox Horizon |  |
Lidar |
| MV-CE060-10UC |  |
Camera | |
| Inertial Sense uINS |  |
RTK | |
| Manifold2C |  |
Onboard-Computer | |
| Scout-mini |  |
Robot Chassis |
Hard synchronization is performed with all of these sensors by a trigger signal of 10 Hz. The camera output at each trigger signal(10 Hz). The lidar keeps a clock (synced with GPS-RTK) and continuously outputs the scanned point with an accurate timestamp. In the meantime, the IMU outputs at a frequency of 200 Hz synced with the trigger. The Hardware Synchronization diagram is as follows.
Connect to your PC to Livox Horizon lidar by following Livox-ros-driver installation.
cd ~/catkin_ws/src/CamVox/
chmod +x run.sh
./run.sh
We open sourced our dataset in SUSTech campus with loop closure. You can download the bag file from google drive CamVox.bag or zenodo CamVox.bag. (Updated)
Other two main framework data formats for comparison. VINS-mono.bag | livox_loam_horizon.bag | Groundtruth.bag (Updated)
We provide a rosbag file with static scenes to test the automatic calibration thread. calibration.bag. (Updated) When the car detects more than 10 frames of still images (about 1 second), the automatic calibration thread starts to work. The thread will be interrupted to enter the SLAM mode if the car starts to move before the end of calibration. The effects of automatic calibration is shown as follows.
An example of RGB camera and point cloud overlay after calibration. (a) not calibrated. (b) automatically calibrated. (c) the best manual calibration. The automatic calibration algorithms is verified at various scenes, (d) outdoor road with natural trees and grasses, (e) outdoor artificial structures, (f) indoor underexposed structures. (g-i) represent the cost value evolution in the optimization process corresponding to the scenes on the left.
roscore
cd CamVox/isee-camvox
rosrun online camvox Vocabulary/ORBvoc.bin camvox/online/Livox.yaml
rosbag play CamVox.bag (or calibration.bag) roscore
cd CamVox/isee-camvox
rosrun online camvox Vocabulary/ORBvoc.bin camvox/online/Livox.yaml 1000
rosbag play CamVox.bag (or calibration.bag)The authors are grateful for the pioneering work from ORB_SLAM2, ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras. The authors also sincerely thank colleagues at Livox Technology for help in data aquisition and discussion. This work is from ISEE Research Group at SUSTech.
The source code is released under GPLv2.0 license.
If you use CamVox in an academic work, please cite:
(**Updated**) @misc{zhu2020camvox,
title={CamVox: A Low-cost and Accurate Lidar-assisted Visual SLAM System},
author={Yuewen Zhu and Chunran Zheng and Chongjian Yuan and Xu Huang and Xiaoping Hong},
year={2020},
eprint={2011.11357},
archivePrefix={arXiv},
primaryClass={cs.RO}
}