SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net

Helin Cao and Sven Behnke
University of Bonn and Lamarr Institute, Bonn, Germany
ICRA 2024
Arxiv | Project page

Teaser

Table of Content

• Preparing
  • Setup
  • Datasets
• Running
  • Training
  • Evaluation
  • Inference
  • Visualization
• Citation
• License and Acknowledgement

Preparing

Setup

We recommend you to use anaconda to manage dependencies. You may need to change the torch and cuda version according to your computer.

1. Create conda environment:

conda create --name slcf python=3.7
conda activate slcf

2. Please install PyTorch:

conda install pytorch=1.13.0 torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

3. Install the additional dependencies:

cd SLCF-Net/
pip install -r requirements.txt

4. We use dvis for visualization, which is a lightweight but efficient tool with a web server. We recommend you to use another conda environment to visualize the result.

conda create --name dvis python=3.8 requests matplotlib pyyaml tqdm imageio
conda activate dvis
pip install visdom
git clone [email protected]:SirWyver/dvis.git
cd dvis
pip install .

Datasets

1. You need to download

• The Semantic Scene Completion dataset v1.1 (SemanticKITTI voxel data (700 MB)) from SemanticKITTI website
• The KITTI Odometry Benchmark calibration data (Download odometry data set (calibration files, 1 MB)) and the RGB images (Download odometry data set (color, 65 GB)) from KITTI Odometry website.
• The dataset folder at /path/to/semantic_kitti should have the following structure:

  └── /path/to/semantic_kitti/
    └── dataset
      └── sequences
        ├── 00
        | ├── image_2
        | ├── labels
        | ├── velodyne
        | ├── voxels
        | ├── calib.txt
        | ├── poses.txt
        | └── times.txt
        ├── 01
        ...
  

2. Create a folder to store SemanticKITTI preprocess data at /path/to/kitti/preprocess/folder.

3. Store paths in environment variables for faster access (Note: folder 'dataset' is in /path/to/semantic_kitti):

$ export KITTI_PREPROCESS=/path/to/kitti/preprocess/folder
$ export KITTI_ROOT=/path/to/semantic_kitti 

Running

All the scripts is controlled by the /SLCF-Net/slcfnet/config/slcfnet.yaml

Preprocess

Before starting the training process, some preprocess is necessary

python preprocess.py

Training

python train.py

Evaluation

Put the checkpoints in the /path/to/kitti/logdir/trained_models/kitti.ckpt, then run:

python evaluation.py

Inference

Put the checkpoints in the /path/to/kitti/logdir/trained_models/kitti.ckpt, then run:

python generate_output.py

Visualization

Please follow the guide of dvis, you need to setup the server before running the script.

python visualization.py

Citation

If you find this work or code useful, please cite our paper and give this repo a star :)

@inproceedings{cao2024slcf,
  title = {{SLCF-Net}: Sequential {LiDAR}-Camera Fusion for Semantic Scene Completion using a {3D} Recurrent {U-Net}},
  author = {Cao, Helin and Behnke, Sven},
  booktitle = {IEEE Int. Conf. on Robotics and Automation (ICRA)},
  pages = {2767--2773},
  year = {2024},
}

License and Acknowledgement

SLCF-Net is released under the MIT license. Our code follows several awesome repositories. We appreciate them for making their codes available to public.

• MonoScene
• DVIS
• LMSCNet
• SemanticKitti