Temporal-Consistent-RGBT-Segmentation

The official implementation of Temporal Consistency for RGB-Thermal Data-based Semantic Scene Understanding. (IEEE RA-L).

Introduction

We propose a temporally consistent framework for RGB-T semantic segmentation, which includes a method to synthesize images for the next moment and loss functions to ensure segmentation consistency across different frames.

Dataset

Download MF dataset or Our preprocessed version and place them in 'datasets' folder in the following structure:

<datasets>
|-- <MFdataset>
    |-- <RGB>
    |-- <Thermal>
    |-- <Label>
    |-- train.txt
    |-- val.txt
    |-- test.txt

Pretrained weights

Download the pretrained segformer here pretrained segformer and place them in 'pretrained' folder in the following structure:

<pretrained>
|-- <mit_b0.pth>
|-- <mit_b1.pth>
|-- <mit_b2.pth>
|-- <mit_b3.pth>
|-- <mit_b4.pth>
|-- <mit_b5.pth>

Usage

• Clone this repo

$ git clone https://github.com/lab-sun/Temporal-Consistent-RGBT-Segmentation.git

• Build docker image

$ cd ~/Temporal-Consistent-RGBT-Segmentation
$ docker build -t docker_image_tcfusenet .

• Download the dataset

$ (You should be in the Temporal-Consistent-RGBT-Segmentation folder)
$ mkdir ./datasets
$ cd ./datasets
$ (download our preprocessed MFNet.zip in this folder)
$ unzip -d .. MFNet.zip

• To reproduce our results, you need to download our pretrained weights

$ (You should be in the Temporal-Consistent-RGBT-Segmentation folder)
$ mkdir ./pretrained
$ cd ./pretrained
$ (download the pretrained segformer weights in this folder)
$ mkdir ./TCFuseNet_pth
$ cd ./TCFuseNet_pth
$ (download our pretrained CMX weights in this folder)
$ docker run -it --shm-size 8G -p 1234:6006 --name docker_container_tcfusenet --gpus all -v ~/Temporal-Consistent-RGBT-Segmentation:/workspace docker_image_tcfusenet
$ cd /workspace
$ python3 eval.py

• To train our method

$ (You should be in the Temporal-Consistent-RGBT-Segmentation folder)
$ mkdir ./pretrained
$ cd ./pretrained
$ (download the pretrained segformer weights in this folder)
$ docker run -it --shm-size 8G -p 1234:6006 --name docker_container_tcfusenet --gpus all -v ~/Temporal-Consistent-RGBT-Segmentation:/workspace docker_image_tcfusenet
$ (currently, you should be in the docker)
$ cd /workspace
$ python3 train.py

• To see the training process

$ (fire up another terminal)
$ docker exec -it docker_container_tcfusenet /bin/bash
$ cd /workspace
$ tensorboard --bind_all --logdir=./CMX_mit_b2/tensorboard_log/
$ (fire up your favorite browser with http://localhost:1234, you will see the tensorboard)

The results will be saved in the ./CMX_mit_b2 folder.

Results

We offer the pre-trained weights of our method modified based on CMX and RTFNet.

CMX

Architecture	Backbone	mIOU	Weight
Our-dice	MiT-B2	60.8%	CMX-B2-dice
Our-con	MiT-B2	59.9%	CMX-B2-con
Our-con-acc	MiT-B2	60.0%	CMX-B2-con-acc

RTFNet

Architecture	Backbone	mIOU	Weight
Our-con	ResNet-50	53.3%	RTF-50-con
Our-con	ResNet-152	54.1%	RTF-152-con

Citation

If you use our work in your research, please cite:

    @article{li2024temporal,
      title={Temporal Consistency for RGB-Thermal Data-based Semantic Scene Understanding},
      author={Li, Haotian and Chu, Henry K and Sun, Yuxiang},
      journal={IEEE Robotics and Automation Letters},
      year={2024},
      publisher={IEEE}
    }

Acknowledgement

Some of the codes are borrowed from CMX.