• [2024-7-8] We have released the arXiv paper of OSP.
• [2024-7-2] OSP is accepted by ECCV 2024!

In this paper, we explore a novel point representation for 3D occupancy prediction from multi-view images, which is named Occupancy as Set of Points. Existing camera-based methods tend to exploit dense volume-based representation to predict the occupancy of the whole scene, making it hard to focus on the special areas or areas out of the perception range. In comparison, we present the \textit{Points of Interest} (PoIs) to represent the scene and propose OSP, a novel framework for point-based 3D occupancy prediction. Owing to the inherent flexibility of the point-based representation, OSP achieves strong performance compared with existing methods and excels in terms of training and inference adaptability. It extends beyond traditional perception boundaries and can be seamlessly integrated with volume-based methods to significantly enhance their effectiveness. Experiments on the Occ3D-nuScenes occupancy benchmark show that OSP has strong performance and flexibility.

1. Prepare conda environment referring to the documentation of BEVFormer

1. Download nuScenes and prepare annotations referring to the documentation of 3D Occupancy Prediction Challenge at CVPR 2023

The pretrained weight of fcos3d can be downloaded here

1. Training

   bash train.sh

   • Replace the default config file as needed.
   • Config osp_minibatch.py represents mini dataset of nuScenes.

2. Evaluation

   bash test.sh

   • Replace the default config file as needed.
   • Replace the checkpoint path in the script with your own.

• Model weights will be released later.

@inproceedings{shi2024occupancysetpoints,
      title={Occupancy as Set of Points}, 
      author={Yiang Shi and Tianheng Cheng and Qian Zhang and Wenyu Liu and Xinggang Wang},
      year={2024},
      booktitle={ECCV}
}

Released under the MIT License.