Skip to content

[NeurIPS 2022] "Signal Processing for Implicit Neural Representations" by Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang Wang

License

Notifications You must be signed in to change notification settings

VITA-Group/INSP

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

24 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Signal Processing for Implicit Neural Representations

License: MIT

The official implementation of NeurIPS 2022 paper "Signal Processing for Implicit Neural Representations".

Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang (Atlas) Wang

[Paper] [Website]

Method Overview

Environment

You can then set up a conda environment with all dependencies like so:

conda env create -f environment.yml
conda activate INSP

High-Level structure

  • Fit INR
  • Export gradients for INR
  • Train INSP-Net
  • Inference INSP-Net

Image Processing

For image processing, we experiment on div2k dataset.

wget http://data.vision.ee.ethz.ch/cvl/DIV2K/DIV2K_train_HR.zip &
unzip DIV2K_train_HR.zip
  • Fit multiple INR

    Use --type to specify the type of images you want to train on.

    python gen_div2k.py | zsh

  • Export gradients for INR

    --load is used for glob to filter out corresponding INRs.

    python export_colorray.py --save_dir grad/train_color_noise/ --load 'div2k*.png_color_noise_'
    

    Then, manually divide grad/train_color_noise and put a few of them into grad/test_color_noise (in our case we used first 100 images in DIV2K for training and the following 100 images for testing)

  • Train INSP-Net

    --img_num changes the number of images that are used for training.

    The training should converge in a couple of minutes.

    python experiment_scripts/train_img_grad_offline.py --model_type=sine --experiment_name denoise --noise_level 0 --target denoise --img_num 100 --overwrite --sigma 7 --sz 256 --batch_size 10240 --lr 1e-4
    
  • Inference INSP-Net

    python eval_insp.py --save_path output/denoise --target denoise --ckpt_path logs/denoise/checkpoints/model_current.pth
    

The INRs used in our experiments can be found here.

Audio Denoising

  • Fit INR

    python experiment_scripts/train_audio.py --model_type=sine --wav_path=data/gt_bach.wav --experiment_name audio_noisy_10
    
  • Export gradients for INR

    python export_audio.py
    
  • Train INSP-Net

    python experiment_scripts/train_audio_insp.py --experiment_name audio_denoise --batch_size 10240
    
  • Inference INSP-Net

    python eval_audio_insp.py
    

SDF Smoothing

  • Fit INR

    
    
  • Export gradients for INR

    python export_sdf_ray.py
    
  • Train INSP-Net

    python experiment_scripts/train_sdf_insp.py --experiment_name smooth_armadillo --sz 256 --ti 10 --batch_size 1
    
  • Inference INSP-Net

    python eval_sdf_insp.py
    

Image Classification

Due to the large size of MNIST and CIFAR INRs, we can't provide all of the checkpoints. However, we share the scripts to generate the INRs.

Citation

@inproceedings{Xu_2022_INSP,
    title={Signal Processing for Implicit Neural Representations},
    author={Xu, Dejia and Wang, Peihao and Jiang, Yifan and Fan, Zhiwen and Wang, Zhangyang},
    booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
    year={2022}
}

About

[NeurIPS 2022] "Signal Processing for Implicit Neural Representations" by Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang Wang

Topics

Resources

License

Stars

Watchers

Forks

Languages