Skip to content
/ PreNAS Public
forked from tinyvision/PreNAS

The official implementation of paper PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search

License

Notifications You must be signed in to change notification settings

drcege/PreNAS

 
 

Repository files navigation

PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search

PreNAS is a novel learning paradigm that integrates one-shot and zero-shot NAS techniques to enhance search efficiency and training effectiveness. This search-free approach outperforms current state-of-the-art one-shot NAS methods for both Vision Transformer and convolutional architectures, as confirmed by its superior performance when the code is released.

Wang H, Ge C, Chen H and Sun X. PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search. ICML 2023.

Paper link: arXiv

Overview



Previous one-shot NAS samples all architectures in the search space when one-shot training of the supernet for better evaluation in evolution search. Instead, PreNAS first searches the target architectures via a zero-cost proxy and next applies preferred one-shot training to supernet. PreNAS improves the Pareto Frontier benefited from the preferred one-shot learning and is search-free after training by offering the models with the advance selected architectures from the zero-cost search.

Environment Setup

To set up the environment you can easily run the following command:

conda create -n PreNAS python=3.7
conda activate PreNAS
pip install -r requirements.txt

Data Preparation

You need to download the ImageNet-2012 to the folder ../data/imagenet.

Run example

The code was run on 8 x 80G A100.

  • Zero-Shot Search

    bash 01_zero_shot_search.sh

  • One-Shot Training

    bash 02_one_shot_training.sh

  • Evaluation

    bash 03_evaluation.sh

Model Zoo

Model TOP-1 (%) TOP-5 (%) #Params (M) FLOPs (G) Download Link
PreNAS-Ti 77.1 93.4 5.9 1.4 AliCloud
PreNAS-S 81.8 95.9 22.9 5.1 AliCloud
PreNAS-B 82.6 96.0 54 11 AliCloud

Bibtex

If PreNAS is useful for you, please consider to cite it. Thank you! :)

@InProceedings{PreNAS,
    title     = {PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search},
    author    = {Wang, Haibin and Ge, Ce and Chen, Hesen and Sun, Xiuyu},
    booktitle = {International Conference on Machine Learning (ICML)},
    month     = {July},
    year      = {2023}
}

Acknowledgements

The codes are inspired by AutoFormer.

About

The official implementation of paper PreNAS: Preferred One-Shot Learning Towards Efficient Neural Architecture Search

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 97.4%
  • Shell 2.6%