Awesome-Normalization-Techniques

Papers for normalization techniques, released codes collections.

Any addition or bug feel free to open an issue or pull requests.

2020 - 2019 - 2018 - 2017 - 2016

2020

• A New Look at Ghost Normalization
  • Neofytos Dimitriou, Ognjen Arandjelovic
  • [Paper]
• TaskNorm: Rethinking Batch Normalization for Meta-Learning (ICML 2020)
  • John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, Richard E Turner
  • [Paper]
  • [Python Reference]
• Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise (AAAI 2020)
  • Senwei Liang, Zhongzhan Huang, Mingfu Liang, Haizhao Yang
  • [Paper]
  • [Python Reference]
• Towards Stabilizing Batch Statistics in BackWard Propagation of Batch Normalization (ICLR 2020)
  • Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, Jian Sun
  • [Paper]
• Rethinking Spatially-Adaptive Normalization
  • Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Nenghai Yu
  • [Paper]
• Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks (CVPR 2020)
  • Saurabh Singh, Shankar Krishnan
  • [Paper]
  • [Python Reference]
• Extended Batch Normalization
  • Chunjie Luo, Jianfeng Zhan, Lei Wang, and Wanling Gao
  • [Paper]
  • [C++ Reference]
• Knowledge Distillation via Adaptive Instance Normalization
  • Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos
  • [Paper]
• Four Things Everyone Should Know to Improve Batch Normalization
  • Cecilia Summers, Michael J. Dinneen
  • [Paper]
• Region Normalization for Image Inpainting (AAAI 2020)
  • Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, Sen Liu
  • [Paper]
  • [Python Reference]
• Local Context Normalization: Revisiting Local Normalization (CVPR 2020)
  • Anthony Ortiz , Caleb Robinson, Mahmudulla Hassan, Dan Morris2, Olac Fuentes1, Christopher Kiekintveld, and Nebojsa Jojic
  • [Paper]
• Attentive Normalization for Conditional Image Generation (CVPR 2020)
  • Yi Wang, Ying-Cong Chen, Xiangyu Zhang, Jian Sun, Jiaya Jia
  • [Paper]
  • [Python Reference]
• Cross-Iteration Batch Normalization
  • Zhuliang Yao, Yue Cao, Shuxin Zheng, Gao Huang, Stephen Lin
  • [Paper]
  • [Python Reference]
• SEAN: Image Synthesis with Semantic Region-Adaptive Normalization (CVPR 2020)
  • Peihao Zhu, Rameen Abdal, Yipeng Qin, Peter Wonka
  • [Paper]
  • [Python Reference]
• Evolving Normalization-Activation Layers
  • Hanxiao Liu, Andrew Brock, Karen Simonyan, Quoc V. Le
  • [Paper]

2019

• Differentiable Dynamic Normalization for Learning Deep Representation (PMLR 2019/ICML 2019)
  • Ping Luo, Peng Zhanglin, Shao Wenqi, Zhang Ruimao, Ren Jiamin, Wu Lingyun
  • [Paper]
• Mean Spectral Normalization of Deep Neural Networks for Embedded Automation
  • Anand Krishnamoorthy Subramanian, Nak Young Chong
  • [Paper]
  • [Python Reference]
• An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
  • Vincent Michalski, Vikram Voleti, Samira Ebrahimi Kahou, Anthony Ortiz, Pascal Vincent, Chris Pal, Doina Precup
  • [Paper]
• Generalized Batch Normalization: Towards Accelerating Deep Neural Networks (AAAI 2019)
  • Xiaoyong Yuan, Zheng Feng, Matthew Norton, Xiaolin Li
  • [Paper]
• Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
  • Michał Zając, Konrad Zolna, Stanisław Jastrzębski
  • [Paper]
• Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients (ICML 2019)
  • Zhenwei Dai, Reinhard Heckel
  • [Paper]
  • [Python Reference]
• Unpaired Image Translation via Adaptive Convolution-based Normalization
  • Wonwoong Cho, Kangyeol Kim, Eungyeup Kim, Hyunwoo J. Kim, Jaegul Choo
  • [Paper]
• EvalNorm: Estimating Batch Normalization Statistics for Evaluation (ICCV 2019)
  • Saurabh Singh, Abhinav Shrivastava
  • [Paper]
• Online Normalization for Training Neural Networks (NIPS 2019)
  • Vitaliy Chiley, Ilya Sharapov, Atli Kosson Urs Koster
  • [Paper]
  • [Python Reference]
• Transferable Normalization: Towards Improving Transferability of Deep Neural Networks (NIPS 2019)
  • Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I. Jordan
  • [Paper]
  • [Python Reference]
• Iterative Normalization: Beyond Standardization Towards Efficient Whitening (CVPR 2019)
  • Lei Huang, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
  • [Paper]
  • [Python Reference]
• Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (CVPR 2019)
  • Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, Bohyung Han
  • [Paper]
  • [Python Reference]
• Attentive Normalization
  • Xilai Li, Wei Sun and Tianfu Wu
  • [Paper]
  • [Python Reference]
• Rethinking Normalization and Elimination Singularity in Neural Networks
  • Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan Yuille
  • [Paper]
• Dynamic Instance Normalization for Arbitrary
  • Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, Shilei Wen
  • [Paper]
• Semantic Image Synthesis with Spatially-Adaptive Normalization (CVPR 2019)
  • Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu
  • [Paper]
  • [Python Reference]
• Differentiable Learning-to-Normalize via Switchable Normalization (ICLR 2019)
  • Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, Jingyu Li
  • [Paper]
  • [Python Reference]
• SSN: Learning Sparse Switchable Normalization via Sparsestmax (CVPR 2019)
  • Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, Ping Luo
  • [Paper]
• A Mean Field Theory of Batch Normalization
  • Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz
  • [Paper]
• Restructuring Batch Normalization to Accelerate CNN Training
  • Wonkyung Jung, Daejin Jung, Byeongho Kim, Sunjung Lee, Wonjong Rhee, Jung Ho Ahn
  • [Paper]
• A Novel Convolutional Neural Network for Image Steganalysis with Shared Normalization (TMM 2019)
  • Songtao Wu, Sheng-hua Zhong, and Yan Liu
  • [Paper]
• Training Faster by Separating Modes of Variation in Batch-normalized Models (TPAMI 2019)
  • Mahdi M. Kalayeh, Mubarak Shah
  • [Paper]
• Uncertainty Estimation via Stochastic Batch Normalization (ISNN 2019)
  • Andrei Atanov, Arsenii Ashukha, Dmitry Molchanov, Kirill Neklyudov, Dmitry Vetrov
  • [Paper]
  • [Python Reference]

2018

• Decorrelated Batch Normalization (CVPR 2018)
  • Lei Huang, Dawei Yang, Bo Lang, Jia Deng
  • [Paper]
  • [Lua Reference]
• MegDet: A Large Mini-Batch Object Detector (CVPR 2018)
  • Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, Jian Sun
  • [Paper]
• Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks (NIPS 2018)
  • Hyeonseob Nam, Hyo-Eun Kim
  • [Paper]
• Kalman Normalization: Normalizing Internal Representations Across Network Layers (NIPS 2018)
  • Guangrun Wang, Jiefeng Peng, Ping Luo, Xinjiang Wang, Liang Lin
  • [Paper]
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
  • Ping Luo, Zhanglin Peng, Jiamin Ren, Ruimao Zhang
  • [Paper]
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks (TNNLS)
  • Shuang Wu , Guoqi Li, Lei Deng, Liu Liu, Dong Wu, Yuan Xie, Luping Shi
  • [Paper]
• In-Place Activated BatchNorm for Memory-Optimized Training of DNNs (CVPR 2018)
  • Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder
  • [Paper]
  • [Python Reference]
• Group Normalization (ECCV 2018)
  • Yuxin Wu, Kaiming He
  • [Paper]
  • [Python Reference]
• Spectral Normalization for Generative Adversarial
  • Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida
  • [Paper]
  • [Python Reference]

2017

• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models (NIPS 2017)
  • Sergey Ioffe
  • [Paper]
  • [Python Reference]
• Modulating Early Visual Processing by Language (NIPS 2017)
  • Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, Aaron Courville
  • [Paper]
  • [Python Reference]
• Instance Normalization: the Missing Ingredient for Fast Stylization
  • Dmitry Ulyanov, Andrea Vedaldi
  • [Paper]
  • [PyTorch Reference]
• Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks
  • Luo Chunjie, Zhan Jianfeng, Wang Lei, Yang Qiang
  • [Paper]
• Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization (ICCV 2017)
  • Xun Huang, Serge Belongie
  • [Paper]
  • [Python Reference]
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
  • Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H. Sinz, Richard S. Zemel􏲗
  • [Paper]
• Recurrent Batch Normalization
  • Tim Cooijmans, Nicolas Ballas, César Laurent, Çag ̆lar Gülçehre, Aaron Courville
  • [Paper]

2016

• Layer Normalization

  • Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton
  • [Paper]
  • [PyTorch Reference]

• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

  • Devansh Arpit, Yingbo Zhou, Bhargava U. Kota, Venu Govindaraju, SUNY Buffalo
  • [Paper]

• Weight Normalization: a Simple Reparameterization to Accelerate Training of Deep Neural Networks (NIPS 2016)

  • Tim Salimans, Diederik P. Kingma
  • [Paper]

2015

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
  • Sergey Ioffe, Christian Szegedy
  • [Paper]
  • [PyTorch Reference]