normalization专题

normalization大体分为3类feature normalization、weight normalization和gradient normalization，本文主要讲feature normalization，其他两个涉及的数学较多，一时无法理清

• Feature normalization
  • Batch normalization
  • Layer normalization
  • Instance normalization
  • Group normalization
  • Switchable norma
• Weight normalization
• Gradient normalization

batch normalization

Layer normalization

Instance normalization ##

Group normalization

Switchable normalization

Weight normalization

其实具体的细节不太能完全理解

Gradient normalization

具体细节不展开，仅提供一些最近进展动向

Conclusion

Others

Normalization in GAN

• Virtual Batch Normalization (Salimans et al, NIPS 2016)
• Weight Norm (Xiang et al, 2017)
  Spectral Norm (Miyato, ICLR 2018)
  Normalization in domain adaptation
• adabn
  Normalization in style transfer
• IN
• ada IN

---

• Normalization Methods for Training Deep Neural Networks:
  Theory and Practice

• 【机器学习】covariate shift现象的解释