内容简介:Batch Normalization is Indeed one of the major breakthrough in the field of Deep Learning and is one of the hot topics for discussion among researchers in the past few years. Batch Normalization is a widely adopted technique that enables faster and more st
What are some drawbacks of using batch normalization?
May 15 ·6min read
Batch Normalization is Indeed one of the major breakthrough in the field of Deep Learning and is one of the hot topics for discussion among researchers in the past few years. Batch Normalization is a widely adopted technique that enables faster and more stable training and has become one of the most influential methods. However, despite its versatility, there are still some points holding this method back as we are going to discuss in this article, which shows that there’s still room for improvement for normalization methods.
Why do we use Batch Normalization?
Before discussing anything, first, we should know what batch normalization is, how it works, and discuss it’s use cases.
What Batch Normalization is
During training, the output distribution of each intermediate activation layer shifts at each iteration as we update the previous weights. This phenomenon is referred to as an internal covariant shift (ICS). So a natural thing to do, if I want to prevent this from happening, is to fix all the distributions. In simple words, if I had some problem that my distributions are shifting around, ill just clamp them and not let them shift around to help gradient optimization and prevent vanishing gradients, and this will help my neural network train faster. So reducing this internal covariant shift was the key principle driving the development of batch normalization.
How it works
Batch Normalization normalizes the output of the previous output layer by subtracting the empirical mean over the batch divided by the empirical standard deviation. This will help the data look like Gaussian distribution .
Where mu and sigma_square are the batch mean and batch variance respectively.
And, we learn a new mean and covariance in terms of two learnable parameters γ and β. So in short, you can think of batch normalization is something that helps you control the first and second moments of the distribution of the batch.
Benefits
I’ll enlist some of the benefits of using batch normalization but I won’t get into much detail, as there are tonnes of articles already covering that.
- Faster convergence.
- Decreases the importance of initial weights.
- Robust to hyperparameters.
- Requires less data for generalization.
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持 码农网
猜你喜欢:本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
Tango with Django
David Maxwell、Leif Azzopardi / Leanpub / 2016-11-12 / USD 19.00
Tango with Django is a beginner's guide to web development using the Python programming language and the popular Django web framework. The book is written in a clear and friendly style teaching you th......一起来看看 《Tango with Django》 这本书的介绍吧!