Curse of Batch Normalization

栏目: IT技术 · 发布时间: 4年前

内容简介:Batch Normalization is Indeed one of the major breakthrough in the field of Deep Learning and is one of the hot topics for discussion among researchers in the past few years. Batch Normalization is a widely adopted technique that enables faster and more st

What are some drawbacks of using batch normalization?

May 15 ·6min read

Curse of Batch Normalization

Photo by Freddie Collins on Unsplash

Batch Normalization is Indeed one of the major breakthrough in the field of Deep Learning and is one of the hot topics for discussion among researchers in the past few years. Batch Normalization is a widely adopted technique that enables faster and more stable training and has become one of the most influential methods. However, despite its versatility, there are still some points holding this method back as we are going to discuss in this article, which shows that there’s still room for improvement for normalization methods.

Why do we use Batch Normalization?

Before discussing anything, first, we should know what batch normalization is, how it works, and discuss it’s use cases.

What Batch Normalization is

During training, the output distribution of each intermediate activation layer shifts at each iteration as we update the previous weights. This phenomenon is referred to as an internal covariant shift (ICS). So a natural thing to do, if I want to prevent this from happening, is to fix all the distributions. In simple words, if I had some problem that my distributions are shifting around, ill just clamp them and not let them shift around to help gradient optimization and prevent vanishing gradients, and this will help my neural network train faster. So reducing this internal covariant shift was the key principle driving the development of batch normalization.

How it works

Batch Normalization normalizes the output of the previous output layer by subtracting the empirical mean over the batch divided by the empirical standard deviation. This will help the data look like Gaussian distribution .

Curse of Batch Normalization

Where mu and sigma_square are the batch mean and batch variance respectively.

Curse of Batch Normalization

And, we learn a new mean and covariance in terms of two learnable parameters γ and β. So in short, you can think of batch normalization is something that helps you control the first and second moments of the distribution of the batch.

Curse of Batch Normalization

Feature distribution output from an intermediate convolution layer from VGG-16 Network. 1. (Before) without any normalization, 2. (After) applying batch normalization.

Benefits

I’ll enlist some of the benefits of using batch normalization but I won’t get into much detail, as there are tonnes of articles already covering that.

  • Faster convergence.
  • Decreases the importance of initial weights.
  • Robust to hyperparameters.
  • Requires less data for generalization.

Curse of Batch Normalization

1. Faster Convergence, 2. Robust to hyperparameters

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

信号与噪声

信号与噪声

[美] 纳特•西尔弗 / 胡晓姣、张新、朱辰辰 / 中信出版社 / 2013-8 / 69.00元

【编辑推荐】 从海量的大数据中筛选出真正的信号, “黑天鹅”事件也可提前预知! “本书将成为未来十年内最重要的书籍之一。”——《纽约时报》 “对于每一个关心下一刻可能会发生什么的人来说,这都是本必读书。”——理查德•泰勒 《华尔街日报》2012年度10本最佳非虚构类图书之一 《经济学人》杂志2012年度书籍 亚马逊网站2012年度10本最佳非虚构类图书之一......一起来看看 《信号与噪声》 这本书的介绍吧!

HTML 编码/解码
HTML 编码/解码

HTML 编码/解码

正则表达式在线测试
正则表达式在线测试

正则表达式在线测试