内容简介:May 29, 2020We can’t compute partial derivatives of very complicated functions using just the basic matrix calculus rules we’ve seenBlog part 1. For example, we can’t take the derivative of nested expressions like sum(
Matrix Calculus for DeepLearning (Part2)
May 29, 2020
We can’t compute partial derivatives of very complicated functions using just the basic matrix calculus rules we’ve seenBlog part 1. For example, we can’t take the derivative of nested expressions like sum( w + x ) directly without reducing it to its scalar equivalent. We need to be able to combine our basic vector rules using the vector chain rule.
In paper they have defined and named three different chain rules.
- single-variable chain rule
- single-variable total-derivative chain rule
- vector chain rule
The chain rule comes into play when we need the derivative of an expression composed of nested subexpressions. Chain rule helps in solving problem by breaking complicated expressions into subexpression whose derivatives are easy to compute.
Single-variable chain rule
Chain rules are defined in terms of nested functions such as y=f(g(x)) for single variable chain rule.
Formula is
dy/dx = (dy/du) (du/dx)
There are 4 steps to solve using single variable chain rule
- Introduce intermediate variable
- compute derivatives of intermediate variables wrt(with respect to) their parameters.
- combine all derivatives by multiplying them together
- substitute intermediate variables back in derivative equation.
Lets see example of nested equation y = f (x) = ln (sin(x³ ) ² )
It is to compute the derivatives of the intermediate variables in isolation!
But single variable chain rule is applicable only when a single variable can influence output in only one way. As we see in example we can handle nested expression of single variable x using this chain ruleonly when x can effect y through single data flow path.
Single-variable total-derivative chain rule
If we apply single variable chain rule to y = f (x) = x + x² we get wrong answer, because derivative operator doesnot apply to multivariate functions. change in x in the equation , affects y both as operand og addition and as operand of square. so we clearly cant apply single variable chain rule. so…
we move to total derivatives.
which is to compute (dy/dx) , we need to sum up all possible contributions from changes in x to the change in y.
Formula for total derivative chain rule
Total derivative assumes all variables are potentially co-dependent where as partial derivative assumes all variables but x are constants.
when you take the total derivative with respect to x, other variables might also be functions of x so add in their contributions as well. The left side of the equation looks like a typical partial derivative but the right-hand side is actually the total derivative.
Lets see example,
total derivative formula always sums , that is sums up terms in the derivative. For example, given y = x × x² instead of y = x + x² , the total-derivative chain rule formula still adds partial derivative terms, for more detail see demonstration in paper.
Formula of total derivative can be simplified further.
This chain rule that takes into consideration the total derivative degenerates to the single-variable chain rule when all intermediate variables are functions of a single variable.
Vector chain rule
derivative of a sample vector function with respect to a scalar, y = f (x).
introduce two intermediate variables, g 1 and g 2 , one for each f i so that y looks more like y = f ( g (x))
If we split the terms, isolating the terms into a vector, we get a matrix by vector.
This completes chain rule. In next blog that is part3 we will see how we can apply this gradient of neural activation and loss function and wrap up.
Thank you.
Useful Points:
It is difficult while writing blog in markdown to convert to superscript and subscript so I have listed down , which you can use ( copy paste) in your markdown
super script ⁰ ¹ ² ³ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ᵃ ᵇ ᶜ ᵈ ᵉ ᶠ ᵍ ʰ ᶦ ʲ ᵏ ˡ ᵐ ⁿ ᵒ ᵖ ʳ ˢ ᵗ ᵘ ᵛ ʷ ˣ ʸ ᶻ
subscript ₀ ₁ ₂ ₃ ₄ ₅ ₆ ₇ ₈ ₉ ₐ ᵦ ₑ ₕ ᵢ ⱼ ₖ ₗ ₘ ₙ ₒ ₚ ᵩ ᵣ ₛ ₜ ᵤ ᵥ ₓ ᵧ
# Blog 10
以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网
猜你喜欢:本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
网站转换率优化之道
[美] Khalid Saleh、[美] Ayat Shukairy / 顾 毅 / 人民邮电出版社 / 2012-4 / 45.00元
内容简介: 怎样才能将访问者转化为顾客? 本书提供了一些切实可行的建议,比如如何说服访问者作出购买决定,如何避免用户因信息过量或导航繁琐而离开网站等。不论你是在设计或营销大型电子商务网站,还是在管理中小型在线业务,都可以从本书学会怎样使用市场营销原则、设计方法、可用性原则和分析数据来持续提升网站的转换率。 作者帮助过众多公司吸引在线顾客,有着丰富的实战经验,在书中细致讨论了从访问......一起来看看 《网站转换率优化之道》 这本书的介绍吧!