1. The Vector Chain Rule.
  2. The gradient of the neuron activation function.

The Vector Chain Rule

While I assume that you already aware of the chain rule for differentiation, the vector chain rule deserves VIP status.

Eq.1 : Vector function
Eq.2 : Variables used for substitution
Eq.3 : Substituted new variables in the Vector function
Eq.4 : Vector chain rule for differentiation
Eq.5 : Chain rule applied to the first vector function
Eq.6 : Generalized vector chain rule
Eq.7 : Vector chain rule as product of Jacobians

The gradient of neuron activation

We can now find the derivative of-

Eq.8 : Activation function
Eq.9 : Function to differentiate
Eq.10 : Elementwise product operator
Eq. 11 : Derivative of w.x
Eq.12 : Chain rule to find the derivative of y
Eq.13 : Derivative of w.x
Eq.14 : Solving for the derivative of y
Eq.15 : Multiplying the vectors in Equation 14.
Eq.16 : Derivative with respect to b.
Eq.17 : Activation function again
Eq.18 : Derivative of the activation function with respect to w.
Eq.19 : Derivative of the activation function with respect to b.

The gradient of the neural network loss function

Say we have the following inputs to our neural network:

Eq.20 : Inputs and targets to the neural network
Eq.21 : Cost function
Eq.22 : Intermediate variables
Eq.23 : Derivative of the activation function with respect to the weights and the biases
Eq.24 : Derivative of v with respect to the weights.
Eq.25 : Differentiating C with respect to w.
Eq.26 : Substituting for the partial derivative of v
Eq.27 : Derivative of the cost function with respect to the weights
Eq.28 : Substituting the difference between the actual and target values as the error
Eq.29 : Derivative of the cost function with respect to the bias

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Kaushik Moudgalya

Kaushik Moudgalya

Computer Science Master’s student at the University of Montreal, specializing in Machine Learning.