Types of regularization in ML2020-12-26
What is regularization?
Regularization is used to reduce the complexity of a model. There are 3 types of regularization we can use in Deep NN.
L2 Regularization: We define the complexity of a model by $W=w_0^2 + w_1^2 + ... + w_n^2$ and add this to Loss function to get
$L(data, model) = loss(data, model) + (w_0^2 + ... + w_n^2)$
and try to reduce this.
As seen the derivative of a the $W$ is $2*W$ so backpropagation reduces the weights by penalizing larger weights.
L1 Regularization: This is similar to L2 Regularization, but $W$ is defined as:
$\sum |w_0| + |w_1| + ... |w_n|$
The derivative of $W$ is a constant $k$ this time, so weights can be reduced to zero unlike L2.
Dropout: Unlike the other two, this is a layer in the neural network instead of a loss function.
A dropout layer sets weights of a random set of weights to 0. If we have 0.3 Dropout layer, it sets 30% of the weights to 0.