Regularization is a technique used to reduce the complexity of a model, thereby preventing overfitting. There are three common types of regularization used in Deep Neural Networks (DNN):

L2 Regularization: We define the complexity of a model by the sum of the squares of its weights: $W = w_0^2 + w_1^2 + … + w_n^2$. We add this term to the loss function to obtain:

$L(\text{data}, \text{model}) = \text{loss}(\text{data}, \text{model}) + \lambda \sum w_i^2$

We then aim to minimize this total loss. As the derivative of $W$ with respect to each weight $w_i$ is $2w_i$, backpropagation reduces the weights by penalizing larger values, effectively “decaying” them.

L1 Regularization: This is similar to L2 regularization, but $W$ is defined as the sum of the absolute values of the weights:

$W = \sum |w_i|$

The derivative of $W$ with respect to $w_i$ is a constant ($\pm 1$) this time, so weights can be reduced exactly to zero, unlike in L2 regularization. This often leads to sparse models.

Dropout: Unlike the previous two methods, dropout is implemented as a layer within the neural network rather than a modification to the loss function.

A dropout layer randomly sets a subset of activations to zero during training. For example, a dropout layer with a rate of 0.3 will randomly deactivate 30% of the neurons in that layer for each training step.

Sequence Learning in RNNs

An example of a sequence is a set of words in a sentence. Sequence learning and transformation allow computers to translate one sequence into another language.

Alternatively, if no explicit target exists, RNNs can predict the next element in a sequence. This type of prediction often blurs the line between supervised and unsupervised learning.

Models with State

Autoregressive models calculate the current value based on previous ones:

$$x_t = f(x_{t-1}, x_{t-2}, \ldots)$$

By incorporating hidden states, it becomes much easier to perform complex tasks:

$$x_t = f(h_t, x_{t-1}, x_{t-2}, \ldots)$$

These hidden states are typically nonlinear.

Similarity to Quantum Mechanics

In Feed-Forward Neural Networks (FFNNs), the hidden state is not directly observable. Is this similar to a quantum state?

Two Earlier Models

There are two general types of models worth mentioning.

Linear Dynamical Systems

Used extensively in engineering. The system state is always linear; therefore, Kalman filtering is utilized.

Hidden Markov Models

Stochastic models with discrete states that store $\log(N)$ bits for $N$ states. HMMs have efficient learning and prediction algorithms.

An important limitation of HMMs is their memory. They can only keep $\log(N)$ bits of information. For a full-fledged linguistic application, we might need at least 100 bits of state, which would require $2^{100}$ states—an infeasible number.

Differences Between RNNs, HMMs, and LDSs

Unlike HMMs, RNNs feature a distributed hidden state and complex, nonlinear hidden units. Furthermore, they are typically deterministic.

RNN state behaviors can include:

• Oscillation: Potentially useful for motor control.
• Settling to point attractors: Potentially useful for retrieving memories.
• Chaos: Generally undesirable for information processing.

RNNs can learn to implement many small programs that run in parallel.

A significant disadvantage of RNNs: RNNs are hard to train.