These notes have been gathered from various sources. I provide credits and links whenever possible, but even where omitted, these are certainly not original ideas.

Sequence Learning in RNNs

An example of a sequence is a set of words in a sentence. Sequence learning and transformation allow computers to translate one sequence into another language.

Alternatively, if no explicit target exists, RNNs can predict the next element in a sequence. This type of prediction often blurs the line between supervised and unsupervised learning.

Models with State

Autoregressive models calculate the current value based on previous ones:

$$x_t = f(x_{t-1}, x_{t-2}, \ldots)$$

By incorporating hidden states, it becomes much easier to perform complex tasks:

$$x_t = f(h_t, x_{t-1}, x_{t-2}, \ldots)$$

These hidden states are typically nonlinear.

Similarity to Quantum Mechanics

In Feed-Forward Neural Networks (FFNNs), the hidden state is not directly observable. Is this similar to a quantum state?

Two Earlier Models

There are two general types of models worth mentioning.

Linear Dynamical Systems

Used extensively in engineering. The system state is always linear; therefore, Kalman filtering is utilized.

Hidden Markov Models

Stochastic models with discrete states that store $\log(N)$ bits for $N$ states. HMMs have efficient learning and prediction algorithms.

An important limitation of HMMs is their memory. They can only keep $\log(N)$ bits of information. For a full-fledged linguistic application, we might need at least 100 bits of state, which would require $2^{100}$ states—an infeasible number.

Differences Between RNNs, HMMs, and LDSs

Unlike HMMs, RNNs feature a distributed hidden state and complex, nonlinear hidden units. Furthermore, they are typically deterministic.

RNN state behaviors can include:

• Oscillation: Potentially useful for motor control.
• Settling to point attractors: Potentially useful for retrieving memories.
• Chaos: Generally undesirable for information processing.

RNNs can learn to implement many small programs that run in parallel.

A significant disadvantage of RNNs: RNNs are hard to train.