These notes are gathered from various places. When I can, I give credits and links, but even if I don't, they are certainly not original ideas.
Sequence Learning in RNNs
A example to sequence is a set of words in English. Sequence learning and transforming allows computers to translate this sequence to another language.
Or if no target exists, RNNs predict next element in a sequence. The prediction blurs the line between supervised and unsupervised learning.
Models with State
With hidden states, it's much easier to achieve more complex tasks.
Hidden state is nonlinear.
Similarity to Quantum Mechanics
In Feed Forward Neural Nets, hidden state is not observable. Similar to Quantum state?
Two Earlier Models
There are two general types of models worth mentioning.
Linear Dynamical Systems
Used in engineering. The system state is always linear, therefore Kalman filtering is utilized.
Difference of RNNs from HMMs and LDSs
Unlike HMMs, RNNs have distributed hidden state and non-linear complex hidden units. Also they are deterministic
Behavior of RNN states - Oscillate. Good for motor control? - Settle to point attractors. Good for retrieving memories? - Chaotic. Bad for information processing.
RNNs could learn to implement lots of small programs to run in parallel.
A disadvatage of RNNs: RNNs are hard to train.