I think using graphs for recommendation systems is not a natural choice where temporal patterns and sequences play a role. It is not natural to represent time with nodes that can have multiple edges. Instead, RNNs or Markov chains might be more suitable. Recommendation is almost always sequential. Otherwise, it is about finding embeddings between elements. If so, graphs might not fit these problems. If graphs are not suitable, why do we use them?

The answer is simple: People don’t buy things in a strict sequence; they have options, multiple choices, and various factors that play a role in these choices. So, it is actually natural to model these interactions with graphs.

These datasets are provided in a binary format. There are four files—two for the training set and two for the test set—in gzipped IDX and IDX3 formats. Although it is a simple format, it is non-standard and requires custom code to parse.

For a professional project, I wrote the following two functions to load these files into NumPy arrays.

import struct
import gzip
import numpy as np

def mnist_images_idx_to_array(images_filename):
    images_f = gzip.open(images_filename, mode="rb")
    images_f.seek(0)
    magic = struct.unpack('>I', images_f.read(4))[0]
    if magic != 0x00000803:
        raise Exception(f"Format error: Need an IDX3 file: {images_filename}")
    n_images = struct.unpack('>I', images_f.read(4))[0]
    n_row = struct.unpack('>I', images_f.read(4))[0]
    n_col = struct.unpack('>I', images_f.read(4))[0]

    n_bytes = n_images * n_row * n_col  # each pixel is 1 byte

    images_data = struct.unpack(
        '>' + str(n_bytes) + 'B', images_f.read(n_bytes))

    images_array = np.asarray(images_data, dtype='uint8')
    images_array.shape = (n_images, n_row, n_col)

    return images_array


def mnist_labels_idx_to_array(labels_filename):
    labels_f = gzip.open(labels_filename, mode="rb")
    labels_f.seek(0)
    magic = struct.unpack('>I', labels_f.read(4))[0]
    if magic != 0x00000801:
        raise Exception(f"Format error: Need an IDX file: {labels_filename}")
    n_labels = struct.unpack('>I', labels_f.read(4))[0]
    labels_data = struct.unpack(
        '>' + str(n_labels) + 'B', labels_f.read(n_labels))
    labels_array = np.asarray(labels_data, dtype='uint8')
    return labels_array

You can use these functions by passing the appropriate filenames, as shown below:

training_images = mnist_images_idx_to_array(
    os.path.join(input_dir, "train-images-idx3-ubyte.gz"))
training_labels = mnist_labels_idx_to_array(
    os.path.join(input_dir, "train-labels-idx1-ubyte.gz"))
testing_images = mnist_images_idx_to_array(
    os.path.join(input_dir, "t10k-images-idx3-ubyte.gz"))
testing_labels = mnist_labels_idx_to_array(
    os.path.join(input_dir, "t10k-labels-idx1-ubyte.gz"))

Regularization is a technique used to reduce the complexity of a model, thereby preventing overfitting. There are three common types of regularization used in Deep Neural Networks (DNN):

L2 Regularization: We define the complexity of a model by the sum of the squares of its weights: $W = w_0^2 + w_1^2 + … + w_n^2$. We add this term to the loss function to obtain:

$L(\text{data}, \text{model}) = \text{loss}(\text{data}, \text{model}) + \lambda \sum w_i^2$

We then aim to minimize this total loss. As the derivative of $W$ with respect to each weight $w_i$ is $2w_i$, backpropagation reduces the weights by penalizing larger values, effectively “decaying” them.

L1 Regularization: This is similar to L2 regularization, but $W$ is defined as the sum of the absolute values of the weights:

$W = \sum |w_i|$

The derivative of $W$ with respect to $w_i$ is a constant ($\pm 1$) this time, so weights can be reduced exactly to zero, unlike in L2 regularization. This often leads to sparse models.

Dropout: Unlike the previous two methods, dropout is implemented as a layer within the neural network rather than a modification to the loss function.

A dropout layer randomly sets a subset of activations to zero during training. For example, a dropout layer with a rate of 0.3 will randomly deactivate 30% of the neurons in that layer for each training step.

I’m building a face recognizer with dlib’s frontal face detector, and today, I noticed that some of the results return negative coordinates in face detection. This means the detected face is partial, although it’s a bit of a stretch to use negative coordinates for this.

My code wasn’t checking for negative coordinates and was building the NumPy array incorrectly. However, after reading the issue, although I’m still not convinced that it’s a good approach, I added a few if statements and the problem was solved.

Sequence Learning in RNNs

An example of a sequence is a set of words in a sentence. Sequence learning and transformation allow computers to translate one sequence into another language.

Alternatively, if no explicit target exists, RNNs can predict the next element in a sequence. This type of prediction often blurs the line between supervised and unsupervised learning.

Models with State

Autoregressive models calculate the current value based on previous ones:

$$x_t = f(x_{t-1}, x_{t-2}, \ldots)$$

By incorporating hidden states, it becomes much easier to perform complex tasks:

$$x_t = f(h_t, x_{t-1}, x_{t-2}, \ldots)$$

These hidden states are typically nonlinear.

Similarity to Quantum Mechanics

In Feed-Forward Neural Networks (FFNNs), the hidden state is not directly observable. Is this similar to a quantum state?

Two Earlier Models

There are two general types of models worth mentioning.

Linear Dynamical Systems

Used extensively in engineering. The system state is always linear; therefore, Kalman filtering is utilized.

Hidden Markov Models

Stochastic models with discrete states that store $\log(N)$ bits for $N$ states. HMMs have efficient learning and prediction algorithms.

An important limitation of HMMs is their memory. They can only keep $\log(N)$ bits of information. For a full-fledged linguistic application, we might need at least 100 bits of state, which would require $2^{100}$ states—an infeasible number.

Differences Between RNNs, HMMs, and LDSs

Unlike HMMs, RNNs feature a distributed hidden state and complex, nonlinear hidden units. Furthermore, they are typically deterministic.

RNN state behaviors can include:

• Oscillation: Potentially useful for motor control.
• Settling to point attractors: Potentially useful for retrieving memories.
• Chaos: Generally undesirable for information processing.

RNNs can learn to implement many small programs that run in parallel.

A significant disadvantage of RNNs: RNNs are hard to train.