I then came across the cached crate. It caches the results of functions for memoization. This is exactly what I need, I thought. Internally, it does what I was planning to do.

Later, I noticed the possible bugs that could arise. I’m thinking of using the function in a short-running process, so the metadata is not expected to change during the run. For some reason, suppose the runtime of the process began to get longer, or I decided to add a web server on top of it. At that time, probably many moons from now, I’ll have forgotten the decision I made about caches and my assumption that the metadata won’t change during the run. It will cause some weird bugs when file timestamp changes aren’t detected.

No one will notice that I’m fixing bugs if they never appear, but I believe this is the best kind of software engineering.

Commentary (2022-08-01)

• It looks like my assumption that RAM is significantly faster than disk access may also be wrong. SSDs are fast, and for parallel access, they may perform as fast as RAM.

Binary Classification

Given an image, classify it as cat or non-cat. The result is $\hat{y} = P(y=1 | x)$. In other words, given $x$, we calculate the probability that this data represents a cat.

Feature Vector from Image

We convert an image, e.g., a (64, 64, 3) image, into a (64 * 64 * 3, 1) feature vector.

Sigmoid

$$\hat{y} = \sigma(w^Tx + b)$$

$w$ represents the weights, $w^T$ is the transpose of $w$, $x$ is the input, and $b$ is the bias.

Loss Function

The loss function measures the error between the real value of $y$ and our prediction $\hat{y}$ for a single training example.

$$ L(\hat{y}, y) = \frac{1}{2}(\hat{y}-y)^2 $$

Cost Function

The average of the loss across all training examples.

Learning Rate

The learning rate $\alpha$ is the coefficient applied to update weights:

$$w’ = w - \alpha \frac{dd(w)}{dw}$$

If it is too small, learning occurs slowly; if it is too large, it may overshoot the optimum point. Thus, it should be selected wisely.

IDEA: Can we use a vector instead of a single value for the learning rate?

For Leaky ReLU activation functions, this seems possible, though likely overkill. Using $\alpha$ as a vector adds a layer of complexity and requires more computation to adjust (allowing some features to update more slowly than others).

The Feature Vector

$X$ is a feature matrix with $m$ columns, each representing a different training example. Each example has $n$ features. So, X.shape = (n, m).

Y.shape = (1, m).

The Computation Graph

Any formula can be converted into a graph where operands are vertices and operations are edges.

In neural networks, computation flows forward from inputs to output, while the derivatives are calculated from output back to input.

The Notation for Layers and Training Examples

$X^{(i)}$ refers to the $i^{th}$ training example.

$z^{[i]}$ refers to the $z$ values in the $i^{th}$ layer.

$z^{[1]}$ is the $z$ value for layer 1.

$a^{[2]}$ is the $a$ value for layer 2.

$a^{[1]} = [a^{[1]}_1, a^{[1]}_2, a^{[1]}_3, a^{[1]}_4]^T$ for a neural net having 4 nodes in hidden layer 1.

Steps of Computation

There are two steps of computation:

1. $z^{[1]} = w^{[1]T}x + b^{[1]}$
2. $a^{[1]} = \sigma(z^{[1]})$

In step 1, weights and inputs are dot-multiplied and the bias is added. In step 2, an activation function is applied to this result. While $\sigma$ is used here, other functions can also serve this purpose.

Backpropagation Algorithm

This is the fundamental algorithm that updates weights to find a solution.

In the forward pass, the cost function $L(\hat{y}, y)$ is computed.

In backpropagation, weights and biases are adjusted based on their derivatives multiplied by the learning factor $\alpha$.

While the actual formulas can be complex, the general idea is to calculate derivatives from the last layer back to the first and adjust weights accordingly.

Vectorization Notation

Multiple training examples are denoted by the superscript $(i)$.

The vectorized representation of the $Z$ matrix for multiple inputs and multiple layers is:

$$Z = \begin{pmatrix} z^{1} & z^{1} & z^{1} & \dots & z^{1} \ z^{2} & z^{2} & z^{2} & \dots & z^{2} \ z^{3} & z^{3} & z^{3} & \dots & z^{3} \ \dots & \dots & \dots & \dots & \dots \ z^{n} & z^{n} & z^{n} & \dots & z^{n} \ \end{pmatrix}$$

The $A$ matrix is structured similarly.

Since $w^{[i]T}x + b^{[i]}$ is a column vector, we concatenate these column vectors for multiple inputs.

Activation Functions

There are roughly four types of activation functions:

Sigmoid

This is the historic default. There is generally no need to use it except in the final output layer for binary classification (0 or 1).

Tanh

It is generally superior to the sigmoid function. It is asymptotic between 1 and -1, which often leads to better training behavior.

ReLU

A simple function, $r = \max(0, x)$, which has become very popular.

Leaky ReLU

Since ReLU is not differentiable for $x < 0$, this version adds a small slope ($r = \max(0.01x, x)$) for negative values.

Training Set and Test Set

Traditionally, with datasets of 100 to 10,000 elements, splits like 70/30% or 60/20/20% were common.

However, in the era of Big Data™, where datasets may contain 10,000,000 elements, such percentages are unnecessary. It is more reasonable to keep a fixed number, such as 10,000 elements, for the dev and test sets to speed up development.

Crucially, this data should come from the same distribution. Dev and test sets are used to check for overfitting and evaluate real-world performance.

Bias and Variance

High bias indicates the model is underfitting. High variance indicates the model is overfitting.

High Bias

Example: Train Set Error Rate: 15%, Dev Set Error Rate: 16%. The model is too simple to learn the data. Solutions: Use a larger network, train longer, or change the NN architecture.

High Variance

Example: Train Set Error Rate: 1%, Dev Set Error Rate: 14%. The model overfits the training data and fails to generalize. Solutions: Get more data, use regularization, or change the NN structure.

High Bias and High Variance

While classical models often face a bias/variance tradeoff, Deep Learning models can suffer from both simultaneously (e.g., 15% Train Set error and 30% Dev Set error). Solutions: Larger network, more data, regularization, and architecture changes.

Regularization

Used to decrease variance and prevent overfitting. Two main types: L2 Regularization (adds a factor to the weights) and Dropout Regularization (randomly sets some weights to zero).

L2 Regularization

Without regularization: $$w^{[l]} \leftarrow w^{[l]} - \alpha (dw^{[l]})$$ With regularization, we add a decay factor: $$ -\alpha (\frac{\lambda}{2m} w^{[l]}) $$ The full formula becomes: $$w^{[l]} \leftarrow (1 - \alpha \frac{\lambda}{2m}) w^{[l]} - \alpha dw^{[l]}$$ This is called weight decay because it pulls $w$ closer to zero, effectively making the network “smaller.”

Dropout

Dropout is a technique where the algorithm randomly “knocks out” nodes during training. These nodes are temporarily ignored during weight updates.

In each iteration, a random subset of nodes is removed based on a keep-prob. For example, a keep-prob of 0.5 means roughly half the nodes are ignored.

Data Augmentation

Generating variations of existing images (e.g., flipping, rotating, adding noise) is also a form of regularization. Note: Changes shouldn’t alter the image’s meaning (e.g., a flipped “4” might not be a “4”).

Early Stopping

As training progresses, the cost function $J$ for the training set continues to decrease, but the test set $J$ eventually begins to increase, indicating overfitting. We can stop training at that inflection point.

Normalization

Normalization involves bringing all feature values $X$ into a similar range. If $x_1$ is between 1 and 1000 and $x_2$ is between 0 and 1, the network may struggle. A common approach is $x_i = \frac{x_i - \mu_i}{\sigma_i}$, where $\mu_i$ is the mean and $\sigma_i$ is the standard deviation.

Exploding and Vanishing Gradients

In deep networks, weight values may exponentially increase or decrease. If all weights are 2, the final activation in an $l$-layer network becomes $2^l$, leading to exploding gradients. Conversely, weights less than 1 can lead to vanishing gradients where the network fails to learn.

Weight Initialization to Alleviate Exploding/Vanishing Gradients

To prevent these issues, it is best to initialize weights with a variance of $\frac{1}{n}$, where $n$ is the number of nodes in a layer.

For ReLU:

W[i] = np.random.randn(shape) * np.sqrt(2 / n[i-1])

For tanh:

W[i] = np.random.randn(shape) * np.sqrt(1 / n[i-1])

Another option is Xavier initialization:

W[i] = np.random.randn(shape) * np.sqrt(2 / (n[i] + n[i-1]))

Initialization by Zero

Initializing weights to all zeros fails to break symmetry, and the network will not learn. However, it is perfectly fine to initialize biases to zero.

Initialization with Random Numbers

W[i] = np.random.randn(layer_dim[l], layer_dim[l-1]) * FACTOR

If FACTOR is too large (e.g., 10), the network may converge slowly or suffer from exploding gradients. Using np.sqrt(2 / layer_dim[l-1]) is known as He initialization, while np.sqrt(1 / layer_dim[l-1]) is Xavier initialization.

Mini-Batches

When dealing with massive datasets (e.g., 10 million images), we cannot process them all at once. Instead, we divide them into mini-batches.

• Stochastic Gradient Descent: Batch size of 1.
• Batch Gradient Descent: Batch size equals the entire training set.
• Mini-Batch Gradient Descent: Sizes typically range from 32 to 512.

Optimization: Momentum

When training with mini-batches, it helps to track previous gradients to smooth out updates. A hyperparameter $\beta$ determines the influence of past gradients.

Optimization: RMSprop

Instead of simple updates, RMSprop uses the squared gradients to scale the updates:

$$s_{dW} \leftarrow \beta_2 s_{dW} - (1 - \beta_2) (dW)^2$$ $$ W \leftarrow W - \alpha \frac{dW}{\sqrt{s_{dW} + \epsilon}} $$

This reduces oscillations and speeds up convergence.

Optimization: Adam

Adam combines Momentum and RMSprop. It uses two hyperparameters, $\beta_1$ and $\beta_2$, and is generally the most effective optimization algorithm, converging much faster than others.

Learning Rate Decay

A fixed learning rate $\alpha$ can make convergence difficult. We can decay the learning rate over time: $\alpha = \frac{1}{1 + d * t} \alpha_0$, where $d$ is the decay rate and $t$ is the epoch number.

Hyperparameter Tuning

Key hyperparameters include the learning rate $\alpha$, optimization coefficients $\beta_1$ and $\beta_2$, number of layers, units per layer, activation functions, and mini-batch sizes.

Heuristics:

• Random search is often better than a uniform grid.
• Search on a logarithmic scale for parameters like $\alpha$ or $\beta$ (e.g., searching $1-10^r$ for $\beta$).

Batch Normalization

Batch normalization normalizes layer activations using two learnable parameters, $\beta$ and $\gamma$. It helps information flow to deeper layers and makes the network less sensitive to initial weight scales.

Softmax Activation

For multi-class classification, the final layer uses a softmax function to output a probability vector: $$a_i^{[l]} = \frac{e^{z_i^{[l]}}}{\sum_{j=1}^C e^{z_j^{[l]}}}$$ where $C$ is the number of classes.

Metrics

Single-number metrics like F1 Score or mAP make evaluation much easier.

• Optimizing Metric: The primary number you try to improve.
• Satisficing Metric: A threshold that must be met (e.g., running time < 100ms).

Error Analysis

Regularly inspect a subset of misclassified examples to understand where the model is failing.

Data Mismatch

If your training and test data come from different distributions, use a training-dev set to identify if performance drops are due to the model failing to generalize or a fundamental difference in data.

Transfer Learning

You can take a model trained on task A and repurpose it for task B. This is especially useful when task B has limited data.

Multi-Task Learning

One network can be trained to perform multiple tasks simultaneously (e.g., detecting pedestrians AND traffic signs), provided the network is large enough.

End-to-End Learning

In end-to-end learning, the network maps raw input directly to the final output, bypassing traditional hand-engineered feature extraction steps. This requires significant amounts of data to be effective.

Convolutional Neural Networks (CNNs)

CNNs are the standard for image processing. They apply filters (kernels) to images to extract features. The resulting matrix size is determined by the input size $n$ and filter size $f$ (result size: $n-f+1$).

Padding

Padding adds a border of pixels (usually zeros) around the input to prevent the image from shrinking after each convolution layer.

Classic Networks

LeNet-5

A 1998 network designed for digit recognition (32x32x1). It is small (60k parameters) but foundational.

AlexNet and VGG-16

AlexNet (2012) popularized Deep Learning. VGG-16 (138M parameters) used a very consistent architecture of 3x3 filters and max-pooling.

ResNets

Residual Networks use “shortcuts” or “skip connections” to allow gradients to flow through very deep networks (100+ layers) without vanishing.

1x1 Convolutions

1x1 convolutions are used to reduce or increase the number of channels (depth) in a volume while adding non-linearity.

Inception Network

Inception modules apply 1x1, 3x3, 5x5 convolutions and pooling in parallel, then concatenate the results.

Object Detection

YOLO Algorithm

“You Only Look Once” divides an image into a grid (e.g., 19x19) and predicts bounding boxes and class probabilities for each cell simultaneously.

Anchor Boxes

Anchor boxes allow a single grid cell to detect multiple objects (e.g., a person standing in front of a car).

Non-Max Suppression

NMS filters out overlapping bounding boxes, keeping only the most confident ones.

Face Recognition

One-Shot Learning

Learning to recognize a person from just one image. This is often solved using a Siamese network that learns a similarity metric.

Similarity Metric

A function $f(A)$ that maps an image to a feature vector. The distance $d(A, B) = ||f(A) - f(B)||^2$ tells us how similar two faces are.

Visual Style Transfer

Style transfer creates a new image $G$ that combines the content of image $C$ with the artistic style of image $S$. This is achieved by minimizing a cost function with both content and style components.

Word Embeddings

Word embeddings represent words as dense vectors where similar words have similar vectors.

• Analogies: $e_{king} - e_{man} + e_{woman} \approx e_{queen}$.
• Cosine Similarity: A common measure of vector similarity.

word2vec

A popular algorithm for learning word embeddings by predicting a word from its neighbors (or vice versa).