🐢 Now, I have two classes. This one is a copy of the ONNX Patcher that’s used to convert ColQwen2 to ONNX format. But this one is valid only for the underlying model, that is Qwen2VLForConditionalGeneration, and ColQwen2 also has some modifications to call it.

🐇 What are these modifications?

🐢 The forward method looks something like this:

ColQwen2

    def forward(self, *args, **kwargs) -> torch.Tensor:
        kwargs.pop("output_hidden_states", None)

        # Handle the custom "pixel_values" input obtained with `ColQwen2Processor` through unpadding
        if "pixel_values" in kwargs:
            offsets = kwargs["image_grid_thw"][:, 1] * kwargs["image_grid_thw"][:, 2]  # (batch_size,)
            kwargs["pixel_values"] = torch.cat(
                [pixel_sequence[:offset] for pixel_sequence, offset in zip(kwargs["pixel_values"], offsets)],
                dim=0,
            )

        position_ids, rope_deltas = self.get_rope_index(
            input_ids=kwargs["input_ids"],
            image_grid_thw=kwargs.get("image_grid_thw", None),
            video_grid_thw=None,
            attention_mask=kwargs.get("attention_mask", None),
        )
        last_hidden_states = self.inner_forward(
            *args, **kwargs, position_ids=position_ids, use_cache=False, output_hidden_states=True
        )  # (batch_size, sequence_length, hidden_size)

        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

        # L2 normalization
        proj = proj / proj.norm(dim=-1, keepdim=True)  # (batch_size, sequence_length, dim)
        proj = proj * kwargs["attention_mask"].unsqueeze(-1)  # (batch_size, sequence_length, dim)

        if "pixel_values" in kwargs and self.mask_non_image_embeddings:
            # Pools only the image embeddings
            image_mask = (kwargs["input_ids"] == self.config.image_token_id).unsqueeze(-1)
            proj = proj * image_mask
        return proj

🐇 How about the inner_forward method? Does it just directly call Qwen2VLForConditionalGeneration’s forward?

🐢 No, it has some conditionals:

        if inputs_embeds is None:
            inputs_embeds = self.model.embed_tokens(input_ids)
            if pixel_values is not None:
                pixel_values = pixel_values.type(self.visual.get_dtype())
                image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw)
                image_mask = (input_ids == self.config.image_token_id).unsqueeze(-1).expand_as(inputs_embeds)
                image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)

            if pixel_values_videos is not None:
                pixel_values_videos = pixel_values_videos.type(self.visual.get_dtype())
                video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw)
                video_mask = (input_ids == self.config.video_token_id).unsqueeze(-1).expand_as(inputs_embeds)
                video_embeds = video_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_embeds)

            if attention_mask is not None:
                attention_mask = attention_mask.to(inputs_embeds.device)

        outputs = self.model(
            input_ids=None,
            position_ids=position_ids,
            attention_mask=attention_mask,
            past_key_values=past_key_values,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]
        return hidden_states

🦊 The patcher only modifies the past_key_values_args and moves them to a DynamicCache. There are no other changes for patching.

🐢 But the return types are also different. Qwen2VLForConditionalGeneration.forward returns a Union[Tuple, Qwen2VLCausalLMOutputWithPast], but ColQwen2.forward returns a torch.Tensor.

🐇 How is that torch.Tensor calculated from the return type of Qwen2VLForConditionalGeneration? ❓

🦊 There are multiple return types in Qwen2VLForConditionalGeneration. It’s determined by the return_dict argument.

        if not return_dict:
            output = (logits,) + outputs[1:]
            return (loss,) + output if loss is not None else output

        return Qwen2VLCausalLMOutputWithPast(
            loss=loss,
            logits=logits,
            past_key_values=outputs.past_key_values,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
            rope_deltas=self.rope_deltas,
        )

🐇 What’s that argument in ColQwen2?

🐢 It’s passed from the caller; it’s a kwarg.

🐇 Is there any modification to this in the ONNX patcher?

🐢 Nope.

🐇 Then we can consider it as the default. What’s the default?

🐢 The default is None. Hence it returns (logits,) + outputs[1:].

🐇 Then, logits are the first parameter by default.

🐢 Yes, we can assume this.

🐇 What does ColQwen2 do with these logits?

🦊 By the way, it calls inner_forward with use_cache=False and output_hidden_states=True. What do these change in Qwen2VLForConditionalGeneration?

🐢 These hidden states are actually output. In ColQwen2.forward, the inner_forward call is actually:

        last_hidden_states = self.inner_forward(
            *args, **kwargs, position_ids=position_ids, use_cache=False, output_hidden_states=True
        )  # (batch_size, sequence_length, hidden_size)

and the output is the hidden states.

🦊 It then runs:

        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

and calculates a set of projections.

🐢 custom_set_proj is defined as:

        self.custom_text_proj = nn.Linear(self.model.config.hidden_size, self.dim)

hence these projections are fully connected layer calculations.

🦊 It returns these after L2 normalization and checks if there are pixel_values to consider.

🐢 In this case, the output from ColQwen2 is this single tensor.

🐇 Ok. Then it’s actually simpler than wrapping up the whole Qwen2VLForConditionalGeneration.

🐢 It looks so, yes. But the example for ColQwen2 also has a post-processing step:

scores = processor.score_multi_vector(query_embeddings, image_embeddings)

🐇 I think we can leave that post-processing for the time being. We only need multi-vector embeddings for FastEmbed.

🐢 Maybe it’s convertible to a forward method that we can use with ONNX.

🐇 Let’s see how this scoring works, then.

🐢 score_multi_vector receives two tensors or tensor lists and compares query vectors with passage vectors. It’s a rather straightforward implementation that requires Torch, but not Transformers.

🐇 In theory, we can also convert this to an ONNX model.

🦊 We can also write a custom model for this.

🐢 It has two for loops for comparisons. Can we convert all of these?

🦊 The loop indices can be seen as dynamic_axes, and it’s possible to convert the whole thing as a Torch model, then use torch.onnx.export just as we do for the model itself.

🐢 I see, but I don’t think that’s what we must do now.

🐇 Yes, let’s skip that for the time being and convert the model itself. We’ll have multivectors for patches and queries at the end.

🐢 So, we’ll keep the processing part, send BatchFeature objects that are output from the processor, and send this to two models: one for images and one for text.

🐇 Yep, that’s the plan. In the end, we’ll have two ONNX models that require BatchFeatures.

🐢 Then, we’ll modify past_key_values in this argument to use DynamicCache.

🐇 Yes, that’s alright. We can start by moving past_key_values_converter to a method in PatchedColQwen2.

🐢 past_key_values wasn’t used much, so I completely removed it. I also began to use processor outputs as dummy input in the exporter. However, when using BatchFeature, we get:

RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: BatchFeature

🐇 So, we can try Dynamo first, I think. If that doesn’t work, we can just collect items as tensors and build up a BatchFeature inside the patcher.

🐢 Let’s try dynamo=True for this first.

🦊 This time, it’s about BatchFeature again, but the error is different: KeyError: 'Indexing with integers is not available when using Python based feature extractors'

🐇 In this case, we can just create BatchFeature inside the patcher.

PatchedColQwen2

class PatchedColQwen2(ColQwen2):
    def forward(self, *args):
        (
            input_ids,
            inputs_embeds,
            attention_mask,
            position_ids,
            *past_key_values_args,
        ) = args
        # Convert past_key_values list to DynamicCache
        if len(past_key_values_args) == 0:
            past_key_values = None
        else:
            past_key_values = DynamicCache()
            for i in range(self.config.num_hidden_layers):
                key = past_key_values_args.pop(0)
                value = past_key_values_args.pop(0)
                past_key_values.update(key_states=key, value_states=value, layer_idx=i)

        breakpoint()
        o = super().forward(
            input_ids=input_ids,
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            # position_ids=position_ids,
            past_key_values=past_key_values,
        )

        flattened_past_key_values_outputs = {
            "logits": o.logits,
        }
        output_past_key_values: DynamicCache = o.past_key_values
        for i, (key, value) in enumerate(
            zip(output_past_key_values.key_cache, output_past_key_values.value_cache)
        ):
            flattened_past_key_values_outputs[f"present.{i}.key"] = key
            flattened_past_key_values_outputs[f"present.{i}.value"] = value

        return flattened_past_key_values_outputs

It’s possible to use “missing” as a label for categorical variables when their frequency is high or relevant. Otherwise, it may be seen as adding another rare categorical variable to the dataset.

Using random values for missing data

Random sample imputation can be used to fill missing entries by selecting random values from those already present in the dataset.

Using this approach feels a bit unusual to me. However, when you are looking for a prediction and not all values are present in your sample, you can estimate a value based only on the supplied data.

For random variables, we need to generate values by seeding the random number generator with the given values. This ensures we obtain the same values for random variables whenever the same inputs are supplied, providing a degree of consistency.

Binary Classification

Given an image, classify it as cat or non-cat. The result is $\hat{y} = P(y=1 | x)$. In other words, given $x$, we calculate the probability that this data represents a cat.

Feature Vector from Image

We convert an image, e.g., a (64, 64, 3) image, into a (64 * 64 * 3, 1) feature vector.

Sigmoid

$$\hat{y} = \sigma(w^Tx + b)$$

$w$ represents the weights, $w^T$ is the transpose of $w$, $x$ is the input, and $b$ is the bias.

Loss Function

The loss function measures the error between the real value of $y$ and our prediction $\hat{y}$ for a single training example.

$$ L(\hat{y}, y) = \frac{1}{2}(\hat{y}-y)^2 $$

Cost Function

The average of the loss across all training examples.

Learning Rate

The learning rate $\alpha$ is the coefficient applied to update weights:

$$w’ = w - \alpha \frac{dd(w)}{dw}$$

If it is too small, learning occurs slowly; if it is too large, it may overshoot the optimum point. Thus, it should be selected wisely.

IDEA: Can we use a vector instead of a single value for the learning rate?

For Leaky ReLU activation functions, this seems possible, though likely overkill. Using $\alpha$ as a vector adds a layer of complexity and requires more computation to adjust (allowing some features to update more slowly than others).

The Feature Vector

$X$ is a feature matrix with $m$ columns, each representing a different training example. Each example has $n$ features. So, X.shape = (n, m).

Y.shape = (1, m).

The Computation Graph

Any formula can be converted into a graph where operands are vertices and operations are edges.

In neural networks, computation flows forward from inputs to output, while the derivatives are calculated from output back to input.

The Notation for Layers and Training Examples

$X^{(i)}$ refers to the $i^{th}$ training example.

$z^{[i]}$ refers to the $z$ values in the $i^{th}$ layer.

$z^{[1]}$ is the $z$ value for layer 1.

$a^{[2]}$ is the $a$ value for layer 2.

$a^{[1]} = [a^{[1]}_1, a^{[1]}_2, a^{[1]}_3, a^{[1]}_4]^T$ for a neural net having 4 nodes in hidden layer 1.

Steps of Computation

There are two steps of computation:

1. $z^{[1]} = w^{[1]T}x + b^{[1]}$
2. $a^{[1]} = \sigma(z^{[1]})$

In step 1, weights and inputs are dot-multiplied and the bias is added. In step 2, an activation function is applied to this result. While $\sigma$ is used here, other functions can also serve this purpose.

Backpropagation Algorithm

This is the fundamental algorithm that updates weights to find a solution.

In the forward pass, the cost function $L(\hat{y}, y)$ is computed.

In backpropagation, weights and biases are adjusted based on their derivatives multiplied by the learning factor $\alpha$.

While the actual formulas can be complex, the general idea is to calculate derivatives from the last layer back to the first and adjust weights accordingly.

Vectorization Notation

Multiple training examples are denoted by the superscript $(i)$.

The vectorized representation of the $Z$ matrix for multiple inputs and multiple layers is:

$$Z = \begin{pmatrix} z^{1} & z^{1} & z^{1} & \dots & z^{1} \ z^{2} & z^{2} & z^{2} & \dots & z^{2} \ z^{3} & z^{3} & z^{3} & \dots & z^{3} \ \dots & \dots & \dots & \dots & \dots \ z^{n} & z^{n} & z^{n} & \dots & z^{n} \ \end{pmatrix}$$

The $A$ matrix is structured similarly.

Since $w^{[i]T}x + b^{[i]}$ is a column vector, we concatenate these column vectors for multiple inputs.

Activation Functions

There are roughly four types of activation functions:

Sigmoid

This is the historic default. There is generally no need to use it except in the final output layer for binary classification (0 or 1).

Tanh

It is generally superior to the sigmoid function. It is asymptotic between 1 and -1, which often leads to better training behavior.

ReLU

A simple function, $r = \max(0, x)$, which has become very popular.

Leaky ReLU

Since ReLU is not differentiable for $x < 0$, this version adds a small slope ($r = \max(0.01x, x)$) for negative values.

Training Set and Test Set

Traditionally, with datasets of 100 to 10,000 elements, splits like 70/30% or 60/20/20% were common.

However, in the era of Big Data™, where datasets may contain 10,000,000 elements, such percentages are unnecessary. It is more reasonable to keep a fixed number, such as 10,000 elements, for the dev and test sets to speed up development.

Crucially, this data should come from the same distribution. Dev and test sets are used to check for overfitting and evaluate real-world performance.

Bias and Variance

High bias indicates the model is underfitting. High variance indicates the model is overfitting.

High Bias

Example: Train Set Error Rate: 15%, Dev Set Error Rate: 16%. The model is too simple to learn the data. Solutions: Use a larger network, train longer, or change the NN architecture.

High Variance

Example: Train Set Error Rate: 1%, Dev Set Error Rate: 14%. The model overfits the training data and fails to generalize. Solutions: Get more data, use regularization, or change the NN structure.

High Bias and High Variance

While classical models often face a bias/variance tradeoff, Deep Learning models can suffer from both simultaneously (e.g., 15% Train Set error and 30% Dev Set error). Solutions: Larger network, more data, regularization, and architecture changes.

Regularization

Used to decrease variance and prevent overfitting. Two main types: L2 Regularization (adds a factor to the weights) and Dropout Regularization (randomly sets some weights to zero).

L2 Regularization

Without regularization: $$w^{[l]} \leftarrow w^{[l]} - \alpha (dw^{[l]})$$ With regularization, we add a decay factor: $$ -\alpha (\frac{\lambda}{2m} w^{[l]}) $$ The full formula becomes: $$w^{[l]} \leftarrow (1 - \alpha \frac{\lambda}{2m}) w^{[l]} - \alpha dw^{[l]}$$ This is called weight decay because it pulls $w$ closer to zero, effectively making the network “smaller.”

Dropout

Dropout is a technique where the algorithm randomly “knocks out” nodes during training. These nodes are temporarily ignored during weight updates.

In each iteration, a random subset of nodes is removed based on a keep-prob. For example, a keep-prob of 0.5 means roughly half the nodes are ignored.

Data Augmentation

Generating variations of existing images (e.g., flipping, rotating, adding noise) is also a form of regularization. Note: Changes shouldn’t alter the image’s meaning (e.g., a flipped “4” might not be a “4”).

Early Stopping

As training progresses, the cost function $J$ for the training set continues to decrease, but the test set $J$ eventually begins to increase, indicating overfitting. We can stop training at that inflection point.

Normalization

Normalization involves bringing all feature values $X$ into a similar range. If $x_1$ is between 1 and 1000 and $x_2$ is between 0 and 1, the network may struggle. A common approach is $x_i = \frac{x_i - \mu_i}{\sigma_i}$, where $\mu_i$ is the mean and $\sigma_i$ is the standard deviation.

Exploding and Vanishing Gradients

In deep networks, weight values may exponentially increase or decrease. If all weights are 2, the final activation in an $l$-layer network becomes $2^l$, leading to exploding gradients. Conversely, weights less than 1 can lead to vanishing gradients where the network fails to learn.

Weight Initialization to Alleviate Exploding/Vanishing Gradients

To prevent these issues, it is best to initialize weights with a variance of $\frac{1}{n}$, where $n$ is the number of nodes in a layer.

For ReLU:

W[i] = np.random.randn(shape) * np.sqrt(2 / n[i-1])

For tanh:

W[i] = np.random.randn(shape) * np.sqrt(1 / n[i-1])

Another option is Xavier initialization:

W[i] = np.random.randn(shape) * np.sqrt(2 / (n[i] + n[i-1]))

Initialization by Zero

Initializing weights to all zeros fails to break symmetry, and the network will not learn. However, it is perfectly fine to initialize biases to zero.

Initialization with Random Numbers

W[i] = np.random.randn(layer_dim[l], layer_dim[l-1]) * FACTOR

If FACTOR is too large (e.g., 10), the network may converge slowly or suffer from exploding gradients. Using np.sqrt(2 / layer_dim[l-1]) is known as He initialization, while np.sqrt(1 / layer_dim[l-1]) is Xavier initialization.

Mini-Batches

When dealing with massive datasets (e.g., 10 million images), we cannot process them all at once. Instead, we divide them into mini-batches.

• Stochastic Gradient Descent: Batch size of 1.
• Batch Gradient Descent: Batch size equals the entire training set.
• Mini-Batch Gradient Descent: Sizes typically range from 32 to 512.

Optimization: Momentum

When training with mini-batches, it helps to track previous gradients to smooth out updates. A hyperparameter $\beta$ determines the influence of past gradients.

Optimization: RMSprop

Instead of simple updates, RMSprop uses the squared gradients to scale the updates:

$$s_{dW} \leftarrow \beta_2 s_{dW} - (1 - \beta_2) (dW)^2$$ $$ W \leftarrow W - \alpha \frac{dW}{\sqrt{s_{dW} + \epsilon}} $$

This reduces oscillations and speeds up convergence.

Optimization: Adam

Adam combines Momentum and RMSprop. It uses two hyperparameters, $\beta_1$ and $\beta_2$, and is generally the most effective optimization algorithm, converging much faster than others.

Learning Rate Decay

A fixed learning rate $\alpha$ can make convergence difficult. We can decay the learning rate over time: $\alpha = \frac{1}{1 + d * t} \alpha_0$, where $d$ is the decay rate and $t$ is the epoch number.

Hyperparameter Tuning

Key hyperparameters include the learning rate $\alpha$, optimization coefficients $\beta_1$ and $\beta_2$, number of layers, units per layer, activation functions, and mini-batch sizes.

Heuristics:

• Random search is often better than a uniform grid.
• Search on a logarithmic scale for parameters like $\alpha$ or $\beta$ (e.g., searching $1-10^r$ for $\beta$).

Batch Normalization

Batch normalization normalizes layer activations using two learnable parameters, $\beta$ and $\gamma$. It helps information flow to deeper layers and makes the network less sensitive to initial weight scales.

Softmax Activation

For multi-class classification, the final layer uses a softmax function to output a probability vector: $$a_i^{[l]} = \frac{e^{z_i^{[l]}}}{\sum_{j=1}^C e^{z_j^{[l]}}}$$ where $C$ is the number of classes.

Metrics

Single-number metrics like F1 Score or mAP make evaluation much easier.

• Optimizing Metric: The primary number you try to improve.
• Satisficing Metric: A threshold that must be met (e.g., running time < 100ms).

Error Analysis

Regularly inspect a subset of misclassified examples to understand where the model is failing.

Data Mismatch

If your training and test data come from different distributions, use a training-dev set to identify if performance drops are due to the model failing to generalize or a fundamental difference in data.

Transfer Learning

You can take a model trained on task A and repurpose it for task B. This is especially useful when task B has limited data.

Multi-Task Learning

One network can be trained to perform multiple tasks simultaneously (e.g., detecting pedestrians AND traffic signs), provided the network is large enough.

End-to-End Learning

In end-to-end learning, the network maps raw input directly to the final output, bypassing traditional hand-engineered feature extraction steps. This requires significant amounts of data to be effective.

Convolutional Neural Networks (CNNs)

CNNs are the standard for image processing. They apply filters (kernels) to images to extract features. The resulting matrix size is determined by the input size $n$ and filter size $f$ (result size: $n-f+1$).

Padding

Padding adds a border of pixels (usually zeros) around the input to prevent the image from shrinking after each convolution layer.

Classic Networks

LeNet-5

A 1998 network designed for digit recognition (32x32x1). It is small (60k parameters) but foundational.

AlexNet and VGG-16

AlexNet (2012) popularized Deep Learning. VGG-16 (138M parameters) used a very consistent architecture of 3x3 filters and max-pooling.

ResNets

Residual Networks use “shortcuts” or “skip connections” to allow gradients to flow through very deep networks (100+ layers) without vanishing.

1x1 Convolutions

1x1 convolutions are used to reduce or increase the number of channels (depth) in a volume while adding non-linearity.

Inception Network

Inception modules apply 1x1, 3x3, 5x5 convolutions and pooling in parallel, then concatenate the results.

Object Detection

YOLO Algorithm

“You Only Look Once” divides an image into a grid (e.g., 19x19) and predicts bounding boxes and class probabilities for each cell simultaneously.

Anchor Boxes

Anchor boxes allow a single grid cell to detect multiple objects (e.g., a person standing in front of a car).

Non-Max Suppression

NMS filters out overlapping bounding boxes, keeping only the most confident ones.

Face Recognition

One-Shot Learning

Learning to recognize a person from just one image. This is often solved using a Siamese network that learns a similarity metric.

Similarity Metric

A function $f(A)$ that maps an image to a feature vector. The distance $d(A, B) = ||f(A) - f(B)||^2$ tells us how similar two faces are.

Visual Style Transfer

Style transfer creates a new image $G$ that combines the content of image $C$ with the artistic style of image $S$. This is achieved by minimizing a cost function with both content and style components.

Word Embeddings

Word embeddings represent words as dense vectors where similar words have similar vectors.

• Analogies: $e_{king} - e_{man} + e_{woman} \approx e_{queen}$.
• Cosine Similarity: A common measure of vector similarity.

word2vec

A popular algorithm for learning word embeddings by predicting a word from its neighbors (or vice versa).

Distributions

Suppose variable $A$ has 2, $B$ has 2, and $C$ has 3 possible values. Their joint probability distribution will contain $2 \times 2 \times 3 = 12$ values.

We can condition the distribution by setting a variable to a specific value.

We can also marginalize the distribution over a subset of variables to examine the distribution of a single variable.

Factors

A factor $\phi$ is a function that maps values of variables (e.g., $A$, $B$, and $C$) to a real number.

The set of variables that the factor takes as arguments is called the scope of the factor.

Both normalized and unnormalized measures are examples of factors, although their scopes may vary.

A Conditional Probability Distribution (CPD) is another common example of a factor.