The feature is called patch histogram, and it represents the histogram of 3x3 patches of black and white pixels. We collect all 3x3 patches and count their frequencies.

A 3x3 patch for a binary image contains $2^9 = 512$ different combinations. For each of these combinations, we assign a unique ID. I wrote the implementation in Python; here is a lookup table that converts all possible 3x3 patches to their IDs.

patch_histogram_dict = {}
for f in range(0, 512):
    i22 = f & 1
    i21 = (f >> 1) & 1
    i20 = (f >> 2) & 1
    i12 = (f >> 3) & 1
    i11 = (f >> 4) & 1
    i10 = (f >> 5) & 1
    i02 = (f >> 6) & 1
    i01 = (f >> 7) & 1
    i00 = (f >> 8) & 1
    patch_histogram_dict[ma(i00, i01, i02, i10, i11, i12, i20, i21, i22)] = f

I was looking for a skeleton feature that can be applied to components after a medial axis transform. However, in this version, we will use binarized connected component images directly and explore the skeletal version later.

Above, we used a function ma to convert a set of numbers to a tuple of tuples. It’s as simple as:

def ma(i00, i01, i02, i10, i11, i12, i20, i21, i22):
    return ((i00, i01, i02),
            (i10, i11, i12),
            (i20, i21, i22))

The first step in feature generation is marking each pixel of the image with its ID. Then, a simple np.histogram call generates the histogram.

for i in range(1, rows - 1):
    for j in range(1, cols - 1):
        s = framed[(i-1):(i+2), (j-1):(j+2)]
        marks[i-1, j-1] = patch_dict[ma(s[0, 0], s[0, 1], s[0, 2],
                                        s[1, 0], s[1, 1], s[1, 2],
                                        s[2, 0], s[2, 1], s[2, 2])]

framed is a copy of the original image with a 1-pixel frame, so that boundary pixels are also counted without much boundary checking. patch_dict is a parameter that the function receives. Its default value is the dictionary that we created above, but any kind of patch dictionary can be used.

The feature is generated using a call to numpy.histogram as follows:

histogram = numpy.histogram(marked_points,
                            hist_range,
                            density=True,
                            bins=bins)

marked_points are the results of the pixel counting loop above. hist_range is the minimum and maximum value for the patch dictionary. The density parameter is set to True so that the histogram is normalized to 1, making the feature size-invariant. bins is a parameter we can set freely, but by default, it’s equal to the range of elements.

Comparisons can be performed using standard histogram comparison methods, such as EMD (Earth Mover’s Distance) or Chi-Square distance metrics.

A moment is defined as:

$$m_{p,q}(x,y) = \int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} x^p y^q f(x,y) dxdy$$

In other words, it is the summation of the figure with respect to the function $f$ for both axes.

Q: What are Zernike moments?

Zernike moments are complex polynomial functions used to sum the elements of a shape. They were first introduced in the 1930s. The higher the order, the more complex the shape that can be represented. Order 1 Zernike moments (ZMs) are ellipsoid planes where one side is higher than the other.

Q: What is the difference between Hu moments and Zernike moments?

The importance of Zernike moments lies in their rotational invariance. However, Hu moments are also said to have these properties. Thus, Hu moments appear to be simpler alternatives to Zernike moments.

Q: What are their properties?

Zernike moments use polar coordinates, making it easy to describe rotational invariance.

There are two problems here. In Arabic-based writing systems, dots play a significant role, much more so than in Latin-based scripts. Therefore, these dots should be classified correctly.

The second problem is that the connected components are not always reliable. There are unduly divided components which are part of a single component. We can’t label them as they are, and uniting them into a uniform component requires manual intervention—something we try to avoid.

In the coming days, I’ll try to exemplify these problems and how we treat them.

Keywords:

• ridge
• printed text
• non-text segmentation
• gaussian-filter bank
• reading order

Q1: How is line skew determined?

There is a $\theta$ parameter in the Gaussian kernel which is used to produce ridges. This may be used in detecting the skew, but since it’s constant for an entire page, a varying line skew will probably decrease its performance.

Q2: How are non-text portions detected?

The paper does not include a description but cites “S. S. Bukhari, F. Shafait, and T. M. Breuel, ‘Improved document image segmentation algorithm using multiresolution morphology,’ in Proc. SPIE Document Recognition and Retrieval XVIII, San Jose, CA, USA, Jan. 2011” as a source for an improved technique.

Q3: Which heuristics are used in reading order determination?

Breuel is reported to have an algorithm in “T. M. Breuel, ‘High performance document layout analysis,’ in Symposium on Document Image Understanding Technology, Greenbelt, MD, USA, April 2003.” The paper says the authors modified the algorithm for right-to-left scripts. No further details are provided.

Q4: How large is the dataset, and what does it contain?

25 Arabic documents and 20 Urdu documents are used.

Q5: Are there any techniques applicable to divans?

There might be, if any of them were described in detail. We already have more sophisticated text line detection techniques. For the others, I’ll need to read the cited works.

Keywords:

• handwritten
• paleography
• fragments
• SIFT
• sparse coding
• dictionaries

Q1: What is the ultimate goal of the authors?

The two main goals are providing tools to bring together fragments of the same page (specifically from the Cairo Genizah) and trying to classify handwriting and dates.

Q2: How is SIFT used?

SIFT is used at various points of a letter to generate descriptors. There are 100,000 descriptors overall before inputting them into k-means. SIFT serves as the main classification technique.

Q3: How did they produce the letter dictionaries?

They produced letter dictionaries using the generated SIFT descriptors and k-means clustering to find representative visual words.

Q4: What is sparse coding and its importance?

Sparse coding is used to code documents (or any other codable thing) with a separating/descriptive code which also shows the similarity between items. An example might be the bag of visual words approach.

Q5: Are there any relevant techniques for our research?

This is relevant to the Divan matching problem. This work might be cited in historical document matching, although it covers techniques that are generally well-known in the field.

Keywords:

• LMS estimator
• O(n logn)
• bracelet
• slab
• random
• approximation
• quantiles

Q1: What is LMS?

Given a set of points $p_0, …, p_n$, LMS finds a line $q_0, q_1$ that minimizes the median of the square of distances of $p_0, …, p_n$. This is in contrast with summing up all the squared distances and minimizing them as in OLS (Ordinary Least Squares).

Q2: How are approximations done using LMS?

There are exact solutions for the LMS problem. This paper presents an algorithm with lower complexity. It tries to find an LMS approximation in a band defined by a parameter $\epsilon_r$.

Q3: On which parameters does the algorithm depend?

The algorithm approxLMS depends on a set of lines, a set of quantiles, and two error bounds $\epsilon_r$ and $\epsilon_q$.

Q4: How can this help in our line approximations?

This paper is aimed towards providing an efficient algorithm for random approximations. Since we need repeatability in our keypoint detection, it might be harder (or impossible) to prove that the algorithm produces the exact same line endings in each run with a similar set of points. Hence, it’s not usable in our studies.

Keywords:

• Stereo image
• descriptor
• circle
• quantization
• formalization
• binary mask
• Depth estimation

Q1: How is depth estimation related to object recognition?

Objects are located in a 3D environment, and in order to recognize them correctly, we need to be able to recreate their layout in a scene. With such an aid, we can successfully determine the object boundaries.

Q2: What does the descriptor contain?

It is a concatenation of vectors. The first vector is the Gaussian of the center point with a $\Sigma_0$, the second set of vectors are circles lying on circle $R_1$, the third set of vectors are circles lying on circle $R_2$… Each vector contains orientation maps after a Gaussian convolution.

Q3: On which datasets did the authors try the technique?

As far as I can tell, it is a custom dataset that contains the view of the same scene from many perspectives.

Q4: What is the salience criterion for keypoints?

The aim of the technique is not matching these keypoints to each other by selecting the most appropriate ones. The computation is done on all pixels/keypoints. Hence, no criterion for keypoint filtering is reported.

Authors: Alexandre Alahi, Raphael Ortiz, Pierre Vandergheynst

Keywords:

• Keypoint
• Binary descriptor
• Retina
• Sampling
• Saccadic
• Coarse-to-fine
• Orientation

Q1: What is the formula for the retina pattern?

The one difference from BRISK is that the pattern has overlapping circles. In BRISK, they were tangential. Redundancy increases recognition.

The circles are log-polar. In this case, it is similar to Shape Context descriptors, but we do not divide into regions; we create increasingly larger circles on polar lines.

Q2: What do the descriptors contain?

A binary descriptor is a string of bits. A bit corresponds to a pair of receptive fields. If the intensity of the first receptive field is larger than the second, the bit is set to 1; otherwise, it is zero.

Q3: How does the sampling work?

Each circle in the pattern is called a receptive field. A Gaussian kernel is applied to these fields, and their intensities are calculated.

Q4: Is there scale invariance? How?

There is no discussion of scale invariance, but scale invariance seems to arise from the building of the descriptor. Since the circles are created in log-polar orbits and bits are put into the descriptor according to their contribution to recognition, scale invariance follows these.

Q5: How is rotation invariance achieved?

Orientation is calculated using 45 symmetric pairs from the center. It has larger steps than those of BRISK and thus needs lower memory.