There are basically two problems to address: NumPy arrays often have data types with more than 8 bits, and OpenCV reads images in BGR format rather than the more common RGB.

The following Python (3.5+) code demonstrates the solution:

import PySide2.QtGui as qtg
import numpy as np

def get_qimage(image: np.ndarray):
    assert (np.max(image) <= 255)
    image8 = image.astype(np.uint8, order='C', casting='unsafe')
    height, width, colors = image8.shape
    bytesPerLine = 3 * width

    image = qtg.QImage(image8.data, width, height, bytesPerLine,
                       qtg.QImage.Format_RGB888)

    image = image.rgbSwapped()
    return image

The feature is called patch histogram, and it represents the histogram of 3x3 patches of black and white pixels. We collect all 3x3 patches and count their frequencies.

A 3x3 patch for a binary image contains $2^9 = 512$ different combinations. For each of these combinations, we assign a unique ID. I wrote the implementation in Python; here is a lookup table that converts all possible 3x3 patches to their IDs.

patch_histogram_dict = {}
for f in range(0, 512):
    i22 = f & 1
    i21 = (f >> 1) & 1
    i20 = (f >> 2) & 1
    i12 = (f >> 3) & 1
    i11 = (f >> 4) & 1
    i10 = (f >> 5) & 1
    i02 = (f >> 6) & 1
    i01 = (f >> 7) & 1
    i00 = (f >> 8) & 1
    patch_histogram_dict[ma(i00, i01, i02, i10, i11, i12, i20, i21, i22)] = f

I was looking for a skeleton feature that can be applied to components after a medial axis transform. However, in this version, we will use binarized connected component images directly and explore the skeletal version later.

Above, we used a function ma to convert a set of numbers to a tuple of tuples. It’s as simple as:

def ma(i00, i01, i02, i10, i11, i12, i20, i21, i22):
    return ((i00, i01, i02),
            (i10, i11, i12),
            (i20, i21, i22))

The first step in feature generation is marking each pixel of the image with its ID. Then, a simple np.histogram call generates the histogram.

for i in range(1, rows - 1):
    for j in range(1, cols - 1):
        s = framed[(i-1):(i+2), (j-1):(j+2)]
        marks[i-1, j-1] = patch_dict[ma(s[0, 0], s[0, 1], s[0, 2],
                                        s[1, 0], s[1, 1], s[1, 2],
                                        s[2, 0], s[2, 1], s[2, 2])]

framed is a copy of the original image with a 1-pixel frame, so that boundary pixels are also counted without much boundary checking. patch_dict is a parameter that the function receives. Its default value is the dictionary that we created above, but any kind of patch dictionary can be used.

The feature is generated using a call to numpy.histogram as follows:

histogram = numpy.histogram(marked_points,
                            hist_range,
                            density=True,
                            bins=bins)

marked_points are the results of the pixel counting loop above. hist_range is the minimum and maximum value for the patch dictionary. The density parameter is set to True so that the histogram is normalized to 1, making the feature size-invariant. bins is a parameter we can set freely, but by default, it’s equal to the range of elements.

Comparisons can be performed using standard histogram comparison methods, such as EMD (Earth Mover’s Distance) or Chi-Square distance metrics.

A moment is defined as:

$$m_{p,q}(x,y) = \int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} x^p y^q f(x,y) dxdy$$

In other words, it is the summation of the figure with respect to the function $f$ for both axes.

Q: What are Zernike moments?

Zernike moments are complex polynomial functions used to sum the elements of a shape. They were first introduced in the 1930s. The higher the order, the more complex the shape that can be represented. Order 1 Zernike moments (ZMs) are ellipsoid planes where one side is higher than the other.

Q: What is the difference between Hu moments and Zernike moments?

The importance of Zernike moments lies in their rotational invariance. However, Hu moments are also said to have these properties. Thus, Hu moments appear to be simpler alternatives to Zernike moments.

Q: What are their properties?

Zernike moments use polar coordinates, making it easy to describe rotational invariance.

Here, I describe the transliteration system. The system is organized as a pipeline in which the tools at a stage produce the input for the next stage. The input to the system is a set of historical document images. The output is either a search result or a textual representation of these documents.

The sections below describe these stages briefly.

Binarize Color Images to Binary Images

Binarization is the process of converting color or grayscale images to black-and-white binary images. Document images come in various flavors, mostly as color images. Color information is mostly noise for further stages. It is better to remove the color while keeping the textual representation intact.

Although seemingly easy at first, this stage includes challenges like determining the ink color or removing ink stains from images.

In the literature, the standard idea is to use a mathematical model, like Otsu’s method, to convert color to binary. We approach this problem differently, as a classification problem. The idea is briefly as follows:

Color images consist of 3 channels. The ink color of a region should be persistently present or absent in these channels. For example, a dark blue ink should be represented within similar numeric ranges in each of these channels, and a red ink should be represented more in the red channel than others. A standard binarization approach tries to come up with a cumulative ink color value using all three channels. We do it differently.

Instead of trying to find a cumulative threshold for binarization, we detect edges in each channel, considering each channel as a separate binary image. When components in each channel are found, they are evaluated by various features (like size, presence in other channels) and classified as text or non-text.

After classification, the text elements are drawn to a canvas in black, and the document hence becomes binarized.

Extract Components: Convert Binary Images to Components

Although the components are extracted during the binarization stage, we extract them in an independent stage to have a definite input and output. The primary reason for this is to evaluate the performance of different binarization options. This stage works even if the binarization part uses another (standard) approach.

Binarized document images are converted to sets of components by finding their edges. Each component is recorded with its location and binary image.

Extract Features: Find Features of Components for Comparison

Each component can have several different features, such as height, width, number of holes, number of ascenders and descenders, etc. Some of these features work better than others in classification. However, we don’t know beforehand which ones work better than others.

In order to find a set of good features, we extract all the features we can think of and run a Principal Component Analysis (PCA) on them. Previously, since we lacked classifications for these components, it was impossible to find good features, and we had to check the outputs manually. However, now that we have labeled around 11,000 components from 50 handwritten pages, we can determine which features are better than others.

extract-features extracts a large set of features from the component set. These features are stored in CSV files and analyzed for their classification value.

Keywords:

• dominant points
• consecutive vectors
• toothbrush shape
• distance metric
• smallest perpendicular distance

Q1: How usual calculation of distance is done?

Minor DPs are deleted in approximation. A minor DP is a DP where the perpendicular distance between the point and the straight line is minimum.

a a
 b

Here b is deleted when its distance to the line a-a is minimum.

The perpendicular distance is calculated using

[ d_i = \sqrt{\frac{((x_i - x_a) (y_b - y_a) - (y_i - y_a) (x_b -x_a))^2}{(x_a - x_b)^2 + (y_a - y_b)^2}} ]

for lines between points $p_a$ and $p_b$ and the point $p_i$.

Q2: What is a toothbrush shape?

It’s something like

aaaaaa
bbbbbbbbbbbbbbbbbbbbbb

Hence the toothbrush.

Though I don’t get why is this particularly important.

Q3: Which information is included in distance metric?

Angle acuteness is added to the information described above.

Q4: How the distance metric differs from others?

It includes angle acuteness in the metric and the more acute the angle, the less likely it’s removed from DP set.

Q5: What’s baseline for performance and how does this improve it?

As the number of DPs decrease, RMSE of the new metric decreases. For large number of DPs it doesn’t matter much. (So performance penalty may not pay off)

URL: http://arxiv.org/pdf/0704.1267.pdf

Keywords:

• page segmentation
• overlapping components
• image quality
• document complexity
• preprocessing
• projection based
• smearing based
• grouping based
• hough transform based
• repulsive attractive
• stochastic
• touching components

Q1: What are the most usable techniques for Ottoman divans?

Likforman-Sulem and Faure’s technique, which uses Gestalt criteria to associate text elements, might be of use. Feldbach and Tennies’ work, which was tested on Church Registers, may also be helpful. The Hough transform can be used. The Repulsive-Attractive method of Öztop et al. is also applicable. Stochastic methods by Tseng and Lee, which use a probabilistic Viterbi algorithm, can also be utilized.

Q2: How are touching components successfully delimited?

A touching component can be detected by its size. Subsequently, it should either be assigned to a lower or upper line, or be separated. Successful separation requires letter images or skeletons (which we lack).

Q3: How is the Hough transform used?

Centroids of the connected components (CCs) are used as units of the Hough transform. Line hypotheses are developed in the Hough domain and verified in the image domain.

Q4: What are the problems specific to non-Latin texts?

The baseline of Hebrew is at the upper part of the letters because of their box shape. Devanagari and similar scripts also have a headline on top of them. Diacritics and inter-letter shapes pose problems for Arabic.