In the past, I have used FOMA and TRmorph as a building block and basis for Ottoman conversion. However, I saw that writing something on top of a morphological analyzer to convert Ottoman to Turkish requires almost another morphological analyzer. (This is also true for Turkish to Ottoman conversion as well, because spelling rules of Ottoman require another layer of FSTs.)

After that initial failure, I decided to write an ad-hoc algorithm that uses surface-level representations of Turkish and Ottoman. The result has been running in dervaze for two years now.

But as it can be observed, it’s slow. I was working to replace it with a C library that will also be used in mobile applications. I have completed the dictionary part that’s used to search for words. In this version of the library, I was using trie structures.

I have decided, however, that using FSTs to translate may be more robust and formal in the long run. When you use tries, it’s almost like using half of FSTs without much thought on states. As I’m developing this solo and my main concern is expandability, converting ad-hoc rules to states seemed easier than writing a C library.

I will be writing about this experience in the coming days.

I’ve learned that there is a specialized Common Lisp implementation for mobile devices, mocl.

It should also be possible to write C-compatible shared libraries in CL and use them in iOS and Android. Although I began to prefer writing libraries in plain C (instead of any other language), I may switch to CL if there is enough motivation.

The technology for Ottoman OCR was mostly ready before my family obligations interrupted the project. I need to re-evaluate what is currently available, but a more pressing problem for me is the speed of translation; currently, it is so slow that it’s barely usable.

I have written the dictionary as a C library without any database dependencies and integrated it into the Android version. I’m currently updating the search functionality in the mobile app to use this library instead of a web service. It will be orders of magnitude faster than the current version because it’s offline and uses a trie to store the words—making it both small and fast.

In my experience, having a single data structure with dedicated functions for transforming and indexing is much simpler than managing multiple data structures. In my case, the core structure is:

typedef struct _dervaze_lexical_item {
  int index;
  bstr latin_search_key;
  bstr latin;
  bstr visenc_search_key;
  bstr visenc_dotless_search_key;
  bstr visenc;
  bstr annotation;
  bstr meaning;
  bstr abjad;
  lexical_role role;
  int last_vowel;
  int props;
} dervaze_lexical_item;

visenc is our abbreviation for visual encoding, which is used to represent Arabic/Ottoman/Farsi words using basic ASCII letters. It is documented on its own page.

index is a unique identifier assigned to each word. Search keys for Latin, Visenc, and Dotless Visenc (e.g., searching for ﺥ using letters like ح, چ, or ج) are used to locate these lexical items via the tries mentioned earlier.

I’m intentionally avoiding UTF-8 or other Unicode encodings because terminal output is often not well-suited for displaying Arabic text correctly.

When I began writing this software, one of my goals was to enable searching for words by their traditional numeric values (Abjad). These are often used in classical Ottoman poetry to encode a date within a verse. For example, the letter Alif corresponds to 1, Ba (ب) to 2, and so on. To search for words by these numerals—for instance, typing 246 to find words with that total value—we store the value here as well.

The lexical_role is used during translation; currently, we use two roles to distinguish between noun and verb suffixes in Turkish.

last_vowel, as the name implies, represents the last vowel of the word. Since vowel harmony in Turkish is not always reflected in Ottoman spelling, we check the last vowel when converting an Ottoman word to its Latin Turkish equivalent to ensure the correct suffix is added (e.g., gemiler instead of gemilar).

props is a bit field used to represent various properties of the lexical item, such as:

#define HAS_FINAL_VOWEL 0x01
#define HAS_SINGLE_VOWEL 0x02
#define IS_LAST_VOWEL_HARD 0x04
#define IS_FINAL_CONSONANT_HARD 0x08
#define HAS_CONSONANT_SOFTENING 0x10

These properties are particularly important when converting Ottoman text to modern Turkish.

The current version of the translation engine is written in Python, but I plan to rewrite it in C this week. As this is the third rewrite, I don’t expect it to pose a major algorithmic challenge, although debugging the C version may be more demanding.

Here, I describe the transliteration system. The system is organized as a pipeline in which the tools at a stage produce the input for the next stage. The input to the system is a set of historical document images. The output is either a search result or a textual representation of these documents.

The sections below describe these stages briefly.

Binarize Color Images to Binary Images

Binarization is the process of converting color or grayscale images to black-and-white binary images. Document images come in various flavors, mostly as color images. Color information is mostly noise for further stages. It is better to remove the color while keeping the textual representation intact.

Although seemingly easy at first, this stage includes challenges like determining the ink color or removing ink stains from images.

In the literature, the standard idea is to use a mathematical model, like Otsu’s method, to convert color to binary. We approach this problem differently, as a classification problem. The idea is briefly as follows:

Color images consist of 3 channels. The ink color of a region should be persistently present or absent in these channels. For example, a dark blue ink should be represented within similar numeric ranges in each of these channels, and a red ink should be represented more in the red channel than others. A standard binarization approach tries to come up with a cumulative ink color value using all three channels. We do it differently.

Instead of trying to find a cumulative threshold for binarization, we detect edges in each channel, considering each channel as a separate binary image. When components in each channel are found, they are evaluated by various features (like size, presence in other channels) and classified as text or non-text.

After classification, the text elements are drawn to a canvas in black, and the document hence becomes binarized.

Extract Components: Convert Binary Images to Components

Although the components are extracted during the binarization stage, we extract them in an independent stage to have a definite input and output. The primary reason for this is to evaluate the performance of different binarization options. This stage works even if the binarization part uses another (standard) approach.

Binarized document images are converted to sets of components by finding their edges. Each component is recorded with its location and binary image.

Extract Features: Find Features of Components for Comparison

Each component can have several different features, such as height, width, number of holes, number of ascenders and descenders, etc. Some of these features work better than others in classification. However, we don’t know beforehand which ones work better than others.

In order to find a set of good features, we extract all the features we can think of and run a Principal Component Analysis (PCA) on them. Previously, since we lacked classifications for these components, it was impossible to find good features, and we had to check the outputs manually. However, now that we have labeled around 11,000 components from 50 handwritten pages, we can determine which features are better than others.

extract-features extracts a large set of features from the component set. These features are stored in CSV files and analyzed for their classification value.