The reason I had to stop was the lack of data, since without a considerable amount of data, statistical methods for both Natural Language Processing and Computer Vision fail. Producing and maintaining data seemed a much greater burden than providing technical solutions, so I mostly gave up the idea that a working solution is obtainable with classical OCR techniques. The research is still waiting for me to finish.

I am beginning this series of explorations in Computer Vision, Machine Learning, and related fields to document my achievements and provide a basis for further research. From time to time, I will provide results regarding the ideas discussed here. Currently, this is mostly a hobby/side project.

My current endeavor is to write a robust translation engine between Ottoman and Turkish in Dart.¹

Instead of performing a full-fledged morphological analysis like TRMorph, Google Research Morphological Analyzer, or Starlang Morphological Analyzer, our aim is to provide a surface-level to surface-level translation between Arabic and Turkish scripts.

It is true that it’s possible to use something like:

+-----------------+                +----------------+                 +---------------+
|                 |                |                |                 |               |
|                 |                |                |                 |               |
|    kelimeler    |    +------->   |  kelime+PLU    |  +----------->  |   کلمهلر      |
|                 |                |                |                 |               |
|                 |                |                |                 |               |
+-----------------+                +----------------+                 +---------------+

to translate between Turkish Latin and Ottoman, but there are two problems here:

1. There is no Ottoman morphological analyzer, and although grammatically Turkish, surface-level forms should be translated. Even morphological analyzers for Turkish in the Latin alphabet are relatively recent.

2. Translating the output of a Turkish Latin morphological analyzer to Ottoman seems like more work than writing a translation method itself.

For example, for the query kelimeler TRMorph gives

    kelime<N><pl>
    kel<Adj><0><N><p1s><dat><0><V><cpl:pres><3p>
    kel<Adj><p1s><Prn><dat><0><V><cpl:pres><3p>
    kelime<N><0><V><cpl:pres><3p>
    kelime<N><pl><0><V>
    kelime<N><pl><0><V><cpl:pres><3p>
    kelime<N><pl><0><V><cpl:pres><3s>

and although most of the items’ surface forms are identical, we need to work through all suffixes and their different kinds of connections. Also, as the surface form of Ottoman Turkish has less information, Ottoman morphological analysis would yield many more results than its corresponding counterpart.

Because of these hindrances and trying to come up with a quick-and-dirty yet workable solution, I have made the following observations:

1. Our part-of-speech system does not need to be highly specialized. Actually, we only need to have three classes: Nouns, verbs, and proper nouns. Proper nouns are grammatically nouns, but their orthography may require different rules. We will call these three word classes.

2. We can get away with a set of surface-level rules to translate suffixes for each word class. These rules use attributes that can be derived from Turkish Latin orthography.

These rules are:

• Part of Speech: The root class we discussed above.
• Last Vowel: The last vowel in a Turkish root that is required to find the actual suffix when vowel harmony is utilized.
• Last Consonant: The last consonant of the Turkish root. This is needed to find the palatalization/softening of certain consonants.
• Ends with Vowel: Whether the root ends with a vowel, like ata, or not.
• Has Single Vowel: Whether the root has a single vowel. This is important for aorist inflection irregularities in verbs.
• Last Vowel Hard: Whether the last vowel is one of a, ı, o, u (back vowels) or not.
• Last Consonant Hard: Whether the last consonant is one of p, ç, t, k and undergoes softening when it receives a suffix starting with a vowel.
• Has Consonant Softening: The inverse of the last consonant hard rule, occurring when a root ends with b, c, d, g and receives a suffix.

From these rules, which can be derived from the Turkish Latin forms of the words using regular expressions, we can write rules to translate Turkish to Ottoman and vice versa.

To be continued.

1. Dart has the little benefit of being able to run both on mobile (in Android and iOS) and servers with support for the web, so write once, run on Linux, iOS and Android is feasible without much architecture jumping. I hope Flutter gains more traction and becomes the mainstream way of writing mobile applications. ↩

In the past, I have used FOMA and TRmorph as a building block and basis for Ottoman conversion. However, I saw that writing something on top of a morphological analyzer to convert Ottoman to Turkish requires almost another morphological analyzer. (This is also true for Turkish to Ottoman conversion as well, because spelling rules of Ottoman require another layer of FSTs.)

After that initial failure, I decided to write an ad-hoc algorithm that uses surface-level representations of Turkish and Ottoman. The result has been running in dervaze for two years now.

But as it can be observed, it’s slow. I was working to replace it with a C library that will also be used in mobile applications. I have completed the dictionary part that’s used to search for words. In this version of the library, I was using trie structures.

I have decided, however, that using FSTs to translate may be more robust and formal in the long run. When you use tries, it’s almost like using half of FSTs without much thought on states. As I’m developing this solo and my main concern is expandability, converting ad-hoc rules to states seemed easier than writing a C library.

I will be writing about this experience in the coming days.

I’ve learned that there is a specialized Common Lisp implementation for mobile devices, mocl.

It should also be possible to write C-compatible shared libraries in CL and use them in iOS and Android. Although I began to prefer writing libraries in plain C (instead of any other language), I may switch to CL if there is enough motivation.

The technology for Ottoman OCR was mostly ready before my family obligations interrupted the project. I need to re-evaluate what is currently available, but a more pressing problem for me is the speed of translation; currently, it is so slow that it’s barely usable.

I have written the dictionary as a C library without any database dependencies and integrated it into the Android version. I’m currently updating the search functionality in the mobile app to use this library instead of a web service. It will be orders of magnitude faster than the current version because it’s offline and uses a trie to store the words—making it both small and fast.

In my experience, having a single data structure with dedicated functions for transforming and indexing is much simpler than managing multiple data structures. In my case, the core structure is:

typedef struct _dervaze_lexical_item {
  int index;
  bstr latin_search_key;
  bstr latin;
  bstr visenc_search_key;
  bstr visenc_dotless_search_key;
  bstr visenc;
  bstr annotation;
  bstr meaning;
  bstr abjad;
  lexical_role role;
  int last_vowel;
  int props;
} dervaze_lexical_item;

visenc is our abbreviation for visual encoding, which is used to represent Arabic/Ottoman/Farsi words using basic ASCII letters. It is documented on its own page.

index is a unique identifier assigned to each word. Search keys for Latin, Visenc, and Dotless Visenc (e.g., searching for ﺥ using letters like ح, چ, or ج) are used to locate these lexical items via the tries mentioned earlier.

I’m intentionally avoiding UTF-8 or other Unicode encodings because terminal output is often not well-suited for displaying Arabic text correctly.

When I began writing this software, one of my goals was to enable searching for words by their traditional numeric values (Abjad). These are often used in classical Ottoman poetry to encode a date within a verse. For example, the letter Alif corresponds to 1, Ba (ب) to 2, and so on. To search for words by these numerals—for instance, typing 246 to find words with that total value—we store the value here as well.

The lexical_role is used during translation; currently, we use two roles to distinguish between noun and verb suffixes in Turkish.

last_vowel, as the name implies, represents the last vowel of the word. Since vowel harmony in Turkish is not always reflected in Ottoman spelling, we check the last vowel when converting an Ottoman word to its Latin Turkish equivalent to ensure the correct suffix is added (e.g., gemiler instead of gemilar).

props is a bit field used to represent various properties of the lexical item, such as:

#define HAS_FINAL_VOWEL 0x01
#define HAS_SINGLE_VOWEL 0x02
#define IS_LAST_VOWEL_HARD 0x04
#define IS_FINAL_CONSONANT_HARD 0x08
#define HAS_CONSONANT_SOFTENING 0x10

These properties are particularly important when converting Ottoman text to modern Turkish.

The current version of the translation engine is written in Python, but I plan to rewrite it in C this week. As this is the third rewrite, I don’t expect it to pose a major algorithmic challenge, although debugging the C version may be more demanding.

Since conventional systems aim to represent phonemes, a correct reading is necessary and this requires expertise in the language being represented. For Ottoman, this is a deeper problem since the writing system is not actively used.

Labels should correspond to classes in the classification of visual elements. For example, in some cases we label ى, ي and ـيـ identically as y but their visual features are distinct from each other. In other cases, an identical letter is coded as y, e or i albeit represented identically in visual terms, because of their difference in reading. This makes visual classes mixed and fuzzy, and any measurement of performance gives little clue about the effectiveness of features or classifiers.

Here we document a new transliteration system for visual items in Ottoman Turkish. In this work, our aim is to provide a simple visual approach to transliteration and then devise necessary conversion tables into phonetic transliteration or transcription systems.

Except for numerals, words in Arabic-based writing systems are composed of items in two categories. The first category is a large letter group composed of a continuous movement of the pen, like کلمه. The other category is the smaller elements like dots and diacritics found around these larger items.

A visual transliteration should represent distinct visual elements differently. In order to keep simplicity in labeling and application, dotless letters will be used as base letters and the other letters will be written in terms of these dotless items.

A full letter code is composed of one base letter code and an optional set of diacritic codes. The base letter code is a small Roman letter. The diacritic codes have two parts: the first part is one of o, u or i, which mean over, under or ligature respectively. The second part of the diacritic code shows the type of diacritics and is a string of digits or letters. A full letter code conforms to the following regular expression: [a-h,j-n,p-t,v-z](([ou][0-9]+)|(i[a-h,j-n,p-t,v-z]))*

The letters o, u and i are not used as base letter codes, so there is no ambiguity in parsing the elements.

The tables for all visual elements can be found in Tables tab:baseletters, tab:diacritics and tab:numerals.

Letter Shape Transliteration Used in Letters

ا e ا

ٮ b ب, ت, ث, ن, پ, یـ

ح x ح, خ, ج, چ

د d د, ذ

ر r ر, ز

س s س, ش

ص z ص, ض

ط t ط, ظ

ع a ع, غ

ٯ f ف, ق

ک k ک, گ

ل l ل, ك

م m م

و w و

ه h ه, ة

ی y ی

ﺀ c ﺀ

Table: The Transliterations of Base Letters

Transliteration Description

1 Dot of ب, ن or خ

2 Dots of ق, ت or یـ

3 Dots of ث, ش or چ

5 ء hamza and also in ك

8 ـّ shadda

0 ـْ sukun

6 ـٓ madda

7 / sign above گ

4 ـَ fatha and ـِ kasra

9 ـُ damma

44 ـً fathatan and ـٍ kasratan

99 ـٌ dammatan

Table: Transliterations for Diacritics

Description Transliteration

۱ n1

۲ n2

۳ n3

۴ n4

۵ n5

۶ n6

۷ n7

۸ n8

۹ n9

۰ n0

Table: Transliterations for Numerals

The advantages of using this transliteration instead of a phonetic transliteration can be summarized as follows:

Representing visual elements in a phonetic way is not an optimal representation for Computer Vision research. The information from the phonetic representation slips into visual information which results in classes that bear no direct connection with visual features. In turn, these classes become harder to classify and understand.

The transliteration system described in this paper does not need expertise in the language. Anyone who recognizes the letters should be able to transliterate word images.

It is possible to denote single elements which do not represent a sound in the language with visual transliteration. No transliteration system for Arabic and similar languages represents diacritics as in Table tab:diacritics. This is important for Computer Vision, since these items are as legitimate visual elements on a paper as others. Without independent representation, these items would have to be represented along with others, and their classification could not be done independently.

The system is much more flexible than a phonetic transliteration system. It allows for the development of new letter signs by combining existing diacritics with base letters. For example, ﭪ is a letter not found in historical documents, but can be seen in modern Arabic to represent the v sound. Although not thought of beforehand, this can be represented as fo3 in the system we describe. There are also writing variations, for example, in Maghribi (Western) Arabic, the letter ف is written as ڢ but is phonetically identical. In a phonetic transliteration this difference is lost, but the visual transliteration is able to represent the usual case with fo1 and the specific case with fu1.

It is common in handwriting to attribute diacritics of one letter to another. For example, three dots in the middle of سر may be read as شر or سژ. In a phonetic transliteration system, the first might be represented as şr and the second as sj. This leads to complexity in describing such middle cases. In a visual transliteration system, these two are represented as so3r and sro3, in which we are able to write rules that exchange o3 between neighboring letters and decide for the best reading in later stages.

In handwriting and print, diacritics are especially written loosely. For example, three dots above ث are written as three separate dots in print, but usually contracted as a single shape in handwriting. For the print case, we can have so1o1o1 denoting three dots separately while for the handwriting case so3 is used and in an upper level, the former is converted to the latter by a rule specifying three separate dots mean a single three dot.

Table tab:ottoman-letters shows all letters of the Ottoman alphabet.

**Ottoman **Transliteration** Ottoman Transliteration Letter Letter** \\

ا e اَ eo4

اِ eu4 اُ eo9

أ eo5 إ eu5

آ eo6 ب bu1

پ bu3 ت bo2

ث bo3 ج xu1

چ xu3 ح x

خ xo1 د d

ذ do1 ر r

ز ro1 ژ ro3

س s ش so3

ص z ض zo1

ط t ظ to1

ع a غ ao1

ف fo1 ق fo2

ك lo5 گ ko7

ل l م m

ن bo1 ڭ lo5o3

ه h و w

ی y ـیـ bu2

لا lie ک k

Table: Ottoman Letters and Transliterations