Natural Language Processing And the Finnish Language

Post Reply
Posts: 116
Joined: Wed Jun 13, 2018 9:36 am
Has thanked: 0
Been thanked: 1 time

Natural Language Processing And the Finnish Language

Post by john » Wed Feb 27, 2019 11:08 am


It's interesting to observe the evolution of natural language processing when my own native language (Finnish) is such an adversarial case that it breaks every system. I haven't seen even a functioning spell checker. Here some reasons why this is the case.
First, every word has dozens of conjugations. For example nouns are conjugated in case (15), number (2). So, in even basic situations each noun can appear in approximately thirty forms. This means that each word occurrence is extremely rare. New words can be formed by compounding, just like in German.

For example:
tietokone = knowledge machine (literally) = computer
kämmentietokone = palm knowledge machine (literally) = tablet

This causes many word occurrences of other languages very rare.
Even non-compound words can be changed with many modifiers.

For example:
syödä = to eat
syömättäkinköhän = Does he/she mean even without eating, I wonder.

These tags can be added independently, which causes a combinatorial explosion making many everyday words ultra rare.
To make the combinatorial explosion even worse. Many tags can be permuted within the word.

The following mean roughly the same:
Compounding some times changes the meaning from literal to figurative:

luotaantyöntävä = unappealing
luotaan työntävä = something that literally thrusts you away

For the latter Google Translate gives the nonsense translation "I trust the pushing".
These features of the Finnish language, and many more, make it very badly compatible with #NLP systems. In fact Finnish isn't very well compatible even with any IT system. You can observe that in how Finnish speakers have to adapt the language to using hashtags.
All in all, I don't see deep learning being the way to #AGI before I see even a well functioning Finnish language spell checker. Additionally, machine translation to and from Finnish is usually just garbage. (Which luckily protects us from some foreign disinformation.)

Thanks for reading

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests