Hybridization of Rule-Based and Statistical-Based Ranking Models for Best Translation Candidates in Yorùbá Language

Abstract

Language is the only channel by which human beings abstract reality and interacts. The Yorùbá language is very rich in culture and developing continuous natural language processing (NLP) resource tools for its research is a necessity in this age. A lot of machine translation approaches have been employed by researchers in the automatic translations of English to Yorùbá language. However, all these works have not fully solved the problem of translations in the language. This research hybridized the rule-based approach and the statistical-based approach to machine translations respectively to take advantage of their strengths and improve the accuracy of translation works in English to Yorùbá. For the rule-based part of the system, Context-Free Grammars (CFG) formalisms were used to formulate grammar rules for various cases of grammars. The grammar of the language was modeled using Finite State Automata (FSA) whose operations were based on the first and follow sets techniques. For the statistical-based approach, monolingual corpora of the Yorùbá language were extracted from some online resources; unigram and bigram frequencies tables were formulated from the monolingual corpus. The total number of extracted unigram was 11,532 with only 7,907 valid tokens while the total number of extracted bigram was 81,362 with 70,108 valid tokens. Comparison analysis of the two models with previous related works and Google translate and the evaluation of the system with BLEU metrics show better accuracy. Accuracy of 70.5% was recorded on a test set of 442 sentences. The final implementation of the system was carried out using Python 2.7 programming language with Natural Language Toolkits (NLTK) for lemmatization/stemming and parts of speech tagging. The proposed model produced promising results which can be used for translations works.