A Graph-based Word Segmentation Algorithm for Dialectal Arabic: Libyan Dialect as a Case Study

Abstract

Arabic language and its dialects both have a very rich and complex morphology, and they face the same challenge, which is called agglutination, where the words might be attached to one or more affixes. However, word segmentation has become a very important preprocessing procedure for many natural language processing tasks that deal with agglutinative languages to improve their performance. Besides Arabic, Arabic dialects are known for their complex agglutination system, which makes word segmentation challenging. To address this challenge, this paper presents an out-of-context full word segmentation algorithm that is based on weighted directed graph theory. The main purpose of this algorithm is to tackle the agglutination phenomena observed in dialectal Arabic. To illustrate the efficacy of the algorithm, the Libyan dialect is selected as a case study for testing its feasibility. A test dataset of 1,200 Libyan dialect words was used to manually evaluate the algorithm for accuracy. The experimental results show that the proposed algorithm achieves good out comes on the test dataset.