Multi-Hop Arabic LLM Reasoning in Complex QA

Abstract

The introduction of Large Language Models (LLMs), and generative AI has significantly transformed the field of natural language processing. These models have exhibited profound reasoning capabilities, marking considerable progress across diverse general knowledge reasoning tasks. Consequently, the deployment of LLMs in domain-specific contexts has become a prime objective for governments and corporations eager to leverage the generative AI revolution. However, the Arabic language has notably lagged in attention and development compared to other languages in this arena. This research endeavors to delve into various facets of Arabic closed-domain question and answering systems that emulate the reasoning requirements of private enterprise data. Our study focuses on the practical deployment of Arabic LLMs in targeted applications, specifically utilizing the ACQAD (Arabic Complex Question Answering Dataset), which exhibits multi-hop reasoning. Different strategies are experimented using Long Context Window (LCW) and Retrieval Augmented Generation (RAG). Results showed that decomposing complex questions using Chain-of-Thought reasoning considerably improved the performance from 75% to 92% using LCW, but at much higher token cost compared to RAG. Trade-of between cost and performance showed that 80% accuracy can be attained using only 30% of the cost using RAG Sentence - level embeddings. Microsoft E5 embedding model is used and OpenAI GPT4-turbo LLM which proved superior reasoning performance compared to other Arabic LLMs