Irish-based Large Language Model with extreme low-resource settings in machine translation

Thumbnail Image
2024_UCCIX_MT_loresmt.pdf(532.83 KB)
Accepted version
Tran, Khanh-Tung
O'Sullivan, Barry
Nguyen, Hoang D.
Journal Title
Journal ISSN
Volume Title
Published Version
Research Projects
Organizational Units
Journal Issue
Large Language Models (LLMs) have demonstrated exceptional performances in a wide range of natural language processing tasks. However, their success does not always extend to machine translation, particularly in challenging scenarios such as translating low-resource languages. This study investigates the multilingual capability of LLMs, with a case study on Irish, an extremely low-resource language, focusing on translation tasks between English and Irish. We propose a dynamic, efficient language adaptation framework for English-centric LLMs, which involves layer-specific adjustments and subsequent fine-tuning for machine translation. Our findings highlight several key insights: (1) different layers in the LLM serve distinct functions such as language understanding and task reasoning, (2) effective translation requires extensive pre-training on both source and target languages, and (3) targeted fine-tuning for machine translation leads to significant improvements of 36.7% for English to Irish and 133.4% for Irish to English compared to the previous state-of-the-art.
Large Language Models (LLMs) , Natural Language Processing (NLP) , Translation , Machine translation , Language technologies , Accessibility , Irish language
Tran, K.-T., O'Sullivan, B. and Nguyen, H. D. (2024) 'Irish-based Large Language Model with Extreme Low-Resource Settings in Machine Translation', LoResMT 2024: The Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages, @ACL2024, Bangkok, Thailand, August 11–16.
Link to publisher’s version
© 2023 Association for Computational Linguistics