Irish-based Large Language Model with extreme low-resource settings in machine translation

Tran, Khanh-Tung; O'Sullivan, Barry; Nguyen, Hoang D.

Irish-based Large Language Model with extreme low-resource settings in machine translation

Files

2024_UCCIX_MT_loresmt.pdf(532.83 KB)

Accepted version

Date

2024-08-11

Authors

Tran, Khanh-Tung

O'Sullivan, Barry

Nguyen, Hoang D.

Publisher

ACL

Abstract

Large Language Models (LLMs) have demonstrated exceptional performances in a wide range of natural language processing tasks. However, their success does not always extend to machine translation, particularly in challenging scenarios such as translating low-resource languages. This study investigates the multilingual capability of LLMs, with a case study on Irish, an extremely low-resource language, focusing on translation tasks between English and Irish. We propose a dynamic, efficient language adaptation framework for English-centric LLMs, which involves layer-specific adjustments and subsequent fine-tuning for machine translation. Our findings highlight several key insights: (1) different layers in the LLM serve distinct functions such as language understanding and task reasoning, (2) effective translation requires extensive pre-training on both source and target languages, and (3) targeted fine-tuning for machine translation leads to significant improvements of 36.7% for English to Irish and 133.4% for Irish to English compared to the previous state-of-the-art.

Keywords

Large Language Models (LLMs) , Natural Language Processing (NLP) , Translation , Machine translation , Language technologies , Accessibility , Irish language

Citation

Tran, K.-T., O'Sullivan, B. and Nguyen, H. D. (2024) 'Irish-based Large Language Model with Extreme Low-Resource Settings in Machine Translation', LoResMT 2024: The Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages, @ACL2024, Bangkok, Thailand, August 11–16.