UCCIX: Irish-eXcellence Large Language Model
| dc.check.date | 2024-10-25 | en |
| dc.check.info | Access to this article is restricted until after the conference has taken place | en |
| dc.contributor.author | Tran, Khanh-Tung | en |
| dc.contributor.author | O'Sullivan, Barry | en |
| dc.contributor.author | Nguyen, Hoang D. | en |
| dc.contributor.funder | Science Foundation Ireland | en |
| dc.date.accessioned | 2024-08-12T09:07:32Z | |
| dc.date.available | 2024-08-12T09:07:32Z | |
| dc.date.issued | 2024 | en |
| dc.description.abstract | The development of Large Language Models (LLMs) has predominantly focused on high-resource languages, leaving extremely low-resource languages like Irish with limited representation. This work presents UCCIX, a pioneering effort on the development of an open-source Irish-based LLM. We propose a novel framework for continued pre-training of LLMs specifically adapted for extremely low-resource languages, requiring only a fraction of the textual data typically needed for training LLMs according to scaling laws. Our model, based on Llama 2-13B, outperforms much larger models on Irish language tasks with up to 12% performance improvement, showcasing the effectiveness and efficiency of our approach. We also contribute comprehensive Irish benchmarking datasets, including IrishQA, a question-answering dataset, and Irish version of MT-bench. These datasets enable rigorous evaluation and facilitate future research in Irish LLM systems. Our work aims to preserve and promote the Irish language, knowledge, and culture of Ireland in the digital era while providing a framework for adapting LLMs to other indigenous languages. | en |
| dc.description.sponsorship | Science Foundation Ireland (12/RC/2289-P2) | en |
| dc.description.status | Peer reviewed | en |
| dc.description.version | Accepted Version | en |
| dc.format.mimetype | application/pdf | en |
| dc.identifier.citation | Tran, K.-T., O'Sullivan, B., Nguyen, H. D. (2024) 'UCCIX: Irish-eXcellence Large Language Model', 27th European Conference on Artificial Intelligence (ECAI), 19-24 October 2024, Santiago de Compostela, Spain. | en |
| dc.identifier.uri | https://hdl.handle.net/10468/16188 | |
| dc.language.iso | en | en |
| dc.relation.project | info:eu-repo/grantAgreement/SFI/SFI Centres for Research Training Programme::Data and ICT Skills for the Future/18/CRT/6223/IE/SFI Centre for Research Training in Artificial Intelligence/ | en |
| dc.rights | © 2024, the Authors. | en |
| dc.subject | Large Language Model (LLM) | en |
| dc.subject | Open-source Irish-based LLM | en |
| dc.subject | UCCIX | en |
| dc.subject | Irish language | en |
| dc.title | UCCIX: Irish-eXcellence Large Language Model | en |
| dc.type | Conference item | en |
