A study on text classification in the age of large language models
| dc.contributor.author | Trust, Paul | en |
| dc.contributor.author | Minghim, Rosane | en |
| dc.contributor.funder | Science Foundation Ireland | en |
| dc.date.accessioned | 2025-11-26T12:55:00Z | |
| dc.date.available | 2025-11-26T12:55:00Z | |
| dc.date.issued | 2024-11-21 | en |
| dc.description.abstract | Large language models (LLMs) have recently made significant advances, excelling in tasks like question answering, summarization, and machine translation. However, their enormous size and hardware requirements make them less accessible to many in the machine learning community. To address this, techniques such as quantization, prefix tuning, weak supervision, low-rank adaptation, and prompting have been developed to customize these models for specific applications. While these methods have mainly improved text generation, their implications for the text classification task are not thoroughly studied. Our research intends to bridge this gap by investigating how variations like model size, pre-training objectives, quantization, low-rank adaptation, prompting, and various hyperparameters influence text classification tasks. Our overall conclusions show the following: 1—even with synthetic labels, fine-tuning works better than prompting techniques, and increasing model size does not always improve classification performance; 2—discriminatively trained models generally perform better than generatively pre-trained models; and 3—fine-tuning models at 16-bit precision works much better than using 8-bit or 4-bit models, but the performance drop from 8-bit to 4-bit is smaller than from 16-bit to 8-bit. In another scale of our study, we conducted experiments with different settings for low-rank adaptation (LoRA) and quantization, finding that increasing LoRA dropout negatively affects classification performance. We did not find a clear link between the LoRA attention dimension (rank) and performance, observing only small differences between standard LoRA and its variants like rank-stabilized LoRA and weight-decomposed LoRA. Additional observations to support model setup for classification tasks are presented in our analyses. | en |
| dc.description.sponsorship | Science Foundation Ireland (Grant No. SFI 18/CRT/6222) | en |
| dc.description.status | Peer reviewed | en |
| dc.description.version | Published Version | en |
| dc.format.mimetype | application/pdf | en |
| dc.identifier.citation | Trust, P. and Minghim, R. (2024) ‘A study on text classification in the age of large language models’, Machine Learning and Knowledge Extraction, 6(4), pp. 2688–2721. https://doi.org/10.3390/make6040129 | en |
| dc.identifier.doi | 10.3390/make6040129 | en |
| dc.identifier.eissn | 2504-4990 | en |
| dc.identifier.endpage | 2721 | en |
| dc.identifier.issued | 4 | en |
| dc.identifier.journaltitle | Machine Learning and Knowledge Extraction | en |
| dc.identifier.startpage | 2688 | en |
| dc.identifier.uri | https://hdl.handle.net/10468/18281 | |
| dc.identifier.volume | 6 | en |
| dc.language.iso | en | en |
| dc.publisher | MDPI | en |
| dc.rights | © 2024, the Authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/) | en |
| dc.rights.uri | http://creativecommons.org/licenses/by/4.0/ | en |
| dc.subject | Text classification | en |
| dc.subject | Language models | en |
| dc.subject | Fine-tuning | en |
| dc.subject | Prompting | en |
| dc.title | A study on text classification in the age of large language models | en |
| dc.type | Article (peer-reviewed) | en |
