Improve Performance of Fine-tuning Language Models with Prompting

Yang, Zijian Győző and Ligeti-Nagy, Noémi (2023) Improve Performance of Fine-tuning Language Models with Prompting. INFOCOMMUNICATIONS JOURNAL, 15 (SI). pp. 62-68. ISSN 2061-2079


Download (452kB) | Preview


This paper explores the effectiveness of prompt programming in the fine-tuning process of a Hungarian language model. The study builds on the prior success of prompt engineering in natural language processing tasks and employs the prompting method to enhance the fine-tuning performance of a huBERT model on several benchmark datasets of HuLU. The experimentation involves testing 45 prompt combinations for the HuCoPA dataset and 15 prompt variations for the HuRTE and HuWNLI datasets. The findings reveal that the addition of an instructional text consistently produces the best results across all winning cases, and that the [CLS] token produces the best results in the separator token experiments. The most significant enhancement was observed in the HuWNLI dataset, with an increase in accuracy from 65% to 85%. These results demon- strate that the addition of instruct text is crucial and sufficient in enabling the language model to effectively interpret and solve the Winograd Schemata problem. These results showcase the potential of prompt programming in enhancing the performance of language models in fine-tuning tasks, and highlight the importance of incorporating task-specific instructions to improve model interpretability and accuracy.

Item Type: Article
Uncontrolled Keywords: BERT, prompting, fine-tuning
Subjects: P Language and Literature / nyelvészet és irodalom > P0 Philology. Linguistics / filológia, nyelvészet
Q Science / természettudomány > QA Mathematics / matematika > QA75 Electronic computers. Computer science / számítástechnika, számítógéptudomány
Q Science / természettudomány > QA Mathematics / matematika > QA76.76 Software Design and Development / Szoftvertervezés és -fejlesztés
Depositing User: MTMT SWORD
Date Deposited: 07 Sep 2023 12:59
Last Modified: 07 Sep 2023 12:59

Actions (login required)

Edit Item Edit Item