Under the hood : An inside look at PULI models

Yang, Zijian Győző and Stajer, Lili Anna and Lukács, Gergely (2025) Under the hood : An inside look at PULI models. In: Proceedings of the International Conference on Formal Methods and Foundations of Artificial Intelligence. Eszterházy Károly Katolikus Egyetem Líceum Kiadó, Eger, pp. 233-242. ISBN 9789634963035

Preview

Text fmfai2025_pp233-242.pdf - Published Version Download (1MB) | Preview

Abstract

Understanding the internal structure and behavior of large language models remains a key challenge in natural language processing. In this work, we present a comprehensive analysis of the PULI family of Hungarian generative large language models. Our study combines static analysis of model parameters with dynamic visualization of model behavior during inference. The static analysis reveals patterns in parameter distributions and dimensionality across layers, offering insight into how different layers specialize. The dynamic analysis integrates an adapted version of BertViz into a webbased interface that enables interactive exploration of attention mechanisms for arbitrary prompts and generated responses. This dual approach advances interpretability and facilitates further research on the internal mechanics of transformer models tailored for low-resource languages like Hungarian.

Actions (login required)

Edit Item