Use of open and closed items in automation of evaluation systems

T. Kárász, Judit and Takács, Szabolcs (2023) Use of open and closed items in automation of evaluation systems. APPLIED PSYCHOLOGY IN HUNGARY, 25 (3). pp. 33-54. ISSN 1586-7382

Preview

Text APA_2023_3_3.pdf Download (504kB) | Preview

Abstract

Background and Aims: The design of automated evaluation systems raises the problem of open-ended questions or tasks that require living labour. Coding open-ended questions is a costly, time- and labour-intensive task. Reviewing and selecting CVs reduces the amount of time spent on face-to-face interviews. Our research question is: which subjects are affected by the omission of open-ended questions, and what are the consequences for evaluating results? Our study was conducted on data from the National Assessment of Basic Competencies, which can also be understood as an assessment system currently in the automation process. Methods: The original proportions were restored by weighting according to the measurement methodology. In this study, we compared achievement scores and proficiency levels calculated based on the whole test booklet and the basis of closed items only. Results: Ability scores calculated from the entire test and closed items show a strong correlation. Discussion: Our calculations demonstrate that open-ended items are needed in ability ranges where fewer items are available in the first place. By omitting the open-ended items, a significant “loss” is typically incurred by those for whom we have less information, who are classified as “very high performers” or “very low performers”.

Actions (login required)

Edit Item