AI-Driven Mixed Reality for Scalable and Adaptive Training Solutions

Nagy, Bálint György and Bihari, Bence and Sonkoly, Balázs (2025) AI-Driven Mixed Reality for Scalable and Adaptive Training Solutions. 2025 9th International Conference on Artificial Intelligence and Virtual Reality (AIVR2025). (In Press)

Preview

Text aivr25.pdf - Accepted Version Download (2MB) | Preview

Abstract

Training in industrial environments is often expensive and time-consuming. While self-learning approaches, customized curricula, and automation offer promising solutions, the creation of new training materials remains a major bottleneck—especially in rapidly evolving industries where processes change frequently. Mixed reality (MR) and virtual reality (VR) technologies introduce a new generation of immersive learning platforms, but their widespread adoption is limited by the expertise and effort required to generate content. In this paper, we present a novel AI-powered system designed to simplify and accelerate the development of MR-based training applications. Using a HoloLens 2 headset, an expert needs to demonstrate and explain a complex task only once. Our system captures this demonstration, interprets the instructions using various AI techniques, and automatically produces a step-by-step tutorial. The resulting guide includes time-aligned video snippets and descriptive text, allowing trainees to visually follow each subtask. This approach significantly reduces the cost and complexity of training content creation, opening the door to scalable and efficient instructorless learning. The system was evaluated in terms of usability, user experience, and the accuracy of the generated content, the results of which confirm the correct functioning and applicability of our approach.

Actions (login required)

Edit Item