Enriching Scene-Graph Generation with Prior Knowledge from Work Instruction

Jeskó, Zoltán Dominik and Tran, Tuan-anh and Halász, Gergely Lajos and Abonyi, János and Ruppert, Tamás (2024) Enriching Scene-Graph Generation with Prior Knowledge from Work Instruction. In: Advances in Production Management Systems. Production Management Systems for Volatile, Uncertain, Complex, and Ambiguous Environments. IFIP Advances in Information and Communication Technology (729). Springer Nature Switzerland AG, Cham, pp. 290-302. ISBN 9783031658938; 9783031658945

Text APMS2024___Extended_scene_graph2.pdf - Published Version Restricted to Registered users only Download (3MB) | Request a copy

Abstract

With the current focus on human resources in Industry 5.0, analysing the work movements of industrial operators is the important first step in optimising labour performance. Thanks to the popularity of camera sensors, vision-based Human Activity Recognition models have become useful engines for real-time monitoring tools, in which scene- graphs play an important role. Traditional scene-graph generation meth- ods rely primarily on visual data for perception, neglecting a valuable source of process-oriented prior knowledge: the work instruction. There- fore, an extension of the scene-graph paradigm by integrating ground truth elaborated on elements from the work instruction is elaborated to complement and enhance the understanding of human activities in industrial environments, and improve the tracking capability with mi- cro and repetitive movements. This conceptual paper discusses the basic design of this approach with potential applications in industrial envi- ronments, which is validated by a simulated use case of an electronic assembly process. Based on the proposed extension, the Human Activity Recognition model can be lightweight and robust. Further integration of multi-modal sensory inputs beyond visual cues, such as environmental and human-centric data, can enrich scene interpretation and provide a more comprehensive understanding of work behaviour, paving the way for more effective labour utilisation and improved productivity.

Actions (login required)

Edit Item