Deep Learning-Based Vision System for Intelligent Object Detection and Robotic Sorting Solution

Manufacturing and production enterprises encounter digital development in the scope of Industry 4.0. Car workshops as part of the automotive industry aftermarket are in the transition from manual labour to automation and robotics solutions that aim to facilitate the workspace productivity by decreasing workers’ fatigue and errors. Automotive industry expects efficiency increase by 30% due to the deployment of autonomous robotics in both production and operation. Machine vision serving as the perceptive brain for automation systems allows robots to navigate in the unstructured environments of car workshops. Recent advancements in deep neural networks allow models to learn hierarchical features and patterns extracted from visual data and hence contribute to automated solutions in car workshops’ unstructured environments that present a certain challenge for machine learning. This thesis does not aim to replace human workers, but to provide aid in necessary but secondary task of the post-work tool sorting and arranging in a car service garage. In this project, a deep learning-based vision system was developed and evaluated. The primary task of the project included automatic identification, instance segmentation, and pose estimation of metallic wrenches in automotive workshop environment. Metallic tools represent a certain machine vision challenge due to the high specular reflection that causes significant segmentation mask fragmentation and affects performance of 3D cameras. This issue was addressed by utilizing a deeplearning-based keypoint detection as geometric anchors to interpolate and repair fragmented masks. The vision framework utilizes custom trained Detectron2 segmentation framework. The primary technical contribution of this thesis is the developed keypoint-guided mask reconstruction algorithm. Unlike standard frameworks that encounter fragmented segmentation masks on reflective objects, this system exploits detected semantic keypoints as stable geometric anchors that are further utilized to interpolate the mask and repair incomplete visual data. In this system, Zivid2 M70 industrial 3D camera was used for perception task to ensure high precision of depth values. The developed vision system was experimentally validated through three testing cases: static repeatability, rotational accuracy, and robustness under occlusion. The test results have shown high system stability: angular standard deviation of 0.17° and depth variance of 0.02 mm. The system achieved MAE of 1.33° and RMSE of 1.69° in a 180° rotational sweep. Furthermore, the developed algorithm to reconstruct fragmented masks has shown precise orientation calculation in occluded scenes with angular variance of 0.08°. Therefore, the proposed vision system provides accurate and reliable data for robotic manipulation and automated tool arranging tasks under unstructured conditions of car workshops.