Deep Learning Object Detection Models in Robotic Process Automation
Abstract
In recent years, businesses have become increasingly aware of their need to automate previously manual business processes. Adoption rates of robotic process automation (RPA) tools have increased a lot, and this has generated a great deal of value for companies. But, alas, especially some older legacy applications are still extremely problematic from the point of view of RPA platforms, and they cannot often be interfaced in a same fashion as some more modern applications. This is especially lamentable, as it is often the older applications, which tend to be used in the more manual business processes. This thesis aims to build a foundation for building and evaluating commercializable computer vision (CV) object detection models, that resolve the issues faced by RPA applications that arise from the lack of operating system (OS) level user interface (UI) visual element data. This kind of models could be used as a basis for building completely new RPA tools, or for augmenting existing solutions. The research questions discussed by this study are as follows: 1. Is using machine learning models for the detection of user interface elements viable? 2. What are the implications to existing RPA applications? 3. Can we build a model that is able to generalize from data from another domain? This thesis discusses the theoretical background of object detection and machine learning, and presents three potential families of machine learning model architectures that could be used for the aforementioned tasks. These models were then trained and compared to the CV capabilities of a commercial RPA platform, UiPath. The results show that the most simple object detection models cannot compete with the current commercial offering. There is, however, a clear path to further research, and it is clear that building a object detection system for UI elements is viable. In addition, this thesis presents the first publicly available dataset of (manually) annotated desktop UI screenshot images, and a tool for generating artificial user interface data.
Keywords
Not available
Specialisations
Similar theses
- Deep Learning-Based Vision System for Intelligent Object Detection and Robotic Sorting Solution Anastasiia Stepanova · Tampere University · 2026
- Computer Vision for Automated Robotic Device Disassembly: Object Detection, Pose Estimation, and Action Prediction Ruiz, Sebastian · Georg-August-Universität Göttingen · 2024
- Distributed Intelligence for 3D Object Tracking in Autonomous Vehicles, A Multi-Sensor Approach to Object Tracking and Planning for Self-Driving Cars, Real-Time 3D Perception and Path Planning in Autonomous Systems,,Vision-Based Localization and Distributed Architecture for Driverless Cars,A Hybrid Deep Learning Framework for Object Detection and Trajectory Planning in Autonomous Vehicles Manas Madine · University of Massachusetts Amherst · 2023
- Object detection, recognition and classification using computer vision and artificial intelligence approaches Gharib Ismail Al Matroushi · Loughborough University · 2018
- Towards Few-Annotation Learning in Computer Vision : Application to Image Classification and Object Detection tasks Quentin Bouniot · 2023
- Real-Time Object Detection and Grasping for a Robotic Arm Thanh Thoi Nguyen · 2021
