Benchmarking machine learning models at the edge

Recently the use of resource-constrained edge devices to perform machine learning tasks has gained substantial research attention. The need to perform such tasks arises from the vast amounts of sensors, and IoT and smart devices that generate large swaths of data at an enormous rate. To process such data in a real-time manner, cloud computing has proven to be suboptimal. As an alternative, edge devices paired with ML models can execute demanding tasks in diverse environments. This study explored a range of machine learning models coupled with several low and high-end edge devices to test their suitability for deployment in an edge environment. Unlike previous works, this experiment takes into account the initial heavy lifting that machine learning models and devices go through before performing the actual task of classification or detection. To systematically benchmark and gather results, this study divides the ML pipeline into initial and execution phases while thoroughly measuring each phase’s time and resource usage. This type of rigorous performance metric analysis enables decision-making about which machine learning model and devices have better suitability for latency and speed. Moreover, to add a device in the suit of existing ones and replicate the benchmark workflow, this study has automated the benchmark with the Ansible automation tool. The thesis research presents its findings from different devices. The division of phases allows for a clear understanding of the bottlenecks experienced by the devices and their capabilities. Moreover, an underlying crucial detail such as the warmup was revealed by scrutinizing the performances of the devices from the very beginning. The thesis concludes with performance highlights and limitations of the benchmark, identifying possible future research directions that can be developed further.