Rethinking image formats for computer vision: JPEG sRGB, linear, and log RGB; object detection and shadow removal

From driverless cars to generative AI, computer vision is making a larger impact in our lives every day. Computer vision models are powered by data, but for many datasets, the image processing pipeline is unknown. Could there be important patterns in the data lost to image processing? Computer vision datasets in common use today are comprised of images formatted for human viewing, not machine learning. As a result, physical relationships dictated by the laws of optics are not readily available to computer vision models. Is it time we re-examined where and how we source our data? After all, we live in a time when edge devices and phones are more common than ever, and every camera is capable of capturing RAW data. In this work, I quantify differences in performance between models trained on sRGB, JPEG-compressed, linear RGB, and log RGB data, both with strictly controlled data processing pipelines and with unknown data processing pipelines. The results suggest there are improvements to be had by using more physically accurate data and that further research is warranted.--Author's abstract