Self-Supervised Learning for Geometry
Abstract
This thesis focuses on two fundamental problems in robotic vision, scene geometry understanding and camera tracking. While both tasks have been the subject of research in robotic vision, numerous geometric solutions have been proposed in the past decades. In this thesis, we cast the geometric problems as machine learning problems, specifically, deep learning problems. Differ from conventional supervised learning methods that using expensive annotations as the supervisory signal, we advocate for the use of geometry as a supervisory signal to improve the perceptual capabilities in robots, namely Geometry Self-supervision. With the geometry self-supervision, we allow robots to learn and infer the 3D structure of the scene and ego-motion by watching videos, instead of expensive ground-truth annotation in traditional supervised learning problems. Followed by showing the use of geometry for deep learning, we show the possibilities of integrating self-supervised models with traditional geometry-based methods as a hybrid solution for solving the mapping and tracking problem. We focus on an end-to-end mapping problem from stereo data in the first part of this thesis, namely Deep Stereo Matching. Stereo matching is one of the oldest problems in computer vision. Classical approaches to stereo matching typically rely on handcrafted features and a multiple-step solution. Recent deep learning methods utilize deep neural networks to achieve end-to-end trained approaches while significantly outperforming classic methods. We propose a novel data acquisition pipeline using an untethered device (Microsoft HoloLens) with a Time-of-Flight (ToF) depth camera and stereo cameras to collect real-world data. A novel semi-supervised method is proposed to train networks with ground-truth supervision and self-supervision. The large scale real-world stereo dataset with semi-dense annotation and dense self-supervision allow our deep stereo matching network to generalize better when compared to prior arts. Mapping and tracking using a single camera (Monocular) is a harder problem when compared to that using a stereo camera due to varies well-known challenges. In the second part of this thesis, We decouple the problem into single view depth estimation (mapping) and two view visual odometry (tracking) and propose a self-supervised framework, namely SelfTAM, which jointly learns the depth estimator and the odometry estimator. The self-supervised problem is usually formulated as an energy minimization problem consist of an energy of data consistency in multi-view (e.g. photometric) and an energy of prior regularization (e.g. depth smoothness prior). We strengthen the supervision signal with a deep feature consistency energy term and a surface normal regularization term. Though our method trains models with stereo sequence such that a real-world scaling factor is naturally incorporated, only monocular data is required in the inference stage. In the last part of this thesis, we revisit the basics of visual odometry and explore the best practice to integrate deep learning models with geometry-based visual odometry methods. A robust visual odometry system, DF-VO, is proposed. We use deep networks to establish 2D-2D/3D-2D correspondences and pick the best correspondences from the dense predictions. Feeding the high-quality correspondences into traditional VO methods, e.g. Epipolar Geometry and Prospective-n-Points, we can solve visual odometry problem within a more robust framework. With the proposed self-supervised training, we can even allow the models to perform online adaptation in the run-time and take a step toward a lifelong learning visual odometry system.
Keywords
Not available
Specialisations
Similar theses
- Hybrid Optimization and Learning-Based Techniques for Autonomous Driving Kürsat Petek · Universitätsbibliothek Freiburg · 2026
- Multi-Scale Deep Learning Frameworks for Trajectory Prediction, Scene Understanding, and Perception in Autonomous Driving Xingnan Zhou · Concordia University · 2026
- Interaction-Aware Maneuver and Lane Change Prediction Using Deep Learning Architectures for Autonomous Driving Petrit Rama · Kaiserslautern University of Technology · 2026
- A Novel Explainable Artificial Intelligence Framework with Improved Learning Mechanisms Kabir, Sami · Luleå University of Technology · 2025
- Leveraging deep learning for visual understanding of videos Clement Xian Ren Tan · Nanyang Technological University · 2024
- Distributed Intelligence for 3D Object Tracking in Autonomous Vehicles, A Multi-Sensor Approach to Object Tracking and Planning for Self-Driving Cars, Real-Time 3D Perception and Path Planning in Autonomous Systems,,Vision-Based Localization and Distributed Architecture for Driverless Cars,A Hybrid Deep Learning Framework for Object Detection and Trajectory Planning in Autonomous Vehicles Manas Madine · University of Massachusetts Amherst · 2023
