Learning to Drive: Exploiting Deep Models for Autonomous Driving

Building self-driving vehicles is exciting and promising. It is going to transform the way we live and provide safety, efficiency, and mobility for everyone. In this thesis, I present a collection of our work in the direction of building smart self-driving vehicles from understanding convolutional neural networks (CNNs) to the full autonomy stacks, including perception, prediction, and planning. First, we study the property of CNNs and provide theoretical analysis on the receptive field. We use Fourier transform and center limit theorem to show that the effective receptive field only occupies a fraction of the theoretical one, especially in deep models. This can provide insights for future CNNs design. Next, we push the state of the art for tasks in the autonomy stacks using deep CNNs. For depth estimation with stereo cameras, we develop a deep matching network that simplifies previous approaches by using a dot-product layer and incorporates a multi-class classification loss, allowing the network to calibrate scores among larger contexts. These greatly improve the runtime and achieve much better matching results. Furthermore, we extend the idea to optical flow where the matching is done in 2D space. Combined with an instance segmentation algorithm, we achieve state-of-the-art results on KITTI optical flow benchmark. Third, we propose new formulations for the self-driving system, i.e. using a joint model for 3D detection, prediction, and tracking. Our model takes a sequence of consecutive LiDAR sweeps and predicts the bounding boxes for vehicles at the current time step as well as one second into the future. Tracking is done by greedily checking the overlap between current detections and predictions from the past. This approach bridges the gap between perception and prediction to achieve better performance both in accuracy and runtime. Then, we take this one step further to construct a deep structured model for interpretable neural motion planning, where the CNN also predicts a 3D cost volume, expanding on time and x-y spatial dimensions. The final planning trajectory is generated with a sampling-based inference algorithm. We conduct offline testing and show that our approach is better than several baselines.