Constrained deep networks for medical image segmentation

Weakly supervised image segmentation, in the form of partially labeled images, is attracting significant research attention as it can mitigate the need for laborious pixel annotations required by deep learning models. Enforcing high-order, global inequality constraints on the network outputs can leverage unlabeled data by guiding the training with prior knowledge, restricting the search space during training to anatomically feasible solutions. A range of possible values (such as a lower/upper bounds on the size of a organ) can be very valuable to guide training. However, in the context of deep neural networks, standard Lagrangian optimization has been largely avoided, mainly due to the instability and computational complexity ensuing from alternating explicit dual updates and stochastic optimization. Interior point methods, despite their popularity in convex optimization, are not applicable neither, as they require a feasible starting point, which is itself a difficult constrained problem for deep neural networks. In this thesis, we investigate hard inequality constraints in the context of deep networks with both quadratic penalties and more principled log-barrier extensions. We also investigate methods to mitigate class-imbalance in segmentation problems, such as in brain lesions dataset, by constraining the boundary of the predicted segmentation to match the ground-truth boundary. This thesis produced five different publications as first author, and four papers as co-author. Our papers received several awards, and we were invited to publish extended versions of our works in two special issues of Medical Image Analysis (MedIA). In our first contribution, we propose to introduce a differentiable penalty, which enforces inequality constraints directly in the loss function, avoiding expensive Lagrangian dual iterates and proposal generation. From constrained-optimization perspective, our simple penalty-based approach is not optimal as there is no guarantee that the constraints are satisfied. However, surprisingly, it yields substantially better results than Lagrangian-based constrained convolutional neural networks, while reducing the computational demand for training. By annotating only a small fraction of the pixels, our approach reaches performances comparable to full supervision, on three separate tasks. While our experiments focused on basic linear constraints such as the target-region size and image tags, our framework can be easily extended to other non-linear constraints, e.g., invariant shape moments and other region statistics. In our second contribution, we propose log-barrier extensions, which approximate Lagrangian optimization of constrained-CNN problems with a sequence of unconstrained losses. Unlike standard interior-point and log-barrier methods, our formulation does not need an initial feasible solution. We report comprehensive weakly supervised segmentation experiments, with various constraints, showing that our formulation outperforms substantially the existing constrained-CNN methods, both in terms of accuracy, constraint satisfaction and training stability. In our third contribution, we enforce constraints on the boundary of predicted segmentation. Widely used loss functions for CNN segmentation, such as Dice or cross entropy, are based on integrals over the segmentation regions. Unfortunately, for highly unbalanced segmentations, such regional summations have values that differ by several orders of magnitude across classes, which affects training performance and stability. We propose a boundary loss, which takes the form of a distance metric on the space of contours, not regions. This can mitigate the difficulties of highly unbalanced problems because it uses integrals over the interface between regions instead of unbalanced integrals over the regions. Furthermore, a boundary loss complements regional information. Inspired by graph-based optimization techniques for computing activecontour flows, we express a non-symmetric L2 distance on the space of contours as a regional integral, which avoids completely local differential computations involving contour points. This yields a boundary loss expressed with the regional softmax probability outputs of the network, which can be easily combined with standard regional losses and implemented with any existing deep network architecture for N-D segmentation. We report comprehensive evaluations on different unbalanced problems, showing that our boundary loss can yield significant increases in performances while improving training stability. In a fourth contribution, we investigates a curriculum-style strategy for semi-supervised CNN segmentation, which devises a regression network to learn image-level information such as the size of the target region. These regressions are used to effectively regularize the segmentation network, constraining the softmax predictions of the unlabeled images to match the inferred label distributions.