Interpretability of a Deep Learning Model for Semantic Segmentation : Example of Remote Sensing Application

Understanding a black-box model is a major problem in domains that relies on model predictions in critical tasks. If solved, can help to evaluate the trustworthiness of a model. This thesis proposes a user-centric approach to black-box interpretability. It addresses the problem in semantic segmentation setting with an example of humanitarian remote sensing application for building detection. The question that drives this work was, Can existing methods for explaining black-box classifiers be used for a deep learning semantic segmentation model? We approached this problem with exploratory qualitative research involving a case study and human evaluation. The study showed that it is possible to explain a segmentation model with adapted methods for classifiers but not without a cost. The specificity of the model is likely to be lost in the process. The sole process could include introducing artificial classes or fragmenting image into super-pixels. Other approaches are necessary to mitigate identified drawback. The main contribution of this work is an interactive visualisation approach for exploring learned latent space via a deep segmenter, named U-Net, evaluated with a user study involving 45 respondents. We developed an artefact (accessible online) to evaluate the approach with the survey. It presents an example of this approach with a real-world satellite image dataset. In the evaluation study, the majority of users had a computer science background (80%), including a large percentage of users with machine learning specialisation (44.4% of all respondents). The model distinguishes rurality vs urbanization (58% of users). External quantitative comparison of building densities of each city concerning the location in the latent space confirmed the later. The representation of the model was found faithful to the underlying model (62% of users). Preliminary results show the utility of the pursued approach in the application domain. Limited possibility to present complex model visually requires further investigation.