Face sketch recognition using deep learning

Face sketch recognition refers to automatically identifying a person from a set of facial photos using a face sketch. This thesis focuses on matching facial images between front face photos and front face hand-drawn sketches, and between front face photos and front face composite sketches by software. Because different visual domains, different image forms, and different collection methods exist between the matching image pairs, face sketch recognition is more difficult than traditional facial recognition. In this thesis, three novel deep learning models are presented to increase recognition accuracy on face photo-sketch datasets. An improved Siamese network combined with features extracted from an encoder-decoder network is proposed to extract more correlated features from facial photos and the corresponding face sketches. After that, attention modules are proposed to extract features from the same location in the photos and the sketches. In the third method, in order to reduce the difference between different visual domains, the images are transferred into a graph to increase the relationship for different face attributes and facial landmarks. Meanwhile, the graph neural network is utilized to learn the weights of neighbors adaptively. The first is to fuse more image features from the Siamese network and encoder-decoder network for increased the recognition results. Moreover, the attention modules can fix the similarity positions from different domain images to extract the correlated features. The visualized feature maps exhibit the correlated features which are extracted from the photo and the corresponding face sketch. In addition, a stable deep learning model based on graph structure is introduced to capture the topology of the graph and the relationship after images have been mapped into the graph structure for reducing the gap between face photos and face sketches. The experimental results show that the recognition accuracy of our proposed deep learning models can achieve the state-of-the-art on composite face sketch datasets. Meanwhile, the recognition results on hand-drawn face sketch datasets exceed other deep learning methods.