Numerous methods designed for heterogeneous face recognition, particularly face photo-sketch recognition, operate by extracting features from a face sketch image and comparing them to features extracted from face photos, with the aim of finding the identity of the person depicted in the sketch. Most current approaches use hand-crafted features, which may not be optimal for the task of face photo-sketch recognition. An alternative would be to use methods to learn the optimal set of features, given the great popularity and successes of deep learning-based methods. However, such methods typically require vasts amounts of data for robust model training, in terms of both the number of classes (i.e. human subjects in the case of face recognition) to be recognised, and the number of examples of each class; such vast amounts of data are unavailable in the case of face photo-sketch recognition, since most publicly available datasets only contain 1 sketch image per subject. Consequently, a network would struggle to robustly learn intra-class similarities whilst differentiating between different subjects (inter-class differences). As a result, the use of deep learning for face photo-sketch recognition has been fairly limited in the literature, and such methods tend to use relatively shallow networks which have been shown to be inferior to deeper networks.
The DEEP (face) Photo-Sketch System (DEEPS) framework has been designed to counteract these drawbacks, by applying transfer learning to a state-of-the-art deep convolutional neural network (DCNN) pre-trained for face photo recognition with a set of synthetic face images created by fitting a 3D face morphable model to the original sketch and photo images and then adjusting several facial attributes (in terms of local changes to the eyes, nose, mouth and the rest of the face and global changes which modify the subjects’ appearance of weight, height, gender, and age). In other words, the resultant images depict a subject under different appearance scenarios. The exact parameters used in the 3D morphological model to obtain the synthesised images may be found in the next page.
The method has been described in the paper entitled ‘Matching Software-Generated Sketches to Face Photos with a Very Deep CNN, Morphed Faces, and Transfer Learning‘ when using software-generated sketches in the UoM-SGFS database. It was also implemented for viewed and real-world forensic hand-drawn sketches, as described in the paper entitled ‘Forensic Face Photo-Sketch Recognition Using a Deep Learning-Based Architecture‘, where the synthetic images were also used during the testing stage following the observation that they may bear a closer resemblance to the corresponding photo image than the original sketch. Fusion with a method using engineered features was also shown to improve performance.
A demo of DEEPS is available for download by filling in the resource request form here. The demo consists of a MATLAB code which uses a model trained on software-generated sketch-photo pairs to match a sketch in the UoM-SGFS database to 3 photos (one of which is the subject depicted in the sketch). The results are also shown graphically to facilitate interpretation. Other models (e.g. trained on hand-drawn sketch-photo pairs) are available on request by filling in the contact form here.
Christian Galea 