Introduction early modern portraits prints

The study of portraits opens opportunities to study cultural, social and artistic dimensions of the early modern period. The use of modern image processing technologies allows to enter new directions in research e.g. the analysis of social and artistic traditions.

This article briefly reports on experiments with image processing systems for a well curated collection of 35,000 portraits mainly from the 15 th to the 19 th century.

The study of portraits has been used for studying art history as well as social and cultural history. The focus of research has been mainly on the individual rather than on its historical and economic context, the media environment and its materiality. Portrait research in art history has often focused on paintings and thus neglected the large quantities of traditions of printed portraits. Portraits can show how individuals wanted to be depicted and which visual traditions and social norms they followed.

Face detection and face recognition have become established fields in computer vision. A variety of applications are based on large pre-trained models consisting of millions of realistic images. In the experiments reported here, we use a state-of-the-art face detection model called OpenFace (Baltrusaitis et al. 2018) to analyse the applicability of those models on printed portraits from the early modern period until the 19 th century.

First, an overview for face detection projects will be given. A short introduction to the portrait collection will follow. In the next part, we describe the setup of the experiment including the pre-processing. The results part first discusses the applicability of the OpenFace model based on different observations. We show notable influences of the printing technique on the face detection rate and observe the model’s confidence based on head size and the number of faces in the image. In the second part, we report trends and correlations in the portrait collection like the gaze direction compared to the body’s orientation. In the conclusion, we discuss the applicability of this approach and the validity of the observed trends.

Related work

OpenFace 2.0 implements multiple tasks from the basic face detection itself to eye and head pose estimation and even the interpretation of facial expressions. State of the art landmark detection models are based on cascading regression models in combination with convolutional neural networks which got enhanced over the last few years (Valle et al. 2019). While OpenFace’s gaze estimation had a mean angular error of 7.9° (Wood et al. 2015) modern approaches like FewShot are able to get as low as 3.14° angular error (Park et al. 2019).

Faces in art have been automatically analysed before. The beauty of faces depicted between the 13 th and 18 th century was studied by de la Rosa and colleagues. For face detection analysis, the authors used 120,000 portraits and received 25,000 portraits with 47,000 faces detected. They observed that the symmetry of faces is the highest between the 15 th and 18 th century. (De la Rosa et al. 2015) This is in line with the FACES study which used a 15 th to 18 th century portrait corpus to compare two different feature detection approaches, due to the more naturalistic representations. The authors also mention that the detection rate depends on the artistic style and artistic quality (Rudolph et al. 2017).

Portraits and image analysis

The Herzog August Library owns a collection of over 35,000 portraits which are used as a foundation for the creation of an image recognition model. The goals of ongoing research is to find and use similarities to identify origins of portraits, their composition and other important visual features. We hope to find similarities between collections of portraits, conventions of representations, group specific symbolism and submittals for derived works and copies. We use OpenFace which was previously tested in a pre-study (Üresin 2020). With this tool we hope to gain first insights which might be useful in later applications.

Experiment setup

Before the analysis, portraits needed to be cropped, so that unnecessary elements in the digital image do not distort the evaluation. Cropped portraits were extracted based on the most outer portrait frame. A non-pretrained DLA-34 Neural Network (Yu et al. 2018) was used for this process. Due to the variety in the dataset about 20,000 portraits from about 27,000 have been successfully cropped and were used further for the face detection.

OpenFace offers different combinations of face and landmark detectors where the recommended detectors have been used based to the most promising results. As a result, OpenFace created a csv file containing the confidence of the model, eye position, gaze direction, eye landmarks, face landmarks, face position and face rotation for every found face. The metadata of the portraits provides useful information regarding the year the portrait was created, the body orientation of the person in the portrait and the printing technique if available.

Face detection results

From the 19,913 portraits OpenFace detected 6,267 potential faces. For 518 portraits multiple faces were detected. This results in a detection rate of about 28.9% which is in line with previous studies (De la Rosa et al. 2015) and an evaluation of 250 protraits in a pre-study (Üresin 2020).

The detection rate can further be dissected using insights of the different printing techniques from the metadata. Of the 26 printing techniques present inside the corpus, only 18 techniques led to successful identifications of faces. Table 1 shows the detection rate per technique for techniques over 100 examples. It is clear, that Steel engraving has the highest detection rate with 59.3% whereas Lithography and Mezzotint have a detection rate above 40%. Copper Engraving, which represents by far the biggest portion in the dataset, has a detection rate of 28%. It is also important to notice that Wood Cut portraits have a detection rate of only 3.4%. This indicates that there are substantial differences between the detectability of different techniques by a modern computer vision approach.

Table 1

In a next step, the certainty of the model is measured with a confidence value. The confidence over all detected faces measured 88.6%.

Figure 1

91.6% of all found faces have a confidence above 90% and 6.7% below 10% confidence which indicates that the model is either very confident or extremely unconfident. Comparing the confidence with the head size a correlation can be observed where the smallest head sizes have the lowest confidence and contain the most false-positives, like Figure 1 (based on manual evaluation). We can conclude that the confidence value is helpful for identifying errors.

For multiple faces per portrait the biggest head size is the first recorded detection. The confidence for the first detection has an average of about 90.1% confidence whereas for every next detection it drops to 71.1%. Based on these results we filtered out detections with a confidence below 40% due to their unreliability. Using this approach, we filtered out 443 portraits.

Face Orientation

The orientation of a portrait is one of the artist’s decisions and an important feature for analysing artistic traditions. Besides the position of the eyes in the portrait OpenFace also gives the gaze direction for each eye, visually indicated by a gaze vector. This vector has x,y,z coordinates to indicate the direction the person is looking outgoing from the eye of the portraited person. In x-dimension 3448 (59,1%) persons look to the left and 2384 to the right based on the observer’s point of view. Most of the gazes are around -15° to +15°. In y-dimension 95,7% look down with the majority between -30° and -15°. The horizontal head rotation is measured by the Pose_Ry (yaw) where 3431 (58,8%) heads are oriented to the right as seen in Figure 2. For both orientations, the focus lies between 5° and 20°.

Figure 2

Looking at the head pose distribution over the years from about the 15 th to the 19 th century no visible trend is observable (Figure 3). The general head pose distribution of 60% to 40% is still visible based on the opacity of the point clouds.

Figure 3

Lastly, a correlation between the eye, face and body orientation can be seen. Comparing the eye to the face orientation a strong negative correlation can be calculated, which indicates that the gaze direction is opposite to the face orientation. Comparing the body orientation to the gaze direction (Figure 4a) this opposing trend can also be observed, while the face orientation is equal to the body orientation as seen in Figure 4b. This implies that the body and the face are oriented in one direction and the eyes in the opposite direction, which would make sense of the eyes were to look at the viewers point of view. This correlates with the observations made like in Figure 2.

Figure 4

Conclusion

Our analysis showed that state-of-the-art approaches like OpenFace can be applied on very different kinds of dataset like printed portraits out of the box. Even though the overall detection rate is very low with only every fourth portrait detected, the detected faces have a low error rate and basic features of faces can be correctly identified and used for mass processing and trend analysis. It also showed that there are substantial differences regarding the detection rate based on the printing technique. Therefore, further explorations in this field could be useful.