We summarize the results of the object detection by looking at the 10 most frequent objects overall and per movie. Table 2 and 3 show the objects starting with the most frequent per unit. Freq is the absolute number of detected instances while % is the percentage of frames at least one of the specific objects was detected.

Table 2. Detected objects per movie and overall (part 1).

Table 3. Detected objects per movie and overall (part 2).

Persons are the most frequently detected “objects” (figure 1). Other frequent objects are mostly furniture (book, chair), clothes (tie, handbag) and drinking objects (cup, wine glass).

Figure 1. Frame with the most detected persons ( Metropolis).

Comparing the movies, we identified that movies below 90% of frames with persons are indeed the more action-oriented movies ( Avengers, Metropolis) or include fantasy/animal-like characters ( Wizard of Oz). Many modern objects (e.g cell phones and airplanes) are more frequent in the contemporary movie Avengers (figure 2). One outlier we identified is the clock-object in Metropolis, which is not a frequent object in the other movies but represents a well-studied reoccurring motif of this specific movie (figure 3; cf. Cowan 2007).

Figure 2. Detected airplanes in Avengers.

Figure 3. Clocks as a reoccurring motif in Metropolis.

While we did not perform a systematic evaluation, but we identified a lot of mistakes in the prediction e.g. guns were predicted as handbags or the character “Cowardly Lion” in Wizard of Oz was oftentimes predicted as dog (figure 4).

Figure 4. The “Cowardly Lion” in Wizard of Oz detected as „dog“.

Nevertheless, we see potential in the method of object detection to explore specifics of the mise-en-scène as well as motif-like reoccurring objects in movies (Zaharieva / Breiteneder 2012). Furthermore, as object classes of the COCO dataset are not necessarily fitting for movies, we recommend exploring the possibilities of post-training via Detectron to analyze objects that are not part of the pretrained models.

For the emotion recognition we decided to create an average for a frame if multiple faces are detected. If no face is detected, we mark the frame with missing values. Table 4 summarizes the results. Maximums and minimums are marked in bold.

Table 4. Emotion values per movie and overall (M=mean, Max=maximum, Sd=standard deviation).

Overall, highest averages for emotions are the neutral (M=0.24) and the sad class (M=0.29). Surprise (M=0.11) and disgust (M=0.00) are rather rare among the movies. The two comedies in the movie corpus (Wizard of Oz, Some Like it Hot) do indeed have the highest happy-averages (M=0.13) (figure 5).

Figure 5. Frame with maximum happy value (Some Like it Hot).

However, the results are rather inconsistent since Wizard of Oz has also the highest sad- and angry-averages and therefore is the movie with generally the strongest emotional expressions. Breakfast at Tiffany’s on the contrast is the most neutral movie (M=0.37; figure 6).

Figure 6. Frame with highest neutrality value in the corpus (Breakfast at Tiffany’s).

Additionally, we performed a Welch-ANOVA to investigate if the movies differ to each other significantly (all requirements for the test are met according to Field (2009)). Indeed, we do find significant differences (p<0.05) for all emotion categories but rather small effects according to Cohen (1988) defining η²<0.01 as weak, <0.06 as moderate and <.14 as strong effect. We report the p-, F- and η²-value (table 5).

Table 5. Results of Welch-ANOVA-Tests for all emotion categories

The strongest effect can be seen for neutral. Performing post-hoc tests and inspecting a box-plots graph (figure 7) we identified Breakfast at Tiffany’s as interesting outlier. This might be due to the fact that the main characters of the movie try to stay rather “unaffected” up until the ending of the movie while Wizard of Oz, as a musical, consist of strong emotional outbursts.

Figure 7. Box-plots graph for the emotion class neutral

Table 6 illustrates the descriptive statistics for the gender- and age-detection.

Table 6. Descriptive statistics for age and average gender.

The average age is for most movies is around 40 which is a rather consistent over-estimation since most leading actors in the selected movies are around 30. Performing a Welch-ANOVA shows that the difference between the movies is significant (p<0.001, F=336.07, η²=0.09) with a moderate effect. The strongest outlier movie, as shown with post hoc tests, is Wizard of Oz with a child/teenager as leading actor that gets correctly detected as around 14-16 years old (figure 8).

Figure 8. Lowest age in the corpus (Wizard of Oz).

An average score for gender below 0.5 points to more male detections and it is striking that all movies point below 0.5, thus a more frequent representation of males which is in line with the reality of the movies. There is a significant difference considering gender but with a smaller effect compared to age (p<0.001, F=251.36, η²=0.06) and with the strongest differences concerning Wizard of Oz. The differences become apparent regarding the distribution of gender-classes (table 7). We assigned every frame with male if average gender > 0.6 and female if <0.4. We decided to include a class androgynous for in-between-values pointing to either multiple genders on one screen or uncertainty by the model.

Table 7. Frequency distributions of gender classes.

Wizard of Oz has the most frames classified as androgynous. In general, this means that female and male characters are equally on the frame but in this case the classification is due to the high number of human-like fantasy creatures for which the model is unsure to pick a gender (figure 9).

Figure 9. An “androgynous” face (Wizard of Oz).