Size, range and representativity of the full sample of faces

Although the studies in our review relied on relatively large samples of observers (M = 602; SD = 686), the size of the original sample to create the stimuli was much smaller. Indeed, all the papers included in our final dataset (21Footnote ¹ out of 21) exclusively used the prototypes from the Faceaurus dataset, which were created from a small original sample: just 33 men and 48 women.Footnote ²

The main issue with such a limited sample size is range restriction. Simply put, it is unlikely that with a small number of men (N = 33), we will have respondents with very high scores of psychopathy, for example. This is problematic, as studies using the Faceaurus typically aim to make inferences about ‘psychopathic’ behaviour. For example, Lyons et al. (Reference Lyons, Marcinkowska, Helle and McGrath2015: 157) write: ‘Thus, if information about these types of behaviours can be gleaned from the facial structure, it would be adaptive to avoid these men due to risk of violent injury, or in extreme cases, death’. However, these violent men may be nowhere to be found in the stimuli set. A study with 1,510 Belgian participants (48% men) argued that 43 respondents (2.9%; 2 SD from the mean) scored extremely high on psychopathy (Gordts et al., Reference Gordts, Uzieblo, Neumann, Van den Bussche and Rossi2017). In a sample of 33 men, we can thus expect around one or two male participants to show such an extreme score. We can generate high and low prototypes on a trait which significantly differ in appearance, but it is unlikely that they will represent the extreme of a trait. Thus, while such prototypes are argued to represent, for example, psychopathy, there were (probably) no psychopaths in the stimulus sample to begin with. The same logic applies to other traits collected in the Faceaurus: with small samples the extremes of any given trait of the population at large are probably not represented. If we take as an arbitrary heuristic that 2.5% of the sample score extremely, then around one male and female participant per trait is expected to score extremely high or low on any given trait in the Faceaurus. For some traits, this will imply that there are no extremes included in the generation of any given prototype. Because a sample of population extremes might look very different than one using the top or bottom 10 from a sample of 33 men and 48 women, there is also no guarantee that the features will correspond between a prototype generated from ‘true’ psychopaths drawn from a representative male population vs. a prototype derived from 33 men. This issue is accentuated by the fact that the original sample for the Faceaurus consisted of students, as the range of personality traits in students is expected to be more restricted than a sample of the same size taken from the general population. In sum, for studies using the Faceaurus, it is important to acknowledge its limited range – the true, extreme, trait of interest (e.g. psychopathy) might not be captured at all. Only one publication (one out of 21) discussed the restricted range of personality scores in the original sample (Lyons & Blanchard, Reference Lyons and Blanchard2016). Surprisingly (and worryingly), none of the publications in our dataset (zero out of 21) mentioned the small size of the full original sample as a limitation. Moreover, only one of the publications in our literature review (one out of 21, namely Holtzman Reference Holtzman2011) reported the size of the full original sample from which the faces used for the prototypes were drawn.

Size of the subsamples of faces

Independently of the size, range or representativity of the full original sample, the external validity of the prototypes would be limited by the even smaller subsamples of faces selected to create the prototypes. All the studies in our review used prototypes from the Faceaurus dataset, which are made by averaging 10 individual faces (e.g. the faces of the 10 individuals with the higher scores on extraversion in the full original sample). A sample size of 10 would be considered insufficient for the vast majority of behavioural studies, and we argue that there is no valid reason why a sample used to create facial prototypes should be an exception. Importantly, this issue would remain even if the full original sample size was larger, or if the sampling method was intentionally targeting individuals scoring extremely high or low on a trait (by recruiting clinical populations for example), as a sample of 10 individuals to generate prototypes remains too small to be representative of any group. None of the publications in our dataset (zero out of 21) mentioned the small size of the subsamples as a limitation, and only two-thirds of them (14 out of 21) reported the number of faces used per prototype (i.e. per group). The remaining seven publications did refer to at least one previous publication mentioning this information (for example, Holtzman Reference Holtzman2011 or one of their own previous publications), which makes this critical information technically available but more difficult to retrieve or evaluate.

To conclude, although the lack of external validity of the prototypes from the Faceaurus is both manifest and problematic, this was never mentioned as a limitation in the publications in our dataset. This is intriguing, as external validity was something acknowledged in several of these publications in reference to the sample of observers – which can be seen through the use of large and diverse samples of observers and direct mentions in the text (see Lyons & Blanchard, Reference Lyons and Blanchard2016: 43, for example) – but never regarding the stimuli set.

One prototype per group

In all of the 21 publications included in our final dataset, only one prototype per group (e.g. high extraversion group) was used (the one available in the Faceaurus dataset) to test hypotheses about the whole population itself (e.g. extravert individuals). This creates dependence in the observations and constitutes a case of ‘simple pseudoreplication’ (McGregor, Reference McGregor2000). To illustrate the concept of pseudoreplication with an obvious example (borrowed from McGregor and expanded), imagine you wanted to test whether the height of men differed from that of women. You measure the height of one man six times and of one woman six times. You then perform a t-test using the number of measurements made, N = 12. This is incorrect, as the true number of statistically independent replicates for the test is two and not 12 since there was only one man and one woman. This is a case of pseudoreplication. Now, instead of measuring the height of your subjects six times yourself, let us assume that you ask a sample of 200 participants to come into the lab and measure the height of the two same subjects (each participant measures the height of the woman once and the height of the man once). Now that you have 200 different participants measuring height, your height measurement might be more precise, but the number of statistically independent replicates is still two (one man and one woman). This is because the question you wished to answer was not whether your two subjects differed in height, but the more general question of whether men (in the population) were taller than women (in the population). Assuming that the sample size is 200 per measurement would be committing pseudoreplication. Studies using the ‘prototype method’ do not use one single individual as stimulus, but they do use one single prototype. Although a composite stimulus (e.g. a prototype made of several faces scoring high on extraversion) is better than a single individual face used as a stimulus (e.g. the face of one single individual with a high extraversion score), this approach still constitutes simple pseudoreplication, because only one stimulus is used to make inferences to a whole population (e.g. all extraverts). In our height example, it would correspond to creating a ‘prototype body’ by averaging the body shapes from 15 men and another prototype body averaged from 15 women, then having each prototype body measured once by 200 participants. The height difference will be more accurate than when using only one individual man and one woman, but the sample size for each measurement still is not N = 200. Note that this is an issue only because the research question is not about the perception of these two specific composite stimuli, but about the broader population they represent (e.g. all extraverts). As such, in this case, the non-independence of observations is intrinsically linked to the issue of external validity discussed above. None of the publications in our dataset (zero out of 21) discussed the risk of pseudoreplication or non-independence of observations when using a single prototype per group.

The same prototypes feature in different stimuli

In some publications, the prototypes were directly used as the stimuli presented to the participants (see Alper et al., Reference Alper, Bayrak and Yilmaz2021a for an example). In other publications, more than one facial stimulus was created for each group by using the low and high prototypes to transform a new set of faces (see Brown et al., Reference Brown, Sacco and Medlin2019 for an example). In order to achieve this a new set of individual faces (sometimes called the ‘bases’) was transformed (or ‘morphed’) with one single pair of prototypes from the Faceaurus. As a result, there was a new pair of faces for each base, comprising one ‘high’ and one ‘low’ version, which were used as stimuli. However, these different facial stimuli were always created using the same single prototype per group, meaning that the facial stimuli were not independent of one another. Indeed, this technique would be comparable with presenting the same prototype several times to the participants, but each time with a different background colour: although the overall facial information differed from one pair to another, the variable of interest (in this case the facial information linked to the personality trait of interest) was exactly the same for all the pairs of stimuli. As a result, the same difference in facial information was evaluated several times by each observer (i.e. repeated measures design). The base faces (the individual faces which were transformed to create the new set of stimuli) might of course have impacted the observers’ perception, but this should only shift the baseline, and the difference between the two faces of each pair should remain unchanged. To be fair, in the studies in our dataset, the observers’ responses were aggregated at the prototype level (e.g. averaging the ratings for all the stimuli transformed with the same prototype), removing the statistical issue of non-independence of the observations. Nonetheless, we believe that the used methodology remains a concern, as it gives the false reassuring impression that several independent stimuli were used, thereby inflating the confidence that one can have in the results. This is particularly the case when the description of the method used to create the stimuli is either incomplete or unclear, which was often the case in our dataset.

The same individual faces make up different prototypes

In the Faceaurus dataset, there is some overlap between the prototypes, as the same faces appear in several different prototypes (if we include the Dark Triad and the Big Five only, a male face appears in five prototypes on average, and up to eight prototypes), and some prototypes use similar sets of faces (up to 70% of faces in common). As a result, a prototype for one personality trait is often not independent from the prototype for another personality trait (see Figure 1 for Low Dark Triad male prototypes). This overlap is the result of both the correlation between some personality traits and the small size of the original sample (N = 33 for male faces). This raises two issues: (1) a few individual faces may be responsible for most of the results, even across traits, which makes the results less generalisable (see the ‘External validity’ section); (2) the observations made with one pair of prototype might not be independent of the observations made with another pair, and this non-independence is impossible to detect without going back to the original dataset used to create the prototypes; and (3) resulting from (1) and (2) combined, it is hard to know exactly what is being measured or evaluated by the observers. Given that some of these prototypes have 70% of faces in common (e.g. the high agreeableness and low psychopathy prototypes have seven faces out of 10 in common, see Figure 2), it is difficult to disentangle if participants are detecting high agreeableness or low psychopathy, for example. Although Holtzman explicitly acknowledged this limitation of the Faceaurus dataset (Holtzman, Reference Holtzman2011: 650), only one other publication in our dataset (Alper et al., Reference Alper, Bayrak and Yilmaz2021b) mentioned the fact that the same faces appear in multiple different prototypes.

Figure 1. Venn diagram for the individual faces used to create the Low Dark Triad male prototypes in Faceaurus (N = 33 men). Prototypes consisted of 10 faces each. The three prototypes ‘low psychopathy’, ‘low narcissism’ and ‘low Machiavellism’ have five faces in common. The ‘low psychopathy’ and ‘low narcissism’ prototypes have six faces in common. Similarly, the ‘low psychopathy’ and ‘low Machiavellism’ prototypes share six faces in common, so do the ‘low narcissism’ and ‘low Machiavellism’ prototypes.

Figure 2. Male prototypes examples from the Faceaurus (Holtzman, Reference Holtzman2018). High agreeableness on the left and low psychopathy on the right. These two prototypes have seven faces (out of 10 faces) in common.

Different studies using the same stimuli

In August 2021, we found 21 published papers using prototypes from the Faceaurus to create their facial stimuli, and this number continues to grow. This intensive use of the same dataset adds another layer of dependence in the units of information. Indeed, additional studies finding similar results contribute to giving a false impression that these results are robust. The problem is that these new studies are not independent replications (in that sense they are ‘pseudo-replications’, although this will not usually fall into the classical definition of pseudoreplication; Hurlbert, Reference Hurlbert1984), as they do not use new stimuli datasets. Thus, any new study using the same prototypes only gives us more confidence in the fact that new observers react similarly toward this specific stimuli set (Westfall et al., Reference Westfall, Judd and Kenny2015). It does not, however, give us any additional confidence in the fact that the personality trait can be detected in faces in the population, as it is often assumed. Even when two different studies use different prototypes from the Faceaurus, they might still not represent an independent data point because of the overlap in faces used across prototypes (see above). This leads to inflated confidence in the fact that different personality traits can be detected in faces, as different studies (which could be published in different papers by the same or different authors) using different prototypes from the same dataset give the false impression that they contribute independent data.

While the authors of the publications in our dataset mentioned that they used prototypes from a previous study, this was never mentioned as a limitation, and future replications using new stimuli is suggested as a future direction in only one publication (one out of 21). This is particularly intriguing as the publications we reviewed did sometimes acknowledge the need for replication, but exclusively applied this statement to observers (e.g. Lyons & Blanchard, Reference Lyons and Blanchard2016: 42), and not to the stimuli set used.