Introduction
Discrimination has been defined as the behavioural aspect of stigma; the enactment of problems of knowledge (ignorance or misinformation) and problems of attitudes (prejudice) (Thornicroft, Rose, Kassam, & Sartorius, Reference Thornicroft, Rose, Kassam and Sartorius2007). Discrimination represents a pervasive global violation of the human rights of individuals who are experiencing disability due to mental illness (Drew et al., Reference Drew, Funk, Tang, Lamichhane, Chávez, Katontoka and Saraceno2011).
It is well-evidenced that parity of esteem is not reflected in the funding of mental health services and access to both general medical and mental healthcare treatment for individuals with a mental illness (Angermeyer, Matschinger, Link, & Schomerus, Reference Angermeyer, Matschinger, Link and Schomerus2014; Docherty & Thornicroft, Reference Docherty and Thornicroft2015; Hilton, Reference Hilton2016). This has led to a global treatment gap in conditions such as major depressive disorder where only 1 in 5 people in high-income and 1 in 27 in low/lower-middle-income countries receive a minimally appropriate level of treatment (Thornicroft et al., Reference Thornicroft, Chatterji, Evans-Lacko, Gruber, Sampson, Aguilar-Gaxiola and Borges2017). In this regard, mental illness stigma and discrimination impact notably on population health (Hatzenbuehler, Phelan, & Link, Reference Hatzenbuehler, Phelan and Link2013).
Structural and social capital barriers are also present in employment, education, housing, child custody and criminal justice settings (Brouwers et al., Reference Brouwers, Mathijssen, Van Bortel, Knifton, Wahlbeck, Van Audenhove and Ballester2016; Jeffery et al., Reference Jeffery, Clement, Corker, Howard, Murray and Thornicroft2013; Webber et al., Reference Webber, Corker, Hamilton, Weeks, Pinfold, Rose and Henderson2014). In personal relationships individuals also experience discrimination ranging from micro-aggressions such as invalidation of experience to physical, emotional, financial and sexual abuse and exploitation (Barber, Gronholm, Ahuja, Rüsch, & Thornicroft, Reference Barber, Gronholm, Ahuja, Rüsch and Thornicroft2020; Bhavsar, Dean, Hatch, MacCabe, & Hotopf, Reference Bhavsar, Dean, Hatch, MacCabe and Hotopf2019; Drew et al., Reference Drew, Funk, Tang, Lamichhane, Chávez, Katontoka and Saraceno2011). This is further reflected in the role of fear in characterising the experiences of mental health services users (Sweeney, Gillard, Wykes, & Rose, Reference Sweeney, Gillard, Wykes and Rose2015).
To evidence the discrimination experienced by individuals with a mental illness globally, a robust measurement approach is required. The Discrimination and Stigma Scale (DISC) was developed as part of the International Study of Discrimination and Stigma Outcomes (INDIGO) (Thornicroft, Brohan, Rose, Sartorius, & Leese, Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009), a global research collaboration to evidence these experiences. The most recent version of this scale, DISC-12, is a 32-item structured interview designed to assess the experienced and anticipated discrimination in people with a mental illness. DISC has shown strong psychometric properties (Brohan et al., Reference Brohan, Clement, Rose, Sartorius, Slade and Thornicroft2013) and has been widely used in over 50 countries with individuals with schizophrenia (Thornicroft et al., Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009), depression (Lasalvia et al., Reference Lasalvia, Zoppei, Van Bortel, Bonetto, Cristofalo, Wahlbeck and Reneses2013) and bipolar disorder (Farrelly et al., Reference Farrelly, Clement, Gabbidon, Jeffery, Dockery, Lassman and Howard2014). In the study by Farrelly et al. (Reference Farrelly, Clement, Gabbidon, Jeffery, Dockery, Lassman and Howard2014), similar levels of experienced discrimination were reported by those with depression, bipolar disorder or schizophrenia, with 87% experiencing discrimination in at least one area of life in the previous year. DISC-12 also collects qualitative examples of the experiences that participants are referring to when they provide a rating for their experience of discrimination. Further detail on the conceptual underpinnings of DISC, and similarities and differences between DISC and other measures of stigma and discrimination, are available elsewhere (Brohan et al., Reference Brohan, Clement, Rose, Sartorius, Slade and Thornicroft2013; Brohan et al., Reference Brohan, Milenova, Bakolis, Evans-Lacko, Kohrt, Thornicroft, Vogel and Wadein press; Brohan, Slade, Clement, & Thornicroft, Reference Brohan, Slade, Clement and Thornicroft2010).
The ability to distil a longer scale into a short-form which retains the core aspects of the original, and demonstrates appropriate psychometric properties, is highly advantageous when the administration of the full measure is not feasible or necessary. This is particularly pertinent in low resource settings and a necessary step in ensuring that barriers to the global assessment of stigma and discrimination are reduced by shortening measures where possible (Bakolis et al., Reference Bakolis, Thornicroft, Vitoratou, Rüsch, Bonetto, Lasalvia and Evans-Lacko2019; Brohan et al., Reference Brohan, Milenova, Bakolis, Evans-Lacko, Kohrt, Thornicroft, Vogel and Wadein press). Beidas et al. (Reference Beidas, Stewart, Walsh, Lucas, Downey, Jackson and Mandell2015), suggest that the following properties are key in measures for use in low resource setting: freely and easily accessible; brief; have established psychometric properties; and high relevance. DISC-Ultra Short (DISCUS) aligns closely with these requirements by reducing participant burden, while remaining free to use, easily accessible and highly relevant. This paper provides evidence on the psychometric properties.
A full discussion of the psychometric properties of stigma and discrimination measures is presented elsewhere (Brohan et al., Reference Brohan, Milenova, Bakolis, Evans-Lacko, Kohrt, Thornicroft, Vogel and Wadein press). A short-form version of the DISC, DISCUS, has recently been developed based on a secondary analysis of INDIGO network data with the use of novel meta-analytic factor analysis methods to perform item reduction and examine measurement invariance in a global sample. This resulted in an 11-item version which was further tested in a secondary analysis of a diverse London-based sample (Farrelly et al., Reference Farrelly, Clement, Gabbidon, Jeffery, Dockery, Lassman and Howard2014). Excellent agreement was observed between corresponding experienced discrimination scores on the 21-item experienced discrimination scale of DISC-12 and short-form DISCUS (Pearson's correlation ρ = 0.95) (Bakolis et al., Reference Bakolis, Thornicroft, Vitoratou, Rüsch, Bonetto, Lasalvia and Evans-Lacko2019). DISCUS was also found to have appropriate internal consistency (α = 0.87) and convergent validity [Brief Psychiatric Rating Scale (BPRS) r = 0.35, Internalised Stigma of Mental Illness (ISMI), r = 0.35] (Bakolis et al., Reference Bakolis, Thornicroft, Vitoratou, Rüsch, Bonetto, Lasalvia and Evans-Lacko2019). The anticipated discrimination aspects of DISC-12 have previously been developed into a stand-alone measure of anticipated discrimination and are not the focus of this current work (Gabbidon, Brohan, Clement, Henderson, & Thornicroft, Reference Gabbidon, Brohan, Clement, Henderson and Thornicroft2013).
This paper seeks to evaluate whether the psychometric properties the DISCUS can be replicated in a sample of individuals with a diagnosis of depression, bipolar disorder, schizophrenia or an anxiety disorder in 15 countries/territories across six global regions. This is the first time that primary data have been collected using the short-form DISCUS and builds on the secondary analysis conducted by Bakolis et al.
The aims of this study are:
1. To evaluate the psychometric properties of the DISCUS in a primary data sample including: (i) scaling assumptions or the extent to which each DISCUS item contributes to a unidimensional scale producing a total experienced discrimination score; (ii) construct validity or the ability of DISCUS to yield consistent, reproducible estimates of the construct by assessing hypothesised relationships with similar constructs (convergent validity) and within distinct groups e.g. clinically different subgroups (known groups); (iii) reliability or the ability of DISCUS to yield consistent estimates of the construct under consideration across item responses (internal consistency).
2. To provide evidence on the appropriateness of using DISCUS to measure the discrimination experienced by individuals with mental illness in each of six global regions and five diagnosis categories.
Methods
Participants
Participants were recruited from 15 countries/territories: Brazil, China, Czech Republic, Germany, Hungary, India, Italy, Netherlands, Nigeria, Portugal, South Africa, Spain, Taiwan, Tunisia, Turkey. Sites were participating in the INDIGO anti-stigma programme and recruited participants locally through health services or service user organisations as has been done previously in the original DISC study (Lasalvia et al., Reference Lasalvia, Zoppei, Van Bortel, Bonetto, Cristofalo, Wahlbeck and Reneses2013; Thornicroft et al., Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009). More details on sites are presented in online Supplementary Table S1. Each site recruited a minimum of 30 participants. Inclusion criteria specified that participants were at least 18 years of age, were using secondary mental health services or had a clinician-diagnosed mental illness, and had the capacity to consent. Individuals with comorbid mental illness were included, but the sample excluded those who had a primary diagnosis of substance abuse, personality disorder, dementia or other neurological condition without a major mental illness.
Data collection
Data collection took place in the period 2018–2019. Translation and cross-cultural adaptation of study measures were completed in line with the principles adopted in previous INDIGO network and partnership studies (Lasalvia et al., Reference Lasalvia, Zoppei, Van Bortel, Bonetto, Cristofalo, Wahlbeck and Reneses2013; Thornicroft et al., Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009). This included translation, back translation and concept checking (Knudsen et al., Reference Knudsen, Vázquez-Barquero, Welcher, Gaite, Becker, Chisholm and Thornicroft2000; Thornicroft & Evans-Lacko, Reference Thornicroft and Evans-Lacko2016). Fourteen different language versions of DISCUS were developed in this study and all translated versions are freely available from the Indigo website (http://www.indigo-group.org/stigma-scales/).
The study was approved by the appropriate ethical review board in each of the sites and by King's College London. Participants provided written informed consent. Sites could choose to administer DISCUS by interview or in a patient-completed written format. The majority were completed by interview with only two sites (Netherlands, China) reporting some use of the patient-completed option. Participants completed the DISCUS and a range of additional measures to establish known groups and convergent validity. All measures are listed below.
Measures
Discrimination and Stigma Scale (DISCUS)
The DISCUS contains 11 items which assess aspects of experienced discrimination (Bakolis et al., Reference Bakolis, Thornicroft, Vitoratou, Rüsch, Bonetto, Lasalvia and Evans-Lacko2019). It is a short-form version of DISC-12. Further detail on the development and psychometric validation of DISC-12 and the process of item-reduction and development of DISCUS is provided elsewhere (Bakolis et al., Reference Bakolis, Thornicroft, Vitoratou, Rüsch, Bonetto, Lasalvia and Evans-Lacko2019; Brohan et al., Reference Brohan, Clement, Rose, Sartorius, Slade and Thornicroft2013; Thornicroft et al., Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009). Each DISCUS item is rated on a 4-point Likert scale anchored between 0 = not at all and 3 = a lot. A non-applicable response option was also available.
Stopped contact
Three items, adapted from DISC-12, were included to assess the extent to which individuals were socially excluded in various aspects of their personal life including stopped contact by friends; partners (boyfriend/girlfriend/husband/wife) and family members (Thornicroft et al., Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009). Each item rated on a 4-point Likert scale anchored at 0 = not at all and 3 = a lot. Higher scores indicate a greater level of reduced contact.
Stigma stress
A 2-item short version of the original 8-item Stigma Stress Scale (Rüsch et al., Reference Rüsch, Corrigan, Wassel, Michaels, Olschewski, Wilkniss and Batia2009) assessed perceptions of stigma as a stressor, based on the perception of harm and coping resources. One item assessed perceived stigma-related harm, the other item perceived resources to cope with the threat of stigma. Each item was rated on a 7-point anchored numerical response scale (NRS): 1 = strongly disagree to 7 = strongly agree. Stigma stress was then calculated as the difference score (harm item minus coping resource item), with higher scores from −6 to +6 indicating higher stigma stress.
Stigma consciousness
Three items were included to assess stigma consciousness or the extent to which an individual considers that there is a public stigma towards people with a mental health problem and whether it impacts their interactions. The items were adapted from the Brief Stigma Consciousness Scale (Pinel, Reference Pinel1999). Each item was rated on a 4-point Likert scale anchored at 1 = strongly agree and 4 = strongly disagree. Higher scores indicate a higher level of stigma consciousness.
Patient Health Questionnaire (PHQ-2)
The PHQ-2 is a short-form measure of depression (Kroenke & Spitzer, Reference Kroenke and Spitzer2010). It contains the first 2 items of PHQ-9, which constitute the two core DSM-IV items for major depressive disorder. Each item is rated on a 4-point Likert scale anchored at 0 = not at all and 3 = nearly every day. The total score ranges from 0 to 6. Higher scores indicate higher levels of depressive symptoms. A score of >3 has been used as a cut-point when screening for major depressive disorder (Löwe, Kroenke, & Gräfe, Reference Löwe, Kroenke and Gräfe2005).
Beck Hopelessness Scale (BHS-4)
The BHS-4 is a short-form measure of hopelessness derived from the 20-item item BHS (Aish & Wasserman, Reference Aish and Wasserman2001; Beck, Weissman, Lester, & Trexler, Reference Beck, Weissman, Lester and Trexler1974). Each item is rated on a 6-point Likert scale anchored at 1 = strongly agree and 6 = strongly disagree. Total scores range from 4 to 24. Higher scores indicate a higher level of hopelessness. A score of 11 has been used as a cut-point indicating a higher level of hopelessness (Yip & Cheung, Reference Yip and Cheung2006).
Suicidal Ideation Attributes Scale (SIDAS)
The SIDAS is a five-item measure of suicidal ideation (Van Spijker et al., Reference Van Spijker, Batterham, Calear, Farrer, Christensen, Reynolds and Kerkhof2014). Each item is rated on an 11-point NRS anchored at 0 = never to 10 = always. Total SIDAS scores range from 0 to 50. A score of 21 has been used as a cut-point to indicate a high risk of suicide behaviour (Van Spijker et al., Reference Van Spijker, Batterham, Calear, Farrer, Christensen, Reynolds and Kerkhof2014).
Internalised Stigma of Mental Illness (ISMI-10)
The ISMI-10 is a 10-item short-form measure of internalised stigma derived from the 29-item ISMI (Boyd, Otilingam, & DeForge, Reference Boyd, Otilingam and DeForge2014). Each item is rated on a 4-point Likert scale anchored at 1 = strongly agree and 4 = strongly disagree. Total scores range from 0 to 4 with higher scores indicating higher internalised stigma. A two-category cut-point method for interpretation of scores has been proposed with a total score of 1.00–2.50 indicating lower internalised stigma and 2.51–4.00 indicating high internalised stigma for the 29-item ISMI (Ritsher, Otilingam, & Grajales, Reference Ritsher, Otilingam and Grajales2003).
Warwick-Edinburgh Well-being Scale (WEMWBS)
The WEMWBS is a 14-item scale which assesses mental wellbeing (Tennant et al., Reference Tennant, Hiller, Fishwick, Platt, Joseph, Weich and Stewart-Brown2007). Each item is scored on a 5-point frequency response Likert scale anchored at 1 = none of the time and 5 = all of the time. Total scores range from 14 to 70 with higher scores indicating higher positive mental well-being. A score of ≤42 on WEMWBS can be used as a cut-point to indicate low mental wellbeing consistent with estimates of clinically relevant levels of depression in the UK population as assessed in the CES-D (Stewart-Brown, Samaraweera, Taggart, Kandala, & Stranges, Reference Stewart-Brown, Samaraweera, Taggart, Kandala and Stranges2015).
Socio-demographic and illness-related variables
Participants completed self-report questions on socio-demographic and clinical variables. Socio-demographic variables included: gender, age, education, housing situation and employment. Illness-reported variables included self-reported diagnosis, age at first diagnosis and disclosure of diagnosis.
Psychometric evaluation of DISCUS
Descriptive statistics were calculated for all included measures. For DISCUS, scores were calculated at both the item and the scale level. The psychometric analysis of DISCUS focused on three aspects: (1) confirming the scaling assumptions of DISCUS; (2) evaluating scale reliability; (3) evaluating construct validity.
When >0 <2 item scores were missing (49/1195 cases) then a total score was still calculated. When >2 item scores were missing the mean score was not calculated (10/1195 cases). As a sensitivity analysis, scores were also calculated using median imputation for non-applicable and missing responses where the median response of all answered items for that individual item were used in place of non-applicable and missing responses.
Country-level data were grouped into six regions according to the United Nations statistics division geoscheme (http://unstats.un.org/unsd/methods/m49/m49regin.htm). Grouping decisions were guided by maximising the sample size available within each region.
Analysis was performed using SPSS version 26 and Stata version 16.
Discus scaling assumptions
DISCUS is assumed to be unidimensional based on previous research (Bakolis et al., Reference Bakolis, Thornicroft, Vitoratou, Rüsch, Bonetto, Lasalvia and Evans-Lacko2019). Polychoric correlations were calculated between each item pair to ensure that all items are providing distinct information. Items were then summed and averaged to calculate a DISCUS total score. Corrected item-total correlations <0.30 were used to indicate the unacceptable fit of the items with the DISCUS total score (Terwee et al., Reference Terwee, Bot, de Boer, van der Windt, Knol, Dekker and de Vet2007).
Multi-group confirmatory factor analysis was conducted to identify if the same construct is being measured across regions. To evaluate the overall model fit, the comparative fit index (CFI) and the root mean square error of approximation (RMSEA) were calculated. A CFI value of greater than 0.90 indicates an adequate fit to the data (Hu & Bentler, Reference Hu and Bentler1999). A value of RMSEA < 0.05 indicates close fit, values between 0.05 and 0.08 suggest adequate model fit, and values >0.10 suggest poor model fit (Hu & Bentler, Reference Hu and Bentler1999). The standardised root means squared residual (SRMR) was also calculated to examine how the model functions to reproduce the relationships from the input covariance matrix with an SRMR of <0.1considered acceptable (Kline, Reference Kline2015). Postestimation modification indices were examined to identify pairs of variables where allowing correlation of the error terms may improve model fit e.g. modification index <3.84 (Acock, Reference Acock2013). The selection of included correlated error terms was not purely data-driven, and the conceptual rationale for each inclusion was discussed by the authors. A final model was then run.
To examine whether the same construct is being measured across diagnostic groups, a second model was run which used the diagnosis as the grouping variable rather than region. The diagnosis was categorised as: Depression, Anxiety Disorder, Schizophrenia, Bipolar Disorder or Do not know diagnosis. Participants who reported other primary diagnoses were excluded from this analysis, though are included elsewhere in this paper. This model included the correlated error terms identified in the region model.
Reliability
The reliability of DISCUS was assessed by considering internal consistency using Cronbach's α with a criterion of α > 0.70 indicative of appropriate internal consistency for each subscale (Cronbach, Reference Cronbach1951). α > 0.90 were also flagged, as this may indicate item redundancy.
Construct validity
Construct validity provides evidence that scores on an instrument are related to other measures, or participant characteristics in a hypothesised manner. Two aspects of construct validity were assessed: (1) convergent validity; (2) known groups method [See (Brohan et al., Reference Brohan, Milenova, Bakolis, Evans-Lacko, Kohrt, Thornicroft, Vogel and Wadein press)]. In convergent validity analyses, the following relationships were hypothesised [high correlation ⩾0.50 <0.80, moderate correlation ≥0.30 <0.50, and lower correlation <0.30 (Cohen, Reference Cohen1988)]:
1. Moderate/high positive correlation between mean DISCUS score and ISMI-10 self-stigma total score
2. Moderate/high positive correlation between mean DISCUS and stopping-self total score
3. Moderate positive correlation between mean DISCUS score and stigma stress difference score
4. Moderate positive correlation between mean DISCUS score and stigma consciousness total score
The known-groups method assessed differences in scores between participants who differed on identified clinical variables. The criterion was considered to be met when significantly different DISCUS scores (defined as p value <0.05) were obtained between the sample subgroups. The following groups were considered:
1. PHQ-2 (lower depression score <3 v. higher depression score ⩾3)
2. BHS-4 (lower hopelessness <11 v. higher hopelessness >11)
3. SIDAS (not high-risk suicide behaviour <21 v. high-risk suicide behaviour ⩾21)
4. WEMWBS (higher mental wellbeing ⩾42 v. lower mental wellbeing <41)
Results
Sample characteristics
1195 people participated in this study. Socio-demographic and clinical characteristics are reported in Table 1. These are available by region in online Supplementary Table S1. Table 1 also presents descriptive statistics for each study measure with a categorical interpretation of scores based on cut-points is further presented where available. Scores suggest that 46.4% of the sample had low mood indicative of major depressive disorder using PHQ-2. According to BHS-4, 54.4% had higher levels of hopelessness and 12.4% were at high risk of suicidal behaviour using SIDAS. 31.3% reported high internalised stigma. WEMWBS scores suggested that 40% had low mental wellbeing.
Table 1. Demographic and clinical characteristics for study sample (n = 1195)
a Note as an inclusion criterion all patients had a clinician-reported primary diagnosis of either schizophrenia, depression, bipolar disorder or anxiety disorder. The categories provided here represent the diagnoses reported by participants.
b Median and inter-quartile range presented rather than mean (s.d.) due to non-normality of scores.
The median DISCUS score was 0.45 (Interquartile range: 0.82) with a possible range of 0–3. This ranged from 0.38 (0.41) in Southern Europe to 0.88 (0.69) in Latin America. The most frequently reported experiences of discrimination were being shunned or avoided at work by (49.6%) and discrimination in making or keeping friends (47.3%). 13 participants (1.09%) reported no experiences of discrimination across any all items. The frequency and percentage of response category endorsement for each DISCUS item by region is displayed in online Supplementary Table S2.
Psychometric evaluation of DISCUS
The results of the psychometric evaluation of DISCUS are presented in Table 2. No differences in the interpretation of results were seen in the sensitivity analysis using median imputation to assign scores for non-applicable and missing responses.
Table 2. Summary of psychometric properties of DISCUS by region
a See Fig. 2 for results of multi-group confirmatory factor analysis.
b α increased to 0.78 with the removal of DISCUS item 4: housing. No other increases with item deletion were seen in any region.
c Spearman's ρ was used for convergent validity analysis due to non-normality of DISCUS mean scores.
d **Indicates p < 0.001, *indicates p < 0.05, NS indicates non-significant.
e Wilcoxon rank-sum test was used to compare median scores for known groups analysis due to non-normality of DISCUS mean scores.
f Median DISCUS scores are displayed with the significance level indicated on the upper result for each pair.
DISCUS scaling assumptions
Polychoric correlations did not indicate item-redundancy with an average correlation of 0.41 in the total sample (average range 0.33–0.47 across regions). The corrected item-total correlations were generally in the moderate range (0.36–0.57 in the total sample).
The 11 DISCUS items all loaded significantly on a single dimension with standardised loadings by region ranging from 0.29 to 0.49 for Item 4: Education to 0.49–0.74 for Item 7: Social life and 0.49 to 0.75 for Item 11: Shunned at Work. Significant loadings were seen for each item across each of the six regions. The one factor CFA solution for DISCUS had a reasonable fit, as represented in Fig. 1. The RMSEA value of 0.08, and SMSR <0.10 suggest acceptable model fit while the CFI is slightly lower than the threshold of 0.90, at 0.87. The significant χ2 test indicates the model is not a perfect fit. This model includes correlation of the following error terms (Item 2: Dating and Items 8: Privacy, 9: Safety and 10: Children), (Item 3: Housing and Items 7: Social life, 8: Privacy, 9: Safety and 10: Children), (Item 5: Finding work and Item 6: Keeping work), (Item 6: Keeping work and Item 8: Privacy, 10: Children), (Item 8: Privacy and Item 9: Safety). SMSR was also examined at the regional level with all regions having scores <0.1 (range 0.053–0.095) apart from Asia, SMSR = 0.157.
Fig. 1. Multi-group CFA model for DISCUS by region1. 1The ranges for the loadings and errors reflect the standardised figures for each of the six regions. RMSEA, Root mean square error of approximation; CFI, Comparative fit index; SRMR, Standardised root mean square residual.
When this model was run with diagnosis as a grouping variable, rather than region, significant loadings were seen for each item across each of five diagnostic categories (Depression; Anxiety Disorder; Bipolar Disorder; Schizophrenia; do not know mental diagnosis). The one factor CFA solution for DISCUS is graphically represented in Fig. 2. The RMSEA of 0.070, CFI of 0.904 and SMSR <0.10 suggest acceptable model fit. SMSR was also examined by diagnosis category with all categories having scores <0.1 (range 0.060 to.084).
Fig. 2. Multi-group CFA model for DISCUS by region and diagnosis category1. 1The ranges for the loadings and errors reflect the standardised figures for each of the five diagnosis groups. RMSEA, Root mean square error of approximation; CFI, Comparative fit index; SRMR, Standardised root mean square residual.
Reliability
Cronbach's α coefficient for the eleven-item DISCUS was strong at 0.82. This ranged from 0.74 in Southern Europe to 0.84 in Africa. Alphas did not substantially increase with item deletion across any region.
Construct validity
There was evidence of convergent validity in the total sample and in most regions. Moderate/high correlation was seen between mean DISCUS and stopping-self total score and moderate correlation between mean DISCUS score and stigma consciousness total score was achieved in the total sample and all regions fully meeting the threshold. Moderate correlations between mean DISCUS score and stigma stress/coping total score were achieved in the total sample but not all regions with scores lower in Western and Eastern Europe and non-significant in Africa. Moderate/high correlations were seen between mean DISCUS score and ISMI-10 self-stigma score in the total sample and all regions apart from Eastern Europe where scores were slightly lower at ρ = 0.28 and Eastern Europe ρ = 0.29.
The Wilcoxon rank-sum test was used to compare median scores for known groups analysis. There was evidence of construct validity using the known groups method in the total sample with a significant difference between groups seen for each hypothesised relationship. In each region, a minimum of 50% of the hypothesised group differences were observed.
Discussion
DISC has been widely used to assess mental illness related experienced discrimination worldwide and provides a comprehensive picture of these experiences (Farrelly et al., Reference Farrelly, Clement, Gabbidon, Jeffery, Dockery, Lassman and Howard2014; Lasalvia et al., Reference Lasalvia, Zoppei, Van Bortel, Bonetto, Cristofalo, Wahlbeck and Reneses2013; Thornicroft et al., Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009). Previous research suggests that DISCUS, an 11-item, short-form version developed and confirmed using secondary data, produces an experienced discrimination score which is equivalent to DISC-12 scores (Bakolis et al., Reference Bakolis, Thornicroft, Vitoratou, Rüsch, Bonetto, Lasalvia and Evans-Lacko2019). This current study advances this work by providing evidence that the unidimensional nature of experienced discrimination, as assessed using DISCUS, is replicated in a primary data sample from 21 sites in 15 countries/territories encompassing six global regions. Unidimensionality was also supported across five diagnosis categories (Depression, Anxiety Disorder, Schizophrenia, Bipolar Disorder and ‘Do not know diagnosis)’. Further support for convergent validity was provided in relation to self-stigma, stopping oneself from engaging in social contact and stigma stress, across all six global regions. Internal consistency was strong across all regions. The known groups method also provided evidence that experienced discrimination is greater among those experiencing higher levels of depression (PHQ-2: p < 0.001), higher hopelessness (BHS-4, p < 0.001), higher risk of suicidal behaviour (SIDAS, p < 0.001) and lower mental wellbeing (WEMWBS, p < 0.001). This work is further supported by a sensitivity analysis using median imputation of DISCUS item scores. This resulted in a very limited change to the interpretation of results suggesting that the non-applicable and missing responses were not providing meaningfully different information whether they were scored as 0 ‘not at all’ or using the median value.
This suggests that DISCUS provides a short-form, reliable and valid approach for measuring experienced discrimination which performs similarly across a range of global regions and diagnosis groups. It is suitable for use where full information on the content of discrimination experiences (as provided by DISC) is not required or not feasible within the setting. It offers detail on the content of experiences within the most key items, as well as a mean item score which is highly comparable with that produced by the full version.
DISCUS draws on macro, meso and micro-level experiences of discrimination with a focus on micro-level experiences. The most frequently reported experiences in this study were ‘being shunned or avoided at work’ by 48.7% of global participants (Item 11) and ‘discrimination in making or keeping friends’ by 47.2% (Item 1). This is in keeping with previous studies (Bakolis et al., Reference Bakolis, Thornicroft, Vitoratou, Rüsch, Bonetto, Lasalvia and Evans-Lacko2019). There is some variation across regions with ‘discrimination in keeping a job’ most frequently reported in Latin America. This provides support for interventions that focus on addressing loneliness [e.g. (Castelein, Bruggeman, Davidson, & Gaag, Reference Castelein, Bruggeman, Davidson and Gaag2015; Vogel et al., Reference Vogel, Swart, Slade, Bruins, van der Gaag and Castelein2019)] or employment challenges [e.g. (Janssens, van Weeghel, Henderson, Joosen, & Brouwers, Reference Janssens, van Weeghel, Henderson, Joosen and Brouwers2020)] among individuals with a mental illness.
It is important to note that although a unidimensional model of experienced discrimination is appropriate across all regions, this does not mean that the nature of experiences is the same. This is supported by the region-level fit statistics which suggest that the model may fit less well for Asia with greater residual variance here, as indicated by the higher SRMR. A different pattern of non-applicable responses was noted in Asia which may contribute to this. For example, for the Education-related item (Item 4), 63.6% of responses were non-applicable with the next highest 46.9% for Southern Europe. Further examination of the open-text responses collected using DISCUS could help to explain some of the reasons for this difference but is beyond the scope of this current paper. Discrimination experiences are compounded in an intersectional way for individuals who experience discrimination because of other minority characteristics as well as mental illness including race/ethnicity, sexuality, gender, age, and comorbid mental or physical disabilities (Hall et al., Reference Hall, Farrelly, Hatch, Henderson, Williams, Bhugra and Clement2014). Further work is still required to understand how these multiple discrimination experiences influence the safety, health and wellbeing of individuals with mental illness.
The correlated error terms most commonly included in the CFAs relate to Item 8: Privacy (correlated with Item 2: Dating, Item 3: Housing, Item 6: Keeping a job and Item 9: Safety) and Item 9: Safety (correlated with Item 2: Dating, Item 3: Housing, Item 8: Privacy). This fits with evidence that fear is a pervasive theme in understanding mental health service users experiences and suggests that privacy and safety violations may underlie experienced discrimination in other areas (Sweeney et al., Reference Sweeney, Gillard, Wykes and Rose2015). Further analysis of the qualitative responses to understand the underpinning nature of safety and privacy concerns on other experiences of discrimination is an area for future research.
Limitations
As diagnoses were self-reported by participants recruited through clinical settings, the extent to which diagnoses correspond to those recorded in clinical notes is unknown. However, as the focus of this work is to assess discrimination experiences then arguably it is most important to understand participants own understanding of their diagnosis as this is the information that they use in making sense of their experiences of discrimination.
Analysis using the known groups method did not reach significance across all regions. The unequal distribution of responses across categories within certain regions is likely to be a limiting factor. This is particularly true for SIDAS where low numbers selected a response indicating ‘high risk' e.g. n = 4 in Africa. As this is the most severe indicator of mental ill-health in the study then an unequal split in subgroup membership is to be expected, however, this is magnified at the regional level. Although significance was still found in 3 of the 6 regions, with scores in the expected direction for a further two regions, it may be most accurate to consider the total sample for these analyses.
There are also limitations in the use of a convenience sample of participants and in the regional groupings. Although established regional groupings have been used, these are broad categorisations to allow for an adequate sample size in each region (range 118–315). The validity of grouping diverse countries such as India, Taiwan and China under ‘Asia’ or Tunisia, Nigeria and South Africa under ‘Africa’ is uncertain; however, it reflects the regions currently used by the WHO. Moreover, confirmation of robust psychometrics across such broad groups is a strength.
Further work is also required to establish the ability to detect a change, and interpretation of scoring including minimal important change, for DISCUS if it is to be used as an outcome measure in interventional settings. However, information on these aspects is available for the full DISC.
Conclusions
The 11-item DISCUS is a reliable, unidimensional, measure of experienced discrimination for use in global settings among individuals with a range of mental disorders. It is highly relevant with over 98% of participants reporting at least one experience of discrimination. It has confirmed construct validity in these groups using convergent validity and known groups methods, replicating earlier findings. It offers a brief, free and easily accessible, person-reported, measure for use in estimating the discrimination experienced by individuals with a mental illness. It is recommended as an evaluation tool in global settings to assess the impact of discrimination. DISCUS offers a scalable solution for discrimination measurement in busy, real-world, and resource-limited settings.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291722000630
Acknowledgements
Our thanks to all the participants in this study and to the researchers at each site who enabled data collection and entry by conducting interviews or completing data entry.
Author contributions
Ioannis Bakolis (I. B.), Graham Thornicroft (G. T.) and Sara Evans-Lacko (S. E. L.) designed the study and wrote the protocol. S. E. L. managed data collection and Elaine Brohan (E. B.) and I. B. undertook the statistical analyses. E. B. wrote the first draft of the manuscript. All authors contributed to and have approved the final manuscript. Jie Li (J. L.) collected data for Chinese site.
Financial support
IB is supported by the NIHR Biomedical Research Centre (BRC) at South London and Maudsley NHS Foundation Trust and by the NIHR Applied Research Collaboration South London (NIHR ARC South London) at King's College Hospital NHS Foundation Trust. GT is supported by the National Institute for Health Research (NIHR) Applied Research Collaboration South London at King's College London NHS Foundation Trust, and by the NIHR Asset Global Health Unit award. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. GT also receives support from the National Institute of Mental Health of the National Institutes of Health under award number R01MH100470 (Cobalt study). GT is supported by the UK Medical Research Council in relation to the Emilia (MR/S001255/1) and Indigo Partnership (MR/R023697/1) awards. JL is supported by the programme of community mental health service model in Guangzhou (20161A031002), which was supported by the Health and Family Planning Commission of Guangzhou Municipality.
Conflict of interest
No conflicts of interest are declared.
Open access
