Dear Editor,

In nursing home (NH) residents, outcomes such as well-being and depression are often based on observable behaviors or signs reported by someone other than the resident. While previous studies have reported on the agreement between proxy-reported scores and self-reported scores (e.g. Leontjevas et al., Reference Leontjevas, Teerenstra, Smalbrugge, Koopmans and Gerritsen2016), and between categories of proxy, such as relatives and professional caregivers (e.g. Robertson et al., Reference Robertson, Cooper, Hoe, Hamilton, Stringer and Livingston2017), studies assessing agreement between professional caregivers acting as observers are scarce. Furthermore, limited attention has been paid to reporting agreement indices stratified by the level of residents’ cognitive functioning. As professional caregivers commonly act as observers for resident outcomes, knowledge about the inter-rater reliability of observer-reported outcomes is important. Therefore, secondary analysis was performed on a dataset containing observer-reported outcomes in residents with and without dementia in Dutch and Flemish (Dutch speaking part of Belgium) NHs.

Eighty-one residents of 21 NHs were evaluated for depression (Nijmegen Observer-Rated Depression scale for detection of depression in nursing home residents [NORD]) (Leontjevas et al., Reference Leontjevas, Gerritsen, Vernooij-Dassen, Teerenstra, Smalbrugge and Koopmans2012), well-being (adapted version of the Social Well-being Of Nursing home residents scale [SWON-3]) (Gerritsen et al., Reference Gerritsen, Steverink, Frijters, Ooms and Ribbe2010), and quality of life (two subscales of the QUALIDEM, namely “social relations” and “having something to do”) (Ettema et al., Reference Ettema, Dröes, De Lange, Mellenbergh and Ribbe2007) by two professional caregivers (registered nurse or certified nurse assistant) who were involved in caring for the resident about whom the questions were answered. Most caregivers (46 out of 71) filled out the questionnaires for one resident (median, 1; range, 1–9). To assess the agreement between the pairs of caregivers, we calculated Gwet’s AC1 or AC2 coefficients (Gwet, Reference Gwet2021) for individual items of the questionnaires using the irrCAC R package (Gwet, Reference Gwet2019). Individual item’s agreement was calculated for more insight into whether individual items could be adjusted or deleted for improving the psychometric characteristics of an instrument. In addition, coefficients were calculated for subjective judgment regarding residents’ depressive symptoms (“no,” “yes, mild or light,” or “yes, severe”) and for caregivers’ knowledge of whether a depression diagnosis had been established (“yes,” “no,” or “don't know”). Intraclass correlation coefficients (ICC [1,1] and [1,2]) (Koo and Li, Reference Koo and Li2016) were calculated for the scale mean scores of the NORD (total scale), SWON-3 (three subscales and the total scale), and QUALIDEM (two subscales) using the irr R package (Gamer et al., Reference Gamer, Lemon, Fellows and Singh2012).

For the total sample, Gwet’s coefficients ranged from 0.29 to 0.63 for items of the NORD, from 0.32 to 0.75 for the items of the SWON-3, and from 0.45 to 0.74 for items of the QUALIDEM (see Table 1). Most items were characterized as “fair” or “moderate.” Gwet’s coefficients for the subjective judgment of residents’ depressive symptoms and a depression diagnosis were 0.41 (fair) and 0.84 (good) respectively.

Table 1.

Agreement statistics of nursing home caregivers’ observations of residents’ depression, well-being, and quality of life

Note: % Obs = percentage observed; CI = confidence interval; ICC = intraclass correlation coefficient; GDS = global deterioration scale; N = valid number of caregiver pairs.

1. NORD: Nijmegen Observer-Rated Depression scale for detection of depression in nursing home residents. Response options: “Yes,” “No.”

2. SWON-3: Social Well-being Of Nursing home residents scale.

3. QUALIDEM: Response options: “Never,” “Rarely,” “Sometimes,” “Frequently.”

4. Symptoms: 1-item (“Do you think this resident has depressive symptoms?”).

5. Diagnosis: 1-item (“Has a depression diagnosis been established? ”). Response options: “Yes,” “No," “Don't know” (the option “don't know” was treated as missing).

6. Gwet’s AC1 was used for calculating unweighted coefficients, and Gwet’s AC2 was used for calculating weighted coefficients.

7. Altman’s benchmarking 5-point scale ranging from "poor" to "very good" was used to interpret the magnitude of the AC1 and AC2 coefficients. A cumulative probability of above 0.95 was applied to determine the lowest expected agreement level.

8. ICC (1,1): Intraclass correlation coefficient for absolute agreement, 1-way random effects model, single rater. This is informative for planning measurements from a single rater.

9. ICC (1,2): Intraclass correlation coefficient for absolute agreement, 1-way random effects model, two raters. This is informative for the use of a mean value of two raters as the basis of the actual measurement (Koo and Li, Reference Koo and Li2016).

Although comparison of coefficients across different subsamples must be interpreted with caution due to relatively small sample sizes and, consequently, broad confidence intervals, the results point toward lower levels of agreement for observer-reported scores of residents with moderate to severe cognitive decline (N = 18, 14 of 24 analyzed items were characterized as “poor”), compared to residents with no to mild cognitive decline (N = 58, the most frequent item coefficients were characterized as “fair” [8 items] or “moderate” [9 items]).

Under the assumption of multiple raters, all (sub)scales showed at least moderate agreement (ICC [1,2] ≥ 0.50) for the total sample and for the subsample of residents with no to mild cognitive decline. For residents with moderate to severe cognitive decline, poor agreement (ICC [1,2] < 0.50) was found for all (sub)scales but the NORD and the subscale “social relations” of the QUALIDEM.

The limited agreement between caregivers concerning residents with moderate to severe cognitive decline underscores challenges for measurements in this population. One possible explanation is that interpretation of items or response options may be extra challenging when residents are less able to express themselves. Another explanation may be that accurate observations can be challenging if the symptoms of the outcome variables overlap with those of severe dementia (Leyhe et al., Reference Leyhe2017).

We believe that practitioners and researchers should be aware of these challenges when using and interpreting observer-reported outcomes for residents with dementia. Moreover, understanding why different raters reach different conclusions regarding the same resident is important for interpreting observer-reported outcomes. We therefore argue that future research should explore the reasons why observer-reported scores may differ between caregivers, especially concerning residents with moderate or severe dementia. Both cognitive interviewing and other forms of in-depth interviews with caregivers are recommended to better understand their interpretation of items and to discover the actual reasons for differences between caregivers’ scores.