Introduction
Stroke is the leading cause of neurological disability in adults Reference Feigin, Nichols and Alam1 and survival after stroke is increasing. Reference Lakshminarayan, Berger and Fuller2–Reference Waziry, Heshmatollah and Bos4 In addition to physical post-stroke deficits, Reference Verstraeten, Mark and Sitskoorn5 approximately 30–50% of stroke survivors are affected by each of depression, obstructive sleep apnea (OSA) and cognitive impairment (DOC). Reference Herrmann, Seitz and Fischer6–Reference Kapoor, Lanctôt and Bayley9 These DOC comorbidities are all associated with poorer functional outcomes Reference Kapoor, Lanctot, Bayley, Herrmann, Murray and Swartz10 and an increased risk of mortality. Reference Swartz, Bayley and Lanctôt11
The DOC screen was developed as a feasible and valid tool to screen and stratify stroke patients into high, intermediate and low risk groups for DOC comorbidities to facilitate detection and management in high-volume stroke clinic settings. Reference Swartz, Cayley and Lanctôt12 The screen is efficient, yet designed to maintain the construct validity of a delayed recall task. Eighty-nine percent of patients in stroke prevention clinics are able to complete the tool in <6 minutes (mean = 4.2 minutes, SD = 1.5). Reference Swartz, Cayley and Lanctôt12 In validation studies, the cognitive component of the DOC score is helpful to quickly stratify people into “cognitively normal,” “cognitively impaired” and “need more assessment” groups, compared to more detailed cognitive testing. Reference Swartz, Cayley and Lanctôt12 Although the DOC completion time was originally collected as a way to assess feasibility, practitioners can record this measure when administering the DOC screen in clinical settings. Several studies have reported the average time taken to complete other well-known cognitive screens as feasibility demonstrations, including the Montreal Cognitive Assessment (MoCA; means ranging from 9.5 minutes to 11 minutes) Reference Lees, Hendry and Broomfield13,Reference Barnay, Wauquiez and Bonnin-Koang14 and the Mini-Mental State Examination (MMSE; means ranging from 8 minutes to 13.4 minutes). Reference Barnay, Wauquiez and Bonnin-Koang14,Reference Molloy and Standish15 However, few studies have assessed the utility of using a cognitive screen’s completion time as a metric to evaluate underlying cognitive abilities, such as executive functioning.
Executive dysfunction and delays in speed of processing are the most commonly reported cognitive impairments after stroke. The DOC screen specifically examines mood symptoms, cognitive (executive, memory and abstraction) dysfunction and OSA/fatigue – all of which could be associated with cognitive or psychomotor slowing. Reference Schrijvers, Hulstijn and Sabbe16
Aim
Screen completion time is an immediately available metric, requiring no additional effort from either patients or clinicians, which might reflect executive function. The objective of this study was to determine whether completion time for the DOC screen is a reliable reflection of cognitive dysfunction and whether a single completion time cut-point could indicate cognitive impairment.
Methods
All patients were recruited from the DOC feasibility and validity study. Reference Swartz, Cayley and Lanctôt12 This study included English-speaking (or English-fluent) patients newly referred to stroke prevention clinics between April 23, 2012, and April 30, 2014 (n = 1504), who could complete the screen independently (with the administrator, but without third-party support). We excluded patients with severe aphasia, severe motor dysfunction (unable to hold a pen and draw a clock) and patients who were not fluent in English. Each eligible participant was administered the DOC screen (Figure 1) as a brief screen of DOC. All DOC screens were timed from the beginning of the memory registration (first task) until the end of the five-word free recall (final task). Chart abstractions by trained research members captured demographic and clinical data on all participants from patient charts using previously published and validated methods. Reference Hall, Khan and O’Callaghan17,Reference Kapral, Fang and Hill18
The depression, obstructive sleep apnea and cognitive impairment (DOC) screen (freely available for download at www.docscreen.ca).
To reduce sampling bias, all consecutive patients from stroke prevention clinics who completed the DOC screen were asked to complete more detailed neuropsychological assessments, including a cognitive battery and formal mood assessments as outlined in the DOC feasibility study. Reference Swartz, Cayley and Lanctôt12 All patients who completed the detailed assessments provided written informed consent. Only the site PI could access the information that could identify individual participants, all the other authors were given anonymized study IDs that were created upon the completion of the informed consent process. A complete list of all mood and cognitive assessments completed as part of the DOC study is reported elsewhere. Reference Swartz, Cayley and Lanctôt12 In this analysis, cognition was assessed using the 30-minute neuropsychological battery recommended by the NINDS-CSN. Reference Hachinski, Iadecola and Petersen19 This cognitive battery consists of the Controlled Oral Word Association Test (phonemic fluency), Animal Naming task (semantic fluency), California Verbal Learning Test (CVLT), Digit Symbol Coding and Trail Making Tests (TMT-A and TMT-B). All scores were normalized (z-score or scaled score) for age using age-matched norms from each respective test manual. CVLT and Animal Naming were also education-standardized. Reference Tombaugh20,Reference Delis, Kramer, Kaplan and Ober21 The study was approved by the Sunnybrook Research Ethics Board (approval number SUN-2312).
Statistical analysis
Statistical analysis was performed using IBM SPSS Statistics for Windows version 24. Descriptive statistics, including means and standard deviations, were calculated for age, screen completion time and number of years of education.
Multivariable linear regression analyses of the relationship between time-to-completion and NINDS-CSC standardized scores
To assess whether screen time reflects cognitive function, independent linear regression models were used to examine the association between DOC completion time and the scaled or z-scores of all neuropsychological subtests. Data from all participants were used in the regression models. A sensitivity analysis was performed using a complete case approach to assess whether missing variables affected the models. All models controlled for age, education, modified Rankin Score (mRS) and sex. Due to the established relationship between the DOC cognitive sub-scores and detailed cognitive assessments, Reference Swartz, Cayley and Lanctôt12 we also controlled for the DOC-Cognition score in all models. To adjust for multiple (7) linear regressions, Bonferroni correction (0.05/7 = 0.0071) was used to define significance at p < 0.007 for all analyses.
ROC and logistic regression analyses to identify cutoffs associated with high likelihood of cognitive impairments
To identify whether a single cut-point (in seconds) for screen time could be found with high specificity and likelihood ratios for cognitive impairment, receiver operating characteristic (ROC) curves were used. ROC analyses were run for each neuropsychological assessment significantly associated with the DOC screen completion time. A logistic regression with screen time completion (as a continuous variable) and the cognitive impairment classification on the NINDS-CSN assessments were applied to the ROC curves. The classification of cognitive impairment of NINDS-CSN was defined as scores >2.0 standard deviations from expected norms, on two or more cognitive tasks. This required participants to have completed all tests in the detailed cognitive battery; thus, a complete case approach was used for all ROC analyses. First, a single, specific cut-point (time in seconds) was defined based on the ROC curve output for patients with an overall classification of impaired on the NINDS-CSN battery. The cut-point was pre-specified to have 95% specificity for cognitive impairment. This cut-point was then applied to ROC curves from each individual assessment and evaluated using likelihood ratios.
Results
A total 437 patients completed the cognitive and mood gold standard assessments within a maximum of 3 months of screening, with an average time interval of 3 days Reference Swartz, Cayley and Lanctôt12 (Supplemental Table 1). Of these, 213 (48.7%) participants were male, with a mean (± standard deviation) age of 62.7 ± 15.6 years and a mean years of education of 15.6 ± 3.9 years (Table 1). Additionally, 387 patients were able to complete all assessments in the battery; 13.7 % of these were classified as impaired based on the NINDS-CSN classification. The DOC screen completion mean was 3.8 ± 1.3 minutes (range: 1.9–9.6 minutes). Among the patients, 134 (31%) had an ischemic stroke, 138 (32%) had a probable/possible TIA and the remainder (37%) were diagnosed with other conditions (Table 1). Non-stroke/transient ischemic attack (TIA) diagnoses included patients referred with possible stroke symptoms, but whose further investigations revealed alternative diagnoses, as well as patients without specific stroke/TIA symptoms referred for either vascular risk reduction or assessment of incidental abnormal imaging findings.
Demographics for participants completing detailed cognitive and neuropsychological assessments (n = 437)
† TIA = transient ischemic attack; ICH = intracerebral hemorrhage; IVH = intraventricular hemorrhage; SAH = subarachnoid hemorrhage; SDH = subdural hemorrhage; DOC = depression, obstructive sleep apnea and cognitive impairment.
We performed linear regressions with DOC screen completion time (in seconds) as a predictor for each neuropsychological assessment score (Table 2). In all models, we controlled for age, sex, years of education, screening score of cognitive function (DOC-cognition score) and overall function (mRS). All regression models for screen completion time were significant (p < 0.001) (Supplemental Table 2). Additionally, model summaries showed that screen completion time was a significant predictor (p < 0.005) of verbal fluency semantic score (95% confidence interval (CI) of beta-coefficient from linear regression: −0.006 to −0.001), verbal fluency phonemic score (95% CI: −0.018 to −0.006), digit symbol coding (95% CI: −0.016 to −0.004) and the Trail Making Tests (TMT-A 95% CI: −0.017 to −0.005; TMT-B 95% CI: −0.016 to −0.004). In all cases, these were negative correlations (i.e., longer completion times correlated with poorer cognitive scores). DOC screen completion time was not a significant predictor of memory performance on the CVLT Short Delay Free Recall (p = 0.713, 95% CI: −0.003–0.002) or the CVLT Long Delay Free Recall (p = 0.790, 95% CI: −0.002–0.003). Results did not differ in the sensitivity models with complete case data (see Table 2 compared to Supplemental Table 3 with complete case data). Neither DOC mood and apnea screening scores nor SCID-D or polysomnogram scores were associated with DOC screen completion time in any multivariable regression.
Linear regression results showing the effect of the DOC screen completion time on individual neuropsychological assessments
DOC = depression, obstructive sleep apnea and cognitive impairment.
* All models controlled for by age, sex, years of education, DOC-cognition score and modified Rankin Scale (mRS).
†Significant results bolded and set at p < 0.007.
† TMT = Trail Making Test; CVLT = California Verbal Learning Test.
Using the single cutoff point approach on the overall cognitive impairment ROC curve (Figure 2, Table 3A), the point with 95% specificity for cognitive impairment was 332.5 s. When this time was applied to ROC models for each individual cognitive task (Table 3B), the same cut-point had high specificity on all executive and speed of processing tasks. The area under the curve was greater than 0.7 for all executive and speed of processing tasks. Likelihood ratios for predicting abnormal results on executive and speed of processing tasks ranged from four to six – that is, people taking more than 332.5 s to complete the DOC screen were 4–6 times more likely to have severe cognitive impairment on executive and speed of processing tasks than those with faster completion times (see Table 3). Scatterplots demonstrating the predicted probability of impairment on each domain by completion time, derived from the logistic regression analysis can be found in the supplemental material.
Receiver operating characteristic (ROC) curve, model for overall cognitive impairment with a cutoff set at 95% specificity.
Receiver operating characteristic (ROC) model outputs comparing DOC screen completion time with full neuropsychological assessments, with a cutoff set at 332.5 s (95% specificity) obtained from the model for overall cognitive impairment
† Trails = Trail Making Test; DOC = depression, obstructive sleep apnea and cognitive impairment.
Discussion
Several studies Reference Swartz, Stuss, Gao and Black22 have shown that post-stroke cognitive impairments can be separated into independent cognitive factors including language, memory and executive function, with deficits in executive functioning and speed of processing being the most common. Reference Zinn, Bosworth, Hoenig and Swartzwelder23 Screening tests for executive function and speed of processing are limited and rarely used in routine clinical care. These results demonstrate that DOC screen completion time is an independent predictor of executive function (semantic fluency, Reference Maseda, Lodeiro-Fernández, Lorenzo-López, Núñez-Naveira, Balo and Millán-Calenti24 TMT-B Reference Arbuthnott and Frank25 ), speed of processing (digit symbol coding, Reference Deloire, Bonnet and Salort26 TMT-A and TMT-B28) and verbal fluency Reference Whiteside, Kealey and Semla28 after stroke, even after controlling for age, sex, education, DOC-cognition score and stroke severity. Completion time did not predict CVLT scores, a verbal test primarily affecting verbal memory (learning/registration and recall). Reference Jacobs and Donders29 Verbal fluency, while reflecting language function, is also reflective of executive function. Reference Aita, Beach, Taylor, Borgogna, Harrell and Hill30 Moreover, we have demonstrated that a 332.5 s (roughly 5.5 minutes) cutoff has 95% specificity and high likelihood ratios for predicting both overall cognitive and executive function impairment. This can be used as a quick and easily obtainable measure to identify people at risk for impairment on executive and speed of processing tasks. Certainly, other timed tasks, whether pen-and-paper (like Trails) or digital (e.g., Creyos), can be used to assess executive and speed of processing deficits in detail; however, detailed cognitive batteries are too onerous for routine clinical use. Simply timing the DOC screen as it is administered provides additional information, beyond the actual DOC cognitive screening score, which can flag people at high risk of having multi-domain cognitive impairment and executive/speed of processing dysfunction.
A few notable neuropsychological measures have used completion time to assess specific cognitive functions. For instance, Trail Making Tests (TMTs) are a set of widely accepted timed neuropsychological measures that provide insight into executive abilities. Reference Muir, Lam and Honjo27 Processing speed is highly associated with performance on TMT Part B (a task reflecting attention and executive functions such as set-shifting) and with performance on TMT Part A (which is more closely related to motor speed and attention). Reference Arbuthnott and Frank25,Reference MacPherson, Cox and Dickie31,Reference Fishman, Ashbaugh and Swartz32 Similarly, Woods et al. discovered that a patient’s question completion time on self-paced questionnaires could be used as a measure of executive functioning. Reference Woods, William Yund, Wyma, Ruff and Herron33 Question completion time measures processing and decision-making speeds, providing insight into motivation, effort and cognitive ability that is not measured by existing tests. Reference Woods, William Yund, Wyma, Ruff and Herron33 These studies support the notion that timed measures may be useful as a measure of executive dysfunction in addition to their use as screening instruments. The findings presented in our study correspond well to those reported by Woods et al. Their analyses showed that complex tasks, akin to our DOC-cognitive tasks, were strongly related to executive function and processing speed. Their neuropsychological tests (including TMT-B and Digit Span) also correlated significantly with self-paced question completion time. Their research process was similar to ours, wherein completion time was compared to existing screens to validate completion time as a metric; both studies suggest that completion time of self-paced complex assessments may be valid markers of executive function.
Few studies use the completion time of a neuropsychological screening tool as a cognitive marker. Most timed tasks examine processing speed directly (e.g., Trails, symbol-digit modalities test Reference Zaidi, Rich and Sunderland34 ) and have been studied in clinical settings, for example, for HIV-induced cognitive dysfunction Reference Davis, Skolasky, Selnes, Burgess and McArthur35,Reference Van Harten, Courant, Scheltens and Weinstein36 and in multiple sclerosis. Reference Ezegbe, Zarghami, van der Mei, Alty, Honan and Taylor37,Reference Wishart, Everest, Morrow, Rose, Shen and Feinstein38 However, these types of tasks are more detailed and time-consuming, and while they can be performed in a clinic in isolation, they are more often done as part of larger batteries. In contrast, screening tasks like the MoCA or MMSE are not routinely timed when applied in clinical settings. By simply timing the DOC screen, in addition to the information generated by the screen on mood, apnea and cognitive function, the time taken to complete the entire screen is itself an indirect measure that can highlight people at risk for cognitive impairment, especially executive, speed of processing and attentional issues. Moreover, executive function deficits are not often assessed in stroke patients; these deficits are subtle, challenging to test for and often go unrecognized. Reference Zinn, Bosworth, Hoenig and Swartzwelder23 The NINDS-CSC battery is recommended as a research battery, but it requires a trained administrator and at least 30 minutes per person plus scoring. This is not feasible for routine clinic use. The DOC screen, in contrast, takes less than 5 minutes, can be performed by clinical staff (students, administrative assistants, nurses and physicians) and can help to highlight people at risk for impairments in mood, apnea and cognition.
The interpretation of our findings is limited by our sample population. Compared to the total number of patients who were asked to volunteer from the stroke prevention clinic (n = 1504), consenting participants (n = 437) tended to be slightly younger and with slightly milder neuropsychological deficits (healthy participant bias). Reference Swartz, Cayley and Lanctôt12 However, our sample also included a wide range of patients across the full spectrum of severity. As expected from stroke/TIA clinic samples, 62% had a diagnosis of stroke and/or TIA, and the rest had alternative diagnoses common in stroke prevention clinics (mimics, multiple vascular risk factors, abnormal imaging). This heterogeneity reflects the pragmatic nature of the screening and its broad generalizability to the population of patients referred to stroke prevention clinics. TIA patients are well recognized to share similar long-term risk profiles Reference Edwards, Kapral, Fang and Swartz39 and are also at risk for cognitive impairment, Reference Pendlebury, Wadling, Silver, Mehta and Rothwell40 compared to those with imaging-confirmed strokes. While the strongest associations to DOC completion were with tests of executive function, processing speed and verbal fluency, other domains that were less well represented in the NINDS-CSC battery could also impact screen completion time. For example, visuospatial function was not specifically assessed in the NINDS-CSC battery, and while language function could also affect completion time, there was no relationship with score on the California Verbal Learning Task (a verbal memory task). Since many tasks have more than one cognitive construct underlying them (e.g., phonemic and semantic fluency tasks each require language, attention and executive functions), DOC screen time cannot be considered a reflection of only one underlying domain. However, the tasks associated with DOC screen time all share underlying cognitive constructs of attention, executive dysfunction and/or speed of processing. The relationship between DOC completion time and gold standard testing was found across a range of severity from normal function to severely impaired. It should also be noted that there is not a single perfect cut-off score for DOC completion time that indicates executive dysfunction. To facilitate clinical utility, and because this is intended as a screen in high-volume clinics, we chose to explore a cutoff with high specificity so clinicians could be confident there was a high likelihood of true cognitive impairment beyond this time; however, this cutoff will have a low sensitivity and will miss some people with cognitive impairments. Previous work has already established that the DOC-cognition score can also be a sensitive screen, effectively ruling out cognitive impairment in people who score highly. Reference Swartz, Cayley and Lanctôt12 Finally, it is important to note that although screen completion time may be a useful tool to identify people at risk for executive dysfunction, it is still not equivalent to a detailed neuropsychological assessment.
Conclusion
Clinical cognitive screening tools have not commonly used completion time as a metric. We aimed to determine whether the DOC screen completion time could provide clinically relevant information on patients’ cognitive function. DOC screen completion time reflects executive function, speed of processing and verbal fluency. When administering the DOC screen, completion time requires no additional time or patient burden to collect. This convenience is vital in busy stroke prevention clinic settings, where there is minimal time for detailed cognitive assessments. Exploring whether screen time can act as a predictor of future outcomes would provide further support for the utility of this measure in clinical settings.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/cjn.2024.303.
Data availability statement
DOC screening for mood, cognition and apnea was performed in stroke prevention clinics under waiver of consent. Patients provided written consent to undergo detailed cognitive testing and to relate their screening results to the detailed neuropsychological testing. However, the public release of data was not part of the patients’ consent.
Acknowledgments
We would like to acknowledge the DOC participants who volunteered their time for the study.
Author contributions
Sajeevan Sujanthan and Alisia Southwell contributed equally to this work.
SS – Formal analysis, writing – original draft, writing – reviewing and editing.
AS – Data collection, formal analysis, writing – original draft.
TA – Formal analysis, writing – original draft.
EX – Formal analysis, writing – original draft.
AK – Formal analysis, writing – original draft.
XL - Formal analysis, writing – original draft.
KLL - Resources, supervision, writing – review and editing.
NH - Resources, supervision, writing – review and editing.
BJM - Resources, supervision, writing – review and editing.
KET - Resources, supervision, writing – review and editing.
MLC – Data collection, writing – review and editing.
MNS – Data collection, writing – review and editing.
KL – Data collection, writing – review and editing.
DS - Resources, supervision, writing – review and editing.
RHS – Funding, conceptualization, resources, supervision, writing – original draft, writing – reviewing and editing.
Funding statement
This study was supported by Canadian Institutes of Health Research (CIHR) (grant ref no. 137038) and Heart and Stroke Foundation of Canada (HSF) (project no. 000392). RHS received salary support for research from an Ontario Clinician Scientist (Phase II) Award from the HSF Canada and from the Department of Medicine (Neurology) at Sunnybrook and the University of Toronto. In-kind support for data storage was provided by the Ontario Brain Institute and the Ontario Neurodegenerative Disease Research Initiative.
Competing interests
RHS reports ownership shares in FollowMD Inc., a virtual vascular risk reduction clinic. None of the other authors have any conflicts of interest to disclose.
Data access
Data is not available to share publicly, as patients did not consent to public data release. Clinical Trials Registration Identifier: NCT02363114. Clinical Trials URL: https://clinicaltrials.gov/ct2/show/NCT02363114.
Open access
