Validated computerized assessments for cognitive functioning are crucial for older individuals and those at risk of cognitive decline. The National Institutes of Health (NIH) Toolbox Cognition Battery (NIHTB-CB) exhibits good construct validity but requires validation in diverse populations and for adults aged 85+. This study uses data from the Assessing Reliable Measurement in Alzheimer’s Disease and cognitive Aging study to explore differences in the factor structure of the NIHTB-CB for adults 85 and older, Black participants versus White participants, and those diagnosed as amnestic Mild Cognitive Impairment (aMCI) vs cognitively normal (CN).

Subtests from the NACC UDS-3 and NIHTB-CB were administered to 503 community-dwelling Black and White adults ages 55–99 (367 CN; 136 aMCI). Confirmatory factor analyses were used to investigate the original factor structure of NIHTB-CB that forms the basis for NIHTB-CD Index factor scores.

Factor analyses for all participants and some participant subsets (aMCI, White, 85+) substantiated the two anticipated factors (Fluid and Crystallized). However, while Black aMCI participants had the expected two-factor structure, for Black CN participants, the List Sorting Working Memory and Picture Sequence tests loaded on the Crystallized factor.

Findings provide psychometric support for the NIHTB-CB. Differences in factor structure between Black CN individuals and Black aMCI individuals suggest potential instability across levels of cognitive impairment. Future research should explore changes in NIHTB-CB across diagnoses in different populations.

The ability to remotely monitor cognitive skills is increasing with the ubiquity of smartphones. The Mobile Toolbox (MTB) is a new measurement system that includes measures assessing Executive Functioning (EF) and Processing Speed (PS): Arrow Matching, Shape-Color Sorting, and Number-Symbol Match. The purpose of this study was to assess their psychometric properties.

MTB measures were developed for smartphone administration based on constructs measured in the NIH Toolbox® (NIHTB). Psychometric properties of the resulting measures were evaluated in three studies with participants ages 18 to 90. In Study 1 (N = 92), participants completed MTB measures in the lab and were administered both equivalent NIH TB measures and other external measures of similar cognitive constructs. In Study 2 (N = 1,021), participants completed the equivalent NIHTB measures in the lab and then took the MTB measures on their own, remotely. In Study 3 (N = 168), participants completed MTB measures twice remotely, two weeks apart.

All three measures exhibited very high internal consistency and strong test-retest reliability, as well as moderately high correlations with comparable NIHTB tests and moderate correlations with external measures of similar constructs. Phone operating system (iOS vs. Android) had a significant impact on performance for Arrow Matching and Shape-Color Sorting, but no impact on either validity or reliability.

Results support the reliability and convergent validity of MTB EF and PS measures for use across the adult lifespan in remote, self-administered designs.

Symptom clustering research provides a unique opportunity for understanding complex medical conditions. The objective of this study was to apply a variable-centered analytic approach to understand how symptoms may cluster together, within and across domains of functioning in mild cognitive impairment (MCI) and dementia, to better understand these conditions and potential etiological, prevention, and intervention considerations.

Cognitive, motor, sensory, emotional, and social measures from the NIH Toolbox were analyzed using exploratory factor analysis (EFA) from a dataset of 165 individuals with a research diagnosis of either amnestic MCI or dementia of the Alzheimer’s type.

The six-factor EFA solution described here primarily replicated the intended structure of the NIH Toolbox with a few deviations, notably sensory and motor scores loading onto factors with measures of cognition, emotional, and social health. These findings suggest the presence of cross-domain symptom clusters in these populations. In particular, negative affect, stress, loneliness, and pain formed one unique symptom cluster that bridged the NIH Toolbox domains of physical, social, and emotional health. Olfaction and dexterity formed a second unique cluster with measures of executive functioning, working memory, episodic memory, and processing speed. A third novel cluster was detected for mobility, strength, and vision, which was considered to reflect a physical functioning factor. Somewhat unexpectedly, the hearing test included did not load strongly onto any factor.

This research presents a preliminary effort to detect symptom clusters in amnestic MCI and dementia using an existing dataset of outcome measures from the NIH Toolbox.

As the older adult population increases in the coming decades, the number of persons that develop dementia of the Alzheimer’s type (DAT) will increase accordingly. Though curative treatment for Alzheimer’s disease remains elusive, early detection of cognitive decline allows for initiation of pharmacological treatment to slow disease progression and non-pharmacological approaches to support quality of life and well-being of affected individuals and their care partners. Streamlined approaches that bridge the gap between brief screenings and comprehensive neuropsychological evaluation are needed. The NIH Toolbox Cognition Battery (NIHTB-CB) is a brief, easily administered, computerized cognitive battery that assesses various aspects of both fluid and crystallized cognitive abilities. ARMADA (Advancing Reliable Measurement in Alzheimer’s Disease and Cognitive Aging) is a multi-site study that aims to validate the NIHTB across the spectrum from normal aging to DAT. The current study utilized longitudinal data from ARMADA to determine whether performance on the NIHTB-CB detects cognitive decline in persons with normal cognition (NC), mild cognitive impairment (MCI), and mild DAT over the span of two years. We predicted that scores would decline for the MCI and DAT groups, but not for the NC group.

Participants were 191 participants drawn from the larger ARMADA cohort aged 65-84 (nNC = 118, nMCI = 47, nDAT = 26) that completed the NIHTB-CB at baseline and 12 months. The clinical groups were significantly older than the NC group at baseline (MNC = 72.72, MMCI = 76.63, MDAT = 75.42; p < .001) and the NC and MCI groups had significantly more years of education than the DAT group (MNC = 17.03, MMCI = 16.83, MDAT = 15.54; p = .008).

Mixed model ANOVAs determined differences in uncorrected NIHTB-CB scores between clinical groups at baseline and 12 months, controlling for age and education. There were significant interactions between time and clinical group for Flanker (p < .001), Pattern Comparison (p < .001), and Picture Vocabulary (p = .001), such that the DAT group demonstrated a more negative slope of change than the NC and MCI groups. For Oral Reading, the MCI group demonstrated a more negative slope of change than the NC and DAT groups (p = .01).

Differential score trajectories were found for the Flanker task, with a more negative pattern of change in scores in the DAT group compared to the NC and MCI groups. Contrary to expectation, scores decreased for the two crystallized subtests across groups, which may reflect regression to the mean given high baseline scores, especially for Picture Vocabulary; however, these results were also moderated by group with less decline in scores in the NC group, which may indicate involvement of non-crystallized abilities in executing a single word comprehension task. Group differences were subtle, which may in part reflect the relatively short period of follow up. The Flanker task appears to be most sensitive to decline in mild DAT compared to MCI and NC. Results provide preliminary support for the utility of NIHTB-CB in detecting cognitive decline along the cognitive aging to DAT spectrum.

To present the Mobile Toolbox (MTB), comprised of an expandable library of cognitive and other tests, including adapted versions of NIH Toolbox® measures. The MTB provides a complete research platform for app creation, study management, data collection, and data management. We will describe the MTB project and MTB research platform and demonstrate examples of assessments.

MTB is the product of an NIH-funded, multi-institutional effort involving Northwestern University, Sage Bionetworks, Penn State, University of California San Francisco, University of California San Diego, Emory University, and Washington University. The MTB assessment library is a dynamic repository built upon Sage Bionetworks mobile health platform. All MTB measures are created or adapted for a mobile interface using iOS and Android smartphones. Guided by the principles of open science, many components are open source to allow researchers and developers to integrate externally developed tests, including supplemental scales (e.g., passively collected contextual factors) assessing variables such as mood and fatigue that might influence cognitive test performance.

The current MTB library includes eight core cognitive tests based on well-established neuropsychological measures: two language tasks (Spelling and Word Meaning), two executive functioning tasks (Arrow Matching and Shape-Color Sorting), an associative memory task (Faces and Names), an episodic memory task (Arranging Pictures), a working memory task (Sequences) and a processing speed task (Numbers and Symbols). Additional cognitive assessments from other popular test libraries including the International Cognitive Ability Resource (ICAR), Cognitive Neuroscience Test Reliability and Clinical Applications for Schizophrenia (CNTRACS) and Test My Brain are currently being implemented, as are non-cognitive measures from the NIH Toolbox Emotion Battery and the Patient-Reported Outcomes Measurement Information System (PROMIS). The MTB library includes measures suitable for use in research studies incorporating point-in-time and burst designs as well as ecological momentary assessment (EMA).

The MTB was created to address many of the scientific, practical, and technical challenges to cognitive assessment by capitalizing on advances in technology measurement and cognitive research. Initial psychometric evaluation of measures has been performed, and additional clinical validation is underway in studies with persons at risk for cognitive impairment or Alzheimer’s disease (AD), diagnosed with mild cognitive impairment (MCI) or AD, Parkinson’s disease, and HIV-associated Neurocognitive Disorders. Calculation of norms and reliable change indicators is in progress. The MTB is currently available to beta testers with public release planned for Summer, 2023. Clinical researchers will be able to use the MTB system to design smartphone-based test batteries, deploy and manage mobile data collection in their research studies, and aggregate and analyze results in the context of large-scale norming data.

To present validation evidence for the first eight cognitive measures available through Mobile Toolbox (MTB). These measures use a remote self-administered platform to assess language, working memory, episodic memory, executive function, and processing speed.

We used two separate samples, recruited as part of a larger study, to validate MTB measures. Sample I, comprised of 92 English-speaking adults ages 18-85, was used to assess internal consistency and construct validity. Participants were first administered “gold standard” cognitive measures (Wechsler Memory Scale-IV Verbal Paired Associates I and II; Wechsler Adult Intelligence Scale-IV Symbol Search, Digit Span, Coding, and Letter-Number Sequencing; Delis-Kaplan Executive Function System Color-Word Interference Test, Peabody Picture Vocabulary Test, Wechsler Individual Achievement Test-4 Spelling, and the Wisconsin Card Sorting Test), after which they completed MTB (pre-loaded on a study-provided smartphone) on their own. Internal consistency was evaluated using measure-appropriate indices (split-half reliability, Cronbach’s alpha or IRT-based indices). Pearson correlation coefficients between MTB tests and measures of similar constructs were used to evaluate concurrent validity. For two tests with timing-dependent scores, Arrow Matching and Shape-Color Sorting, separate analyses were performed for iOS and Android devices. Sample II, with 1,120 English-speaking participants ages 18-90, was used to evaluate age-related change. Participants completed MTB measures remotely on their own smartphones, in a preset order, within a 14-day period. Spearman correlation coefficients, corrected for education, were calculated to evaluate relationships between age and test scores.

Sample I participants were 67% female, 52% white, 99% non-Hispanic; average age=48 (SD= 17). Education was: < high school (1%); high school (55%); some college (21%); college (15%); graduate degree (8%). Internal consistency estimates ranged from 0.81 to 0.99. Pearson correlations between MTB and external measures ranged from 0.41 to 0.86 (all p < .01). Of the timed tests, only Shape-Color sorting showed significant score differences between Android and iOS devices. Sample II was 57% female, 13% Hispanic, 72% white, mean age = 45 (SD = 21). Education distribution was: < high school (2%); high school (34%); some college (34%), college (20%); graduate degree (11%). Measures of executive function (r = -0.50; r=-0.57) and processing speed (r= -0.61) showed the expected negative correlation with age (all p <0.001). Negative correlations, although weaker, were also seen on measures of working memory (r=-0.2) and episodic memory (r=-0.2, r=-0.37; p.<.001). Vocabulary performance improved with age (r=0.4; p<.001), while spelling scores remained stable (r=0.09).

Initial studies support the validity and reliability of the first eight MTB cognitive measures in two diverse samples. MTB tests showed satisfactory construct validity, as demonstrated by the associations between MTB and well-established tests. Furthermore, most MTB measures correlated with age in the expected directions. Executive function, processing speed and memory typically decrease with age and this decrease was reflected in MTB test performance. In contrast, spelling and vocabulary, typically preserved as we age, did not decrease in our sample. Our results support the use of MTB in cognitive aging research.

A critical need in the neuropsychology field is development and validation of efficient, scalable assessments of cognition. The Mobile Toolbox (MTB), a novel suite of mobile device-compatible, app-based cognitive assessments, was developed to address this need. The goals of this study were (1) To collect longitudinal normative data for the MTB assessments in a large, ethnoculturally and educationally diverse cohort; (2) To assess the feasibility and usability of remote assessment using MTB.

Participants were recruited from the UCSF Brain Health Registry (BHR), an online cohort (N>100,000) that collects longitudinal cognitive, functional, behavioral, and health data using online neuropsychological tests and self- and study-partner report surveys. BHR participants who opted to learning about additional research opportunities were sent automated email invitations to enroll in the MTB study. Those who indicated study interest were provided instructions within the BHR online portal for downloading the MTB app. All participants had the opportunity to complete a single baseline administration of MTB (Word Meaning, Sequences, Spelling, Arranging Pictures, Arrow Matching, Faces and Names, Shape-Color Sorting, Number Match). Those who completed the baseline assessment within three days were invited to continue into the longitudinal study, where they complete MTB assessments at a single, short-term timepoint (day 7, 14, or 21; study arms sequentially assigned), and then at 6-month intervals. Enrollment across demographic groups was monitored, and study invitations were sent to specific demographic groups, with the goal of enrolling a sample of 800 participants in the longitudinal study: equal distribution across eight, 10-year age bands (ages 18-80+); 60% with <16 years of education; 10% non-Latinx Black, 15% Latinx, and 5% non-White other ethnocultural identity.

Between January-June 2022, 48,110 BHR participants were invited to the MTB study. Of those, 8294 (17%) expressed interest, 3401 (7%) completed the baseline assessment, 850 (1.8%) were assigned to the longitudinal study, and 782 (1.6%) completed a short-term longitudinal assessment. Study staff received 797 help tickets submitted by participants asking for email support to complete MTB. The baseline cohort had and average age of 64 years and an average of 16.6 years of education, 76.2% female, 2.1% non-Latinx Black, 7.1% Latinx, 86.8% non-Latinx White, and 4% from other ethnocultural groups. The longitudinal cohort had an average age of 62.3 years and an average of 16.1 years of education, 80% female, 2.8% non-Latinx Black, 8.5% Latinx, 83.5% non-Latinx White; and 5% other ethnocultural group. Compared to those invited to the study, those who enrolled in the longitudinal study were older, had higher educational attainment, and were more likely to be female and self-identify as non-Latinx White (p<0.05 for all).

Efficient enrollment and task completion of a large cohort in a novel, app-based mobile cognitive assessment is feasible in a completely remote setting. Most participants were able to complete MTB without individual support, indicating good usability. This approach can be scaled up to efficiently assess cognition in many research and healthcare settings. A remaining challenge is achieving robust ethnocultural and educational diversity.

To describe advantages and disadvantages of using digital assessments remotely and in-person to inform clinical and research practice.

As part of a larger study,1120 adults completed a battery of remotely administered tests (Mobile Toolbox) and a subset of this sample completed examiner administered in-person testing (NIH Toolbox® Cognition Battery). Attention was given to making the sample reflective of the US 2020 Census during participant recruitment. Of the 1120 participants, the majority of the sample were female (57%) and Caucasian (72%) and had a mean age of 45 (SD = 21). In terms of education, equal percentages had high school (34%) or some college (34%).

NIH Toolbox cognitive tests of processing speed, language, executive function, attention, and episodic memory were administered via a trained examiner and correlates of these tests were self-administered remotely via a smartphone. Using examples, we will show which aspects of cognitive assessment had the best correlations between remote self-administration and face-to-face examination and which had lower correlations.

Digital remote assessments can help overcome barriers by enabling repeated testing in naturalistic conditions, reducing participant burden and expense, and increasing research accessibility for populations currently under-represented. Moreover, the ubiquity of internet-connected devices vastly increases opportunities to remotely monitor other dimensions relevant to cognition using smartphone apps and wearable sensors. In addition to improving access to testing, digitally administered assessments dramatically improve some individual’s tolerance to testing with shorter tests that can be administered via computer adaptive testing (CAT). Despite these benefits, some aspects of the cognitive assessment cannot be adequately replicated remotely and thus yield lower correlations to their examiner-administered alternatives. Clinical and research implications are discussed.

We conducted a population-based study using Ontario health administrative data to describe trends in healthcare utilization and mortality in adults with epilepsy during the first pandemic year (March 2020–March 2021) compared to historical data (2016–2019). We also investigated if changes in outpatient visits and diagnostic testing during the first pandemic year were associated with increased risk for hospitalizations, emergency department (ED) visits, or death.

Projected monthly visit rates (per 100,000 people) for outpatient visits, electroencephalography, magnetic resonance, computed tomography, all-cause ED visits, hospitalizations, and mortality were calculated based on historical data by fitting monthly time series autoregressive integrated moving-average models. Two-way interactions were calculated using Quasi-Poisson models.

In adults with epilepsy during the first quarter of the pandemic, we demonstrated a reduction in all-cause outpatient visits, diagnostic testing, ED visits and hospitalizations, and a temporary increase in mortality (observed rates of 355.8 vs projected 308.8, 95% CI: 276.3–345.1). By the end of the year, outpatient visits increased (85,535.4 vs 76,620.6, 95% CI: 71,546.9–82,059.4), and most of the diagnostic test rates returned to the projected. The increase in the rate of all-cause mortality during the pandemic, compared to pre-pandemic, was greater during months with the lower frequency of diagnostic tests than months with higher frequency (interaction p-values <.0001).

We described the impact of the pandemic on healthcare utilization and mortality in adults with epilepsy during the first year. We demonstrated that access to relevant diagnostic testing is likely important for this population while planning restrictions on non-urgent health services.