INTRODUCTION
The novel coronavirus (COVID-19), declared a global pandemic on March 11, 2020 (Cucinotta & Vanelli, Reference Cucinotta and Vanelli2020), placed unprecedented demand on healthcare systems, resulting in delays of nonurgent appointments and widespread closures. Many neuropsychology clinics were abruptly closed, and neuropsychologists were faced with a choice between ceasing to care for patients or finding alternative methods for delivering services.
The remote diagnosis and treatment of patients by means of telecommunications technology, “telemedicine”, originated in the 1960s (Wittson, Affleck, & Johnson, Reference Wittson, Affleck and Johnson1961). Telemedicine use expanded dramatically during the first month of COVID-19 closures, leading to a 683% increase in the use of telehealth (Mann, Chen, Chunara, Testa, & Nov, Reference Mann, Chen, Chunara, Testa and Nov2020). The application of telemedicine practices to neuropsychology and teleneuropsychology (TNP) has proven to be feasible in multiple studies (Harrell, Wilkins, Connor, & Chodosh, Reference Harrell, Wilkins, Connor and Chodosh2014; Hodge et al., Reference Hodge, Sutherland, Jeng, Bale, Batta, Cambridge and Silove2019; Parikh et al., Reference Parikh, Grosch, Graham, Hynan, Weiner, Shore and Cullum2013; Sutherland, Trembath, Hodge, Rose, & Roberts, Reference Sutherland, Trembath, Hodge, Rose and Roberts2019; Turner, Horner, Vankirk, Myrick, & Tuerk, Reference Turner, Horner, Vankirk, Myrick and Tuerk2012; Wadsworth et al., Reference Wadsworth, Galusha-Glasscock, Womack, Quiceno, Weiner, Hynan and Cullum2016), and preliminary reliability studies have been promising (Brearly et al., Reference Brearly, Shura, Martindale, Lazowski, Luxton, Shenal and Rowland2017; Cullum, Hynan, Grosch, Parikh, & Weiner, Reference Cullum, Hynan, Grosch, Parikh and Weiner2014). Parikh et al (Reference Parikh, Grosch, Graham, Hynan, Weiner, Shore and Cullum2013) used patient surveys to evaluate the feasibility and acceptance of neuropsychological evaluations via remote video connection, and 98% of patients reported positive experiences with the evaluation process.
Prior to the initial submission of this manuscript, most studies evaluated assisted-TNP, in which a neuropsychologist interacts with patients in a clinical space at a remote location, ensuring adequate internet connectivity, a distraction-free environment, and in some studies, providing a proctor to manage test materials. The assisted-TNP model has extended the reach of neuropsychology to remote areas (Harrell et al., Reference Harrell, Wilkins, Connor and Chodosh2014; Turner et al., Reference Turner, Horner, Vankirk, Myrick and Tuerk2012), but did not reduce risk of COVID-19 exposure to the same extent as could be accomplished by providing direct-to-home-teleneuropsychology (DTH-TNP). The Inter-Organizational Practice Committee (a collaboration of neuropsychology professional groups) published guidance for DTH-TNP (https://iopc.online/teleneuropsychology-guidelines), and recent update has provided a careful consideration of risks and benefits of neuropsychological evaluation in different settings during the COVID-19 pandemic (Postal et al., Reference Postal, Bilder, Lanca, Aase, Barisa, Holland and Salinas2021). Over the past several months, the feasibility and acceptance of DTH-TNP has been a focus of study in children (Harder et al., Reference Harder, Hernandez, Hague, Neumann, McCreary, Cullum and Greenberg2020), as has validation of a telephone cognitive screening test in patients with Parkinson disease (Benge & Kiselica, Reference Benge and Kiselica2021).
We report preliminary findings regarding clinician feasibility and patient acceptance of DTH-TNP in an academic medical center lifespan neuropsychology practice that serves patients from a broad range of referral sources, the Massachusetts General Hospital Psychology Assessment Center (MGH-PAC). Based on findings in the assisted-TNP literature, we hypothesized that patients would endorse high acceptance rates and that neuropsychologists would attest to the feasibility of DTH-TNP.
Methods
Participants and Process
The MGH-PAC was closed on March 13, 2020. Over the ensuing weeks, a service development process took place (detailed in a separate manuscript under review) to develop DTH-TNP capabilities. Patients underwent an informed consent process reviewing the novel nature of this service and were given the option to be seen in person at a later date. Patients completed DTH-TNP visits, involving interview, formal testing, and/or feedback. This project was undertaken as a Quality Improvement Initiative of the MGH-PAC and followed the policies laid out by the Institutional Review Board.
Patient Surveys
A previously published patient survey (Parikh et al., Reference Parikh, Grosch, Graham, Hynan, Weiner, Shore and Cullum2013) was modified for DTH-TNP and implemented in the Research Electronic Data Capture (REDCap) system. After completion of a DTH-TNP appointment, patients or their caregivers were invited to complete the voluntary satisfaction survey. To reduce any possible feelings of coercion, surveys intentionally did not include patient identifying information (e.g., age ranges only), though patients were given the option to identify the clinician. Surveys, offered in both English and Spanish, examined satisfaction using a five-point Likert scale (see Supplementary Table 1a) and free-text responses.
Clinician Surveys
Following the assessment, the 10 neuropsychologists at the MGH-PAC were encouraged to complete a survey to assess feasibility (see Supplementary Table 1b). Individual clinician identifiers were not captured and surveys were combined in the data set.
Data Analysis
Responses to survey items are reported as percentages. Responses of “strongly agree” and “agree” were combined to indicate agreement, “strongly disagree” and “disagree” to indicate disagreement, and responses of “neutral” were not analyzed. Subgroup comparisons were conducted using chi-square tests for categorical data (e.g., checked vs. not checked items on the survey). Free-text responses were reviewed by three members of the study team (MWP, MG, KP) and are reported qualitatively.
Results
Patient Feedback
The patient feedback survey was launched on April 20, 2020, and 72 surveys were completed by August 19, 2020 (not all respondents completed every item, thus the below totals may not sum to 72). Respondents were primarily the patient themselves (n = 59, 82%), though some were completed by a parent/guardian (n = 7, 10%) or caregiver (n = 5, 7%). One survey was completed in Spanish. The median duration of DTH-NP appointments was 120 min (standard deviation = 80 min; range = 30–440 min). Patients across the lifespan were represented in the survey (Figure 1a). Appointments included multiple activities (interview, testing, and feedback), with 56 (78%) respondents indicating that their appointment included testing (median length = 140 min). The remaining visits were interview (n = 13, 18%) or feedback (n=3, 4%) only (median length = 60 min). All respondents indicated satisfaction with the DTH-TNP visit (Figure 1b), indicated that they felt understood by the examiner, and the vast majority would recommend DTH-TNP to others. There were no differences in the degree of satisfaction (e.g., “strongly agree” vs “agree”) between patients who did vs. did not have testing as part of their appointment, X 2 (1, n = 72) = 2.77; p = .10 (ns). Technical difficulties were noted in some instances, and a small number of individuals felt that there were difficulties with communication. Despite the favorable responses, a majority of patients indicated that they would travel some distance for an in-person appointment (Figure 1c).
Agreement with specific items from the patient survey expressed as percentages of the total number of respondents (n = 72). (a) Age of respondents (note, patients provided age range rather than specific age to protect anonymity of respondents), (b) response to satisfaction questions, and (c) preference for driving distance or virtual assessment. For the purposes of illustration, responses of “strongly agree” and “agree” were combined into the bar indicating agreement and responses of “strongly disagree” and “disagree” were combined into the bar indicating disagreement.
Respondents endorsed a number of favorable features of the virtual assessments compared with their expectations of in-person assessments (59 of 61 surveys included responses and endorsed saved travel time: n = 48, 79%; reduced risk of COVID exposure: n = 47, 78%; reduced anxiety: n = 13, 21%; easier to concentrate: n = 9, 15%), though some also indicated potential advantages of an in-person visit (36 of 61 surveys included responses and indicated that an in-person assessment would: allow more extensive assessment: n = 19; 53%; promote better personal connection with the examiner n = 15; 42%; allow better communication n = 8; 22%; make it easier to express concerns n = 7; 19%).
Thirty-eight respondents provided free-text comments, which were categorized as positive (n = 19, 50%), negative (n = 7, 18%), or a mixture of positive and negative input (n = 12, 32%). Positive comments related to the examiner (e.g., “She was extremely thorough… made sure I was comfortable…”; “the doctor was unbelievably competent, caring, and thoughtful…”) and the convenience of being able to complete testing from home (e.g., “My husband was able to join the session from quarantine…”; “Virtual assessments are less intrusive to my schedule.”). Themes from negative feedback included technical difficulties (n = 10; e.g., “My computer had problems and I lost connection many times”; “There was a small glitch in getting connected…”) and limitations on the testing imposed by the virtual modality (n = 3; e.g., “I didn’t like that the examiner could not see my drawing and writing task”).
Clinician Feedback
Providers completed 134 surveys (of a total possible evaluation count of 237, representing a 57% return rate) between April 22, 2020, and August 21, 2020. The median age of patients for which surveys were completed was 58 years (SD = 21.79; range = 5 to 88) and 47% (n = 64) were female. The median duration of virtual visits was 120 min (SD = 68; range = 30–480). As shown in Figure 2, 82% of cases involved testing. Providers reported that they were able to achieve the goal of the assessment in 90% of cases, and rate of goal achievement did not differ between visits that included formal testing (88%) and those that did not (94%); X 2 (1, n = 134) = 1.25; p = .26 (ns). Neuropsychologists reported factors that interfered with their ability to achieve the goal of assessments included the inability to administer high-level executive or attention tasks in (n = 6; 4%), severe cognitive impairment or behavioral regulation issues on the part of the patient (n = 4; 3%), and technical or logistical issues (poor connection, provider not licensed in patient’s state; n = 4; 3%). The assessment was divided into more than one session in 51% of cases (n = 69); the remainder were completed in a single session. There was no difference in the rate of goal achievement between single session evaluations (86%) or multisession evaluations (93%); X 2 (1, n = 134) = 1.56, p = .21 (ns). Testing assistance (e.g., psychometrist) was used in 30% of cases (n = 40).
Agreement with specific items from the clinician survey expressed as percentages of the total number of respondents (n = 134). (a) Proportion of assessments involving specific activities, (b) methods of assessment, (c) problems endorsed by clinicians, and (d) overall feedback. Only the question regarding achievement of evaluation goals included an option for “partially”.
Clinicians indicated that no modifications to standard procedures were needed in most evaluations (n = 104, 77%), but endorsed using brief instructions (n = 12, 9%), taking frequent breaks (n = 12, 9%), or having another person or family member assist with technical issues (n = 19, 14%) as common strategies. Specific challenges that occurred during the assessments (64 surveys included responses; 47%) included interruptions, technical problems, and patient cognitive or perception problems.
The most common themes in free-text response regarding challenges during the assessment were technical issues (n = 51 comments, e.g., “Patient could not hear audio if they responded at the same time”; “Patient experienced poor internet connection”), poor engagement, distraction, or fatigue on the part of the patient (n = 37 comments, e.g., “Although I asked him to turn off his phone, the patient was clearly getting text messages that distracted him”; “Men were cleaning windows outside of where the patient was sitting”), and delays in the examination (n = 15 comments; e.g., “Patient overslept and I had to call to wake them up”). Strategies used by the clinicians included technical assistance for patients (e.g., family member in the home, 36 comments) and taking extra time to manage the evaluation (e.g., extended planning time, breaking testing into multiple sessions, 58 comments).
DISCUSSION
Patients and clinicians found DTH-TNP to be both acceptable and feasible. One hundred percent of patients who responded to this survey reported satisfaction and 93% indicated that they would recommend this type of appointment to others, despite the fact that 59% of patients noted some issues or limitations. These findings suggest that patients recognized both the benefits and limitations of the DTH-TNP evaluation. Indeed, only 26% of patients stated that they preferred virtual to in-person appointments. Providers were able to achieve the goal of their visit in 90% of the clinical encounters. To our knowledge, this is the first demonstration of acceptance and feasibility of DTH-TNP in a lifespan clinical sample.
DTH-TNP allows neuropsychologists to reach patients when traditional face-to-face approaches are not possible. In our clinical experience, results from the DTH-TNP evaluations that we conducted have been accepted as valid by consumers of neuropsychological services in the medical, educational, and legal communities. Neurologists, psychiatrists, primary care, and therapy providers are using recommendations from these DTH-TNP evaluations to inform differential diagnosis or adjust treatment course. School districts are using information about diagnosis and recommendations to provide services for students. Results from some of these DTH-TNP evaluations are being used in pursuit of legal guardianship. Thus, despite the novelty of the practice, the DTH-TNP evaluations we are conducting seem to be recognized across multiple settings as similar to in-person evaluations.
Limitations include the fact that survey responses were voluntary and were not completed by all patients or providers after every encounter, reflecting the fact that the survey was implemented as an early QI project. We also do not have data regarding individual characteristics of respondents, other than age range. While this step was taken to ensure anonymity and reduce potential bias in the responses, it may be that those patients who were not satisfied with the experience chose not to respond to the survey. We believe this is unlikely, given the relatively balanced feedback we received, which included challenges, concerns, and areas for improvement. Because the patient survey was completed by either a patient or a caregiver, the results may reflect more than one perspective, which cannot readily be disentangled. Our approach does not allow a comparison of satisfaction with DTH-TNP to in-office neuropsychological assessment, since most of these patients had no basis for comparison (though there were some comments suggesting a favorable view of the virtual visit as compared with patients’ prior neuropsychological experience: e.g., “Regarding virtual vs. non-virtual, I would say they were about equal”). The research was conducted during the initial phase of the COVID-19 pandemic, and while there have been intermittent peaks in infection rate since that time, neuropsychologists may now be more informed and equipped to provide in-office services than they were during that early stage. During the period covered by this survey, many states made provisions for service by providers licensed elsewhere. In the future, modifications to licensure laws, such as those initiated by the Association of State and Provincial Psychology Boards (https://www.asppb.net/page/PSYPACT), may be needed to provide DTH-TNP services to patients outside their home state.
Acceptance and feasibility are only the first steps in developing a clinically viable service. The reliability and validity of neuropsychological tests in the DTH-TNP setting have not been established. Although the early studies of reliability and validity using assisted-TNP are promising (Brearly et al., Reference Brearly, Shura, Martindale, Lazowski, Luxton, Shenal and Rowland2017; Cullum et al., Reference Cullum, Hynan, Grosch, Parikh and Weiner2014; Marra, Hamlet, Bauer, & Bowers, Reference Marra, Hamlet, Bauer and Bowers2020; Sutherland et al., Reference Sutherland, Trembath, Hodge, Rose and Roberts2019; Wadsworth et al., Reference Wadsworth, Galusha-Glasscock, Womack, Quiceno, Weiner, Hynan and Cullum2016), the lack of control over the testing environment in DTH-TNP raises concerns for compliance with standardized procedures, additional distractions, technical limitations, and problems with test security (Bilder et al., Reference Bilder, Postal, Barisa, Aase, Cullum, Gillaspy and Lechuga2020; Hewitt & Loring, Reference Hewitt and Loring2020). Further exploration of limitations on the test battery imposed by DTH-TNP conditions, as compared with in-person evaluations, may be important in understanding the limits of this modality. The ability to accurately assess test effort and performance validity in the DTH-TNP context is an unresolved issue. Given that clinicians are likely relying more heavily on experience and clinical judgment when using TNP methods, investigating whether additional training is warranted and how psychometricians and neuropsychology trainees will respond/adapt may also be important issues to study. Perhaps of most concern, some individuals may lack access to the equipment needed for DTH-TNP (e.g., computer with camera) or are in a home that does not provide a suitable environment for testing. Methods to address disparities in access will be critical to the value of DTH-TNP.
These preliminary and qualitative results demonstrate that patients were satisfied with their DTH-TNP experience, and clinicians felt that they were able to successfully conduct evaluations using a DTH-TNP approach. These findings and those of many others (Cullum et al., Reference Cullum, Hynan, Grosch, Parikh and Weiner2014; Cullum, Weiner, Gehrmann, & Hynan, Reference Cullum, Weiner, Gehrmann and Hynan2006; Harder et al., Reference Harder, Hernandez, Hague, Neumann, McCreary, Cullum and Greenberg2020; Hewitt & Loring, Reference Hewitt and Loring2020; Marra et al., Reference Marra, Hamlet, Bauer and Bowers2020; Temple, Drummond, Valiquette, & Jozsvai, Reference Temple, Drummond, Valiquette and Jozsvai2010) suggest that DTH-TNP may be a relevant method for delivery of neuropsychology services and may have the potential to expand access to neuropsychological services over the long term. These results build on TNP research over the past two decades, but work remains to determine the long-term viability of this practice. Given the potential advantages of these methods, we feel that continued exploration of its strengths and limitations is warranted.
ACKNOWLEDGEMENTS
The authors wish to acknowledge the support and assistance of the administrative staff of the MGH-PAC, particularly Vanessa Jillson and Rhina Sariles, who worked well beyond the call of duty to make virtual appointments a reality. The support and encouragement of the MGH Department of Psychiatry was also instrumental in accomplishing these goals. Yakeel Quiroz, PhD, contributed to the conceptualization of the clinical service, coordinated initial information sessions, and supervised the work of Jazmin Rios, who translated the patient survey into Spanish.
FINANCIAL SUPPORT
This work was not funded by any sources of financial support.
CONFLICT OF INTEREST
The authors have nothing to disclose.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S1355617721000436
Open access
