Overview

The initiative to implement CR in outpatient clinics throughout the state required significant organization and administrative support. A 3-tiered model of oversight governed the implementation process, coordinated through the Office of the Medical Director and the Office of State Operated Adult Services; leadership was represented at the state level by Central Administration, as well as within each local facility and clinic. Figure 1 illustrates the responsibilities assumed by each. Prior to implementation, a leadership team (authors AM, MDE, and LIS) first conceptualized the treatment model and presented a white paper and budget plan. During implementation this team identified local and central leaders, content experts, and vendors; located initial OMH state-operated adult outpatient sites for a staggered rollout of CR; and readied facilities for the necessary staffing, training, and infrastructural requirements as required by the treatment model.

Figure 1

Model of implementation oversight.

The conceptual model of CR

A first step of the leadership team was to align the model of CR that would be implemented with the overall goals of outpatient psychiatric care. Conceptualization of the treatment model helped to define a culture of cognitive health at OMH and operationalize methods of client referral, screening, and assessment, as well as staffing needs.

At OMH, a primary goal of outpatient treatment is to provide recovery-oriented, patient-centered treatment to facilitate maximal productive engagement in the community with as much independence as possible. CR services were being provided to further that goal. Research indicates that CR is more likely to impact cognitive symptoms, community engagement, and functional outcome when it is provided in a rehabilitation context.Reference Wykes, Huddy, Cellard, McGurk and Czobor ^{15 ,} Reference Medalia and Saperstein ³³ Rehabilitation programs emphasize a multidisciplinary approach, skills acquisition, independent living, consumer advocacy, and pharmaceutical treatment in combination with behavioral intervention. The preferred model of CR was one that could integrate easily within the outpatient rehabilitation context and that emphasized the improvement of cognition as a way to enhance attainment of overall recovery goals. The CR service was expected to complement and enhance current treatments. Maximizing wellness, addressing cognitive deficits, and improving patient-centered recovery goals were identified as the 3 pillars of the CR initiative. OMH central leadership named the initiative Cognitive Remediation to Promote Recovery (CR2PR), to reflect that it would serve patients with SMI identified as having cognitive impairments that hamper recovery goal attainment. With this name, the initiative tied cognitive health to functional outcome, and squarely placed CR as a recovery oriented treatment.

A second goal of outpatient treatment at OMH is to improve the quality of behavioral and cognitive health of all community-dwelling public sector clients. The OMH leadership team advocated for a CR program with a “public psychiatry” focus, whereby most adults within the state-operated system would be eligible to receive services. The program therefore had to be accessible in rural and urban areas, and available to the multilingual populations served, with few exclusion criteria. The CR program needed to be flexible so it could be adapted to client and clinic alike. Recognizing that avolition is a primary issue for public sector clients with severe mental illnesses,Reference Foussias and Remington ^{34 ,} Reference Strauss, Horan and Kirkpatrick ³⁵ and that low motivation interferes with treatment compliance and learning of skills to promote recovery,Reference Medalia and Saperstein ^{36 ,} Reference Saperstein and Medalia ³⁷ the model of CR needed to address impairments in motivation alongside cognitive dysfunction. The goal that participation in the CR service be experienced as enjoyable by the clients was important to facilitate treatment engagement and learning outcomes.

Finally, feasibility factors were considered when choosing a model of CR implementation. The service needed to fit within the constraints of Current Procedural Terminology (CPT) coding requirements so that it would be billable and therefore self-sustaining. Staffing, technical requirements such as web connectivity, and space requirements also informed the choice of CR model. Research on home-based computer or iPad delivered CR is an exciting frontier, but at this juncture such models of delivery are incongruent with the realities of internet access and codable/billable service implementation in the public sector. On the other hand, models of CR that conform to CBT group-based treatment are codable and thus sustainable as an implemented service.

The Neuropsychological and Educational Approach to Remediation (NEAR)Reference Medalia, Herlands, Saperstein and Revheim ²⁸ is the approach considered to best fit these guidelines. NEAR has been used in multiple clinical settings, geographic regions, cultures, age groups, and languages and thus provides the flexibility and “public psychiatry” mandate of being a CR service that is client inclusive and accessible. With ready-made clinician training curriculums and manuals, inclusive of web-based training platforms, there is a systematic and efficient process to train clinicians. NEAR implementation kits provide training in the set-up of quality assurance and treatment fidelity monitoring so that programs start with a mechanism in place to continually improve service delivery.

The theoretical framework of NEAR is based on neuropsychology, educational psychology, learning theory, and cognitive psychology. In NEAR, psychiatric rehabilitation is viewed as a learning process, and the client is considered a learner—a participant in a learning process. The theoretical foundation of NEAR emphasizes motivation, and the teaching techniques are designed to promote motivation to learn. Each session is designed to be a positive, personalized, recovery-oriented learning experience that targets the cognitive skills identified as impeding progress toward individuals’ recovery goals.

Specifically developed for psychiatric patients, NEAR is a group-based treatment that provides highly individualized learning by allowing each person in the group to work at his or her own pace on tasks carefully chosen to be engaging and to address particular cognitive needs. Sessions are conducted in groups of 6–8 clients, which has both therapeutic and administrative advantages. In a group, a sense of community often develops among the clients because they share the same valued activities of learning and self-improvement. Because clients work at different tasks and rates, the groups have a rolling admission. It is not administratively advantageous or clinically responsive to wait to start a group until all 6 people have been screened for admission. Rather, the groups slowly develop as the clinician enrolls more people. Clients participate in sessions at least twice each week, and the sessions are typically 60 minutes long. About three-fourths of the session involves individual cognitive activities on the computer, and one-fourth is spent in a bridging group, which is a verbal group for all the participants to discuss how the computer activities relate to real-world situations. The bridging group discussions also promote metacognition by teaching how cognitive skills are used in everyday life, and by providing in vivo activities to promote cognitive skill development.

Creating a culture of cognitive health

Implementation of CR in a large system of care begins with creating a culture of cognitive health which brings awareness to cognitive impairment as a barrier to recovery and a key target of intervention. This often requires education to develop a common language around cognition that is shared by administrators, clinical care providers, patients, and their families. Shared understanding of what is meant by cognitive symptoms and their impact in real-world contexts fosters coordinated, person-centered care by providing a unified context for recognizing and addressing cognitive problems from multiple vantage points. Cognitive awareness promotes the integration of cognitive health building practices, including but not limited to CR, while also impacting down-stream care across behavioral health disciplines (eg, attention to pro-cognitive pharmacologic treatment, improved physical health, vocational and educational support). The culture of cognitive health and a common cognitive lexicon facilitates referrals to a CR program, shared decision making for cognitive intervention(s), recovery-oriented treatment planning, the therapeutic alliance, and outcomes assessments to ensure program efficiency and effectiveness.Reference Amado and Sederer ³⁸ The CR2PR initiative sought to create a culture of cognitive health throughout OMH and relied on clinic leadership to uphold the local culture, thereby facilitating the development and growth of each CR program.

OMH created THINKING WELL, a model educational campaign that focuses on the need for cognitive health to make a good recovery. This initiative includes education about cognitive health and wide-scale enhancements for the delivery of treatments that promote optimal cognitive health. Components of THINKING WELL include literature and informational sheets for providers, patients, and families; a website for providers; and statewide archived Grand Rounds, which are web-based free lectures and clinician focus groups to inform OMH staff about implementation efforts.

Gateway to CR services

When implementing a service, key questions concern the treatment gateway: Who will be eligible to receive the service? How will consumers be identified, referred, and screened? What assessments will be needed? In keeping with the recovery orientation of the treatment model, the referral process was to identify all clients with cognitive problems interfering with attainment of recovery goals. The central question was: Does this person have memory, attention, processing speed, or other cognitive problems that are making it difficult to go back to school or work, manage symptoms, or function independently in their home? People with intellectual disabilities or less than a 4th grade reading level were excluded from CR and instead referred to services for people with developmental disabilities. Clients who would be better served by a CR program for people with neurologic disorders (eg, Traumatic Brain Injury) were to be appropriately routed to care and not referred to CR2PR, which was designed for people whose cognitive problems stem from psychiatric disorders. Actively symptomatic clients were welcome as long as they had sufficient behavioral control to participate in a 1-hour group session.

Operationalization of the referral and screening process aimed to facilitate a steady stream of appropriate referrals, with few barriers to compromise accessibility and efficiency. Referrals from all clinical staff as well as clients and support systems would be accepted. The screening process was to adhere to shared decision making guidelines, whereby client and clinician identify and agree on recovery-oriented goals and subsequent steps to promote goal attainment.

The staffing model

The CR program was designed to be sustainable by billing for testing and the CR group, with one group meeting twice weekly with 6–8 clients, requiring a total of .15 full-time effort cumulatively across providers and administrators. The staffing model recognized that each clinic where CR2PR was to be implemented needed a “champion” to be the primary leader for the CR service and an advocate for the integration of CR as a new clinical practice. Psychologists were selected as champions for this initiative, as within OMH they are recognized as the team member with the theoretical and clinical background to easily learn and help other team members understand the importance of addressing cognitive health, as well as having expertise in cognitive assessment and behavioral therapies. Psychologists thus would be both the champions and the persons who performed cognitive assessments to inform the treatment plan. All other clinical staff with at least a master’s level of education in clinical psychiatric care (nurses, physicians, psychologists, social workers, mental health counselors) were eligible to conduct CR sessions. Finally, a clinical administrator was identified as the team member who facilitated the implementation and sustainability of the service.

Identifying sites

The OMH central office leadership team took a stepped approach to identifying which of the 62 outpatient clinics within 16 statewide adult Psychiatric Centers (PCs) were best suited to provide CR2PR. In step 1, phone calls with PC administrators and medical directors identified 13 PCs across the state as having the staffing and interest to implement CR2PR. In step 2, clinics within these PCs were considered for CR2PR implementation based on leadership, staffing, the availability of skills training and recovery-oriented services, the infrastructure to support CR2PR activities, and the need for geographic diversity of sites, as well as socioeconomic, cultural, and educational diversity of the client population.

CR requires specific conditions that affect its implementation in real-world settings that may differ from other evidence-based practices. Clinics interested in being a CR2PR site completed a Facility Needs Assessment (Table 1) which helped to concretize program needs, and alerted sites and central office when the facility had to make adaptations in order to implement CR2PR. Of the conditions queried, the most common adaptations were the need to acquire more computers and establish web connectivity for computer-based cognitive exercise. In large public systems of care, this can be a time-consuming process, since internet security issues must be managed.

Table 1

Facility needs assessment

Ultimately, 16 clinics were identified as CR2PR implementation sites. These 16 clinics range in size, with the smallest serving 23 clients and the largest serving 299 per week. The mean weekly census across clinics is 138 clients and 92–100% of clients are diagnosed with a SMI. On average, the percentage of patients carrying a diagnosis of schizophrenia or other psychotic disorder is 66.3% (range 40–87%). Adults age 18–64 years constitute over 80% of the population served in all but one clinic, where 60% of the clients were ages 65 and over. Ten clinics are in urban areas, and 6 are in rural areas. One clinic serves a majority (58%) of non-English-speaking clients, while the other 15 clinics serve a majority of English-speaking clients (range: 79–100%).

Implementation was done in two phases. In 2015, Phase 1 implementation commenced at 9 OMH-operated adult outpatient clinics that were attached to 8 unique OMH-operated PCs. In 2016, Phase 2 was launched at 7 additional OMH-operated clinics whereby 5 new PCs received the program and 2 pre-existing Phase 1 PCs implemented CR2PR in a second clinic. One Phase 2 clinic chose to delay clinical implementation until 2017 to become a site for a randomized clinical trial, which offered 2 types of CR (see clinicaltrials.gov NCT01945333), structured in the same format as the type of CR used in CR2PR clinics. By 2017, CR2PR was available at 13 of the 16 adult PCs, and 14 of the 16 chosen clinics were operating with CR2PR (including the clinic that transitioned from being a research site). Clinic closures impacted several CR2PR programs.

Staff training

A standardized, efficient staff training plan is needed when implementing CR in large systems of care. Experts in assessment and CR were identified as content leaders to provide training. To inform training needs, focus groups were conducted with staff before starting CR2PR. Clinicians reported an awareness of cognitive problems among clients, but uncertainty about etiology and treatment; although, for those who were aware of CR, the scientific evidence supporting its efficacy was perceived as moderate to strong. Clinicians were unaware of how to use cognitive assessments to create a CR treatment plan, how to conduct CR sessions, how to use web-based cognitive training exercises, and how CR can be a means to promote recovery. Questions about who to refer and who would benefit from CR were reported. The staff training was designed to provide education to define what cognitive health is, what CR is, and who benefits; how to conduct assessments for program entry; and how to conduct CR sessions. To ensure a solid referral base, educational activities for all clinic staff focused on how cognition impacts daily functioning and how CR can promote recovery.

Staff training draws clinicians away from other important duties, so it is essential that it be conducted efficiently. Multiple venues and resources were utilized (Table 2). CR2PR training included 1 in-person day of training for CR clinicians and champions, offered at 2 geographic locations. This was supplemented with webinars; internet-based learning at www.teachrecovery.com, a website built for CR training; monthly telephonic supervision; and lectures and workshops on assessment and CR session skills at Columbia University’s sponsored annual Cognitive Remediation in Psychiatry conference. Clinical administrators were provided written and verbal instructions, and periodic supervision on how to support the program and supervise staff. Because New York is a large state, it was important to minimize expensive and time-consuming travel to in-person training. Furthermore, with a 25% rate of staff turnover across sites, the web-based and telephonic training programs, coupled with train the trainer classes, could accommodate a constant influx of new clinicians.

Table 2

Training model

The training package specific to NEAR includes slide decks to teach didactics; forms for administrator, clinician, and client use; manuals; and competency exams to check knowledge. These training materials were adapted for use at OMH clinics to align with the state-operated quality assurance and record keeping standards. To facilitate dissemination of information for training and supervisory purposes, a website with CR2PR-related teaching and supervision materials was created on OMH’s internal website. Statewide Grand Rounds and site-specific in-services were used to disseminate information that would facilitate clinician knowledge about who to refer to CR2PR.

Program start-up at each site using web-based, face-to-face, and manualized implementation strategies took an average of 6 months. In Phase 1, 36 OMH staffers were trained to understand the details of cognitive health and to identify cognitive impairments. Additional training on cognitive assessment was provided for psychologists. In Phase 2, 42 new CR2PR staff members were trained to provide CR2PR. At least 30 CR2PR staff members attended the OMH sponsored Cognitive Remediation Conference annually from 2015 to 2017.

An essential part of the training was to familiarize the CR2PR clinicians with the use of web-based cognitive exercises and, in complement, how to provide coaching and support. Since CR2PR is highly personalized, the clinician selects CR exercises based on areas of cognitive strength and weakness revealed by the cognitive assessment, thus accounting for individuals’ learning needs and recovery goals. As a standard CR2PR training activity, clinicians sampled and analyzed exercises with respect to the primary cognitive targets (eg, attention, executive functioning, working memory, processing speed, verbal memory), prerequisite skills (eg, computer familiarity), task adaptability, level of therapist involvement, and overall strengths and weaknesses. Their task analyses were reviewed in supervision to ensure they understood how to choose an exercise for any given client, given their cognitive, learning, and recovery needs.

To evaluate this training component and inform further training needs, the ratings of 24 exercises from 3 web-based training programs were examined from 21 clinicians (eg, psychologists, social workers, nurses) in Phase 1 implementation sites. The specific quality improvement focus was whether clinicians could identify the cognitive target of each exercise. For example, did they recognize a verbal memory training task as different from a processing speed task? While the clinicians’ task analyses appropriately labeled exercises as targeting executive functions (eg, reasoning, problem-solving), memory and attention were often inaccurately under- or over-identified as cognitive targets. Without training, clinicians struggled to identify tasks that would help a client improve their memory or attention.

Without better recognition of cognitive skills in practice, clinicians may neglect to select the most appropriate exercises for a treatment plan, or inaccurately employ other exercises to meet specific cognitive needs. Treatment plans may be less efficient, personalized, effective, and therefore less recovery-oriented. Clinicians’ ability to match exercises and cognitive targets is also necessary to facilitate clients’ meta-cognitive awareness of their own cognitive processes during CR and for teaching appropriate strategies for executing cognitive tasks. This training program evaluation study informed a need for ongoing in-depth training about cognition and the value of cognitive task analysis as a training tool.

Staff receptivity to CR

Staff receptivity to incorporating a new treatment modality was encouraged using basic principles of motivation:

1. 1. Clinicians’ perceptions of competency were supported by ensuring experiences of success in meeting feasible training benchmarks and by discussing positive clinical outcomes.

2. 2. Interest was promoted by employing training techniques that use contextualization, personalization, and control (eg, role-plays, discussing clinicians’ clinical experiences).

3. 3. Supervision provided an experience of caring.

4. 4. Clinicians’ autonomy was promoted by giving choices in the benchmarks and use of materials, and by celebrating creativity while maintaining treatment fidelity.

Staff were offered the chance to participate in CR2PR but respected if they declined. Those who went through the motions of engaging in the training or setting up a program without producing clinical results were encouraged to get involved in other activities to give staff with interest the opportunity to participate. Ongoing supervision of staff and management emphasized interest in the program as opposed to compliance with regulations. Opportunities to showcase their work within and outside of the system were provided.

CR2PR program phases

CR2PR was conceptualized as having 3 program phases: entry, treatment, and exit (Figure 2). The entry phase entails a referral, screening, and assessment. Referrals can be made by any clinical team member by completing a referral form, and the CR clinicians are encouraged to remind staff about the CR2PR program during staff meetings and clinical case conferences. Clients are able to self-refer, in which case their care coordinator is asked to complete the referral form. The initial screening is a brief meeting between the CR2PR clinician and client to review their goals, availability, and interest in the program. The cognitive assessment includes a brief history and 40 minutes of cognitive tests.

Figure 2

CR2PR treatment model.

The CR2PR treatment phase generally consists of 30 sessions conducted twice weekly, although length of treatment can be tailored to individual needs. Clients are asked to commit to attending blocks of 10 sessions that are conducted in groups of 6–8 participants. Each session is 60 minutes, including 45 minutes devoted to restorative web-based cognitive training exercises and 15 minutes of bridging group discussion. Admission to the group is on a rolling basis, and during the computer exercise portion of the session, participants work at their own rate on exercises identified as useful to them based on their assessment and recovery goals. One clinician leads the group.

The exit phase is marked with a graduation ceremony where clients are given certificates of completion. Review of progress and next steps engages all group members and serves to promote motivation and engagement in learning.

Formal assessment

The CR2PR program includes a structured assessment of cognitive functions to accompany subjective reports of cognitive problems. The goal of the assessment is to provide objective evidence of cognitive deficits to guide the CR2PR treatment plan. While brief assessment batteries exist, none met the feasibility and clinical criteria for this project. Therefore, the CR2PR clinical director (author AM) worked with an outside content expert to refine a battery for OMH cognitive health services. Brief Assessment of Cognition in Schizophrenia (BACS)Reference Keefe, Goldberg, Harvey, Gold, Poe and Coughenour ³⁹ and MATRICS Consensus Cognitive Battery (MCCB)Reference Nuechterlein, Green and Kern ⁴⁰ subtests are used in a clinician-administered brief assessment of premorbid intelligence, working memory, verbal memory, attention, executive functioning, and processing speed. An outside vendor was contracted to train psychologists in the administration and scoring of the battery, and to develop a scoring platform that would generate an automated report used to identify pertinent cognitive targets. Clinician training in assessment was done entirely over the web, and certificates were given to those psychologists who passed the competency test. Once certified, psychologists were trained to develop treatment plans using the test results. One or 2 psychologists from each of the 16 clinics were certified as testers.

Quality assurance monitoring

Continuous quality improvement (QI) strategies were built into the CR2PR initiative to maintain programmatic fidelity, efficacy, and efficiency. At each CR2PR site, the clinicians designated as “champions” oversaw the implementation of QI data collection. Staff were trained to systematically track appointments, session attendance, and client progress, enabling monthly monitoring and the collection of data to generate a quarterly report on referrals, intakes, utilization, and discharges, as well as outcomes. Clinician and administrator perceptions of the implementation process are discussed during supervision with author AM. Service data and supervision of clinicians are used to monitor fidelity. Consistent with guidelines for program evaluation in clinical settings,Reference Corrigan, Lickey, Campion and Rashid ^{41 ,} Reference Goodman, Ogrinc and Davies ⁴² outcomes include a client-completed satisfaction scale and a clinician report on recovery progress both completed after 30 sessions. The outcome questionnaires, as well as utilization data from Phase 1 were previously published,Reference Medalia, Erlich, Soumet-Leman and Saperstein ⁴³ and the addition of data from Phase 2 supported the perceived effectiveness of the treatment and high level of satisfaction with the program.Reference Soumet-Lehman, Erlich and Medalia ⁴⁴

As of summer 2017, QI monitoring at 13 sites (the 14th site was still transitioning from being a research to a clinical site) determined that 61% of sites were keeping their fill levels at 80%. At each site an average of 4.85 patients were enrolled in the CR2PR program at any given time; the goal was for an average of 6 people to be enrolled. On average each site received 4.15 referrals per quarter, and referrals were twice as likely to come from a non-physician than physician. Attendance at sessions was good, with 70% of scheduled sessions attended.

Treatment fidelity

Implementation of CR in large systems of care requires monitoring of treatment fidelity to ensure a standardized quality of service provision and to provide concrete feedback to clinics and clinicians to remedy service issues. A chosen fidelity measure must operationalize what is done during CR as well as how it is done, reflecting adherence to an empirically based model, as well as clinician competence in enacting the treatment principles and components. A fidelity measure should also capture how CR differs along critical dimensions from other psychosocial interventions. This enables the effects of CR to be specifically evaluated in terms of its effectiveness in routine care settings. While there is at present no standard CR fidelity scale, a measure developed for use in a clinical trial was selected and adapted for CR2PR. This measure was chosen, given a number of factors that will facilitate uptake of evidence-based CR in routine practice settings. First, key elements included in the measure are informed by the available evidence as impacting cognition and generalization. Second, the items and criteria are applicable, or are adaptable, to the realities of OMH clinic settings, while keeping within empirically based guidelines. Third, the scale is feasible to implement in the context of routine quality assurance monitoring and has utility value. The components are identifiable to a rater via records review, supervision, or live observation. Items are ratable at repeated time points, across phases of treatment. The scoring system is straightforward and easy to interpret to provide feedback to CR clinicians. Finally, having been developed for a multi-site clinical trial, there is a standardized procedure for rater training, which is similar that used in training CR clinicians.

The fidelity scale used throughout OMH clinics maintains the original scale format as a checklist of 4 criteria for each of 6 items such that the item score reflects the number of criteria met (0 to 4), yielding a possible total score (0 to 24) and an overall average item score. Three items—treatment structure and session organization, composition of the treatment group, and clinician competence—measure basic design elements informed by the available evidence-base (eg, dosage and treatment intensity). Criteria were, however, modified to reflect clinic-specific parameters, such as the duration of sessions and session components, and group size. Three items—computer-based learning activities, bridging group, and the learning context—reflect empirically based, defining elements of CR. Criteria measure the extent to which there is sufficient cognitive practice with explicit links to daily functioning and rehabilitation goals in support of generalization, empirically derived techniques are used to support intrinsic motivation, and the therapeutic context is conducive to learning. The criteria reflect group content and therapeutic techniques that distinguish CR from other skills-based interventions.

Fidelity ratings were attained by review of electronic utilization reports submitted quarterly and monthly phone supervision with CR clinicians. Within the first quarter of implementation, the most frequent lapse in treatment fidelity concerned identification of rehabilitation goals, which is important for linking CR to psychiatric rehabilitation planning and treatment. This was corrected via clinician supervision, resulting in 100% treatment fidelity at each clinic for all elements by the next quarter. Subsequent lapses in fidelity arose when there were gaps in web connectivity at selected sites, or when there was staff turnover. These temporary problems were corrected with supervision and local and/or central office support. CR treatment fidelity can thus be feasibly assessed in routine care practices and inform clinical supervision to promote systematic implementation of evidence-based CR practice.