Target Audience and Online Format

This course was offered through the University of Pittsburgh’s Institute for Clinical Research and Education (ICRE) within the Clinical and Translational Science Institute (CTSI), and was intended for students, trainees, and faculty from across the University seeking an introduction to IS, particularly those interested in implementation within large health systems. The course was advertised through the Institute of Clinical Research Education (ICRE) website and discussed during meetings of the University of Pittsburgh’s Dissemination and Implementation Science Collaborative (Pitt DISC). Participants registered in advance for the course, which was held over a 4-week period in March–April 2021 and allowed registration of up to 15 participants. Though the class was intended to be delivered in-person, pandemic accommodations necessitated pivoting to an online format.

The course was taught synchronously online and was recorded, such that students could watch the course recording asynchronously if they could not attend. The course was taught as a mixture of didactic lectures and interactive sessions. Students could pose questions in the chat or during discussions during and after each lecture. Students were encouraged to participate with direct questions asked during the lectures. As students worked through their own slides and projects, instructors (n = 3) and other students provided direct feedback in breakout sessions (using breakout rooms).

Educational Methods and Curricular Program

The principles and learning objectives for this course were intended to develop core compentencies in IS and to enable learners to apply these concepts and methods to their own work. Thus, to further enhance learning in a pragmatic way, students were asked to actively apply the GTI framework to an evidence-to-practice gap that was germane to their area of research. Students engaged in a concrete and sequential process of developing an IS research project, starting with defining an implementation problem (i.e., evidence-to-practice gap) and conceptualizing the project to an implementation trial and sustainment of the evidence-based practice.

GTI-Teach Development

GTI-Teach was designed as a step-by-step approach for educators and students based on GTO and GTI. Table 1 compares the steps of GTO, GTI, and GTI-Teach. The steps of GTI-Teach were discussed and iterated by five content experts (SR, MC, CJ, JF, and LEA) over a 3-month period until consensus was reached on the order and content of GTI-Teach steps. GTO is a 10-step process that was previously adapted as a tool to guide healthcare settings to select and apply implementation strategies. Two fundamental changes were required to adapt GTI into a teaching tool. First, GTI is intended to be used by frontline providers, but IS students require a larger focus on the frameworks, models, and theories which underlie IS, and methods for understanding the mechanisms underlying their target behavior(s). Thus, we added GTI-Teach Step 2 – conceptualizing implementation. Conversely, the later GTI steps focus on the work of implementation and iteration, so GTI-Teach was changed to focus on understanding the general concepts and frameworks underlying implementation evaluation and sustainment, two major areas of IS that are also critical to successful implementation. These areas are covered in the GTI-Teach step 6, evaluate implementation, and 7, sustain implementation. GTI-Teach was thus conceived as a seven-step process (Fig. 1). These “steps” were used to develop the class syllabus. A templated slide deck was developed to guide learners through the GTI-Teach steps (Supplemental File 1) as a way of developing an IS project that applied to their area of study or discipline.

Fig. 1.

GTI-Teach steps.

Table 1.

Steps of Getting to Outcomes (GTO) and its evolution to GTI and GTI-Teach

GTO, Getting to Outcomes; GTI, Getting to Implementation.

Mapping IS Competencies to the Steps of GTI-Teach

Table 2 illustrates how IS competencies align with the GTI-Teach steps. Competencies for beginner and intermediate levels were included in our mapping with the expectation that the students started the class with a diverse range of baseline knowledge. The depth with which each topic was covered was tailored to the students’ levels. For example, one student had large-scale implementation science funding, so the methods that were of interest to that student were distinct from the students with no prior experience. Five advanced skill competency topics were acknowledged but not covered in sufficient detail to develop student proficiency (e.g., de-implementation, tailoring implementation strategies).

Table 2.

GTI-Teach and IS competencies

IS, implementation science; GTI, Getting to Implementation; EBP, Evidence-based practice; Competency is mapped to skill level such that (B), Beginner to IS; (I), Intermediate; and (A), Advanced.

Embedding Health Equity and Human-Centered Design Throughout the Course

The planning group decided a priori to emphasize health equity and human-centered design, based on emerging literature regarding their importance in IS and the expertise of the instructors [Reference Yousefi Nooraie, Kwan and Cohn7,Reference Beres, Simbeza and Holmes26–Reference Woodward, Matthieu and Uchendu31]. As such, we presented a lecture by a health equity expert early in the course and illustrated the Health Equity Implementation Framework [Reference Woodward, Matthieu, Uchendu, Rogal and Kirchner32] as an example of a key IS Framework. We then discussed equity as a crucial aspect of proactive planning and tailoring of the evidence-based practice and implementation strategies for known disparities and barriers (determinants) in priority populations and implementation contexts. Moreover, we emphasized the importance of iterative and ongoing measurement and evaluation of health equity over time as an essential implementation outcome that reflects the quality of sustainability capacity to adapt the “fit” of the evidence-based practice to dynamic context. In our course evaluation, we evaluated the extent to which students felt we had addressed equity.

Human-centered design (HCD) is a philosophy and associated set of techniques to uncover unmet patient needs, promote community and organization partners buy-in, and build trust [Reference Norman, Hamm and Schenker33,Reference Wheelock, Bechtel and Leff34]. In the first lecture, students were introduced to Mural (Tactivos, Inc), a software tool that enables visualization of ideas for collaborative problem-solving and innovation. Throughout the course, examples of HCD activities were presented as complementary methods to IS to not only engage community and organization partners in the co-production process of developing implementation interventions, but to also make course content practical and relevant to students. Specifically, HCD enhances GTI-Teach by enabling students to experience how powerful visual thinking activities can enable interdisciplinary teams to sequentially approach the design of complex innovations (e.g., evidence-based practices, implementation strategies) that are desirable, feasible, and viable for intended users.

Paralleling the steps of many IS process frameworks and methods (e.g., Implementation Mapping [Reference Fernandez, Ten Hoor and van Lieshout35]), HCD methods originally developed in engineering and business contexts are useful to harness the creativity of collaborative teams to generate ideas, insights, and options into a health care problem through divergent thinking activities (e.g., brainstorming, affinity diagrams) before using convergent thinking activities to help teams synthesize, analyze, and eliminate options to generate solutions that can be rapidly prototyped for iterative testing [Reference Brown36]. Similarly, the outcomes used by IS to assess the effectiveness of implementation processes (e.g., feasibility, acceptability, appropriateness [Reference Proctor, Silmere and Raghavan37]) are easy to map to the HCD qualities of a well-designed health care innovation for frontline users (e.g., simple, fit, efficient, learnable, satisfying) [Reference Lyon and Koerner29]. For example, students were introduced to an importance-difficulty matrix, where they could work with community partners to rate the relative importance of barriers to implementing an evidence-based practice and then the perceived difficulty of addressing these barriers as an approach to prioritizing implementation barriers (Fig. 2) [Reference Covey38].

Fig. 2.

Human-centered design example: importance-difficulty matrix used to prioritize implementation barriers. ROI, Return on investment.

Course Materials

Materials, including readings, slides, and summary documents were available on a shared drive for learners. Each step in GTI-Teach was assigned a graphical representation (Fig. 1) and the theme of the steps was carried through interactive didactic sessions, lecture slides, and templated homework and presentations. Supplemental File 1 includes a blank slide template that students completed with the aid of information learned in the class. An asynchronous video was also produced to illustrate the steps (though this was made available after the course ended due to limitations with production capacity). See: https://www.youtube.com/watch?v=y6zhwTvwFsU&t=5s

Adult Learning Principles

We adhered to adult learning principles as we built GTI-Teach and the classroom environment [Reference Collins16]. For instance, such theory posits that adults bring experiences to the classroom, so we focused on establishing a classroom environment wherein we were co-learning with students and participants were invited to provide examples of course concepts from their experiences. Students were also treated as co-facilitators and invited to provide feedback to peers. Thus, the steps were geared toward connecting their life and prior experiences to the new material. Because adult learners are goal-directed, we created clearly defined elements and demonstrated how the findings could be used in their current work. Finally, we encouraged group learning and a variety of options for learning the material, including online lectures, recorded videos of class, readings, discussions, participatory activities, and guest lectures.

Course Implementation

The course was conducted virtually on the institutional Zoom platform (Zoom Video Communications, Inc.) over a 4-week period, with two 2-hour classes per week. The course was led by three primary instructors (SR, CJ, and JF) who participated in nearly all classes and small group sessions. The 2-hour sessions were structured as four 25- to 30-minute segments. The first segment was a didactic session introducing an IS concept relevant to the corresponding GTI-Teach step for that lesson. The next segment was a small group activity of four to five students and one instructor, to help solidify the concept or discussion about the topic and how it relates to student projects. The third segment typically involved another lecture introducing the homework assignment and conveying expectations for completing the template slide. Finally, the last segment involved small groups to enable each learner to start working on their homework assignment and receive feedback from their peers and one of the instructors.

All classes concluded with a summary of what was covered, addressed any final comments or answer remaining questions, and provided expectations for the next class. Projects were expected to follow the slide template, with one to three slides per step. Classes introduced several theories, models and frameworks that are commonly employed by implementation scientists while guest lecturers presented on topics including engaging community and organization partners, health equity, and Diffusion of Innovations Theory [Reference Rogers39] as an opportunity to understand the concepts of passive versus active dissemination of innovations as well as why individuals and organizations adopt new innovations (see Supplemental File 2 for syllabus).

Course Evaluation Surveys

The evaluation included two post-course surveys: an end-of-course survey and a 6-month post-course follow-up survey. The first, an “end-of-course survey” was emailed to students on the last day of the class by administrators of the University of Pittsburgh’s Institute of Clinical Research Education (ICRE), with two subsequent reminder emails. This generic, cross-course, 14-item “end-of-course course survey” was developed by ICRE’s educational experts for participants in the University’s Clinical and Translational Science Institute (CTSI) training and educational programs. This survey asked students to rate (on 5-point Likert scales) their degree of agreement or satisfaction pertaining to nine aspects of the course including: clarity of goals and objectives; readings; course content; assignments; respect for diversity; use of inclusive materials; instructor availability; and use of Zoom sessions. Three items used a 3-item Likert-type format for students to rate course workload, pace, and assignment difficulty while two open-ended questions elicited comments on positive aspects of the course as well as suggestions for improvement.

The second survey (Supplemental File 3), a 10-item “6-month post-course follow-up survey” was developed de novo by the course instructors to ask tailored questions about the content specific to GTI-Teach. This second post-course survey was approved by the University of Pittsburgh IRB as an exempt study. Students were asked to think back upon their pre-course knowledge and rate (on 5-point Likert-type scales) knowledge about each of the GTI-Teach Steps/topics before the course and after the course. They were also asked about the usefulness of each Step/topic in their current work. Other questions requested feedback on specific aspects of the course such as the incorporation of user-centered design principles and tools, guest lecturers, slide templates, and specific ways students applied course knowledge after the course (e.g., to grant writing). Open-ended comments and suggestions were also requested on this survey using a free text box.

Survey Analysis

Summary statistics (e.g., means, medians) were used to describe the responses to the two post-course surveys. The retrospective change in knowledge questions on the 6-month, post-course follow-up survey were analyzed using paired t-tests.