Introduction
In order to advance translational science, there is a need for efforts on many fronts to identify and break down barriers to the successful conduct of translational research [Reference Austin1]. Healthcare professionals in fields who are involved in implementation but have not been systematically engaged in translational research are an underutilized resource. Healthcare professionals, such as physical and occupational therapy practitioners, typically work as clinicians within academic health centers but usually do not conduct research in that role. Nevertheless, because of their entrenchment in the clinical setting, allied healthcare professionals have the potential to play an innovative role in clinical and translational research [Reference Arena, Goldberg, Ingersoll, Larsen and Shelledy2].
A recent systematic review found that practicing allied healthcare professionals often lack the capabilities, education, and time needed to engage in translational research [Reference Smith and Johnson3]. For example, although physical therapy practitioners in training in the United States are now required to obtain a Doctor of Physical Therapy degree in order to become a licensed physical therapist [4], and many programs require students to conduct research as part of this degree [5], currently practicing allied healthcare professionals face multiple barriers to conducting research. One important strategy to overcome these barriers identified in the systematic review is to give staff educational opportunities. However, few opportunities are available for currently practicing allied health professionals to receive training that could prepare them to successfully conduct translational research.
Pioneers in filling this gap in training were a group of researchers at the University of Michigan who created the Practice-Oriented Research Training (PORT) program in 2008 to increase the research capacity of allied healthcare professionals [Reference Murphy, Kalpakjian, Mullan and Clauw6]. Program participants received first-hand experience in research training by conducting their own research project. In an evaluation of the first year of the PORT program, participants reported significant improvements in their clinical research skills. The present study builds upon this evaluation of short-term outcomes by evaluating the long-term outcomes of the PORT program, which was active between 2008 and 2018.
Previous studies examining the short- and long-term impact of similar programs designed for allied healthcare professionals [Reference Harding, Stephens, Taylor, Chu and Wilby7–Reference Janssen, Hale, Mirfin-Veitch and Harland10] found that participation was associated with several positive long-term outcomes. These included higher research self-confidence, increased enrollment in PhD programs, and submission of publications and conference presentations.
In the present study, we examined several long-term outcomes. First, as evidence that the PORT program improved the research productivity of these trainees, we collected data on the number of manuscripts accepted for publication authored by PORT participants. We also gathered data on participants’ perceptions of their professional development since their participation in the PORT program using self-report surveys and qualitative interviews.
Methods
Program design
The PORT program was designed for allied health professionals (including clinical faculty members) who had little or no prior formal training in research. Students participating in graduate programs (i.e., doctoral or master’s level) were ineligible for PORT. Applicants to the PORT program were required to have approval from their supervisor and a departmental agreement to a 50% cost share of up to $5,000 for a research project. Applications were reviewed by PORT program faculty.
Before engaging in research, PORT participants attended a series of didactic seminar sessions, which were intended to help them refine their research ideas and write a study proposal. These sessions addressed topics such as grant writing, protection of human subjects, data collection, research methods, and statistics. Over the 10-year span of the PORT program, several improvements were made. Initially, didactic seminars were presented in two distinct phases, known as Phase 1 and Phase 2 (as described in Murphy et al., 2010) [Reference Murphy, Kalpakjian, Mullan and Clauw6]. Phase 1 consisted of three free didactic sessions that were open to all University of Michigan staff and faculty. Interested participants then applied to attend Phase 2, which consisted of nine additional didactic sessions. Over time, the program directors determined that PORT worked best with fewer didactic sessions and benefitted from a flipped classroom approach [Reference Hurtubise, Hall, Sheridan and Han11,Reference Phillips and Wiesbauer12]. This required students to prepare more before didactic sessions by reviewing slide presentations; in the didactic session, the presentation was briefly reviewed with a focus on discussion and the topic’s applicability to the projects that were being concurrently developed. In 2013, Phases 1 and 2 were combined and the first three sessions were no longer open to all staff and faculty. Only participants who applied and were admitted to the PORT program could attend the eleven seminar sessions. The final structure of the program in 2018 is described in Table 1.
PORT timeline (2018)
After completing the 11 didactic sessions of the program, participants were required to submit a grant application to the PORT review committee for a proposed project. Participants began working on their grant application during the didactic phase. Between sessions, participants engaged in homework assignments that consisted of writing different sections of the grant application, which was based on elements of a National Institutes of Health (NIH) R grant application. Applications could be submitted either by individual participants or by teams of participants. A team leader was designated as principal investigator (PI), and other team members were designated as co-investigators (co-I).
The grant application included a project summary, budget, investigator biosketches, mentor biosketch, proposal narrative, and letters of support. The grant applications were reviewed by a team of PORT program faculty and postdoctoral fellows and evaluated on four criteria: research problem formulation, research methodology, scientific writing and grantsmanship, and human subjects protection. Each application was assigned a score and either awarded or denied based on that score. If awarded, participants and teams advanced to the research project stage.
In the research project stage, participants and teams worked with program mentors to carry out their research projects. Each participant received $5,000 to do this, which was typically matched by their home department. Participants were allowed a maximum of two one-hour consultations with the Michigan Institute for Clinical and Health Research (MICHR) Biostatistics unit to obtain statistical support for their projects. Most participants in the research project phase needed two years to complete the program; however, some received no-cost extensions beyond these two years to complete their projects. In order to receive the PORT program’s certificate of completion, participants were required to submit a poster abstract to an annual research conference held at MICHR or any conference of their choosing.
Termination of the PORT program
The PORT program admitted its final cohort of participants in 2018, and active research teams continued to work on their projects until their funding concluded at the end of 2020. We decided to stop offering the program for two primary reasons. First, there was a university and health system restructuring in some healthcare units, ending their ability to offer protected time for staff to conduct research. Second, after 10 years of evolution of the program to optimally support allied healthcare professionals, we recognized that we had reached a saturation point for our ability to continue to innovate learning in this area. Therefore, we focused our efforts on disseminating the program by developing robust training materials and slides for others to use (as of this writing, these materials are freely available from https://www.diamondportal.org/).
Evaluation
The present study builds upon the previous short-term evaluation of PORT [Reference Murphy, Kalpakjian, Mullan and Clauw6] by examining long-term publication, survey, and interview data. We used mixed methods to evaluate the impact of the PORT program on the ongoing engagement of clinician-trainees in research activities. We used surveys to collect data on participants’ perceptions of the program. Later, we conducted one-on-one interviews to develop a deeper understanding of the survey data. To assess long-term outcomes of the program, we collected data on PI’s and co-I’s subsequent publications from PubMed.
In April 2022 (2–12 years after participants completed their participation in the PORT program), participants who received funding for their research were surveyed to collect data on the long-term impact of the program. The survey was sent to the participant’s email address on file and was administered using the Qualtrics platform [13]. Participants were sent up to three reminders if they did not respond to the initial survey distribution.
The survey, which consisted of a mix of closed yes/no items and open-ended items, asked participants about how the PORT program impacted their career. We decided to use a yes/no response format to lower response burden to participants, given the large number of survey items. These questions were arranged into six components: knowledge and skills, grant awards and patents, publications and presentations, collaborations and networks, professional advancement, and institutional culture and practice. These components were selected based on themes identified by previous research [Reference Harding, Stephens, Taylor, Chu and Wilby7–Reference Harvey, Barker and Tynan9]. They were also asked whether they were currently involved in health research activities, their satisfaction with their professional career, and how their career has been different because of the PORT program.
Survey participants who indicated that they were interested in talking more about how their participation in the PORT program contributed to their careers were contacted about participating in a 30-minute interview. Interviews were conducted in November 2022 with eight program participants. Interviews were semi-structured and tailored to the specific participant being interviewed based on their survey responses, with the goal being to fill in gaps in the information and to get a deeper understanding of the participants’ responses to the survey. Participants were asked several questions, including why they got involved with the program, what institutional supports/services enabled them to participate in PORT, what was the most impactful aspect of PORT, how participating in PORT contributed to their career, what short-term and long-term impacts participating in PORT had on their professional work, why they would recommend participating in PORT to other researchers, the most significant barriers that other researchers would face in trying to participate in the program, and any potential improvements to PORT.
Results
Program participants
Between 2008 and 2018, 64 team applications were submitted for review. Forty-eight of these applications were funded (two declined); 16 were denied funding. Seventy-four allied healthcare professionals and eighteen faculty participated in the research phase of the PORT program, either as a PI or Co-I on a proposal (one staff member participant later became a faculty member after participating in PORT; they are counted as staff in this paper). The number of teams and individuals participating in each cohort year is shown in Table 2. Team sizes averaged two people, with a range of 1–4.
Number of team applications and funded teams by cohort year
The composition of program participants changed over time. The program was initially designed for physical therapy and occupational therapy practitioners in the University of Michigan Department of Physical Medicine and Rehabilitation [Reference Murphy, Kalpakjian, Mullan and Clauw6]. Over time, enrollment expanded to include other allied healthcare professionals (e.g., audiologists, orthotists, nurse practitioners, clinical social workers) and clinical faculty interested in the program. Clinical faculty tended to serve on teams with other clinical faculty members, rather than leading teams of staff. For this evaluation, we excluded outcomes data on clinical faculty members who participated in PORT. We chose to exclude faculty because they typically had a research career before their participation in PORT.
Participants had a variety of professional roles. The most common were physical therapist (n = 22), followed by occupational therapist (n = 15), clinical social worker (n = 12), clinical assistant professor (n = 7), clinical instructor (n = 6), and nurse practitioner (n = 3). The remaining twenty-seven participants fell into a job category with two or fewer participants.
Publications
For this analysis, we included only articles that were: 1) published after the participant started the research phase of the PORT program, 2) published by University of Michigan staff members rather than faculty, and 3) those for which the PORT participant was first author. We focused on publications where the participant was first author because it is likely that the PORT participant led the project. There were 154 papers on which staff participants were credited as a coauthor or first author. Fifteen out of 74 staff published one or more first-authored papers after participating in the PORT program, totaling 31 papers. Of these 15, eight were PI’s and seven were co-I’s. Of the 31 first-authored publications, nine (29%) had at least one additional PORT participant serving as a coauthor. Only three of these 15 participants had ever published a paper as a first author before they participated in the program, and twelve of these 15 participants published as a first author for the first time after their participation in PORT.
Publications were analyzed for impact using iCite Bibliometrics [14] and Altmetric data [15]. iCite Bibliometrics were available for all 31 post-PORT first-authored publications. These papers had a total of 263 citations, with a median of 7.0 and a maximum of 44. The average number of citations per year was 1.4 (max = 6.3, median = 1.0). The average Relative Citation Ratio was 0.76 (max = 3.0, median = 0.69). Altmetric data were available for all but one of the staff participants’ post-PORT publications. There were 22 publications with “mentions” indicating attention, with a cumulative total of 152 “mentions.” There were 148 social media mentions, two news/blog mentions, one policy document mention, and one Wikipedia mention. For the 123 publications coauthored by at least one PORT staff participant, the total number of citations was 2,440, with a median of 5.0 citations per publication.
Survey results
In 2022, the 92 PORT participants who participated in the research phase of the program were asked to complete a survey about their current engagement in health research activities, and 25 responded (Table 3).
Percentage of participants (N = 25) who reported gains as a result of participating in the PORT program
In the same survey, participants were also asked about their current engagement in health research activities, and 24 responded. Seventeen (81%) indicated that they were currently involved in clinical and translational research, including active membership in a multidisciplinary health research team, 11 (52%) were currently involved in supporting funded clinical and translational research, 12 (57%) were currently involved in project writing/re-submitting/reviewing clinical and translational research, six (29%) were currently involved in directing clinical and translational research projects, and five (24%) were currently involved in funding, conducting, or managing clinical and translational research.
Qualitative data
To better understand the meaning of our quantitative findings, we conducted interviews with several PORT participants who have successfully published as a first author. Interview transcripts from eight participants were coded by three coauthors (ES, EC, and PI) who have expertise in qualitative analysis. Using the RADaR technique [Reference Watkins16], the coders independently identified relevant passages of text that exemplified the six themes identified from the quantitative results. The coders then met to reconcile the assignment of quotes to themes. The six themes and representative quotes are shown in Table 4.
Themes and sample quotes from interview participants
These sample participant quotes demonstrate that the program had multiple positive impacts on participants. Participants reported that the PORT program helped them gain knowledge and develop skills quickly and efficiently. By enabling participants to engage in research, the PORT program changed participants’ views of institutional culture, enhanced their work with patients through evidence-based practice, and supported quality improvement in the clinical area. One participant stated that their status as a primary investigator was used as part of a departmental application to obtain Magnet recognition for their department. PORT helped participants establish relationships with mentors and develop a professional network. Participants said that PORT gave them the research background required to get jobs that they would not have qualified for had they not taken the program. Lastly, participants reported that PORT gave them the opportunity to submit manuscripts for publication, present their findings at scientific conferences, and help participants obtain grants to conduct research.
Discussion
The results of this study demonstrate that the PORT program had a considerable impact on participants. Publications data showed that participants produced more than five times as many manuscripts after participating in PORT than before they entered the program. Survey data and interviews showed that participants believed the PORT program benefitted them greatly, especially on their self-assessed knowledge and skills. Taken together, we interpret this as evidence that our program was successful in giving healthcare professionals much-needed knowledge and skills to lead their own research projects. These findings are consistent with previous studies of similar programs [Reference Harding, Stephens, Taylor, Chu and Wilby7–Reference Harvey, Barker and Tynan9].
PORT participants were typically not expected to disseminate research as part of their clinical role; however, the PORT program encouraged dissemination of findings and offered mentorship in manuscript writing as needed. This support may have helped bolster the number of publications produced by participants after completing the program. In fact, 12 out of the 15 participants who published manuscripts as first author after participating in PORT had no prior experience writing for publication. The PORT program appeared to have the right support for enabling participants to conduct and lead their own research projects, as well as disseminate their findings. In addition, several participants published with their fellow PORT team members after their participation in the program ended. Nine of the 31 first-authored publications had at least one additional PORT participant serving as a coauthor, demonstrating that this training was successful in catalyzing team science.
Limitations
While the results of the program appear impactful to this group of healthcare professionals, they must be interpreted with caution. The lack of a comparison group prevents us from being able to directly attribute every first-authored publication to the PORT program, especially those articles written well after participation in the program. Nevertheless, we inferred that all first-authored publications produced after participating in PORT could be attributed at least in part to participants’ involvement in the PORT program as they acquired relevant research skills.
The low response rate to the survey represents another limitation. As only about 30% of PORT participants responded to the survey, they may not have been a representative sample. We believe the response rate was low because we surveyed participants who may have completed their participation in PORT up to 12 years prior to the survey date. During that time, PORT participants may have retired from the university or changed jobs and left the university. While we sent the survey to the participants’ email addresses on file, not all emails were current and some bounced. In addition, those who had published as a first author were more likely to respond to the survey. Therefore, their responses may not represent the sentiment of all participants. However, while our results are highlight experiences from a subset of the entire sample, at least this subset had a positive and impactful experience in PORT that supported their professional careers.
Conclusion
These data address an important need in clinical and translational research by expanding the research workforce. We recommend academic medical centers use our PORT program materials to create similar programs to support members of their clinical workforce who are integrating research and practice into their careers. We believe that the PORT program, and programs like it, maybe a feasible alternative to graduate school for currently practicing healthcare professionals. Future research is needed to address whether this program could also be successfully exported to other populations in the medical realm, like physicians. Our results suggest that programs like PORT could give healthcare professionals the skills they need to test hypotheses informed by their real-world experience.
Acknowledgments
The authors thank Pat Walsh, MA of Milwaukee, Wisconsin for providing technical writing support, which was funded by internal discretionary funds.
Funding statement
This work was supported by the National Center for Research Resources and National Center for Advancing Translational Sciences, National Institutes of Health, through award numbers UL1RR024986 and UL1TR002240.
Competing interests
The authors have no conflicts of interest to report.
Open access
