Introduction
Guidelines for the clinical care of patients and standards for conducting human subject clinical research, codified as good clinical practice (GCP), are fundamentally different. Academic clinical researchers must nonetheless be knowledgeable about both. They spend years honing their clinical skills and expertise in the context of formal post-doctoral training and beyond, yet often primarily acquire much of their knowledge of human subject research experientially. Despite few opportunities for formal GCP education, clinician–researchers are held to a high regulatory standard, and when directing a clinical research study in a principal investigator (PI) role, they implicitly and explicitly assume responsibility for all aspects of the research. In particular, the study PI is held accountable by both institutional and external regulatory authorities for research study deficiencies and deviations from GCP.
In fiscal 2012, US Food and Drug Administration (FDA) inspections resulted in the issuance of 183 warning letters to PIs nationally [Reference Kleppinger1]. The FDA found actionable deficiencies related to compliance with the study protocol in 38% of the research studies, to records in 26%, to drug accountability in 9%, to consents in 7%, and to Institutional Review Board (IRB) communication in 4%. The FDA has further identified five common “mistakes” that PIs make in the context of their clinical research increasing the risk of study non-compliance [Reference Kleppinger1]. In a best-case scenario, regulatory deficiencies thus identified are readily addressed and corrected. The cost in other circumstances may however become more onerous and may include harm to study subjects, compromise of the scientific integrity of research data, and significant penalties for the PI, the institution, and the sponsor. To address this unmet educational need, the New York University School of Medicine (NYUSoM) Clinical and Translational Science Institute (CTSI) developed a focused, short-term, seminar-style, small group, educational GCP program based on principles of adult learning designed specifically for PIs. Here we describe our TransCelerate-recognized Principal INvestigator Development and Resources (PINDAR) program developed in response to an institutional need – that reflects national and international needs – for GCP instruction tailored to the PI. We believe PINDAR is easily adaptable to other research-intensive health systems.
PINDAR Program Development
PINDAR’s development, launch, and running required the skills and expertise of six individuals, working collaboratively as summarized in Table 1. In phase one of the program’s development, A and B were responsible for designing and developing PINDAR’s curriculum de novo for our faculty-level PIs. Faculty C and D joined PINDAR in phase two as described below. Two staff members (E and F) provided administrative support throughout. Faculty A is a Clinical Professor of Medicine, within the division of Translational Medicine, and is an experienced clinician, teacher, and mentor. Faculty B is Director of the NYU Langone Health Research Enterprise Training, within the Office of Science and Research (OSR), and is experienced in human subject research coordination and in educational methodologies. B is also the sole faculty instructor for the monthly 2-day “boot camp” training required of all clinical research coordinators (CRCs) at our institution [Reference Speicher2]. A and B were part of PINDAR from its start, having participated in a National Center for Advancing Translational Science (NCATS)-led CTSA-wide initiative to promote GCP, the results of which were published in 2017 [Reference Calvin-Naylor3–Reference Shanley5]. Once the need for the program was recognized, they spent 9 months working on crafting the course materials in phase one, obtaining information from our IRB and Research Regulatory Services (RRS) about the most frequently occurring protocol violations and GCP deficiencies, pooling their aggregate personal knowledge of the research enterprise at NYU, reviewing the available literature, and making use of primary source materials. The latter included the text of the ICH 6 [6,7], our institution’s IRB policy and procedures manual [8], and materials from the non-profit TransCelerate BioPharma organization [9].
Table 1. Faculty title, degree, and PINDAR programmatic role
F answers directly to B (employee supervisor).
E answers directly to A (employee supervisor).
CTSA, Clinical Translational Science Award; IRB, Institutional Review Board.
Once the course materials were crafted, the developers proceeded to phase two and reviewed and revised their content first with Faculty C and D, and then with senior-level research faculty and representatives of our institutional human research regulatory affairs, which includes IRB operations and RRS. As all of our PI researchers are required to complete the Collaborative IRB Training Initiative program (CITI Program, Miami, FL, citiprogram.org) and to take the refresher course every 3 years, the program developers made a concerted effort to not introduce duplicative information and to instead provide each PI with relevant, actionable information, relevant to the “real world” of research studies [Reference Holbein10,Reference Arango11]. The final version of the materials that make up the PINDAR program includes an original 77 slide presentation and presenter study notes, original vignette flash-card exercises, four original mini-cases (relating to the medical care of trial subjects, adequacy of resources, compliance with the protocol, investigational products, records, and reports), two original detailed case scenarios, based on actual events (Twisting the Vine and The Glass Wall), portions of the ICH-6 text [6], two published journal articles (assigned as pre-course reading) [Reference Seife12,Reference Cohen13], and a relevant publicly available FDA-warning letter [Reference O’Reilly14,Reference Shetty and Saiyed15]. Although FDA-warning letters are not specifically indicative of the quality of research performed at medical research institutions with Clinical and Translational Science Awards (CTSA), they do highlight the types of deficiencies most commonly encountered nationally. The letter selected for PINDAR participants’ review and in-person discussion provides details of GCP non-compliance for which a PI was held accountable and serves as an illustrative “real-world” occurrence. Presenter slides include text cited directly from the ICH E6 GCP guidelines when relevant. Table 2 provides an overview of the two detailed case scenarios’ teaching points. All the educational materials were designed to foster interactions, stimulate discussion and active participation using realistic scenarios, open-ended questions, small group breakouts, and adult learning principles [Reference Taylor and Hamdy16]. We also crafted eight questions for participants designed to anonymously provide direct immediate feedback to the course developers and the PINDAR faculty at the completion of each session along with a 3-month post-course follow-up six question survey to help us gauge the perceived longer-term usefulness of participation in the PINDAR program.
Table 2. Original case-based PINDAR learning scenarios
One week before the PINDAR session takes place, the two journal articles [Reference Seife12,Reference Cohen13] along with a companion editorial [Reference Steinbrook and Redberg17] and the FDA letter [18] are e-mailed to enrolled participants as pre-course reading. A booklet containing the ICH E6 text (GMP Publications, Inc., Medford, NJ) [6] is provided to each PI at the time of the course, along with systems materials for their projects (research binder tables, self-assessment quality assurance checklists specific to study type, relevant infographics and brochures, internal website links, and additional research team supports). PINDAR obtained mutual recognition for ICH E6 GCP training from TransCelerate BioPharma in 2016 [19]. All of our PIs who successfully completed the PINDAR program have met minimum criteria for ICH GCP Investigator Site Personnel Training identified by TransCelerate BioPharma as necessary to enable mutual recognition of GCP training among trial sponsors [20].
The PINDAR Program Overview and Participants
A pilot PINDAR session took place in October 2016 as a “test-run” of the course. Pilot participants attended by invitation and represented a range of experienced PIs conducting human subject research and senior IRB and RRS members. All agreed to provide detailed feedback on all aspects of the pilot session.
The final version of PINDAR that we presently use consists of five interlaced units each one with a defined set of teaching slides, educational materials, and interactive exercises, which are all designed to reinforce and illustrate the principles of GCP as they relate to the PI role in human subject research. We plan to update teaching materials as needed based on our experience using them during PINDAR and given the educational requirements of our PIs. The current course structure is schematized in Fig. 1, which also demonstrates the order in which the materials are presented.
Fig. 1. The PINDAR course comprises five units (U-1 to U-5) lasting between 30 and 90 min, such that the overall program takes place from 8:30 AM until 2:30 PM (purple arrows).
PINDAR is structured as a six-consecutive hour, instructor-led, in-person program, co-taught several times a year by Faculty C and D, two well-respected human subject researchers who hold the rank of Professor and are also NYUSoM IRB Chairs (Table 1). Both C and D reviewed and approved all the course materials prior to the October 2016 pilot session and directly contributed to the crafting of the final versions of the Twisting the Vine and The Glass Wall case scenarios. One important aspect of tailoring PINDAR to the needs of our faculty PIs includes identification of instructors PIs deem trustworthy and credible. We anticipated that a PI participating in PINDAR would be more engaged and more likely to value and internalize information communicated by an instructor perceived as both a peer and a reliable content expert. C and D were thus selected as PINDAR faculty instructors because of their hands-on human subject research expertise that is at least on par with, and most often greater than that of any of the PIs in the course, their extensive knowledge of and interest in GCP and of IRB operations as evidenced by their chairing IRB boards, and their pedagogical skill. The PINDAR faculty represent an important “go to” resource and IRB-contact once PIs have completed the program. During each session, the teaching faculty provide their e-mail and direct mobile phone contact information as there is an expectation that PIs may need to get in contact with specific questions or concerns as they proceed with their research. Enhanced PI faculty–IRB collaboration quickly emerged as a secondary benefit of the design of the PINDAR program and it has led to enhanced mutual interactions as well as several faculty expressing interest in joining the IRB.
PINDAR takes place from 8:30 AM to 2:30 PM in a seminar room equipped with audio–visual capabilities at the NYU Translational Research Building. PI participants, whose number at any session is limited to no more than 15, are seated around a conference table to facilitate interactions and dialogue. Sign-up is via the medical center’s password protected integrated learning and performance management platform and is limited to PI-level faculty from our schools of medicine, dentistry, and nursing. As detailed in our Standard Operating Procedures (SOP), PINDAR participation is strongly encouraged for faculty acting in a PI role for the first time at our institution. Those “first-time PIs” are identified when they submit their initial research protocol to the IRB for approval and are subsequently assigned a seat in the course. PINDAR also serves as a required component of corrective and preventive action plans (CAPA) following specific audit findings of non-compliance. In addition, all interested PIs are welcome to electively participate in the program. Information about the program and scheduling is disseminated regularly via institutional communications such as the monthly CTSI e-newsletter and the weekly OSR newsletter sent to all research faculty by e-mail. Each PINDAR session is co-taught by the PINDAR Faculty C and D (Table 1) and attended by the course co-directors (Faculty A and B in Table 1) who serve as facilitators. One or more IRB staff members and one more staff from the Institution’s RRS also attend PINDAR sessions and are thus immediately available as a resource to both the instructional faculty and the participants, as needed.
The PINDAR Program Launch and Progress to Date
PINDAR enrollment opened to PI participants in November 2016. Information about the program was provided to Department Chairs at various meetings, was included in the NYU-H+H CTSI monthly e-newsletter, in our institution’s monthly OSR e-newsletter, and was highlighted in our organizational intranet. Enrollment was initially challenging; two planned sessions, those of February 2017 and of May 2017, were not held as fewer than six PIs signed up, and the March 2017 session was cancelled because of a snowstorm. Once our PIs learned of the existence of the PINDAR program, both via e-mailing, invitations to first-time PIs, direct targeting of department Chairs, and by word-of-mouth from other participants, enrollment became more consistent and more robust. At the time of this writing, upcoming PINDAR sessions have reached capacity.
Between November 2016 and September 2018, 12 PINDAR sessions were held and 117 individual PIs successfully completed the program. Participants included first-time PIs, senior PIs, and PIs with intermediate experience. As the PINDAR program meets the TransCelerate GCP Training Minimum Criteria, all 117 faculty participants to date have obtained TransCelerate BioPharma-recognized GCP training certificates [20].
Although identical instructional materials were used in each of the 12 sessions, the faculty noted that each session had its own “flavor,” likely reflecting the fact that participants differed in terms of their experience of research (newer vs. more experienced), the type of human subject research they performed (investigator-initiated vs. sponsored), their particular interests and individual characteristics, and whether their participation was elective or part of a CAPA requirement. The ability of the PINDAR faculty to “think on their feet” while presenting and to engage the PI participants provided a customized experience for each group while ensuring that the didactic materials were presented uniformly across groups.
At the end of each PINDAR session, each PI participant was asked (but not required) to complete an immediate written feedback form before leaving the seminar room. That eight-question anonymous feedback survey was designed to help the program developers and faculty to best address the educational needs of our PIs related to their knowledge of GCP and to assess the PIs satisfaction with PINDAR. The eight questions included three “yes/no” questions, four 4-point Likert-type ordinal scaled questions, and one open-ended free-text question. In addition, participants received a link to a second six-question survey by e-mail at least 3 months after completing PINDAR. The follow-up survey consisted of four “yes/no” questions, one 4-point Likert-type question, and one open-ended free-text question. All 117 PIs voluntarily completed the immediate feedback survey (100% participation) and 30 PIs have responded to the second survey to date. All course surveys are archived and available to the program faculty and developers.
The responses to the survey questions (detailed below and presented in Fig. 2) indicate a high degree of satisfaction with the course and with the opportunity for participants to interact directly with PI peers, with IRB chairs, and with representatives of our IRB and our RRS groups both during the sessions and beyond. The “D” in PINDAR refers to the PI Development, essentially the classroom content, whereas the “R” refers to Resources for the PI and the team each PI oversees. The resources include materials and organizational aids, both of which are provided at the time of the course. A special resource is that the PINDAR faculty who are IRB chairs (C and D in Table 1) provide their mobile phone number to PINDAR participants so that they can be contacted directly should the need arise. They have been contacted by participants and report that phone inquiries from PINDAR-PIs have been appropriate and greatly appreciated by both the PI and the IRB chair. A similar sentiment was mentioned in the free-text survey responses.
Fig. 2. All PIs (n = 117) completed an immediate eight-question post-course survey on paper at the close of each PINDAR session.
PINDAR PI Participant Surveys
The immediate post-PINDAR feedback survey was completed by each participant anonymously on paper at the close of each session and consisted of eight questions, the last of which was in free text:
1. Have you ever performed human subject research in any role as part of funded research?
2. Have you ever performed human subject research as the sponsor investigator?
3. How well did the PINDAR course meet the intended goals presented in the course summary?
4. How relevant is the information presented in PINDAR to your current work as an investigator?
5. To what extend do you think that you will incorporate information you learned in PINDAR in your human subject research?
6. Would you recommend this program to your colleagues performing human subject research?
7. Overall, the length of the PINDAR session was?
8. Please use the space below to share your comments on any element of PINDAR you would like the course developers and the teaching faculty to know. We welcome feedback on all aspects of the program.
The survey informed us that 91% (n = 107) of PIs taking PINDAR in the first 22 months of the program had previously participated in funded human subject research and that 39% (n = 46) had performed human subject research as a sponsor–investigator. Most found the program relevant and useful in various ways. The majority (83%, n = 97) indicated that the PINDAR course met its intended goals “very well” and that 71% (n = 83) reported that the information presented in PINDAR was “very relevant” to their current work as an investigator. A comparable percentage, 75% (n = 88), thought that they would incorporate information learned during PINDAR in their human subject research “a great deal.” A great majority of participants (98%, n = 115) indicated that they would recommend PINDAR to colleagues performing human subject research and most (76%, n = 89) found the 6-hour length of the course “just about right,” with slightly less than a quarter (23%, n = 27) categorizing its length as “a bit too long” and one participant describing it as “much too long.” Over three-quarters of PIs (77%, n = 90) chose to enter free text in response to the open-ended last survey question. Those entries were favorable and tended to address aspects of the course content and format (case-based, practical, relevant, organized, interesting), the materials handed out, the format (live, interactive, discussion-based, enjoyable), the faculty (qualifications, skill), the opportunity for peer-to-peer PI interaction, and for PI interaction with IRB and RRS staff and directors.
The second follow-up survey was e-mailed to participants 3 months after they completed PINDAR. It has, to date, been completed by approximately a third of participants (n = 30) as of the time of this writing and consists of six questions:
1. Have you recommended additional training to your research staff as a consequence of PINDAR?
2. Have you provided additional resources to your research staff as a consequence of PINDAR?
3. Please rate the change in your oversight of the studies for which you are the PI since you attended PINDAR.
4. Are you now using, or are you planning to use systems and processes from PINDAR that standardize subject management, data, capture, and/or record keeping?
5. Since completing the PINDAR program, do you believe that you have increased and/or improved study-related communication with your study team members?
6. What information and feedback would you like to share about PINDAR at this point in time with the PINDAR developers and faculty?
PIs responding to the survey indicated that after completing PINDAR, about half of respondent PIs (53%, n = 16) had recommended additional training for their research staff while 67% (n = 20) had provided additional resources to their study teams. Two-thirds of PIs (60%, n = 18) reported they were providing “a bit more oversight” and a third (33%, n = 10) “much greater oversight” to their study teams since taking PINDARPlease check the phrase “reported that were providing” in the sentence “Two-thirds of PIs ....” and modify if necessary.. A majority of PIs (87%, n = 26) already were or were planning to use standardized tools from PINDAR. A similar percentage (86%, n = 25) indicated that they had increased improved study-related communication within their individual study teams. Nearly half of PIs (47%, n = 14) completed a free-text response offering additional insights and suggestions.
Discussion
Much has been reported in both the scientific literature and the popular press about research misconduct, which has often been viewed in ethical terms and through a psychological lens [Reference Cohen13,Reference Gogtay21–Reference Garmendia, Epnere and Bhansali24]. Recent attention has been focused on the characteristics of researchers who have committed research misconduct, defined as “fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results” [25]. An individualized remediation program such as The University of Washington in St. Louis’ 3-day Professionalism and Integrity in Research Program that has been attended by 39 researchers from 24 institutions has yielded promising results for those who have had their research privileges suspended [Reference DuBois26,Reference DuBois27]. DuBois and Antes thus define five dimensions of research ethics, including “compliance with regulations, statutes, and institutional policies” as ethical imperatives in the context of the responsible conduct of research [Reference DuBois and Antes28].
The range of ways the research community might address misconduct beyond implementing programs targeting researchers who have had their research privileges suspended remains the topic of intense interest [Reference Kleppinger1,Reference Holbein10,Reference Kalichman29]. Interestingly, a Cochrane review of articles published between 1990 and 2014 points out that “the evidence base relating to interventions to improve research integrity is incomplete” with “very low quality of evidence” and that “the effects of training in responsible conduct of research on reducing research misconduct are uncertain.” [Reference Marusic30]. We nonetheless chose to directly address a perceived unmet need of our PIs engaged in human subject research by designing and implementing a 6-hour “live” educational activity emphasizing and reinforcing the importance of compliance and oversight from the perspective of the PI. PINDAR was thus not designed primarily for researchers in need of remediation; its reach is broader and more inclusive, which we believe is one reason PIs completing the program have expressed a high degree of satisfaction with its relevance, approach, and resources. We geared our program, offered at no cost to participants, to PIs throughout our institution, including any interested PIs, any first-time PIs, and PIs new to our institution. In the 22 months that PINDAR has been up and running, we have met with excellent acceptance of the program and are planning to continue to offer it at approximately 6-week intervals.
Our crafting of the PINDAR program at the NYU-H+H CTSI was performed pragmatically and experientially. Accordingly, the instructional flash card vignettes, mini-cases, and case scenarios were based on real occurrences and “lessons learned,” known to the course developers and faculty, and were selected for their relevance to PIs. The survey questions we wrote and asked our PI participants to answer on paper as they left the PINDAR session were similarly intended for educational purposes, to assess participants’ satisfaction with the session, and to provide us immediate feedback. We thus consider the immediate post-course survey as an inquiry form for opinion-gathering, rather than a research study instrument (designed to address a research question), and consider the survey a quick and practical “real-life” form of communication, as is done in classrooms all over the country. The feedback survey allowed the course developers to review all the submitted responses after each session and to share relevant information with the PINDAR faculty in a timely manner and on a regular basis. Although we deliberately used a consistent, balanced, forced-choice response option for the Likert-like questions, our immediate post-course survey and the follow-up emailed survey are not validated instruments and are thus inherently subject to various forms of bias and to limitations, particularly in terms of reproducibility and external validity. We nonetheless believe that the questionnaires have been invaluable to us in making PINDAR relevant and engaging, and so include them in this descriptive report for interest and to illustrate how we proceeded in the “real-life” setting.
We believe that other CTSA sites could pro-actively adapt our approach to the specific needs of their institutions and implement GCP educational programs such as the 6-hour liveTransCelerate-recognized PINDAR for their PIs, thus expanding their PIs’ knowledge and awareness of their PI role and hopefully advancing the performance of high-quality human subject research.
Author ORCIDs
Claudia S. Plottel, https://orcid.org/0000-0001-6088-6534
Acknowledgments
This work was supported in part by the NYU CTSA grant UL1 TR001445 from the National Center for Advancing Translational Sciences, National Institutes of Health (CSP, SDK, JSH) and by NIH/NCATS UL1 TR000038 from the National Center for Research Resources, National Institutes of Health (SDK). The contents of this work are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. We thank Daniel Cobos for his assistance in distributing and collecting the PINDAR course surveys and in archiving the results, and we thank M’lis Kendricks for assistance in PowerPoint slide creation and for her expert skill in faculty registration management and course completion reporting.
Disclosures
The authors have no conflicts of interest to declare.
Open access
