Introduction
Translation of novel research interventions into clinical applications for improved human health requires a well-prepared, competent workforce. Although numerous competency-based education resources related to clinical and translational research exist, difficulty identifying training needs and accessing resources may exacerbate challenges in conducting effective health research. A central goal of the National Center for Advancing Translational Sciences (NCATS) Clinical and Translational Sciences Awards (CTSA) program is to train the translational science workforce [1,Reference Austin2]. However, members of research teams may face organizational impediments that create inconsistencies in professional development and limit opportunities for career advancement.
In 2016, the NCATS-supported Enhancing Clinical Research Professionals’ Training and Qualifications (ECRPTQ) project [Reference Calvin-Naylor3,Reference Shanley4] sought to harmonize prior specifications of competencies across clinical research education, including the framework from the Joint Task Force (JTF) for Clinical Research [Reference Sonstein5]. Information on the processes, contributors, and deliverables from this project are detailed elsewhere [Reference Calvin-Naylor3–Reference Sonstein5]. As part of this national effort, a critical gap was identified in the availability of high-quality and easily accessible research training. This work informed the creation of the Development, Implementation, and AssessMent of Novel Training in Domain-based Competencies (DIAMOND) project. The purpose of this paper is to 1) describe the development of the DIAMOND project and outline its functionality and impact; 2) identify ways in which DIAMOND could extend workforce development to clinical research professionals (CRPs), study teams, education administrators, and affiliate organizations; 3) discuss the results of a small pilot of an ePortfolio; and 4) share lessons learned in the development of this federated resource.
Method
Development of the DIAMOND Portal
Four CTSA hub institutions (the University of Michigan, the Ohio State University, University of Rochester, and Tufts Clinical and Translational Science Institute) collaborated to develop and launch DIAMOND. Findings from ECRPTQ Phase I, which aimed to standardize Good Clinical Practice (GCP) training for CRPs and investigators [Reference Shanley4], were used to inform the subsequent phase II work. Phase II aimed to apply study team skillsets to clinical trial competency domains, and collect sharable competency-based educational materials [Reference Calvin-Naylor3]. In building upon that work, the purpose of the DIAMOND project is to provide an open, federated learning space that brings together resources from across the CTSA consortium and allows for widespread access to and dissemination of training and assessment materials.
DIAMOND (https://diamondportal.org/) includes a digital portal to collect and catalog publicly available educational resources developed for the clinical and translational workforce [DIAMOND is also accessible from the Center for Leading Innovation and Collaboration website (https://clic-ctsa.org/education-careers)]. The portal includes collections of career development curriculum, instruction, and assessments. The educational resources are centered on eight competency domains identified by the JTF for Clinical Trial Competency (Table 1) [Reference Sonstein5]. DIAMOND promotes uniformity of training by structuring its catalog around this framework and encourages contributors to describe their resources using its elements. Likewise, educators and administrators may search for and select resources based on this information. This structure was intentionally selected to provide an easy way for users to identify areas of need and locate appropriate materials.
Table 1. Joint Task Force competency domains
Source: Joint task force on clinical trial competency
The DIAMOND portal was created using the Digital Commons repository platform developed by BE Press™ (https://www.bepress.com/), which provides academic institutions and libraries with cloud-based, digital repositories to host materials. During development of the portal, four essential functions were identified: 1) provide a simple process for contributing to the portal, while respecting intellectual property rights; 2) produce a comprehensive database of available resources organized by competency domain; 3) enable users to locate training and assessment collections independently; and 4) provide links to the Portfolium™ DIAMOND ePortfolio, a convenient and intuitive platform for CRPs to document professional development.
Because DIAMOND is a federated platform, it is essential to recognize the possibility of barriers to access and address them by ensuring that both users and contributors alike are able to easily navigate the DIAMOND education resources. The team developed and pilot-tested intake forms for both career development curriculum and assessments to facilitate contributor submission of educational resources. Through these forms, contributors can provide important information about the content of their materials, which facilitates a users’ ability to locate them.
Using a set of standards inspired by the Quality Matters framework [6], the team identified seven critical elements to require on all intake forms: (1) a description of the learning activity; (2) keywords; (3) costs for use; (4) public availability; (5) delivery method; (6) learner type; and (7) learning level. Assessments were also differentiated by the level of career development as fundamental, intermediate, or advanced [Reference Kirkpatrick and Kirkpatrick7]. The intake forms were then pilot-tested by the four partner hubs, with additional feedback provided by expert reviewers from three DIAMOND affiliate partner institutions. As a result, both the intake forms and the process for submitting materials to the DIAMOND portal were revised based on this user feedback. The revised site was made available to additional DIAMOND affiliate partners in early September 2018 and delivered broadly to the CTSA Consortium later that month. To enable active use of the portal, the DIAMOND team also created a series of user orientation materials to help users navigate its features, including videos, written guidelines, and user agreements.
The DIAMOND portal was intentionally designed to have career development and assessment resources remain on the websites of the hosting institutions, rather than being uploaded directly to the portal site. Both users and contributors were provided with an agreement explaining how content creators continue to retain all rights to ownership for the resources they upload, as well as all responsibility for the maintenance and upkeep of their links, materials, and web pages. Users can view a profile of each resource on the portal and follow a link to the host institution’s website. Users may also contact a contributor directly to request more information about a resource or request permission to access materials subject to restricted use.
This design is critical to sustainability, as it distributes the work of maintaining the portal across the community of contributors. The intention of DIAMOND is not to rate the quality of materials or rank them on various criteria, but only to catalog existing training and assessment materials. Submissions are reviewed for appropriateness as trainings or assessments only; users ultimately make decisions about the suitability of each item. Metrics on downloads are available for each training.
Development of an ePortfolio
The DIAMOND project also pilot-tested an ePortfolio function provided through PortfoliumTM. The portfolios were designed to encourage workforce development for CRPs. Similar to the DIAMOND portal, the ePortfolio is also structured around the eight JTF competency domains. Self-assessment tools provided by DIAMOND allowed users to identify critical research skills and provided a platform to showcase evidence of acquired experience, training, and education. The DIAMOND team created a video and user guide providing step-by-step instructions, a self-assessment tool, a suggested rubric, and exemplar ePortfolios to ensure CRPs who used this feature maximized their advantage. An educational workshop was also provided to orient CRPs to the ePortfolio.
Evaluation
The DIAMOND portal is continually monitored, with detailed information about user experience collected in several ways. First, detailed analytics related to the portal were collected monthly from Google Analytics, Heap, Portfolium, and Digital Commons. Second, a pop-up survey was installed on the DIAMOND portal in April 2019 to assess overall user experience. The survey asked users about the reason for their visit and whether they were able to achieve their purpose for coming to the site. Results of this survey are continually compiled quarterly. Third, in January 2020, a detailed digital follow-up survey was administered to CRPs at each of the four partner hubs to determine how often they accessed resources in the DIAMOND portal (see Table 2). Finally, registrants who initiated an ePortfolio were invited to participate in brief virtual qualitative interviews to examine the value of DIAMOND and the ePortfolio for this population. The interviews were transcribed and coded by two trained raters, with the resultant data analyzed by members of the DIAMOND team and subject matter experts from the University of Michigan’s Center for Education Design, Evaluation and Research.
Table 2. Survey questions included in the follow-up survey
Results
Usage and Reach of DIAMOND
Since its launch in September 2018, the DIAMOND portal has received more than 275 submissions from 29 contributing organizations, including CTSA institutions and academic medical centers, professional associations, as well as data analysis and marketing organizations. The majority (90%) of the submitted career development resources are available free of charge to any user, with only 10% requiring a fee to access. More than 75% are fully digital online programs, while another 12% are in-person-only events occurring at the contributing institution. Nearly 75% of the trainings currently on the portal are designed as introductory training programs targeted at a “fundamental” learning level.
Training and assessment items are represented in all eight of the competency domains that make up the structure of the DIAMOND portal, with many items identified as touching on more than one domain (see Table 3). The most commonly indicated domains are Study and Site Management, Clinical Trials Operations (GCP), and Scientific Concepts and Research Design.
Table 3. Indications by contributors of competency domains targeted
In addition to career development materials, the portal has also received 15 standalone assessment submissions, with another 41 assessment instruments identified as embedded within the instructional materials they accompanied. Nearly two-thirds (26) of these assessments included measures of participant experience—such as engagement and satisfaction—and just over a one-third (15) include measures of specific learning outcomes. The DIAMOND team also participated in creating and testing a self-assessment aimed at measuring self-perceived self-efficacy in the competency domains called the Competency Index for Clinical Research Professionals [Reference Hornung8,Reference Hornung9]. This work is also available as an assessment in the DIAMOND portal.
Since its release in September 2018, DIAMOND has received over 5,700 visits, with new visits to the site making up 80% of visits each month. Resources in the portal have been accessed by users not only in the United States, but also by users in 33 countries across the globe. The site records an average of 400 active sessions each month with an average of four pages being viewed per session. Visitors have initiated 7,395 metadata page hits and have made 1,310 downloads of resources from the portal.
Results of the DIAMOND Portal Evaluation
A small proportion of users responded to the pop-up survey embedded within the DIAMOND portal (N = 57). Of these respondents, just over half (n = 29, 51%) indicated they were currently working as a CRP. The primary reasons users visited the DIAMOND portal were: to search for training/assessments on specific topics or with specific goals in mind (n = 22, 39%); explore the site generally (n = 13, 23%); because of a recommendation (n = 7, 12%); or for personal career development (n = 6, 10%). Other reasons (n = 9, 16%) included searching for specific resources, investigating DIAMOND’s potential as a dissemination platform, doing research, or to indicate a problem with the site. Over half of the respondents (n = 31, 54%) indicated they achieved the purpose for their visit. The average satisfaction rating (n = 22) for the portal was 7.7 on a scale of 1 (very unsatisfied) to 10 (very satisfied).
Forty-two CRPs from across the four partner hubs completed a follow-up survey. Of those, 26% had accessed the DIAMOND portal and 21% had accessed the ePortolio at any point since September 2018. This group was very active in their professional development, with 60% participating in training related to clinical or translational research (including earning continuing education credits), 19% completing an assessment of their clinical or translational research skills or knowledge, 22% earning one or more new academic degrees or professional certificates, and 52% attending one or more professional conferences.
Results of the ePortfolio Evaluation
DIAMOND users who initiated an ePortfolio (n = 57) were invited by email to participate in 15–30 minute online qualitative interviews to examine the value of DIAMOND and the ePortfolio for this population. The interviews (n = 5) were held in March 2019 and were transcribed and coded by two trained raters. Distinct themes emerged from the interviews in three topic areas: 1) users’ motivation, 2) the potential benefits of the ePortfolio, and 3) helpful features of the ePortfolio platform.
Motivation to use the ePortfolio included the need to prepare for a new position at work (“I wanted to prepare a little bit better for that role which is why I was really interested in the e-portfolio”), and an interest in identifying available resources (“I was intrigued by the DIAMOND-specific potential for it. I just wanted to see what was available and what it was all about”). Regarding benefits of the ePortfolio, users indicated that the platform helped remind them to continue to learn (“I think it serves a purpose of constant improvement….it reminds you that there’s always more room to grow and more room to learn”), as well as increase their knowledge in specific content areas (“If I need to make myself more well-rounded and explore these other topics…or ‘look I’m really lacking in this area. I need to beef up my knowledge over here.’ I think just having the breadth out there would be good”).
Finally, interviewees indicated that one of features of the DIAMOND portal and ePortfolio that they appreciated was its accessibility (“…being able to virtually access training materials I think will be a little more helpful for me… having them accessible in that way will allow me to make better use of them and more frequent use of them, versus kind of a one-time training on an annual basis”). The use of the JTF competency framework was also noted as a feature that enabled users to access resources that were relevant to their professional training needs. Others felt DIAMOND could help them form research networks through collaborations (“just being able to collaborate with other researchers at other establishments and being able to look at best practices”), and identifying and connecting with mentors (“…in kind of a research working world I think it’s really important to have a mentor-mentee relationship so I think it would allow me to build a network in that way and gain more mentors”).
Discussion
Our work demonstrates the reach and impact of the DIAMOND portal. Hundreds of training and assessment resources spanning the entire JTF competency framework have been submitted to the portal since its inception. The relevance and quality of these resources have established DIAMOND as a distinctive resource of training and assessment opportunities for the CTSA Consortium. Evaluations of users’ activity, experience, and professional development outcomes suggest that many are actively engaged, that most are satisfied with their experience, and that, from user perspectives, the DIAMOND portal has the potential to facilitate professional development in meaningful and enduring ways.
We have identified four categories of applications for this resource based on our work developing the DIAMOND portal and evaluating its impact. First, DIAMOND can be included as a valuable resource in enhancing career professional development for the clinical research workforce. Through its easy-to-use format, users can search for and identify training materials for their own professional and career development that may not be readily available at their home institutions. Study team members can use the portal to identify training that may be needed for a particular study, or what types of training activities (within particular competency domains) would most benefit their career advancement. In addition, self-assessments available through the portal can be used by members of research teams to evaluate current skill levels and identify areas for further training and development.
Second, resources from the DIAMOND portal can be used to design targeted, just-in-time training plans for study teams. For example, study team members could use resources from the portal to seek guidance on study design and protocol development, locate resources related to similar research studies, or identify training materials related to particular methods, techniques, and practices. Sharing resources across the CTSA Consortium will help build research capacity and contribute to the conduct of quality health research studies.
Third, education administrators can use the DIAMOND portal to find, create, or refine instructional and career advancement materials. Faculty and staff members in translational science programs are often asked to create courses for study teams or investigators. Resources from DIAMOND can be used as a tool for learner engagement across domains, and to find new tools to develop and supplement training course curriculum. Assessments from DIAMOND can also be used to identify programmatic gaps in training and provide new ways to incorporate assessment into existing training programs. The DIAMOND team has developed guidelines to provide practical suggestions for how to effectively integrate subjective and objective assessments of clinical and translational research knowledge and skills into the evaluations of CRP career development programs [Reference Samuels10].
Fourth, CTSA hubs that contribute materials to the DIAMOND portal will be able to broadly disseminate their resources to institutions and health organizations that may not have ready access to training materials, including Institutional Development Award for Clinical Translational Research sites, hub affiliates, and community partners. In this way, CTSA hubs could help other institutions locate and identify resources, administer and interpret baseline assessment instruments, and learn how to adapt training resources for their organizations. Used in this manner, the DIAMOND portal may facilitate collaboration between large academic medical research centers and affiliated health organizations in their local area or region, or enable universities to extend the reach of their training resources to national or international networks of investigators or other professionals.
Lessons Learned
While DIAMOND has been successful in achieving its aims, three important lessons have been learned that are relevant to promoting its growth and sustainability. First, access to career development programs created by CTSA institutions may be limited because of individual institutional policies or procedures, which inhibits sharing. Career development programs created for a specific institutional context can have content that is useful to others, especially when they address specialized topics and innovative methods. Therefore, we strongly encourage institutions to make novel educational content available to colleagues outside their institution through the DIAMOND portal, as it will expand available resources for all, and demonstrate collaboration in clinical translational sciences education and career development.
Second, significant barriers remain to the widespread adoption of ePortfolios by CRPs. Prior research suggests that implementation of ePortfolios requires addressing operational difficulties [Reference Andre11], logistical barriers [Reference Luera, Brunvand and Marra12], and personal attitudes [Reference Ahmed and Ward13,Reference Haggerty and Thompson14] regarding their use. The small sampling of participants for the qualitative evaluation of this ePortfolio revealed that users sought better integration between the DIAMOND platform and ePortfolio, as well as greater certainty that documenting their development would be valuable for hiring or promotion decisions by future or current employers [Reference Woodley, Fernstrom and Sims15]. Users also wanted to use the ePortfolio as a mechanism for building professional networks. Obtaining broad support of clinical and translational faculty and administration to incorporate an ePortfolio into hiring and promotion decisions may be a necessary prerequisite before an ePortfolio program for CRPs can be successfully implemented. To do this, researchers need to be able to explain the merits of ePortfolios to both CRPs and their supervisors and demonstrate how ePortfolios improve upon other methods used to document training. Ideally, ePortfolios should be incorporated into the process of professional evaluation for the purpose of advancement and promotion.
Third, our work with the DIAMOND portal has shown that having a sufficient number of validated assessments for clinical and translational research skills remains a significant gap in the field. A recent review of instruments for assessing research-based competencies for investigators found only a small number of available instruments, with no consistent competency framework in use among them. Further, only limited validation has been performed on these instruments to date [Reference Ianni16]. The growing collection of assessment instruments on the DIAMOND portal can help rectify this issue, as more users integrate them with professional development programs.
Limitations
Participation rates were low for all three forms of evaluation used for this research, which raises the potential for selection bias. We recognize that this may limit the generalizability of the findings, but we hope these findings will still be informative to the translational research community. Future research should consider identifying strategies that could increase participation rates to ensure that the sample is more representative of the population.
Conclusion
The continued utility and sustainability of the DIAMOND research community requires the continual recruitment of new contributors and users of the portal. DIAMOND can best demonstrate its value for the clinical research workforce by encouraging use of the portal for the purposes outlined in the use cases presented above, by expanding the collection, and by increasing collaborations made between institutions and organizations.
Institutions are invited to be a part of this growing collaboration, which is free to all visitors. On the portal website (https://diamondportal.org) visitors can learn how to become: 1) a user, 2) a contributor, and 3) a partner in the CRP career development community. Institutions and organizations are also encouraged to include links and references to DIAMOND on their clinical research websites and other communication channels, to discuss and recommend DIAMOND in their outreach activities, and to seek input and educational programs from the DIAMOND hubs and existing partner institutions. These efforts will help raise awareness of the DIAMOND portal across the clinical and translational research community.
Acknowledgments
Supported by NIH U01TR002013 (University of Michigan); NIH UL1TR002733 (The Ohio State University); NIH UL1TR001430 (University of Rochester); NIH UL1TR002544 (Tufts University).
Disclosures
There are no conflicts of interest.
Open access
