Introduction
A digital badge is a visual representation of a validated learned skill or competency [Reference Peisachovich, Dubrowski, Da Silva, Kapralos, Klein and Rahmanov1]. A digital badge focuses on one distinct topic as opposed to an academic course which may cover several topics and skills addressed over a semester [Reference Abramovich, Schunn and Higashi2]. The competency-based component of a digital badge allows for opportunities to earn multiple badges that can be taken to increase a competency level sequentially to form a micro-credential.
Traditionally a mainstay of scouting and the military, the modern form of badging can be presented in a digital format in email signature lines and LinkedIn profiles. The concept of digital badging has recently become of greater interest among institutions of higher education [Reference Wilson, Gasell, Ozyer, Scrogan, Ifenthaler and Mah3]. Badging has the potential to be an effective tool for recruitment into academia by serving as a stepping-stone to more comprehensive educational programs offered by the institution backing the badge. For this reason, academia has been increasingly offering digital badges alongside more traditional certificates and degrees. For learners, this trend of digital badges in academia can open the doors to a more focused learning platform since users can earn a university-backed credential without the traditional time commitment, financial investment, or time-related challenges [Reference Ralston4]. Users in this format have the freedom to pick and choose a badging program that matches their educational needs and aligns with their professional goals. They are not required to enroll in the university or take any additional courses. The fact that digital badges can be acquired at an exponentially lower cost option to standard course credits opens doors to users unable to earn standard degrees.
Digital badges are not foreign to medical or health-related education even though barriers to acquiring the training and education needed for a health-related workforce can be challenging. Access to medical or health-related programs and degrees is not available at every institution of higher education. Needed competencies for niche health-related workforce roles like Clinical Research Coordinators can be difficult to acquire [Reference Noyes, Johnson and Carbonneau5]. Competencies such as subject recruitment, enrollment, and consenting are common to this workforce. Administrative competencies including preparing for audits, ensuring regulatory document compliance, reporting adverse events, and maintaining research standards are just a few additional competencies that make the CRC’s role unique to a clinical research team. The fact that digital badges can verify specific skill sets to employers, allows applicants to present the badge as evidence of CRC job-related competencies [Reference Noyes, Johnson and Carbonneau5]. Badging represents a clear, and non-disputable documentation of skills and expert knowledge.
The New Jersey Alliance for Clinical Translational Science (NJ ACTS), a National Center for the Advancement of Clinical and Translational Science’s CTSA hub since 2019, established a Workforce Development core to address barriers and challenges to training and educating clinical research professionals. To provide the evidence-based competencies needed to be a CRC, NJ ACTS designed and developed a Clinical Research Coordinator Digital Badge (CRC Badge). The NJ ACTS CRC Badge was created to build clinical research-related skills to support learners interested in entering or advancing in the role of the CRC.
The purpose of this Special Communication is to describe the development of a digital badge designed to introduce individuals to the CRC position.
Methods
Content development
Seven clinical research professional experts from Rutgers Health with a combined 132 years of experience in clinical and translational science at an academic medical center (AMC) were assembled to identify key competencies needed to run trials. These individuals included: (1) Executive Director of the Clinical Trials Office; (2) Workforce Development Core Leads; (3) Manager of Quality Assurance and Quality Control at NJ ACTS; (4) Associate Director of Clinical Trials Administration at the Cancer Institute of New Jersey; (5) Nurse Manager for Clinical Research at Robert Wood Johnson Medical School; and (6) Operations Manager at the Environmental and Occupational Health and Science Institute.
Incorporating the Delphi Method for collecting perspectives, this panel of experts reviewed the Joint Taskforce for Clinical Trial Competencies (JTF) as the foundation to identify relevant knowledge domains for a Level 1 CRC Position. The group chose 6 of the 8 domains; (1) Scientific Concepts and Research Design, (2) Ethical and Participant Safety Considerations, (3) Investigational Products Development and Regulation, (4) Clinical Study Operations, (5) Study and Site Management, and (6) Data Management and Informatics [Reference Sonstein and Jones6]. The JTF domains relating to Leadership & Professionalism and Communications & Teamwork were excluded since it was felt that these competencies would be more difficult to assess in the proposed badge format. Additionally, 2 JTF domains relating to clinical study operations and site management were merged since these competencies included tasks that may occur simultaneously at the site and with the sponsor. The experts then listed competencies needed by a CRC under each domain that matched an entry-level job role at Rutgers. They identified 5–9 foundational or level 1 competencies per domain that were relevant to this AMC and then proceeded to write course content directed toward teaching these competencies (Table 1).
Table 1. Module competencies
Badging course design
The badging course included 5 educational modules housed within Canvas, our learning management system, each containing video lectures, handouts, a discussion forum, and a corresponding 25-question quiz. The discussion forum served as a repository for questions on content or logistics and was monitored by the team’s project manager.
Pilot testing
To assess the badging course design, implementation, and presentation level (level of difficulty), participants were recruited for pilot testing in 2 phases. Pilot 1 was conducted in July 2022 to review for typographical and content errors, course performance and navigation, as well as overall quality. Pilot 2 was conducted in January 2023 and included updates and recommendations from Pilot 1. The participants were required to complete the modules asynchronously but sequentially and could not progress to the next module until the quiz for that module was completed with a score of 90%. If a participant failed a quiz after three attempts, the plan was to remove the individual from the badging course and enroll them in a future offering to start over.
Data collection points from the pilot testing process included assessing: (1) course difficulty level - based on quiz scores and the questions’ Difficulty Index; and (2) completeness of course competencies from a participant exit survey.
Results
Nine participants were originally referred to Pilot 1 by principal investigators, clinical research administrators, and senior management at a clinical trials solution company that provides staffing solutions for Rutgers University clinical research units. Seven of the 9 testers completed the badging course in its entirety. One participant dropped out before beginning the module. One additional individual dropped out before completing the last module due to personal time commitments. Participants for Phase 2 were recruited through advertisements within the NJ ACTS newsletter and distribution of email flyers to marketing and communications individuals within the medical and health-related schools. Twenty-three participants enrolled in the badging course for Phase 2. However, of the 23 volunteers, 11 did not complete the course, resulting in a total of 19 participants for the pilot phase 1 and 2. Of these 19 volunteers, 4 were male and 15 were female. Eight of the individuals identified as white and another 8 identified as Asian. There were 2 Black/African Americans and 1 individual who identified as Native Hawaiian or Other Pacific Islander. They ranged from no degree (undergraduate students) to having earned a terminal degree and with varying years of experience as a CRC (Fig. 1).
Figure 1. Participant years in the field by age and race.
Participants’ skill level
To determine if the badging course was geared towards foundational versus advanced CRCs, participants with expertise along the continuum between no experience and those with advanced skills had to be recruited. Since participants who volunteered in phase 1 were invited testers, this approach was reserved for phase 2 volunteers who self-selected to participate in this pilot project. Therefore, participants in phase 2 were asked to report their level of expertise in the following areas: data collection and management, enrollment and recruitment, and regulatory activities. Three participants had “no previous experience” in these areas or possessed a mix of “no experience” in some categories and “fundamental expertise” in others. Four participants reported being skilled and advanced in the same areas. The remaining individuals had a combination of fundamental, skilled, and advanced expertise. Figure 1 presents the number of years as a CRC for participants in both phases by age and race.
Assessing quiz questions
Since the badge was geared towards individuals with limited experience it was necessary to determine whether the questions were also constructed on a foundational level. Therefore, an item analysis for each quiz question was performed through Canvas to determine the “difficulty index” (DI). The “difficulty index” refers to the percent of participants who scored correctly on an item [Reference Johari, Sahari and Wahab7]. This score ranged from 0.0 to 1.0 with the more difficult questions scoring low, and easy questions scoring higher. The DI measures question difficulty but also is used to identify questions that were poorly written. Based on their DI score, questions were classified into four categories, “easy” (0.9–1.0), “neutral” (0.6–0.89), “Difficult” (0.3–0.59), and “Very Difficult” (< 0.29), and percentages of each question type was determined for all the modules [Reference Johari, Sahari and Wahab7] (Fig. 2). Out of 125 questions across all modules, 47% had a DI score that placed them in the “easy” range and 42% tested in the “neutral” category.
Figure 2. Question difficulty.
The Discrimination Index (D) is another metric used to assess multiple-choice questions. This index determines how well a question “discriminates” between high-performing participants and lower-performing participants [Reference Kocdar, Karadag and Sahin8]. Questions in which high-performing participants answer correctly and lower-performing participants answer incorrectly would have a high “D.” But if the opposite were true and a high-performing group answered incorrectly but the lower-performing students on average answered correctly, the value for “D” would be a negative number. A highly discriminatory question is considered + 0.25 or above. A score of 0.0 indicates that just as many higher-performing students as lower-performing students answered correctly. A negative number indicates that the higher-performing group scored lower than the lower-performing group on an item and should be rewritten. Out of 125 quiz questions in our badging course, four questions with D values of −0.01, −0.23, −0.01, and −0.03, had to be rewritten.
Survey results
The participants successful in earning the badge (n = 19) completed a survey providing feedback on course mechanics, content accuracy, question clarity, time to completion, competencies met, and whether the content was engaging. Except for the open-ended questions, all questions were scored using a Likert scale: Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree or “Yes” or “No.” See Table 2 for survey results.
Table 2. Exit survey questions
Badging course non-completers
All Pilot 1 participants were allotted 4 weeks to earn the badge. However, on the recommendation from Pilot 1, Pilot 2 participants were granted an additional 2 weeks. Despite the increase in time, 48% (n = 11) of participants either did not start the course or were non-completers. The main reason for not starting or not completing the course was the time commitment.
Discussion
Badges have transcended history from tangible symbols used in the military, sports, and entertainment to digital badges used in business and now academia. However, no matter whether they are physical or digital symbols, they represent motivation, acquired learning, and belonging to a specialized group.
Badges also provide alternative pathways to achievement when a degree may be unattainable for certain population groups. Born from the pandemic, new learners desire a more granular focus as well as an affordable alternative to higher education and a compact program that can be completed at their own pace [Reference Shapiro, Ryu, Huie and Liu9]. Badges are unlikely to eliminate the need for degree offerings but may be a gateway into a program or job by offering foundational knowledge [Reference Onish, Karimi and Hata10,Reference Desmarchelier and Cary11]. Hence, that was the goal behind developing this CRC Level 1 Badge.
Critical to this project was to evaluate whether the badge was presented on a foundational level. To achieve this goal, perspectives from advanced CRCs and those naïve to the role were needed for pilot testing. Having testers with varying experiences ranging from “no experience” to “more than 10 years” improved the accuracy of the Discrimination and Difficulty Indices. If all volunteers were at the advanced level, the item analysis would not discriminate between easy and more difficult questions.
Although the volunteers were never asked to specifically identify the content as either foundational or advanced (which would be an opinion), the Difficulty Index indicated that the majority of quiz questions tested within the “easy” or “neutral” range inferring that the participants understood the material including those with limited exposure to clinical research. Experienced CRCs provided feedback on content accuracy and whether the course covered the competencies, while those individuals with either “No experience” or “Foundational expertise” commented on the course and test question clarity. Feedback obtained from the volunteers indicated that the content was clearly presented and accurate, the course was easy to navigate, and all course competencies were met, plus the item analysis of quiz questions indicated that the badge was designed at a foundational level.
When this project was conceived, the goal was to create a vehicle to introduce individuals from diverse health professions or science fields to clinical research by the creation of a digital badge providing foundational education. Since the initial pilot groups described above, we have conducted three additional badging courses and awarded a total of 47 digital badges representing a 54% completion rate. Further courses have been scheduled through 2024, with some already at capacity with waiting lists. Additionally, the badge will be piloted by first-year medical students and individuals in the MD/PhD program to determine if some of the modules would satisfy new competencies in medical education for clinical research.
The process for conducting the badging course is continuously undergoing improvements. To address the non-completion rate participants are now being charged $35.00 of which $30.00 is refundable upon completion of the badge. Returned funding has been a motivating factor toward improving completion rates. Additionally, the time commitment to earn the badge is thoroughly explained to everyone who inquires about the course.
In addition to process improvements, evaluation is ongoing to establish the usefulness of the badge in facilitating the onboarding process of new hires at Rutgers Cancer Institute of New Jersey. Participants will be monitored to determine if the badge translates to improved job performance. All participants will be surveyed at 6- and 12-months post course completion to retrospectively query if they thought earning the badge was valuable. Additionally, their employer will provide feedback comparing the performance of new hires who earned the badge, with those who did not benefit by enrolling in the course.
Conclusion
We described the development of a digital badge to introduce the foundational knowledge necessary to become a clinical research coordinator at an AMC. Our team of experts in clinical research outlined level one competencies associated with the entry-level position and then subsequently developed an online, asynchronous, five-module badging course addressing those competencies. The badge then underwent 2 pilot testing phases to identify any content or clarity issues. The Difficulty Index and Discrimination Index, plus participants feedback, acknowledged the desired competency level. Research is ongoing to determine the value of the badge on recruitment into the field and on job performance. While the team acknowledges that the concepts covered in the course are not comprehensive enough to enable someone new to the CRC role to immediately practice independently, it should prove to be a useful tool to attract individuals to the field of clinical research and to shorten the onboarding process.
Acknowledgments
The authors wish to thank the following members of the team for their contributions to the badge: Kathy Black, PhD, Clinical Operations Manager, Rutgers Environmental and Occupational Health Sciences Institute; Anthony Gonzalez, MS, Manager, Quality Assurance & Quality Control Rutgers Institute for Translational Medicine & Science; Lisa Palladino Kim, MS, Program Director MS in Clinical Research Management, Rutgers University School of Health Professions; Nancy Reilly RN, MS, CCRC, CHRC, Executive Director Clinical Trials Office Rutgers Health.
Author contributions
Drs. Barbara DeMarco and Barbara Tafuto contributed to the concept, analysis of data, and drafting of the manuscript. Yasheca Ebanks contributed to the editing and data collection.
Funding statement
Research reported in this publication was supported by the National Center for Advancing Translational Sciences, a component of the National Institute of Health under Award Number UL1TR003017. The content is solely the responsibility of the authors.
Competing interests
None.
Open access
