Introduction
The management of modern-day clinical trials is an arduous process that is continuing to increase in complexity. There are many moving parts across a multidisciplinary ecosystem of stakeholders. Effective planning and implementation of a clinical trial greatly influences its success. The integrated trial management systems and electronic health records (EHRs), data interoperability, and best practices to support clinical and translational researchers play a crucial role in improving the quality and efficiency of clinical trial management and research outcomes. In particular, academic medical centers with the Clinical and Translational Science Award (CTSA) maintain an integrated research and training environment for clinical and translational sciences. CTSA hubs play a central role in their local environments where they coordinate and collaborate with multiple spokes such as affiliated hospitals, clinics, and community health centers.
Standardizing and optimizing the management of clinical trials is key to accelerating the clinical trials process, improving study quality, and ensuring the accuracy, reliability, and integrity of the results. While the adoption of clinical trial management systems (CTMSs) has advanced standardizing the work involved in managing clinical trials, CTMSs typically only address a subset of the activities involved in conducting a clinical trial. Organizational leadership needs to regularly evaluate both CTMS and non-CTMS clinical trial activities to effectively run clinical trials and take appropriate action in order to stay up to date with changes in the technology and processes that comprise the broader Clinical Trials Management Ecosystem (CTME). However, self-evaluation tools developed by the CTSA consortium only assess the extent to which an institution has installed a CTMS. No adaptable frameworks or guidance exist to evaluate and track broader organizational clinical trial development, function, or behavior.
Increasingly, leaders turn to maturity models to address such questions [Reference Carvalho, Rocha and Abreu1,Reference Aiwerioghene, Lewis and Rea2]. They use maturity models to assess their team’s/ organization’s performance in a particular domain, recognize where their current methods fall short, identify areas for improvement, set improvement goals, and monitor progress over time. Importantly, these models can guide institutional strategic investment decisions. Maturity models are instruments to define and facilitate organizational management with regard to a particular function or behavior. Maturity models have existed since at least the 1970s [Reference Nolan3]. Initially, they described “Stages of Growth” for process assessment and improvement. Maturity models are “based on the premise that people, organizations, functional areas, processes, etc., evolve through a process of development and growth toward a more advanced maturity accomplishing several stages.” [Reference Carvalho, Rocha and Abreu1] They became more formalized with the development of the Capability Maturity Model (CMM) in the 1980s [Reference Paulk, Curtis, Chrissis and Weber4]. The CMM utilizes a 5-level template, ranging from Initial (unpredictable and poorly controlled) through Optimizing (focus on continuous quality improvement) [Reference Paulk, Curtis, Chrissis and Weber4]. Under the CMM approach, the institution evaluates descriptive statements associated with each of the levels and then chooses the level that most closely describes the organization, realizing that some statements associated with a given level may not be a perfect match. This template has been adopted by many other models. When Garcia-Mireles et al. performed a systematic literature review on maturity models in 2012 across the software engineering literature, over 1,500 maturity model publications were identified [Reference García-Mireles, Ángeles Moraga and García5].
While maturity models have been developed for many heterogeneous domains, none exist for managing the components and complexity of technical, regulatory, financial, and organizational characteristics related to clinical trials. Recently, maturity models have received considerable interest in information technology (IT) for healthcare [Reference Liaw and Godinho6– Reference Zhao, Schalet, Alsalamah, Pujari and Labrique9] and research. Perhaps, the most well known is the HIMSS Electronic Medical Record Adoption Model (EMRAM) [10]. Many other maturity models have been developed, including the Healthcare Analytics Adoption Model [Reference Sanders11], the Enterprise Data Warehouse for Research (EDW4R) model [Reference Knosp, Dorr and Campion12,Reference Knosp, Craven, Dorr, Bernstam and Campion13], the research data sharing maturity model [Reference Champieux, Phuong and Dorr14], Research IT [Reference Knosp, Barnett, Anderson and Embi15], and recently, the Social Determinants of Health Maturity Model [Reference Espinoza, Sehgal and Phuong16].
Several of the above models were developed by the CTSA consortium [17]. Within individual CTSAs, maturity models are used to assess the quality and efficiency of various translational informatics capabilities and activities. Informatics leadership at each CTSA is able to determine the areas that need to be prioritized by analyzing their self-assessment results and benchmarking across CTSA hubs [Reference Espinoza, Sehgal and Phuong16]. When aggregate values across sites are evaluated, the results can be used to identify areas for multi-hub collaboration to improve quality across the consortium.
This article details the development of a CTME maturity model that can help inform research informatics, IT, and clinical trials leadership within institutions in several ways. Key uses include understanding perspectives across the organization, identifying gaps, developing best practices for CTME capabilities at their institution, generating source documents for internal discussion and planning, outlining a roadmap for future growth within the CTME domain, ultimately advancing the field of translational informatics.
Methods
As detailed below, the development of the CTME maturity model was a multi-phase consensus-based process. In this paper, we describe the development of the CTME maturity model across three phases: development, evaluation, and self-assessment (Fig. 1).
Figure 1. Process diagram illustrating the steps in the development of the Clinical Trials Management Ecosystem maturity model. CTSA = Clinical and Translational Science Award, CTM = clinical trials management, CTME = Clinical Trials Management Ecosystem, IRB = Institutional Review Board.
Development Phase
The first step toward developing a multi-axial CTME maturity model was to establish a working group of research informatics leaders from academic medical centers across the USA, especially those with CTSA or Institute for Clinical and Translational Research (ICTR) grant awards. Experts were identified through the CTSA Informatics Enterprise Committee and the American Medical Informatics Association Informatics (AMIA) Clinical Research Informatics Workgroup. A snowball sampling was employed, inviting those participating to suggest others whose experience was relevant. The working group ultimately consisted of 42 experts.
The candidate domain axes were identified based on an internal project at Northwestern University, led by Dr Firas Wehbe, which was aimed at developing a feature matrix for evaluating clinical trials management functions. The axes represented discrete parts of the clinical trials process. An online information session on maturity models was hosted for the working group members in March 2022. After an overview of maturity models, project goals, and expectations, we presented the candidate axes (n = 10) with experts from CTSA hubs. Based on working group recommendations, we modified one of the proposed axes, “Subject management” and divided it into “Subject identification and recruitment” and “Subject management” to provide granularity and specificity to the respective axes. As outlined in Table 1, 11 axes were finalized in April 2022 with expert input to construct the maturity model tool.
Table 1. Clinical Trials Management Ecosystem (CTME) maturity model axes and descriptions
A generic description of maturity levels outlined in Table 2 served as a foundation for developing maturity statements for every axis. We used the standard five levels of the CMM, which is the most common in maturity model development [Reference Wilcox, Dorr and Knosp18,19]. Study data were collected and managed using Research Electronic Data Capture (REDCap) application hosted at Northwestern University. REDCap is a secure, web-based software platform designed to support data capture for research studies [Reference Harris, Taylor, Thielke, Payne, Gonzalez and Conde20,Reference Harris, Taylor and Minor21]. We developed REDCap surveys to solicit candidate statements for each maturity level of each axis. After candidate statements were collected, they were discussed and revised in monthly virtual working group meetings. Because of the need to extensively revise the precise wording of each maturity statement, the consensus process was not possible via online tools only. Each maturity statement was discussed by the working group and refined together until consensus was achieved. This process was first applied to maturity level statements for min/max levels (level 1 and level 5) for each axis of the CTME maturity model. Next, attributes for the intermediate levels, that is, levels 2–4, were collected sequentially via REDCap and iteratively refined for all 11 axes. The statements for every level provide a detailed explanation of the stages from the initial (ad hoc) stage to the optimizing (efficient) level. The goal of this maturity model was to offer institutions a structured means to identify their current level of maturity in clinical trials management, as well as to understand gaps to address in strategic planning.
Table 2. Generic descriptions of maturity levels for Clinical Trials Management Ecosystem (CTME) maturity model
Evaluation Phase
By February 2023, maturity level statements for all axes were defined and the first version of the model was reviewed with the workgroup members (virtually via Zoom and online via REDCap) during March–April 2023 to solicit comments about the maturity model. A REDCap survey for the model’s statements was then created, and the working group members were asked to test the model by completing the survey at their respective institutions. The survey asked respondents to choose the best fit when selecting levels, acknowledging that an organization may satisfy most, but not all, of the statements regarding the best match level. They were also asked to submit feedback about the instrument itself. At this stage, the model was tested by five working group members. No modifications to the model’s design and content were suggested by any of the experts at this stage.
Self-Assessment Phase
This survey for collecting information about the level of institutional CTME maturity was distributed in August 2023 to all working group members. Twenty-three academic medical centers (four from the Northeast, four from the West, seven from the Midwest, four from the Southwest, and four from the South) completed the self-assessment for their institution between August 2023 and October 2023. Although this evaluation was coordinated by an informatician at each academic medical center, anecdotal reports from many sites indicated that the responses for specific axes were solicited from a variety of clinical trials professionals, including Institutional Review Boards, research finance, regulatory oversight, clinical research centers, research administration leadership, IT leaders, and CTMS management.
Ethical Considerations
This project was determined to be exempt by Northwestern University Institutional Review Board (IRB# STU00219631).
Results
A consensus-based approach was used to develop a multi-axial CTME maturity model comprising five advancing levels of maturity across 11 axes of study management, regulatory and audit management, financial management, investigational product management, subject identification and recruitment, subject management, data, reporting analytics & dashboard, system integration and interfaces, staff training & personnel management, and organizational maturity and culture. Maturity level statements were clustered around three conceptual components of standardization, complexity/integration, and monitoring, thus providing an informatics-centric perspective to every maturity level. For example, the first level of regulatory and audit management will typically be characterized by ad hoc, paper-based processes that lack a well-defined, well-documented, controlled, and standardized mechanism, with no established framework for measuring and monitoring performance or milestones. At level 2, some key processes are standardized. Event reporting systems such as adverse event and protocol deviation have been implemented, but response to adverse events is still largely manual. Performance metrics are measured for key areas only. At the next level, institution-wide standardized audit policies exist and there is an established electronic framework for measuring and monitoring performance. Advancement to level 4 would feature the use of standardized metrics for process management and control across the institution, consistent success activating studies on time and reaching study milestones, automatic monitoring of study milestones, though intervention on specific studies would often be ad hoc. At the top level, well-integrated systems exist that provide interactive and actionable information that can be easily accessed by the main institutional clinical research and trials office and audit reports are available for review, decision support, and follow-up.
For each level of each CTME axis, three to five statements were developed under each cluster, resulting in over 200 statements (See Supplementary material 1 for maturity statements for the 11 axes of CTME maturity model). Exemplar statements for the CTME maturity model axes have been outlined in Table 3.
Table 3. Clinical Trials Management Ecosystem (CTME) maturity model axes with exemplar statements
CTME Maturity Self-Assessment Results
Twenty-three academic medical centers representing 22 different CTSA hubs (out of 64 CTSA hubs) [22] and 1 non-CTSA academic medical center across the USA self-assessed their institutional maturity with regard to the CTME. Self-assessment representing a third of the country’s CTSA hubs, with no modifications proposed by any of the participating experts during the evaluation phase, suggests reasonable real-world applicability (see Supplementary material 2 for a summary of CTME maturity scores by institution).
Fig. 2 depicts the institutional responses to the CTME maturity model self-assessment. The average maturity across all axes and all institutions was 2.54 (SD: 0.50). With regard to the axes, the average ranged from 2.57 to 3.09. Fig. 3 shows the number of institutions endorsing a specific maturity level (i.e., self-assessment of the institution at that level) for every axis. The majority of institutions indicated a higher maturity for study management and investigational product management, while the reporting analytics & dashboard axis was the lowest scored axis. The CTME maturity across institutions ranged from 1.27 to 4.45.
Figure 2. Institutional responses (n = 23) to the CTME maturity model self-assessment. CTME = Clinical Trials Management Ecosystem.
Figure 3. Figure depicting the maturity of institutions by axis with regard to the CTME. CTME = Clinical Trials Management Ecosystem.
Discussion
The CTME maturity model defines five levels of evolutionary growth across 11 axes with the levels ranging from level 1, characterized by initial/ad hoc processes, to level 5, characterized by optimizing and efficient capabilities. This assessment can help determine the level of maturity of organizations of various sizes or units within a larger organization, thus demonstrating a balance in the applicability of the CTME maturity model. (The default definition of “institution” or “organization” here was an academic medical center that attempts to function as a cohesive entity regarding clinical research, even if that contains multiple corporate entities.) The CTSA consortium has developed a suite of maturity models to assist CTSA hubs in self-assessment and strategic planning [Reference Knosp, Craven, Dorr, Bernstam and Campion13– 17]. This model fills a gap in that suite by providing a more granular assessment of CTME.
Preliminary discussions of the working group were aimed to determine the range of challenges to improving clinical trials efficiency at the CTSA hubs. The most significant barriers that we identified included lack of IT infrastructure, lack of data systems or lack of integration of existing systems, existence of patient- and protocol-level data in siloed, fragmented systems, administrative and fiscal issues, recruitment issues, lack of institutional leadership commitment, and the availability of needed personnel and expertise [Reference Welch, Tomoaia-Cotisel and Noubary23,Reference Embi and Payne24]. All of these discussion findings informed the development of a formal framework for CTME. For example, the five advancing levels of maturity across system integration and interfaces data, financial management, subject identification and recruitment, subject management, staff training & personnel management, and organizational maturity and culture axes illustrate areas of intervention critical to advancing the respective domain and overcoming the barriers that have been described above. Further, several organizational benefits can be ascribed to the development and implementation of the CTME maturity model: 1) an improved understanding of the components and complexity of technical and organizational issues related to clinical trials management by the CTSA hubs [Reference Bititci, Garengo, Ates and Nudurupati25]; 2) the establishment of consortium-wide benchmarking for CTME [Reference Knosp, Barnett, Anderson and Embi15]; 3) addressing a missing component to the current suite of CTSA maturity models; 4) an improved ability of informatics, clinical trials, and senior institutional leadership to identify and remove the barriers to effective CTM at their respective hubs, provide insights to develop concrete plans, and empower informatics leads to advocate for CTM technology to institutional leadership. Some institutions are already using the CTME maturity model for internal strategic planning (personal communication, Starren and Knosp). Recently, the Iowa CTSA used the CTME maturity model to assess Iowa’s maturity as part of an interdisciplinary clinical trials strategic planning process. The team reported that the model was effective in identifying organizational gaps and improving clinical research operations [Reference Subramain, Sprenger, O’Connell-Moore, Jones-Bitterman and Knosp26]. The long-term impact will be improvement in the quality and comprehensiveness of CTME at CTSA hubs and other academic medical centers.
Defining the evolutionary path consisted of first describing the characteristics of the initial and optimizing maturity stages or the lowest and highest levels (levels 1 and 5) for every CTME axis. This was followed by a description of anticipated, typical, logical, and desired evolution paths (Fig. 4). This top-bottom approach was important to ensure that the CTME MM was designed not only for descriptive purposes but also for wide application in practice, and it reflects the rigor of the intellectual process undertaken by this working group. Overall, the group developed over 200 maturity statements (Supplementary material 1). Further, it is a key factor that differentiates the CTME MM from other informatics maturity models, for example, the Social and Environmental Determinants of Health MM [Reference Espinoza, Sehgal and Phuong16] or the Research IT [Reference Knosp, Barnett, Anderson and Embi15] model that utilized smaller expert development with surveys, or the RIOSM model that was based on extensive interviews across sites.
Figure 4. Figure illustrating the evolution of maturity for the “Data” axis of the CTME maturity model. Note: the illustration represents maturity statements clustered around one conceptual component only, standardization. The figure does not represent maturity statements for complexity/integration and monitoring. CTME = Clinical Trials Management Ecosystem.
An unexpected finding of our assessments was the wide range of maturity across the national CTSA hubs. While most CTSAs fall in the mid-range, some institutions rated all axes as 4 or 5, while some others rated all axes a 1 or 2. Across the CTSA consortium, some CTME axes are less mature than others. None of the participating institutions reported level 5 for four of the CTME axes (regulatory and audit management, financial management, subject identification and recruitment, reporting analytics & dashboard). This highlights the need for modern technologies to consolidate data, accelerate review, and improve time to analysis and insights and sheds light on the inherent challenges in improving maturity for some of these axes. Conversely, the CTSA hubs reported higher levels of maturity for some axes such as study management, investigational product management, and subject management. This could be the result of a confluence of phenomena. First, these areas within the CTME activities tend to be more highly regulated. Second, these activities are the focus of many clinical trials management systems implementations. For example, a web-based research management system that has been in operation at the Medical University of South Carolina, known as SPARC, integrates both research and routine clinical care workflows [Reference Sampson, Glenn, Cates, Scott and Obeid27,Reference Sampson, Shapiro and He28]. Third, novel initiatives have emerged including the NIH Collaboratory [29], which aims to “rethink clinical trials,” and PCORnet, [30] a national clinical trials infrastructure to facilitate national-level clinical trials to be faster and more efficient by using randomization in the course of routine care and utilizing data from EHRs.
Although our work describes a multi-axial maturity framework to measure organizational capability with respect to the CTME, there are limitations. The recruitment of working group members was opportunistic. Also, the work focused on the CTSA consortium and AMIA, which represent only a portion of the clinical trials activity across the nation. Non-academic and community healthcare organizations were not represented. The results are therefore potentially biased. Further, we acknowledge that not all clinical trial stakeholders, such as the clinical trials regulatory staff and leadership, and those with experience in investigational product management, were directly involved in model development. However, the informatics experts involved in model development work closely on a daily basis with all of these groups. Even so, the maturity statements might not accurately reflect all viewpoints of those involved in clinical trials outside of the informatics community at CTSAs. Although our assessments suggest reasonable real-world applicability, future work must evaluate the generalizability and utility of CTME maturity model beyond the national CTSA hubs.
Conclusion
The CTME maturity model gives a framework to research organizations to examine their current maturity level with respect to 11 axes relevant to clinical trials and identify areas for future development. It must be borne in mind that this is not a one-time process. Keeping in mind the evolving nature of clinical trials and the complexity of this ecosystem, organizations will need to assess their capabilities on a regular basis and take action as necessary. Future work must determine the optimal frequency for institutional reassessment, and the utility of the maturity model for related activities, such as strategic planning.
Supplementary material
For supplementary material accompanying this paper visit https://doi.org/10.1017/cts.2024.1168
Acknowledgements
We are thankful to the Center for Leading Innovation and Collaboration (CLIC) and Informatics Enterprise Committee (IEC) for their assistance with workgroup meetings.
Author contributions
JS conceived the study, chaired the maturity working group, analyzed the data, and drafted the original manuscript. SS interpreted the results, analyzed the data, developed the model, and prepared the manuscript. AW, BK, SR, CKC, JT, JF, and ECP contributed to model development and reviewed and edited the manuscript. All authors approved the final draft for submission to JCTS.
Funding statement
This work was supported by multiple grants from the National Center for Advancing Translational Science (NCATS) of the US National Institutes of Health, including UL1TR001422, UL1TR002537, and UM1TR004398.
REDCap is supported at FSM by the Northwestern University Clinical and Translational Science (NUCATS) Institute. Research reported in this publication was supported, in part, by the National Institutes of Health’s National Center for Advancing Translational Sciences, Grant Number UL1TR001422. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Competing interests
The authors declare that they have no relevant competing interests to declare.
Open access
