Introduction
There has been a growing awareness of the role that diversity, health equity, and inclusion play in paediatric clinical care and research. Reference Trent, Dooley and Douge1–Reference Soranno, Simon and Bora5 Advancing diversity, equity, inclusion, and accessibility through research and workforce diversity are also a National Institute of Health priority outlined in its strategic plan.6 Incorporating diversity, health equity, and inclusion into research requires consideration of the following: 1) existing health disparities, 2) the use and depth of appropriate population demographics in research, 3) selection of accurate and high-quality databases, 4) analysis of data beyond race/ethnicity and sex, 5) inclusion of impacted populations in study design with representation from diverse voices, 6) dissemination of results beyond the scientific community, 7) prioritisation of health equity research in funding and publications, and 8) research team diversity.6,Reference O’Connor7
In August 2021, the National Heart, Lung, and Blood Institute convened a workshop to identify paediatric and congenital cardiovascular research opportunities to improve outcomes for individuals with CHD across the lifespan and define a vision for the Pediatric Heart Network, its consortium of high-performing hospitals across the United States, Canada, and other countries that conduct research with a focus on children, adults, and their families living with CHD, and children affected by acquired heart disease. A summary of this workshop was published Reference Opotowsky, Allen and Bucholz3 and identified multiple themes including the recruitment of diverse study participants and support of investigators from diverse backgrounds. Themes provided by this workshop created a framework to expand the scope of the Pediatric Heart Network Scholars programme and launched an initiative to incorporate diversity, health equity, and inclusion into its Funding Opportunity Announcement for 2021.
This manuscript summarises this process, discusses the strengths and weakness of submitted applications, and presents potential process improvement methods. It also makes recommendations for the incorporation of diversity, health equity, and inclusion into future funding cycles of the Pediatric Heart Network.
Materials and methods
Committee approach to novel diversity, health equity, and inclusion addition to the Pediatric Heart Network Scholar grant
Based on the recommendation of the National Heart, Lung, and Blood Institute, the Pediatric Heart Network charged the Scholar grant committee with embedding diversity, health equity, and inclusion as a fundamental component into the 2021 grant cycle. Members of the committee met virtually to discuss the integration and execution of such measures. A decision was made to craft formal criteria for Pediatric Heart Network Scholar grant applications. The committee also discussed the appropriate weighting of diversity, health equity, and inclusion within the grant review and scoring process. Given the novelty of this concept, the committee agreed that the applicants should decide the method(s) of incorporating diversity, health equity, and inclusion into their proposals, highlighting, and justifying their methods to the committee in an application statement. The application announcement would provide examples of incorporating diversity, health equity, and inclusion, without mandating methods. Proposals in all Pediatric Heart Network designated research areas would be considered, while encouraging proposals that address health equity. In addition, the grant application would emphasise the importance of research design with a clear plan to ensure diverse patient enrolment and a diverse research team. All parameters and metrics were voted on by the members of the grant review committee with full agreement (Appendix A).
The committee also enlisted two diversity, health equity, and inclusion experts for process input and application review. Candidates were nominated and two paediatric cardiologists from groups underrepresented in medicine and biomedical science with diversity, health equity, and inclusion expertise were unanimously selected as ad hoc reviewers—one with health disparities research, advocacy, and grant experience, and one with expertise in outpatient clinical work with underserved communities and advancing institutional diversity, equity, and inclusion initiatives. In addition to scoring each application on standard criteria, the ad hoc reviewers provided directed feedback on the strengths and weakness of diversity, health equity, and inclusion in each application. Ad hoc reviewers recused themselves from the evaluation of applications from their home institution.
Process of embedding diversity, health equity, and inclusion into the Pediatric Heart Network Scholar Award
The announcement overview and research objectives were updated to notify applicants of the requirement to address diversity, equity, and inclusion in their proposals (Appendix A). Applicants were also advised that this metric would be evaluated and scored in the review process. Candidates from underrepresented and geographically disadvantaged groups or those with a disability were encouraged to apply.
The announcement’s scored review criteria were also updated with a section on diversity, health equity, and inclusion. Bullets were created to guide candidates on methods of addressing these measures (Appendix A). The announcement made clear that not all bullets would be applicable but provided highlights of what the reviewers would consider strengths. These examples were not exhaustive, and candidates were encouraged to discuss their perceived strengths in the designated section of the application.
The application was expanded to include up to 1 page on how investigator’s career goals/objectives, research environment, approach, and research team addressed diversity, health equity, and inclusion (Appendix B).
The modified National Heart, Lung, and Blood Institute critique template was further adapted to include a diversity, health equity, and inclusion section used to inform the overall score (Appendix C).
The grant review process, designed to resemble a National Heart, Lung, and Blood Institute study section review, was updated such that primary and secondary reviewers provided a written critique and score for their assigned applications on the incorporation of diversity, health equity, and inclusion in addition to standard metrics (Appendix D). The ad hoc reviewers similarly reviewed and scored all applications on these measures. The Scientific Review Officer summarised the strengths and weaknesses of each application and provided written feedback to all PIs.
Results
Strengths and weaknesses of diversity, health equity, and inclusion in submitted grant proposals
Applications were evaluated in the following areas regarding diversity, health equity, and inclusion: applicant and research team demographics; scientific aims, data collection, and analytic plan; patient recruitment; and dissemination of results.
Background of applicant and research team
Data have shown that diverse research teams produce more novel and highly cited papers. Reference Yang, Tian, Woodruff, Jones and Uzzi8 Further, communities historically underrepresented in medicine have cited the importance of having diverse and multilingual research teams in helping to overcome mistrust and increase participation in research. Reference Ford, Siminoff and Pickelsimer9 While diverse candidates have always been encouraged to apply for Pediatric Heart Network Scholar awards, the overall diversity of the research team (applicants, mentors, and collaborators) was now formally evaluated. Inclusion of women, those historically underrepresented in medicine and biomedical research in the United States (i.e., Black, Latino, Native American/Alaskan Native), and immigrants, particularly from a low-income country, and those from geographically diverse locations were considered strengths of the application. Lack of research team diversity was considered a weakness.
In addition to the demographics of the team, an applicant or mentor history of advocacy and mentorship within underserved communities or prior engagement in diversity, health equity, and inclusion work (including health disparities research or membership and/or leadership in local, regional, or national diversity, health equity, and inclusion working groups) was considered an application strength. For applicants and teams with no prior history of diversity, health equity, and inclusion involvement, a plan for training and/or community collaboration in study design, inclusive of those historically underrepresented in research, was viewed as a strength.
Scientific aims, data collection, and analytic plan
The characteristics of having a distinct diversity, health equity, and inclusion focus, and/or metrics embedding diversity, health equity, and inclusion considerations into the research design and analytic plan were considered a significant application strength. Investigation of a disease process or outcome that has been associated with clear and well-described health disparities was considered to have a distinct diversity, health equity, and inclusion focus. Thoughtful incorporation of the roles of sex, race, and ethnicity as well as an emphasis on unequal social determinants of health in the specific aims, data collection, and analytic plan was considered a significant strength. The use of validated metrics to quantify social determinants of health, such as the Centers for Disease Control Social Vulnerability Index, added considerable value to a project. The use of robust data sources to characterise neighbourhoods and identify resource allocation disparities, such as data from the United States Census Bureau, Centers for Disease Control, and Environmental Systems Research Institute, was also considered favourably. Finally, the implementation of novel technologies to enhance access to care and research engagement within the methodology was considered a significant strength for the application.
Applications were viewed less favourably if the research did not propose to collect or analyse factors beyond the individual level (i.e., sex, race) when available (many paediatric cardiology datasets/registries and administrative data did not have SDOH components at the time of this grant mechanism). Proposals where neighbourhood or more societal level factors, such as social determinants of health, were not thoughtfully crafted into the research proposal specific aims or analysis were scored less favourably. As one reviewer stated, using routine demographics is “a missed opportunity” and delineations by broad categories such as race without consideration of associated social determinants of health “have proven to be flawed.” However, it was recognised by the committee that retrospective data collection or data existing in large-scale databases or electronic health records may not have sufficient granularity regarding social determinants of health.
Patient recruitment
Awareness and documentation of the demographics of the study population of interest with a plan for recruitment of a diverse study population relative to the geographic location of the study centre(s) or from large catchment areas was considered a strength. Prior work from a centre demonstrating disparities in outcome and/or care with a plan to recruit affected individuals was noted to be an asset. Identification of potential barriers to study participation, such as language and access to transportation, and inclusion of recruitment strategies to overcome these challenges were noted to be important components of the grant proposals. Some strategies included consent documents in Spanish (or other language appropriate for the study population), use of interpreters during recruitment, establishment of multiple (> 3) contact points for recruitment, employing multiple means to contact patients and families (direct contact within a hospital or clinic, phone calls, and/or emails), and remuneration for participation and transportation costs. Finally, plans to evaluate the reasons potential subjects declined participation was considered a strength, as it allowed for the identification of obstacles to enrolment in future studies.
A lack of granularity regarding geographic diversity was considered a weakness. For example, it was not enough for centres to report having diverse geographic catchment areas. Strategies to ensure research participation of patients from different racial, ethnic, and socio-economic backgrounds needed to be established. However, reviewers recognised that this could be a challenge for retrospective study designs. Lack of attention to potential language barriers was also considered a weakness.
Dissemination of results
A weakness noted in most proposals was the lack of a plan to disseminate study findings to study participants. Given the history of mistrust in the medical establishment by some communities of colour, transparency through dissemination of the study data was noted by the ad hoc reviewers to be necessary for repairing mistrust.
Discussion
Lessons learned and recommendations for future grant application cycles
1. Provide solid definitions and clarity on metrics evaluated
Applicants for the Pediatric Heart Network Scholar Program were asked to highlight diversity, health equity, and inclusion as a formal part of their grant proposals. However, the applicant’s understanding of the concepts and their ability to embed them into a grant proposal was assumed. Explicit definitions to guide applicants were not provided in the proposal instructions. In the future, definitions of diversity, health equity, and inclusion terms and principles should be provided. A potential set of definitions to consider are those used by the American Medical Association and Association of American Medical Colleges in Advancing Health Equity: Guide to Language, Narrative, and Concepts. 10 Further, specific metrics by which each section of the proposal (background of the applicant and research team; scientific aims, data collection, and analytic plan; patient recruitment; and dissemination of results) will be scored should be explicitly stated.
2. Offer guidance on research team diversity
The 2019 National Institute of Health Interest in Diversity Statement recognises the following groups as underrepresented in the biomedical research: Blacks or African Americans, Hispanics or Latinos, American Indians or Alaska Natives, Native Hawaiians, and other Pacific Islanders. The statement also recognises the underrepresentation of individuals with disabilities (as defined by the Americans with Disabilities Act), those from disadvantaged or low socio-economic status backgrounds, and women. 11 Several applications took the approach of engaging diverse teams in the conduct of research. While several included women, evidence for the inclusion of those underrepresented in medicine and biomedical science was lacking. However, the definition of those underrepresented in medicine and biomedical science was not formally stated in the application. Future applicants should receive this definition to gauge the diversity of their current research team. Further, in line with the National Institutes of Health strategic plan of increasing partnership and engagement with historically minority-serving institutions, applicants and their centres should be encouraged to consider opportunities to partner with these institutions and centres typically underfunded by the National Institutes of Health to enhance team diversity. 6 Per National Institute of Health recommendations, applicants should also be allowed to expand the definition of underrepresented groups based on local or regional demographics (e.g., rural, low socio-economic status).
3. Encourage exploration of health disparities
Increasing health provider awareness of racial, ethnic, socio-economic health disparities, and the impact of social determinants of health on these disparities is an important first step in addressing health inequity. However, several studies have shown that physician awareness of the degree of racial, ethnic, and socio-economic disparities in their field or scope of practice and of their own potential contribution to resultant poorer health outcomes is low. Reference Mallinger and Lamberti12–Reference Taylor, Fremont and Jain15 To this end, Pediatric Heart Network Scholar application guidelines should encourage applicants to investigate the intersection of social determinants of health and individual level factors and strategise ways to address existing health disparities within their area of research. Applicants should also be encouraged to engage in health disparities research building upon preliminary data or small studies with an emphasis on social determinants of health, neighbourhood-level factors, etc. In the discussion of research significance, future applications should ask applicants to describe known disparities in their area of research focus and consider how these disparities could be addressed through their research proposal.
4. Emphasise statistical analysis beyond race, ethnicity, and sex
The National Institute of Health Revitalization Act of 1993 requires the assessment of outcomes by sex and race or ethnicity. 16 However, recognising that race is not a biological construct, the impact of social determinants of health, health literacy, patient education, etc. on outcomes must also be considered. Future, Pediatric Heart Network Scholar applications should require an assessment of outcomes by sex, race, and/or ethnicity alongside social determinants of health and other sociodemographic factors in the analysis plan where these data are available. If the primary database utilised in a database study doesn’t capture social determinants of health, applicants should consider the potential to link with other databases capturing this data to enhance their analysis.
5. Advance a distinct plan for the recruitment and retention of diverse populations
National Institute of Health-funded clinical trials are required to include women and minorities as participants. 16 While the inclusion of minoritised groups in clinical trials has increased due to this requirement, racial and ethnic minorities are still vastly underrepresented in biomedical research. Reference Ma, Gutierrez, Frausto and Al-Delaimy17,Reference Geller, Koch, Roesch, Filut, Hallgren and Carnes18 Thus, more intentional methods of recruitment are required. Further, major journals, e.g., New England Journal of Medicine, are now requiring author statements on the demographics of disease and representativeness of study participants. Reference Rubin19 As such, future Pediatric Heart Network Scholar applicants should include a directed plan for enhancing diversity in subject recruitment. Further, studies utilising databases should consider and justify database choice including a discussion on the diversity of subjects captured as it relates to their area of study.
For prospective studies, literature exists to guide diverse subject recruitment and retention. Efforts led by the Association of Black Cardiologists have identified barriers to minority participation in United States of America clinical trials and offered solutions as vetted by patients, referring physicians, investigators, and clinical trial coordinators. Reference Clark, Watkins and Pina20 Similarly, the Food and Drug Administration made recommendations to industry on actions to enhance diversity in clinical trial participation. Reference Clark, Watkins and Pina20,21 Future Pediatric Heart Network Scholar applicants should be referred to these documents for recommendations, and the application should provide examples listed in Appendix E.
6. Encourage expanded data sharing and results dissemination
According to 2023 National Institute of Health policy guidelines, researchers are required to submit a Data Management and Sharing Plan as part of their application. The goal of this policy is to make results and outputs of the National Institute of Health-funded research publicly available. It also intends to accelerate future discoveries by strengthening analysis through combined datasets, allowing for validation of findings, and simplifying the reuse of hard-to-generate data. Plans are now required to outline how scientific data and accompanying metadata will be managed and shared. 22 Similarly, Pediatric Heart Network Scholar applicants should have a distinct data sharing plan, potentially utilising existing Pediatric Heart Network public use datasets and biospecimen repositories as appropriate. Where human subjects are involved, consideration should be given to awareness and authorisation of data sharing through informed consent.
In addition to sharing collected data in repositories for the advancement of science, the National Institutes of Health has placed a focus on transparent communication with its community inclusive of the public. 6 Similarly, Pediatric Heart Network Scholar applicants should develop a plan for disseminating results in a manner accessible to research participants and stakeholders outside of the scientific community. Creation of websites, use of social media, engagement in community events, and other means of communication should be encouraged. If funding is needed to complete this task, it should be included in the budget justification. Consideration should also be given to the development of a space on the Pediatric Heart Network website to share results similar to the pages created for large trials conducted by this entity. 23
7. Recommend targeted training (investigators, mentors, reviewers)
Robust data on the impact of implicit bias or implicit bias training on clinical trials are not readily available. However, a recent secondary analysis of enrolment in a multicentre paediatric randomised trial raised the question of whether physician biases may influence patient enrolment reducing the generalisability of trial results and further perpetuating disparities. While the question may not be answered, it is recommended that processes or training address provider and researcher biases. Reference Coon, Schroeder, Lion and Ray24,Reference Popkin, Taylor-Zapata and Bianchi25 Beyond implicit bias, additional cultural competency and/or cultural humility training should be considered for all investigators and mentors to better understand the communities participating in the proposed research.
Studies have also demonstrated race/ethnicity and gender disparities in funding of National Institute of Health R01 grants. Reference Kolehmainen and Carnes26,Reference Taffe and Gilpin27 Consideration of blinded review and training in implicit bias is recommended as a step to increase awareness and reduce grant reviewer bias. 28 As the National Institutes of Health intends to launch bias training for its reviewers and scientific review officers to enhance the peer review process, Pediatric Heart Network reviewers should also be encouraged to participate in bias training at their home institution or via the current National Institute of Health-developed Implicit Bias Training Course. 6 The Pediatric Heart Network should also consider the development of its own modules addressing implicit bias and cultural humility for researchers and reviewers with assistance of diversity, health equity, and inclusion experts and health equity researchers.
8. Enhance the critical review process
Incorporating new criteria into a review process assumes that all members of the review committee understand how that measure should be judged. In assessing diversity, health equity, and inclusion, reviewers were given binary yes/no questions to consider (e.g., was diversity, health equity, and inclusion incorporated? yes or no). However, no guidance was provided on how to rank the complexity or number of components included in each project. While incorporating experts as reviewers into the process is helpful, development of a formal scoring metric would assist those without expertise in assessing these factors. While there is limited guidance in the literature on how to evaluate diversity, health equity, and inclusion in research proposals and grants, a rubric was created by the University of California-Berkley to assist search committees with assessing candidates’ contributions to diversity, health equity, and inclusion. 29 With the assistance of diversity, health equity, and inclusion experts, the rubric could be modified and merged with the National Institute of Health scoring system to guide reviewers in their assessment. The rubric would need to account for state limitations when assigning weight to some metrics.
On faculty search committees, it is considered best practice to include members of different backgrounds for diversity of thought. Reference Jacobs, Esquierdo-Leal, Smith, Piasecki and Houmanfar30–Reference Hemstreet and Borgelt32 Further, it has been demonstrated that members of diverse hiring committees are more likely to value diversity, acknowledge structural discrimination, and favour inclusive promotion decisions which can lead to the hiring and promotion of more underrepresented persons. Reference Williams33 Similarly, increasing the diversity of the Scholar Committee could enhance the review process and increase the diversity of awarded investigators and research. To encourage this, the call for Scholar Committee nominations could be amended to encourage participating centres to consider nominating those from underrepresented racial and ethnic groups, with a disability, women, and those who will bring diversity of experience and thought to the review process.
Conclusions
A distinct and important initial effort was made by the National Heart, Lung, and Blood Institute-funded Pediatric Heart Network to embed diversity, health equity, and inclusion as a fundamental component of the Pediatric Heart Network Scholar Program for 2021. Our experience in implementing and reviewing the process provides a framework to improve future Pediatric Heart Network Scholar grant cycles and provides guidance that could also be utilised by other funding sources in their grant application and review processes. By further clarifying application guidelines and scoring metrics and encouraging bias and cultural humility training, the Pediatric Heart Network will take an important step in enhancing equity in research. Moving forward, it will be critical for all grant awarding agencies to review and revise their methods and to consider meaningfully embedding diversity, health equity, and inclusion into the review process.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1047951124025587.
Acknowledgements
The views expressed in this manuscript are those of the authors and do not reflect official positions of the National Heart, Lung, and Blood Institute, the National Institutes of Health, or the United States Department of Health and Human Services.
Financial support
This work was supported by Grant Number(s) HL135680, HL135685, HL135683, HL135689, HL135646, HL135665, HL135678, HL135682, HL135666, and HL135691 from NHLBI.
Competing interests
None.
Open access
