Introduction
The National Institutes of Health [1] Data Management and Sharing (DMS) policy (NOT-OD-21-013, effective on January 25, 2023) mandates that a Data Management and Sharing Plan (DMSP) be submitted as a component of grant applications for studies that generate scientific data [1]. Two decades prior, and part of an international groundswell of support for open sharing of resources and results from publicly funded research, the NIH released its initial policy in 2003, requiring a data sharing plan for certain grants. The National Science Foundation (NSF) subsequently revised its policy on Dissemination and Sharing of Research Results, requiring submission of a two-page Data Management Plan starting January 18, 2011. The following years have seen a sharp increase in publications on data management and sharing, many from the library and information science discipline. Examples include reviews [Reference Vitale and Moulaison-Sandy2–Reference Stanciu4], inventory or evaluation of funder requirements for DMPs [Reference Jennifer5–Reference Henning, Silva, Ferreira Pires, van Sinderen and Moreira7], inventory or assessment of tools for DMP creation [Reference Gajbe, Tiwari, Gopal and Singh8,Reference Becker, Hundt, Engelhardt, Sperling, Kurzweil and Müller-Pfefferkorn9], general guidance for DMP creation [Reference Jennifer5,Reference Michener10–Reference Zozus14], discipline-specific guidance for DMP creation [Reference Williams, Bagwell and Nahm Zozus6,Reference Brand, Bartlett and Farley15,Reference Lebedys, Famatiga-Fay, Bhatkar, Johnson, Viswanathan and Zozus16], case studies of DMP creation or implementation [Reference Burnette, Williams and Imker17,Reference Bishop, Neish and Kim18], characterizations of DMP implementations [Reference Spichtinger19], descriptions of programs to help researchers with DMP creation [Reference Mischo, Wiley, Schlembach and Imker20–Reference Karimova22], policy analysis or recommendations [Reference Bishop, Neish and Kim18,Reference Van Tuyl and Whitmire23,Reference Pasek24], evaluation of DMP content [Reference Smale, Unsworth, Denyer and Barr3,Reference Van Tuyl and Whitmire23,Reference Parham and Doty25–Reference Veiga, Ferreira Pires, Henning, Moreira and Pereira33], development of rubrics to assess DMP content [Reference Parham, Carlson, Hswe, Westra and Whitmire29,Reference Bishop, Ungvari, Gunderman and Moulaison-Sandy34], evaluation of DMSP effectiveness resulting in well-shared data [Reference Van Tuyl and Whitmire23], advances toward machine-actionable DMPs [Reference Bakos, Miksa and Rauber35–Reference Romanos, Kalogerini, Koumoulos, Morozinis, Sebastiani and Charitidis37], and a public registry for DMPs [Reference Silva, Monteiro, Mundim Rodrigues, Arellano and Oliveira38].
As a new policy, little has been published regarding the institutional implementation of the NIH DMS policy and its requirements for DMSPs. This study aimed to characterize our institution’s implementation of NIH DMS policy and evaluate structured vs. unstructured approaches to producing policy-conformant DMSPs. The NIH posted a general Microsoft Word DMSP template with headers and brief instructions for each header, along with policy information [39]. In March 2023, a Federal Demonstration Project (FDP) [40] was initiated to test two draft NIH versions of DMSP templates (alpha and bravo). At the same time, our institution initiated a support program to help our researchers comply with the new mandate. The support program produced five study-specific DMSP templates to support policy compliance. The templates averaged five pages in length and followed NIH grant font and margin requirements. They were structured with a section header for each of the six required elements of the NIH DMSP. They provided additional resources, such as links to supplemental institutional and federal policy information. Example text for each DMSP required element was also provided to guide investigators in describing pertinent aspects of their research under the appropriate DMSP section. In this study, we obtained copies of DMSPs submitted with grant applications from our grant management system over an 18-month period. We tracked data related to Just-in-Time (JIT) comments. We created a rubric to assess DMSPs’ compliance with the NIH DMS policy requirements. We surveyed and interviewed investigators who had requested support for preparing their DMSPs. We asked them about usability, usefulness, clarity, ease of understanding, fit to study design, and overall satisfaction with the template they used for their grant applications. The results revealed that unstructured DMSPs omitted significantly more information than the structured ones. Unstructured DMSPs were used as a natural comparison group to assess the relative performance of local and NIH structured templates. The researcher’s feedback overwhelmingly supported the use of structured templates with detailed instructions. The data-related JIT comments were significantly less in submissions using a structured DMSP template. We recommend using structured DMSP templates that guide researchers through each required element and provide example text to support completion. We also recommend the use of a standardized rubric to help researchers and NIH program officials consistently prepare and evaluate DMSPs. Together, these measures could significantly improve compliance, reduce ambiguity, and promote higher-quality data management planning across NIH-funded research.
Beyond the ethical considerations for the public availability of publicly funded research, data sharing may increase the return on investment from such research, for example, by enabling subsequent use of the data to answer new research questions. Thus, the NIH has heavily invested in data repositories to ensure enduring access to these resources. Following the seminal articulation of FAIR principles by Wilkins et al. 2016 [Reference Wilkinson, Dumontier and Aalbersberg41], significant efforts have been undertaken to pursue methods and technology to make shared data and resources Findable, Accessible, Interoperable, and Reusable (FAIR) [Reference Wilkinson, Dumontier and Aalbersberg41] such as the Biomedical Data Fabric (BDF) initiative [42]. Although data sharing in the U.S. is far from achieving the FAIR ideal [Reference Duke43], the NIH DMS policy takes an essential step toward the reuse of research data in clinical translational science by addressing the upfront application of data standards and data management methods.
Methods
Local DMSP templates
We designed the local DMSP templates to reduce cognitive load through their structure, specifically the external representation and the Proximity Compatibility Principle (PCP). Briefly, external representation refers to having the information needed for a task in the world, such as physical constraints, or, in the case of our DMSP templates, instructions and links to relevant information, rather than storing it in memory. The PCP in cognitive engineering states that information required by a task (related information) should be placed physically close to (proximally) or ideally at the location of task performance so that users can more easily integrate the information [Reference Wickens and Carswell44]. For example, adhering to the PCP for DMSPs obviates the need for investigators to conduct web searches for required policy information or requirements.
Data collection
We obtained copies of DMSPs submitted with grant applications from our institutional grant management system for the period from January 1, 2023, to May 23, 2024. Similar to the DART Framework (Data Management Plans as a Research Tool), which is based on NSF data management plan requirements [Reference Parham, Carlson, Hswe, Westra and Whitmire29], and the DMP scorecard developed for the Belmont Forum grants [Reference Bishop, Ungvari, Gunderman and Moulaison-Sandy34], we created a standardized rubric to assess adherence to the DMSP with the NIH policy (Supplementary Material 1). The rubric assessed the six main DMSP elements (Tables 1 and 2 of the rubric in Supplementary Material 1) using nominal binary (Yes/No) items. The rubric also assessed the DMSP sub-elements (Table 3 of the rubric in Supplementary Material 1) using fourteen nominal binary items and seven nominal non-binary items to score each type of data listed in the DMSP. Because the DMS policy requires similar information for all kinds of scientific data generated by a grant, the assessment rubric was designed to be expandable, where a new section could be added for each type of data addressed in the DMSP (Supplementary Material 1). Two independent evaluators examined and scored each DMSP. The reviewers in regular study meetings adjudicated the discrepant assessments. Scores were entered in a scoring worksheet to calculate the adherence/conformance rate and to categorize the results.
Survey and interview
We surveyed and interviewed investigators who had requested support through the DMSP Assistance Program. We expanded the original 14-item questionnaire used in the NIH FDP pilot – comprising 10 quantitative and 4 qualitative items – for local use by adding 8 additional quantitative and 11 qualitative items (Supplementary Material 2). These supplementary questions were included to explore researchers’ perspectives on the template they used, with a specific focus on usability, usefulness, clarity, ease of understanding, fit with the study design, and overall satisfaction. Responses were categorized based on the type of template used, dividing them into three groups: NIH templates, local templates, and investigator-created DMSPs. The survey data were collected and managed using the REDCap electronic data capture tool hosted at the University of Texas Health Science Center in San Antonio (UTHSA) [Reference Harris, Taylor, Thielke, Payne, Gonzalez and Conde45,Reference Harris, Taylor and Minor46]. Survey data were collected two weeks after the investigator submitted their funding application. Non-respondents received up to three follow-up contact attempts via email. Most of the additional qualitative questions focused on identifying opportunities to improve the local DMSP assistance program. Responses to qualitative items were reviewed and thematically categorized for reporting.
Just-in-time (JIT) review
We also waited for the JIT comments from the grant review process. These were used as an additional source of quality control feedback to assess policy adherence.
Statistical analysis
The proportion of completed DMSP elements for each template type was calculated with exact 95% confidence intervals and compared using Fisher’s exact test. Each required element in a DMSP could have multiple sub-elements corresponding to the different data types. These sub-elements were analyzed using Generalized Estimating Equations, with each data type clustered within a DMSP. For meaningful statistical analysis, we grouped the templates into three categories: NIH group templates, local group templates, or idiosyncratic DMSPs. The latter consisted of DMSPs created by investigators combining multiple templates, as well as those with no resemblance to any of the assessed templates. Within this model, the completeness rate of the sub-elements was predicted by a categorical variable representing the three DMSP template categories. No adjustments were made for multiple comparisons.
Results
During the evaluation period, from January 1, 2023, to May 23, 2024, 27% of the funding applications submitted by our institution did not require a DMSP. These grant applications were submitted to NIH for activities that do not generate scientific data, such as training, infrastructure development, or other non-research-focused efforts, or were submitted to funding agencies other than NIH. The remaining 73% of applications fell within the scope of the DMS Policy and therefore required a DMSP. Of the applications requiring a DMSP, 84% (n = 358) included one. For 16.2% (n = 69) of the submissions requiring a DMSP, no plan was retrievable from our grant management system (Figure 1).
DMSPs included in the analysis.
We presumed that these DMSPs were either not submitted at the time of application or were submitted through alternative channels that bypassed our standard institutional tracking system, resulting in their omission from the internal record. We evaluated the content of the 358 DMSPs using the rubric developed for this study (Supplementary Material 1); 79.3% of them addressed all six required elements outlined in the NIH DMS Policy. The percentage of DMSP addressing required DMSP elements varied. Element 1 (Data description) was most frequently addressed (98.9%), while Element 2 (Related Tools, Software, or Code) was the least frequently addressed (82.7%). Similarly, the percent of sub-elements addressed varied from 98.9% for data type description to 49.3% for data generation process description (Table 1).
Addressed DMSP required elements and sub-elements
The study designs represented in the assessed DMSPs were categorized into five groups: clinical interventional studies, clinical observational studies, basic science studies, secondary data or sample use, and other. Overall, the proportion of DMSP elements and sub-elements addressed was generally consistent across these categories (Figure 2). The most notable deviation occurred in the “Other” category, which included miscellaneous study designs not clearly assignable to any primary groups. Excluding this outlier, the percentage of elements addressed by each study category ranged from 76.3% to 92.9%, and the percentage of sub-elements addressed per each study type ranged from 67.0% to 81.7% across the remaining study types (Figure 2). Data aggregation levels and descriptions of data processing were more frequently reported in clinical studies than in basic science studies. In contrast, the element related to data generation was less frequently addressed in studies involving secondary data than in prospective clinical or basic science studies.
Percentage of DMSP elements and sub-elements addressed with 95% confidence intervals per study types.
Types of templates used varied considerably; the most frequently used template was the NIH Generic template (n = 178), followed by the “Other” category (n = 71), the local animal studies template (n = 62), and the local prospective clinical study template (n = 30). In contrast, six of the ten assessed template types – the NIH Alpha and Beta templates, the human tissue only template, the non-animal/non-human studies template, the secondary data use template, and the mixed template – were used too infrequently to allow for meaningful statistical comparison, with usage frequencies of 1, 1, 3, 1, 4, and 7, respectively. To enable meaningful analysis, we grouped the templates into three broader categories: NIH, Local, and Idiosyncratic. The NIH group includes the Alpha, Beta, and Generic NIH templates (n = 180); the local group consists of the five institutionally developed templates (n = 100); and the idiosyncratic group comprises both the “Mixed” and “Other” template categories (n = 78), which reflect unstructured researcher-created formats. We calculated and compared the proportion of completed elements for each template type with exact 95% confidence intervals using Fisher’s exact test (Table 2).
Elements addressed by each DMSP group
Each required element in a DMSP could have multiple sub-elements for each type of data generated. The completion of these sub-elements was analyzed using Generalized Estimating Equations, with each data type clustered within a DMSP (Supplementary Material 3). Within this model, the rate of completeness of the sub-elements was predicted by a categorical variable representing the three DMSP template categories: NIH group template, local group template, or idiosyncratic DMSPs. The significance level (two-sided) was 0.05 for all tests. There were no adjustments for multiple comparisons (Supplementary Material 3). The results showed that the odds of addressing DMSP sub-elements increase significantly using structured templates. Compared to NIH DMSP templates, idiosyncratic (unstructured) templates underperformed, while local (structured) templates performed superiorly in addressing all sub-elements (Figure 3).
Estimating performance of local and idiosyncratic compared to NIH templates in addressing the sub-elements DMSP requirements using the generalized estimating equations. *Sub-element 1.1 is a count of the data types collected for a study. As a continuous variable (and as the denominator for the remainder of the sub-elements), it was excluded from the generalized estimating equations analysis in this table.
Researcher survey and interview results
There were 75 researchers that contacted the DMSP assistance program and accessed the DMSP templates via the provided web link. Of these, 43% (n = 32) responded to the survey, and 27% (n = 20) of those survey respondents also participated in follow-up interviews. In this small sample, we grouped the interview comments from respondents completing a local template together and those completing an NIH template together. Locally developed templates were found to be favorable across all evaluated categories, including usability, clarity, alignment with the study design, and overall user satisfaction. Most respondents (87%, n = 65), regardless of the template used, agreed that research trainees should receive formal data management and sharing training. All participants expressed that any template should be accompanied by instruction sheets tailored to common study types.
Regarding specific template features, 71% found the inclusion of example text within templates helpful, 57% indicated that links to NIH discipline-specific and general data repositories were beneficial, and 50% reported that including information and links related to genomic data sharing requirements was helpful. Researchers selected templates primarily based on how well they aligned with their study design and research aims. For instance, users of the NIH Generic Template cited reasons such as “followed NIH links” and its suitability for “basic science.” The NIH Pilot Alpha Template was chosen because it “fits better with our research type and scopes,” the NIH Pilot Bravo Template user noted a desire to “try the NIH V2 template.” Local templates were favored for their close alignment with specific project needs. Users described them as “most applicable,” “discipline specific,” “aligned with the type of data we are producing,” and “very convenient for my study specifics.” Others emphasized that these templates were “more project specific” and “most closely aligned with the proposed research plan aims.” Other template selections seemed to be based on familiarity or adaptability. Comments included “template used by our lab before,” “simple, clear, and understandable,” and “had to be customized to fit the study needs regarding data management and sharing..”
JIT comments related to data management and sharing
Out of 427 funding applications submitted, 19 (4.4%) received data-related comments or questions as part of the JIT request process. The unstructured templates accounted for the highest number of JIT comments, 52.6% (n = 10), followed by NIH templates, 42.1% (n = 8). Notably, only one data-related JIT request was associated with a submission using a local template, suggesting potential advantages in clarity or completeness for those plans.
Discussion
Of the 358 DMSPs evaluated, 79% addressed all required sections of the NIH DMS policy, while 21% omitted one or more required sections. In our survey results, the majority of respondents indicated that data management training was desirable. Others continue to find lack of knowledge about research data management or DMP creation among researchers [Reference Gajbe, Tiwari, Gopal and Singh8,Reference Parham, Carlson, Hswe, Westra and Whitmire29,Reference Veiga, Ferreira Pires, Henning, Moreira and Pereira33] similarly suggesting that institutional support programs should include training in DMSP creation if not in basic data management principles.
The proportion of DMSPs received data-related JIT comments was significantly lower than the 21% (4.4%), suggesting that incomplete plans did not always trigger follow-up inquiries. There are several possible explanations. Omissions may have gone undetected during NIH program officials’ reviews, may have been deemed inconsequential to the proposed study, or may be attributable to other factors that could not be determined from the available data. These findings suggest room for improvement in the completeness and review of DMSPs. Both investigators preparing DMSPs and NIH program officials reviewing them could benefit from a standardized assessment rubric to promote consistency, completeness, and adherence to policy requirements.
Seven sub-elements were less than 75% complete, meaning that at least 25% of DMSPs failed to address these components (Table 1). While omitting a sub-element does not necessarily indicate that the corresponding data-related tasks will not be performed by the investigator or study team, it does prevent NIH officials from evaluating the adequacy of the plan for completing them. If missing information is not identified and requested upon review, opportunities to intervene in potentially inadequate plans may be lost. Timely intervention is particularly concerning for data generation, collection, and handling aspects that cannot be easily corrected after the fact [Reference Zozus47]. To mitigate this risk, we recommend that DMSP authors use a standardized template and rubric to ensure completeness and a standardized way to explicitly indicate that a required element or sub-element does not apply to a study. NIH program officials could similarly use a rubric during review to consistently evaluate conformance. Completeness – addressing each required element and sub-element – is critical to compliance with the NIH DMS policy. A complete DMSP enhances transparency and fosters early and meaningful discussions about data management and sharing. These early and meaningful discussions can improve planning, implementation, and ultimately, the overall integrity and utility of the research. The completeness of DMSPs, and by extension, compliance with the NIH DMS policy, varied significantly across the most used templates. The local prospective clinical study template (n = 30) had the highest completeness rate at 92.5%, followed by local animal study templates (n = 62) at 84.5%, the NIH Generic template (n = 178) at 75.3%, and “Other” templates (n = 71) at 42.6%. These variations underscore the significance of template structure in facilitating compliance with NIH requirements. While the “Other” template category stood out as distinctly different in having little to no structure, the structured templates provided more specific guidance with respect to section and sub-section content (the six elements and sub-elements), leading to higher completeness rates (Table 2). The survey and interview data supported these findings with majority of respondents indicating that structural elements of the templates were desirable. Others also advocate domain-specific DMP templates or tools [Reference Gajbe, Tiwari, Gopal and Singh8,Reference Veiga, Ferreira Pires, Henning, Moreira and Pereira33].
Interestingly, unstructured DMSPs commonly included data and resource sharing elements consistent with prior NIH resource sharing policies but often omitted those newly required by the NIH DMS policy. This pattern suggests that researchers were generally aware of and responsive to prior sharing expectations, rather than aware of the current policy. Therefore, while researcher-level factors such as awareness and institutional support undoubtedly influence template selection and completeness, our findings also suggest that unstructured templates – lacking prompt for policy-required information – may have also contributed to missing information in DMSPs. The NIH Generic template offered semi-structure, with headers for the six required DMSP sections but no additional framework. This template performed well at the element level (98.9% completeness) but less at the sub-element level (75.4% completeness), reinforcing that greater structure within a template improves overall completeness.
Limitations and generalizability
A randomized controlled trial (RCT) was neither feasible nor appropriate due to the nature of the policy implementation and the real-world constraints. The observational nature of this study limits the strength of evidence. Additionally, the implementation environment itself was evolving throughout the study. Additional challenges included barriers and the time required to gain access to the institutional grant management system and JIT information. These results, from a case study conducted at a single institution, are not directly generalizable to other institutions or settings. However, we believe our context may share significant similarities with other institutions, and thus, some of our findings may be applicable or informative to others in similar settings.
Conclusion
In our case study, the current adherence rate with the NIH DMS policy elements was just over 79%, meaning that approximately 21% of submitted DMSPs missed one or more required elements. This gap highlights the opportunity for targeted interventions to enhance policy adherence, specifically in terms of data management plan completeness. Our findings indicate that structured DMSPs outperform non-structured ones regarding completeness and alignment with policy expectations (Table 2). To address these challenges, we recommend a two-pronged approach: use of structured DMSP templates that guide researchers through each required element and sub-element – with example text to clarify expectations – and implementation of a standardized rubric to assist researchers, institutional officials, and NIH program officials in consistently preparing or evaluating DMSPs. Based on this case study, policy adherence can be significantly improved by adopting these measures, reducing ambiguity, and promoting higher-quality data management planning across NIH-funded research.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/cts.2025.10212.
Author contributions
Muayad Hamidi: Conceptualization, Writing-original draft, Writing-review & editing; Manju Bikkanuri: Data curation, Writing-review & editing; Camille Scott: Data curation, Resources, Writing-review & editing; Monica Carrizal: Data curation, Resources, Writing-review & editing; Mari Martinez: Resources, Software, Writing-review & editing; Andrea N. Schorr: Conceptualization, Methodology, Resources, Writing-review & editing; Liu Qianqian: Data curation, Formal analysis, Validation, Writing-review & editing; Jonathan Gelfond: Data curation, Formal analysis, Writing-review & editing; Joseph Schmelz: Project administration, Resources, Writing-review & editing; Jennifer Potter: Conceptualization, Funding acquisition, Investigation, Methodology, Resources, Writing-review & editing; Meredith Zozus: Conceptualization, Data curation, Formal analysis, Investigation, Project administration, Writing-original draft, Writing-review & editing.
Financial support
Funding source: This work is funded by the National Institute of Health (NIH), National Center for Advancing Translational Sciences (NCAT). The Clinical and Translational Science Award (CTSA) UL1 TR 002645-02 - Element E.
Role of sponsor or funder: The NIH-NCAT awarded The CTSA to the University of Texas San Antonio (UTSA) to pursue integrating clinical and translational research and career development across UTSA schools and diverse public and private partners in South Texas. The CTSA to UTSA provides a broad range of support for research infrastructure to reduce barriers to research and stimulate the transformation of knowledge into improved health care. The funder will have no input on the interpretation or publication of the study results.
Competing interests
All authors report no conflicts of interest.
Open access
