Introduction
The detonation of an improvised nuclear device (IND), among other radiological incidents and emergencies, may result in thousands of casualties. In addition to those who suffer from acute traumatic injuries and potentially high doses of ionizing radiation, others may endure long-term medical complications such as physical disability, mental health disorders, solid tumors, and leukemias.Reference Grant, Brenner and Sugiyama1 After such an event, it is important to identify and monitor appropriate individuals with significant contamination or radiological exposure for long-term medical complications. To follow these individuals, a long–term population health program should be created. However, prior disasters have taught us that the creation of a registry in their aftermath requires significant resources.2 Additionally, it requires collaboration between federal, state, local, tribal, and territorial governments, representatives from affected communities, non-governmental organizations (NGO), partners in the private sector, health care and academic partners, and legislators.
The National Academies of Sciences, Engineering, and Medicine (NASEM) convened a meeting in 2019 to discuss the creation of a long-term monitoring registry after an IND detonation incident. They summarized the key elements in a report titled “Long-Term Health Monitoring of Populations Following an IND detonation in the United States: Proceedings of a Workshop”3 which included: (1) Engaging Stakeholders, (2) Defining the Purpose of a Registry, (3) Agreeing on Roles and Responsibilities, (4) Linking Immediate Response to Long-Term Follow-Up, (5) Implementing Radiation Dose Threshold as a Registry Inclusion Criterion, (6) Screening for Disease, and (7) Preparing for Long-Term Communications. The expert panel also discussed the challenges associated with this project and contemplated potential solutions.
To date, no state public health agency or academic institution has attempted to engage in the planning for the creation of a long-term population health monitoring program while considering the findings and recommendations made by the NASEM in 2019.
Oak Ridge Associated Universities (ORAU)-Directed Research and Development (ODRD)4 is a research and development program that supports collaborations between ORAU researchers and faculty at member universities. The Medical Toxicology Program in the Department of Emergency Medicine at Emory University School of Medicine has had a long-standing relationship with the Radiation Emergency Assistance Center and Training Site (REAC/TS) and is a member of the ORAU consortium of universities. Emory University and REAC/TS have previously collaborated in areas of radiological emergencies, radiation medicine, and public health, and would potentially assist in the event of a need to create a registry after an IND incident.
The authors sought to enhance the preparedness of state and local public health agencies by using subject matter experts (SMEs) to examine the most effective way to conduct a long-term population health study after an IND detonation in the United States (USA). We aimed to determine the consensus of SMEs in the development of a list of subcategories and items within the broader categories listed by the NASEM. These subcategories and items would need to be considered and addressed in the execution of a long-term population health study. This project would be conducted by the affected state with the support of federal agencies after an IND detonation. Given the anticipated scarcity of resources following an event, this task will be daunting and will require a multidisciplinary effort and significant resources.
The outcome of this consensus meeting will be a list of necessary items and expected challenges in the implementation of a long-term population health program after an IND detonation in the USA. This project will augment the knowledge and overall capabilities of private and academic stakeholders in assisting federal, state, local, tribal, and territorial governments (FSLTT) in designing and conducting a long-term health monitoring program in the event of an IND incident. The process utilized in the creation of the list of necessary actions and expected challenges, as well as the results of the workshop, will be available to other state and federal agencies. Consequently, FSLTT government agencies could use the results of this project to guide their planning efforts to create similar programs in the event of an IND detonation emergency.
Material and Methods
Study Design
We conducted a multiphase, modified Delphi study based on the consensus of experts. The Emory University Institutional Research Board (IRB) reviewed the proposal and deemed this study to be exempt.
Study Participants
The experts (listed in Table 1) were selected according to two criteria: professional discipline and agency representation. We invited SMEs in public health, radiological/nuclear emergencies, medical physics, epidemiology, long-term health study registries, communication, behavioral health, oncology, research ethics, medical toxicology, and emergency management to participate in this study. Individuals were identified and contacted by the study team based on their expertise and experience in the field to determine their interest in participation as an SME. There are no formal guidelines on what defines SME in each individual field. Formal invitations were sent to each participant with the expectation that they would participate remotely for the initial 2 rounds and in-person for the remaining 3 rounds of the modified Delphi process.
Experts and area of expertise
Table 1. Long description
The table consists of two columns. The left column is labeled Name and the right column is labeled Discipline or expertise. From the top row downward, the names and corresponding disciplines are: Armin Ansari, Public Health and Health Physics; Adela Salame Alfie, Public Health and Health Physics; Jessica Pavlick, Epidemiology; Kelly Nadeau, Public Health; Eric Jens, Communication; Oleg Muravov, Epidemiology and Registry; Norm Coleman, Medicine; Barty Simonton, Radiation Protection; Kalie Burke, Mental Health; Ashley Golden, Biostatistics, Epidemiology, Registry; Larry Dauer, Epidemiology; Ron Goans, Medicine and Health Physics; Sheri Lynn Russo, Emergency Management; Ginger Martin, Emergency Management; Vivi Siegel, Communication; Tori Jordon, Communication; Travis Mann, Communication; Ricky Schutter, C B R N E Planning; Curt Harris, Disaster Management; Jim Hardeman, Radiation Protection; Henry Falk, Public Health; Theresa Gillespie, Epidemiology; Matthew Bradley, Emergency Management; Betsy Ellis, Epidemiology, Research Ethics, Registry; Adayabalam Balajee, Radiobiology, Cytogenetics, Biodosimetry; Amy Schnall, Epidemiology; Ariana Hanchey, Epidemiology. Each row pairs an expert’s name with their area of expertise, covering a range of disciplines relevant to health, safety, and emergency response.
Study Development
This modified Delphi process was designed to identify, based on SME consensus, a list of the necessary components, actions, and expected challenges in the creation of a long-term health monitoring registry after the detonation of an IND. The team members developed a literature-informed list of topics that would be important in the creation of a long-term population health monitoring program while considering the findings and recommendations made by the NASEM in 2019. Additionally, three radiation and public health experts were interviewed to ensure that the initial list of categories and items did not omit any elements.
Data Collection and Analysis
A resulting list of items organized in categories was incorporated into a survey designed to facilitate the identification of the most important items and determine the difficulty of implementing each item. A modified Delphi process was used, wherein we attempted to build consensus on the most important categories and items across multiple surveys of our expert population. Round 1 generated categories and items by asking SMEs to review the list and add any category or item that they feel was omitted. Round 2 generated additional items by asking experts to add any items they believed were still omitted after reviewing the updated list. Round 3 asked experts to vote on whether they agreed that the individual items should be included in the long-term health registry. Rounds 4 and 5 assessed SME consensus on the items’ importance and difficulty. All Delphi rounds were completed using an anonymous online survey conducted in Qualtrics.
Results
Round 1 was sent out via email 3 weeks prior to the in-person meeting. A list of 39 items believed to be important in the design and implementation of a long-term population health study after the detonation of an improvised nuclear device was generated through a literature review and expert interviews. Each item was categorized into 1 of the following 10 broad categories: Engaging stakeholders, Defining the purpose of a registry, Agreeing on roles and responsibilities, Linking immediate response to long term follow up, Implement geographic boundaries as a registry inclusion criterion before considering additional inclusion criteria, Identifying and implementing the criteria for inclusion in the registry, Screening for disease, Preparing for long-term communication, Finances, and Legislative policies. In round 1, survey participants were first presented with the list of 10 categories and asked to add categories they believed were missing in a free-text box. Then, participants were provided with a list of the 39 items within the 10 categories and asked to add any items they believed were missing in a free-text box. The items suggested by the experts in round 1 were incorporated into the list that was used in round 2.
Round 2 was sent out by email 1 week prior to the in-person meeting. Items that were suggested for inclusion by the participants in the first round of our modified Delphi study were incorporated into 36 new items in the round 2 survey, grouped into 11 categories. Based on round 1 results, 1 new category was added: data management. Participants were provided with an updated list of the 75 items, based on round 1 responses, and asked to add any items they believed were still missing in a free-text box.
Round 3 and all subsequent rounds were completed during the in-person meeting on August 23, 2022. Items that were suggested for inclusion by the participants in the second round of our modified Delphi study were incorporated into 9 new items in the round 3 survey. Based on round 2 responses, one category—“Identifying and implementing the criteria for inclusion in the registry”—and its items were deemed to be redundant and were removed. This led to a resulting list of 79 items within 10 categories. Participants completed the Qualtrics survey online during the in-person meeting, and results were analyzed and presented in real time. In round 3, participants were asked to vote “yes” or “no” for each item’s inclusion in the planning of the long-term health study. Items were assessed for consensus, and consensus was deemed to be reached if 70% of participants agreed upon the importance of an item. All 79 items met the 70% threshold for consensus and progressed to the round 4 survey.
All 79 items met consensus and were included in round 4. In round 4, participants were asked to rate all items on a 7-point Likert scale of importance in the design and implementation of a long-term population health study after the detonation of an IND, with 1 being unimportant and 7 being extremely important. They were then asked to rate all items on a 7-point Likert scale of difficulty, with 1 being not difficult to accomplish and 7 being extremely difficult to accomplish. Consensus was reported to have been reached if the interquartile range (IQR) of responses was less than or equal to 1. The scores used for the Likert scale represented 1—Not at all important/difficult; 2—Low importance/difficult; 3—Slightly important/difficult; 4—Neutral importance/difficulty; 5—Moderately important/difficult; 6—Very important/difficult; 7—Extremely important/difficult.
Results of round 4 were presented to the group, and items were discussed by the experts during a moderated round-table session. The vote was repeated to generate the final list of items and final consensus that was assessed in the same manner as round 4.
Table 2 displays the consensus status of each item in terms of its importance and completion difficulty.
Results of the modified Delphi round 5
Table 2. Long description
The table consists of seven columns: Item, Median, I Q R, Consensus for Importance, Median, I Q R, Consensus for Difficulty. The header row divides columns into Importance and Difficulty. Each row lists a specific task related to registry planning, such as identifying stakeholders, defining roles, building trust, ensuring equity, integrating epidemiology and biology, and securing resources. For each task, the median score for importance is consistently 7, with I Q R values ranging from 0 to 4, and consensus is mostly Yes. Difficulty scores vary from 2 to 7, with I Q R values from 0 to 4, and consensus is often No. Tasks with highest difficulty medians include securing resources, finances, and housing. Some items, such as defining eligible population or screening criteria, show consensus for both importance and difficulty. The table highlights that stakeholder engagement, data protection, communication, and resource allocation are considered highly important but often difficult, with consensus achieved mainly for importance.
Conclusions
We used a multiphase, expert, modified Delphi process to identify the key components that should be included in a long-term health study after an IND detonation. Given the recent increase in threats involving nuclear weapons, this provides an important foundation for the creation of a registry and refinement of the most important elements. We identified several barriers that require action before the occurrence of a nuclear emergency. We have identified several key findings at the conclusion of our expert meeting and Delphi process analysis.
The group of experts reached consensus that the categories and elements identified by the NASEM report should be included in a long-term health study after the detonation of an IND. It is important to note that some minor rearrangements were made when developing the initial questionnaire using the NASEM report. All items were deemed important irrespective of consensus. The median score for importance ranged between 6 and 7 on a Likert scale of 1 to 7. The group of experts reached consensus regarding the level of importance of 59 out of the 79 original elements and deemed them of very or extremely high importance with a median score of 6-7 out of 7. The 20 items for which SMEs did not reach consensus still received a high importance score of 6 or 7 (Table 2).
The lack of consensus on the level of importance regarding 20 items may be due to several factors, such as heterogeneity of disciplines among the SMEs, causing different perspectives on the importance of individual items. Some items may also be poorly defined, or SMEs may be unfamiliar with them.
The group of experts reached a consensus on the level of difficulty for 13 items. The level of difficulty of these items ranged from a median score of 2 to 7. The items where consensus was not reached varied in their median difficulty score from 2 to 7. Notable mentions due to IQR >1 and difficulty score > than or equal to 5 are listed in Table 3.
Notable mentions
Table 3. Long description
Starting from the top row, the left column lists: identify and secure a sufficient workforce in numbers and disciplines; ensure equity by reaching vulnerable, underserved, marginalized, and non-English speaking communities; determine who will own and manage the data; establish screening for exposure assessment and radioactive contamination with priority for special populations such as pregnant individuals, those with mental health issues, elderly in nursing homes, and children; develop a plan to trace registry participants’ locations over time; develop a plan to follow up with participants who move out of state; identify methods to reach qualifying individuals, especially vulnerable groups including low socioeconomic status and pregnant patients; define a plan to screen for mental illness and mental health issues; implement a long-term communication plan. The right column, from top to bottom, lists: implement a strategy to counter disinformation and misinformation; ensure data protection, security, and access protocols to protect registry participants’ privacy; develop a framework for data governance, access, and security, specifying who can access the data and for what purposes; ensure continuity in management and communication among stakeholders over the expected decades-long duration; determine how stakeholders will manage the registry over many decades; develop a plan for documentation of health events over the long term, including pathology and other reports; obtain current medical history; ensure buy-in from influential community awareness groups; define a plan to provide support for patients who screen positive for disease and require resources such as transportation and financial support.
Moving forward, FSLTT agencies could utilize the findings of our study to develop a plan for long-term monitoring of populations post-nuclear detonation. The elements identified by the NASEM panel were validated by our study. Our group of SMEs reached consensus regarding the importance of a sizable proportion of these elements. Consensus was reached on a small proportion of elements regarding the difficulty of executing them. This may be affected by the limitations inherent in our methodology. When planning for a registry, items where consensus was reached that were deemed of high importance and high or moderate difficulty should be prioritized (Table 4).
Consensus regarding difficulty rating and importance rating is useful to assess using the following matrix
Table 4. Long description
The table consists of one row and three columns, anchored at the top-left. The columns are labeled Difficulty High, Difficulty Moderate, and Difficulty Low, with scores 6 to 7, 4 to 5, and 1 to 3, respectively. The single row is labeled Importance High, with a score of 6 to 7. In the Difficulty High column, the cell lists: define and secure resources to operate the registry in a timely and sustained manner with expectation of the long-term needs; secure resources for creation of registry before an event; ensure adequate finances for short-term and long-term staffing; ensure finances to analyze data in the registry; enact necessary legislation to allow rapid initiation of a rapid response registry. In the Difficulty Moderate column, the cell lists: define screening criteria for minimum requirements for inclusion in the registry before considering additional inclusion criteria; integrate epidemiology, including dose and outcomes, and biology, including samples and measures, into the registry. In the Difficulty Low column, the cell lists: determine the need for a long-term monitoring registry.
The authors acknowledge the limitations present in this study. First, other than our federal partners, this is a Georgia-based exercise. We are aware that many of the experts were from and that application of the process to other states or countries may be limited. Next, although we believe we identified salient experts in the field, the results may be influenced by selection and response bias. Given the use of the Delphi method, the results may not reflect the opinions of all radiation experts. Moreover, this study aimed to examine the consensus of experts on the NAS report and did not include in the initial questionnaire lessons learned from other registries, such as the World Trade Center Health Program or the Radiation Effects Research Foundation (RERF), a U.S.-Japan research organization studying health effects on atomic bomb survivors. Future studies could examine what lessons can be applied to other radiological and nuclear scenarios. Lastly, since the study scenario is an IND detonation incident, some of the findings and conclusions do not apply to other radiological or nuclear emergencies such as the Radiological Dispersal Device scenario.
The consequences of an IND detonation will have devastating acute and long-term consequences that will require a health monitoring program. The planning for such a program will be a daunting task that requires a complex multidisciplinary approach and many resources that will likely need Federal support. This study has succeeded in further exploring the findings of the NASEM panel. The results of this project may serve as a resource for planners who will need to consider dozens of elements to ensure comprehensive, effective, and transparent outcomes.
Authors contribution
Ziad Kazzi and Carol Iddins co-led the conception and design of the study and the identification and invitation of experts. They also co-led the grant application and the Delphi process. They directly participated in the review of the data analysis and drafting of the manuscript.
Emily Kiernan, DO, participated in the design of the study, the review of the data analysis, and the drafting of the manuscript.
Leah Goldstein participated in the design of the study, led the data collection and analysis, and participated in the drafting of the manuscript.
Funding statement
The project was funded by a grant from the Oak Ridge Associated Universities Office of Directed Research and Development Grants.
Competing interests
None.
Open access
