During large-scale public health emergencies, the resource-intensive nature of response efforts often leaves health systems with little capacity to systematically assess in real-time how they can perform more effectively. The magnitude and duration of the COVID-19 pandemic has highlighted the need for responders to adopt a more reflective decision-making approach, especially during a protracted public health emergency.
Prior to the COVID-19 pandemic, and increasingly since, there has been an emphasis on the importance of human-centered, learning health systems.Reference Sheikh, Peters and Agyepong 1 –Reference Sheikh, Agyepong and Jhalani 3 Commentary on health system performance, in both academic literature and elsewhere, has been more strident since COVID-19 emerged. As Hanson and colleaguesReference Hanson, Brikci and Erlangga 4 note health systems need to “invest more and invest better.” Other research notes that methods for evaluating health system pandemic preparedness were too narrow.Reference Khalifa, Abbey and Ayeh 5 Different methods and frameworks for health system learning have been suggested and practiced to demonstrate how to improve health preparedness and response (HEPR). There is likely no one size fits all, each country being sui generis with its contexts and complexities. Nevertheless, experiences elsewhere share knowledge shifting previously “unknown unknowns” into “known unknowns,”Reference Tomé, Gromova and Hatch 6 and thereby removing 1 layer of complexity during health emergencies. This paper explores the types of learning uncovered through COVID-19 inter-action reviews.
The World Health Organization (WHO), with its central role in declaring and coordinating responses to public health emergencies of international concern (PHEIC), recognized this need and sought to address it during the COVID-19 pandemic by developing a new tool for countries. In July 2020, WHO published Guidance for conducting a country COVID-19 Intra-action review (IAR), which was subsequently revised in April 2021. 7 The guidance and accompanying templates and tools provide national and subnational health system actors with a flexible methodology for engaging in a structured multisectoral reflective learning process, with findings from this process intended to immediately inform decision-makers and key stakeholders on concrete steps to improve the ongoing COVID-19 responses.
The central purpose of a COVID-19 IAR is to enable responders to draw immediately actionable lessons from their own assessments of measures taken to address the pandemic in their countries. Building on this, countries can also contribute to peer learning by sharing their experiences with other countries and, finally, WHO and partners can learn from countries to further refine and improve their support for country-led efforts. Examples of these 3 types of learning are presented in Table 1.
Examples of 3 types of learning from COVID-19 intra-action reviews (IARs)
Types of Learning
First Type of Learning: Countries Reflecting and Fine-tuning Their Response Through Learning from IARs
WHO’s guidance for conducting IARs is by design a flexible methodology that can be adapted to countries’ diverse needs and contexts. The IAR coordination team can access 14 publicly available IAR tools to plan and conduct facilitated discussions among key COVID-19 response multi-sectoral stakeholders, synthesize findings, and propose recommended activities to improve the ongoing response. 8 –Reference Talisuna, Iwu and Okeibunor 10 The facilitated discussions are structured around the objective of helping participants collectively formulate their insights regarding COVID-19 challenges, best practices, and recommendations.
IARs are country-led and their outcomes country-owned, with WHO and other external partners available to provide technical support as requested. The IAR coordination team can select 1 or more strategic public health response pillars that the country wishes to review among 13 pillars proposed in the IAR guidance and listed in Table 1. The team can further focus the review by selecting the most pertinent trigger questions among the 600+ trigger questions available as prompts in the facilitated discussions for each selected pillar. Countries are also encouraged to modify or propose additional pillars and trigger questions as appropriate to fully meet the needs of stakeholders in their specific national and subnational contexts. Upon the declaration of the end of the PHEIC by the WHO Director-General or the end of the national emergency by competent authorities in their countries, WHO also encourages countries to consolidate learnings from COVID-19 IARs and conduct COVID-19 after-action reviews to galvanize lessons learned from the pandemic response. 11
Second Type of Learning: Countries Learning from Each Other’s Experiences
In accordance with the advice and temporary recommendations from the International Health Regulations (IHR) Emergency Committee during its fourth, fifth, eleventh, and twelfth meetings, in early 2022, WHO performed an analysis of IAR reports and of stakeholder perceptions of IAR processes and outcomes.
The analysis as described in the published report, a global analysis of COVID-19 intra-action reviews, took a mixed-methods approach using data from 83 IAR reports from 57 countries (48 national IARs, 12 subnational IARs, and 23 IARs focused exclusively on COVID-19 vaccinations), from semistructured key informant interviews, and from consultation with health officials in countries via an online survey. 8 The search strategy and selection criteria for this data was all countries and subnational bodies that chose to complete an IAR and shared with WHO the final IAR report by March 2, 2022. Collectively, 83 reports from 57 countries identified 2556 best practices, 2366 challenges, and 2859 recommendations. The most commonly reviewed pillars in IARs included coordination (69%), risk communication (58%), surveillance (55%), and infection prevention and control (62%) (Table 1). Twenty-three IARs focused entirely on the vaccination pillar. Qualitative analysis identified a number of common themes across IAR findings. For example, many IARs described how countries repurposed existing resources, accelerated the uptake of information technology innovations, and employed whole-of-society approaches to strengthen the planning and implementation of various aspects of their COVID-19 responses.
To further explore the role of IARs in strengthening COVID-19 responses, WHO conducted 27 key informant interviews and 29 online surveys with individuals involved in conducting IARs. Both of these data sources indicated that a major perceived value of IARs was the provision of a forum where responders from different sectors could share insights and collectively identify strategies for meeting key challenges during the ongoing response. This approach both yielded immediate benefits in terms of well-informed IAR recommendations and promoted greater long-term multisectoral collaboration among COVID-19 responders. The timing of an IAR was found to be important, with some sources of information describing how IARs had been carried out immediately before the development or revision of strategic plans so that recommendations could be integrated into those plans. As intended by WHO, countries made use of IARs in different ways. Some countries found it beneficial to conduct multiple IARs over time when reflection and re-strategizing of the response were required. The benefits of conducting subnational IARs were emphasized, with sources noting that subnational governments are able to take prompt action to adjust their response strategies based on IAR recommendations.
WHO has strongly encouraged countries to share their IAR experiences and findings for the benefit of informing COVID-19 decision-making in countries with similar situations. Opportunities for peer learning take multiple forms. In November 2020, for example, during a press conference on COVID-19 hosted by the WHO Director-General, the Ministers of Health of Indonesia, Thailand, and South Africa, 3 of the first countries to conduct IARs, were invited to discuss how their countries had drawn on IAR findings to make changes in their COVID-19 responses. Indonesian stakeholders later published a peer-reviewed article describing their first IAR process and the recommendations that it yielded.Reference Wulandari, Hastuti and Setiawaty 12 In May 2021, WHO hosted an online global consultation in which country stakeholders and partners examined the role of IARs in COVID-19 responses, with findings made publicly available in a meeting report. 13 In the second half of 2021, while COVID-19 vaccination campaigns were being rapidly scaled up worldwide, WHO hosted a series of 5 online clinics for countries to share learnings from COVID-19 vaccination IARs that they had conducted, with participants attending from more than 100 countries.
A major resource to support peer learning is the aforementioned analysis of 83 IAR reports that were shared with WHO. 8 The report serves as a compendium of country-defined insights, innovations, and future considerations that can contribute to decision-making in other settings in accordance with COVID-19 responders’ needs and priorities.
Third Type of Learning: Partners Learning from Responders to Improve Support
Besides encouraging countries to conduct IARs to promote reflective learning, WHO has engaged in its own reflective learning process regarding how to better support countries in their preparation for and management of public health emergencies. Requests from countries led WHO to create an online IAR training course with over 9000 learners enrolled and 3000 awarded a certificate of completion. Feedback from countries and reviews of IAR reports have informed the revision and update of the WHO IAR guidance and tools, including changes that support the more flexible use of IARs. WHO also works closely with partners to align their efforts with countries’ IAR-related needs for technical and financial support. These include partners from other intergovernmental agencies, bilateral and multilateral aid agencies, academia, and nongovernmental organizations.
Discussion
The Future of Reflective Learning in Public Health Emergencies
The experience of COVID-19 IARs supports WHO’s continued emphasis on the value of reflective learning in responding to COVID-19 and other concurrent and future public health emergencies. WHO’s efforts in this area recognize multiple aspects of reflective learning while emphasizing the centrality of country-led and whole-of-society responses that are defined by the needs of communities (Figure 1). Two aspects require particular attention from WHO and other stakeholders moving forward. First, as described below, is the institutionalization of continual reflective learning as a core element of multisectoral public HEPR. Second is the translation of the outputs from reflective learning into specific changes to create positive impact on health and social outcomes. It is important to note that when implementing any new intervention, a participatory approach that engages the community is critical to ensure interventions are appropriate and accepted by the community to optimize effectiveness. This then forms the fourth type of learning which brings reflective learning full circle, adopting the whole-of-society approach.
Multiple aspects of reflective learning during and after PHEs.
Reflective learning as an ongoing process
Health emergencies and natural disasters prior to the COVID-19 pandemic provide other existing examples of resource-intensive emergency responses in the past. Notable examples of such events include the 2003 severe acute respiratory syndrome (SARS) epidemic, the 2004 Indian Ocean tsunami, and the 2009 H1N1 influenza pandemic. Some countries have translated learnings from previous health emergencies into institutional changes that enhance ongoing HEPR. Often, however, following emergencies, countries can also easily forget and move onto other priority issues, falling into a panic-and-neglect cycle. In the aftermath of the 2014 Ebola virus outbreak, leading experts issued warnings about insufficient emergency response capacities and called for greater investment in health security.Reference Gates 14 , 15 Yet, the COVID-19 pandemic still revealed unforeseen challenges in health and governance systems around the world. 16
The drivers of the panic-and-neglect cycle are complex, and breaking out of this cycle will require a paradigm change at the highest levels of national and global governance. While numerous sectors of government and society are engaged in responding to large-scale public health emergencies, the health sector has a unique role in coordinating multisectoral efforts to shift countries to a state of safeguarding ongoing HEPR. By institutionalizing reflective learning processes facilitated by health sector actors, policymakers can help to both guide improvements in specific emergency response practices and instill a culture of continuous improvement across all relevant sectors. Such an approach would facilitate agile responses to future health emergencies, especially those that present novel challenges.
Operational research is needed to explore how to encourage greater uptake of ongoing learning practices in HEPR and in broader health system strengthening efforts. The public health field can learn from how reflective learning is undertaken in other contexts. For example, the US Army pioneered the use of after-action reviews to enable soldiers to systematically reflect on what happened, why it happened, and how to improve performance going forward – an approach widely studied and adopted in other fields, including WHO’s after-action reviews.Reference Sawyer and Deering 17 Additionally, in the clinical setting, the role of reflective learning also has been extensively explored in simulation training to improve the skills of health care providers.Reference Abulebda, Auerbach and Limaiem 18
When undertaking learning practices in HEPR, a whole-of-society approach is required that acknowledges different stakeholders. COVID-19 has had a disproportionate impact on vulnerable and marginalized populations, not only in terms of illness and death but also in terms of economic security and access to education. In the IAR global analysis, 8 no IAR specifically reviewed the public health response pillar focusing on vulnerable and marginalized populations. By including representatives of vulnerable and marginalized populations as participants in IARs, countries can also obtain valuable insights on approaches to make HEPR measures more inclusive.
Acting on the outputs of reflective learning
A goal of reflective learning in relation to HEPR is to inform immediate and longer-term policy changes that will strengthen response mechanisms. If reflective learning is to play a larger role in ongoing global HEPR, then countries and partners must recognize and address the challenges associated with moving from reflective learning-based recommendations to action. Evidence about “what works” is often not translated into public policy; rather there are cases of policy-making at the national and subnational levels not being aligned with evidence-based recommendations.Reference Cairney, Oliver and Wellstead 19
It is therefore important to identify strategies to facilitate the translation of findings from reflective learning activities such as IARs into policies that will enhance national and subnational HEPR. Fostering the political will to have reflective learning-based recommendations implemented is an undertaking that should be planned and carried out alongside the reflective learning activities rather than at the end of the process. Some countries have engaged high-level government officials in chairing IARs, increasing the likelihood that the value of the IAR outputs would be recognized by senior policymakers. Involving communities and civil society in reflective learning activities may also encourage the uptake of recommendations. Finally, developing formal quantifiable targets and accountability mechanisms for monitoring and publicly reporting on the implementation of recommendations is also important.Reference Mayigane, Vedrasco and Chungong 20
Beyond policy changes, there is the further work of monitoring whether policies are implemented and what impact these policies have on emergency response processes and outcomes. Communities have a key role to play as implementation partners, and their engagement can contribute to the effective translation of policy measures into nuanced public health initiatives that speak to the needs of diverse populations, including vulnerable and marginalized populations. Communities also can make valuable contributions to monitoring implementation as well as outcomes.
Conclusion
The global community has undergone a remarkable learning experience in response to COVID-19. It is imperative to integrate this new knowledge and new ways of working into emergency preparedness and response planning. Collective action is needed to make a whole-of-society paradigm shift from the panic-and-neglect cycle to an approach that embodies continuous reflective learning as the foundation of strengthening preparedness. Such a shift should encompass not only learning from previous experiences but also learning from peers. External partners supporting a country’s HEPR should recognize the importance of supporting reflective learning activities as well as engaging in reflective learning to enhance their own contributions.
The lessons from COVID-19 IARs suggests that integrating IAR practices into public health emergency response measures can contribute to building the culture of reflective learning that is needed to address emergencies more effectively. WHO calls on all stakeholders to enhance the role of country tailored IARs and similar adapted reflective learning processes during and after public health emergencies, with particular attention to collaborative whole-of society approaches in order to use IARs as a tool to achieve more equitable health outcomes.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/dmp.2025.116.
Data availability statement
Data used for this article are available in: World Health Organization. (2022). A global analysis of COVID-19 intra-action reviews: reflecting on, adjusting, and improving country emergency preparedness and response during a pandemic. World Health Organization. https://apps.who.int/iris/handle/10665/365488.
Competing interests
None declared.
Open access
