Introduction
Pneumococcal disease is major communicable cause of morbidity and mortality, leading to approximately 1.2 million annual deaths worldwide in the absence of vaccination (Reference Drijkoningen and Rohde1;Reference Chen, Cervero Liceras and Flasche2). Infection leads to various conditions: noninvasive pneumococcal diseases (sinusitis, acute otitis media, and community-acquired pneumonia); or, invasive pneumococcal diseases (meningitis and bacteremia) (Reference Drijkoningen and Rohde1). While everyone is susceptible to infection, very young children, older adults, and those with certain comorbidities face higher risks (Reference Drijkoningen and Rohde1).
Several vaccines for pneumococcal disease have been introduced, including those suitable for infants and children. These can be divided into conjugate and polysaccharide vaccines (Reference Berical, Harris, Dela Cruz and Possick3). The first commonly adopted vaccine was the 23-valent pneumococcal polysaccharide vaccine (PPSV23, Pneumovax 23®). It is not immunogenic in young children (Reference Bogaert, Hermans, Adrian, Rümke and De Groot4). The more recently introduced 7-valent (serotypes 4, 6B, 9 V, 14, 18C, 19F and 23F) pneumococcal conjugate vaccine (PCV7, Prevenar®) is effective in infants and children (Reference Berical, Harris, Dela Cruz and Possick3). Most recently, the 10-valent and 13-valent pneumococcal conjugate vaccines (PCV10, Synflorix™ and PCV13, Prevenar 13®, respectively) were developed. PCV10 includes the 7-valent vaccine serotypes plus the 1, 5, and 7F serotypes, whereas PCV13 additionally includes serotypes 3, 6A, and 19A (Reference Berical, Harris, Dela Cruz and Possick3;Reference Principi and Esposito5).
The availability of multiple vaccinations has implications when conducting a health economic evaluation to find the most cost-effective prevention strategy. In order to select the most appropriate health policy, a central premise of cost-effectiveness analysis (CEA) methods is that analyses include all feasible alternatives (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance6). Another important, long-recognized methodological principle within CEA is that strategies are compared to each other incrementally, meaning that strategies are compared to the next best alternative intervention, as opposed to no intervention at all (Reference Gold, Siegel, Russell and Weinstein7). Such comparisons are required for the correct estimation of incremental cost-effectiveness ratios (ICERs). Accordingly, the inclusion of all relevant strategies is important both not only for the assessment of all relevant candidate strategies, but also for the inclusion of sufficient comparator strategies against which to compare candidate strategies. The omission of sufficient comparator strategies can result in biased ICERs, making interventions appear more cost-effective than they are.
Prior research has raised concerns about inappropriate application of CEA methods within the pneumococcal vaccination literature. A systematic review of CEAs of pneumococcal interventions in low- and middle-income countries (LMICs) noted that many did not consider all potential strategies (no vaccination, PCV7, PCV10, or PCV13) and sometimes failed to make appropriate incremental comparisons (Reference Saokaew, Rayanakorn, Wu and Chaiyakunapruk8). It was shown that most studies reported the cost-effectiveness (CE) of vaccines versus no-vaccination, rather than using a correct incremental approach. These observations are consistent with findings of a recent research on pneumococcal vaccination for children in Asian countries, which also found omission of relevant strategies (Reference Zakiyah, Insani, Suwantika, van der Schans and Postma9).
Despite a relatively large body of CEA evidence regarding infant pneumococcal vaccination, there is no clear consensus on which vaccination is optimally cost-effective. These conflicting conclusions regarding CE leaves decision makers facing uncertainty what strategy to adopt. We are assessing to what extent the absence of consensus among CEAs of infant pneumococcal vaccination is potentially due to inappropriate strategy comparisons. In contrast to previous research which focused on LMICs (Reference Saokaew, Rayanakorn, Wu and Chaiyakunapruk8;Reference Zakiyah, Insani, Suwantika, van der Schans and Postma9), our analysis reviews the evidence of economic evaluations of infant pneumococcal vaccination in high-income countries (HICs). To the best of our knowledge, this is the first study to specifically assess questions of strategy comparison in this context. We conducted a systematic review followed by a critical assessment of the recovered studies to gain insight into the reliability of policy recommendations.
Methods
Search strategy
We searched the PubMed, Scopus, Embase, and Web of Science electronic databases for health economic evaluations of infant pneumococcal vaccination. Our search strings are presented in Appendix I of the Supplementary Material. The database searches were restricted to scientific journal articles published since 2010. This was the year after the introduction of PCV10, from when the choice between multiple vaccination strategies became relevant. The search was limited to articles published in English and was conducted on 15 June 2022.
Selection process
As a first step, we screened and selected candidate articles based on title and abstract. Our inclusion and exclusion criteria can be understood within the population, intervention, comparator, outcomes, and study (PICOS) framework as follows (Reference Methley, Campbell, Chew-Graham, McNally and Cheraghi-Sohi10). The population of interest are the general infant population from HICs, as defined by the World Bank classification (11). We excluded non-infant populations and specific groups of elevated risk. The interventions of interest are vaccination against pneumococcal disease of all vaccine types. We excluded articles if they were not related to interventions against pneumococcal disease. The question of what strategies were used as comparators is embedded in our research question and so did not constitute part of our inclusion and exclusion criteria. Regarding outcomes and study type, the inclusion criteria were CEAs reporting outcomes in terms of cost per quality-adjusted life-years (QALYs) or life-years gained (LYG). As a corollary, we excluded study designs that were not health CEAs; which may include reviews, methodological studies, and cost of illness studies. As such, we excluded studies reporting health economic outcomes other QALY or LYG, such as the cost per case prevented.
Data collection and analysis
As our review was focused on the specific question of whether CEAs had included sufficient intervention strategies and compared them appropriately, we did not apply a general quality appraisal tool such as the CHEERS or Drummond and Jefferson checklists (Reference Husereau, Drummond and Augustovski12;Reference Drummond and Jefferson13). Instead, we used our own critical appraisal framework to assess the reliability of each study’s policy recommendations with respect to three criteria: (i) did the analysis report all costs and effects transparently; (ii) did it include all relevant comparator strategies; and (iii) did it perform appropriate incremental comparisons between the costs and effects of the alternative strategies? We explain these in greater detail in the paragraphs below. We considered policy recommendations to be reliable once all three criteria were met. In addition, we recorded the funding reported by each study. Finally, to assess the heterogeneity of identified CEAs, we compiled the costs and effects of all studies in summary CE planes.
Transparency of reporting
The premise of this assessment is that estimated costs and health effects should be reported with sufficient clarity to permit replication of the reported CE estimates. We assessed if studies reported estimates numerically for both costs and effects over all simulated strategies. When the costs and effects were reported, we determined if it was possible to reproduce the reported ICERs.
Inclusion of relevant comparators
The relevant strategies in the context of infant pneumococcal vaccination currently are the no-vaccination strategy, and immunization with PCV7, PCV10, or PCV13. As the vaccines were introduced at different points in time, what vaccines constitute relevant comparators at any prior point in time depends on the year of licensing (Appendix II of the Supplementary Material). We considered the omission of a vaccine type in a study published 2 years or more after the type was licensed to be the exclusion of a relevant comparator. The omission of available intervention strategies might also be justified in certain circumstances. For example, if a strategy has been clearly demonstrated as inefficient in prior studies and cited as such, then it may be justified for omission in future analyses. Accordingly, we examined whether a rationale for the exclusion of an omitted strategy was given within the reviewed studies.
Appropriateness of incremental comparisons
We assessed the appropriateness of incremental comparisons within the published estimates from each CEA. A correct ICER estimate is understood to be the incremental comparison of the differences in costs and effects between strategies that lie on the efficient frontier, with a given intervention being compared to the next most effective efficient strategy (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance6). We evaluated if the reported ICERs were based on the accepted understanding of ICERs or were based on other comparisons, such as those that simply compared to no vaccination or compared to another vaccine strategy that was not the adjacent strategy on the efficient frontier.
Heterogeneity of estimates
We conducted an assessment to determine how heterogeneous the cost and health effects estimates were over the simulated strategies between the reviewed studies. We plotted the CEA results in a series of summary CE planes. These summary CE planes do not report numerical costs and health effects, but give a qualitative summary of whether a given strategy was dominant, dominated, cost-effective or not when compared to a specific comparator. We used a series of these CE planes to summarize the results of different strategy comparisons across multiple studies. Dominant strategies are those with greater effects and lower costs than the reference strategy. These are presented within the south east quadrant of the summary CE planes. Conversely, strategies that are dominated, both more expensive and less effective, are depicted in the northwest quadrant. The north east and south west quadrants are divided between those interventions that are cost-effective or not. We determined the CE with respect to the CE threshold cited within each individual CEA.
Sponsorship association
We also examined the association between reported research sponsorship and the comparator choice and study conclusions. We categorized the data to facilitate quantitative analysis: this included binary categories regarding whether PCV7 was included; whether the no-vaccination strategy was included; whether PCV13 or PCV10 (latest vaccines) was recommended when compared; whether the policy recommendation was considered reliable according to the three criteria described above; and whether the research was reported as funded by a pharmaceutical company. We analyzed the data using STATA Software for Statistics and Data Science (Version 16.1). We applied Fisher’s exact test for the evaluation of categorical data, which is accurate when using a small sample for analysis (Reference Connelly14). A p-value lower than .05 was considered statistically significant.
Results
Description of identified CEAs
As presented in Figure 1, the databases Scopus, Embase, Web of Science, and PubMed returned 253, 169, 151, and 148 articles, respectively. Following pooling and removal of duplicates, we retrieved unique 266 articles for title and abstract screening. Our first screen identified thirty-four studies for full-text screening. Subsequently, we removed three studies from analysis as they differed in study type (retrospective cohorts) (Reference Newall, Reyes and McIntyre15–Reference Perdrizet, Lai, Williams, Struwig and Wasserman17). We excluded another two studies as they focused on dosing schedules for one pneumococcal vaccine alone (Reference Stoecker, Hampton and Moore18;Reference Stoecker, Hampton and Link-Gelles19). We extracted data from the remaining studies, including the publication year, national setting, vaccine intervention, chosen comparators, sponsorship, cost and effect estimates, reported ICERs, and related policy recommendations. In total, we retained twenty-nine CEAs of infant pneumococcal vaccination for further analysis (Reference Ansaldi, Pugh and Amicizia20–Reference Hoshi, Kondo and Okubo48).
Flow diagram of review process.
An overview of the twenty-nine eligible studies is presented in Table 1. The date of publication ranged from 2010 to 2020. The CEAs were conducted in twenty different countries. The number of intervention options varied between studies, ranging from two to four strategies (including no vaccination). The no-vaccination strategy, PCV7, PCV10, and PCV13 vaccines were included in 13, 11, 21, and 26 studies, respectively. Regarding sponsorship, twenty-two studies reported funding from pharmaceutical companies. Of these sponsored studies, eleven reported funding from Pfizer (formerly Wyeth), which manufactures the PCV7 and PCV13 vaccines. Eleven studies reported sponsorship by GlaxoSmithKline (GSK), which manufactures PCV10. Most CEAs included QALYs as a health outcome measure (n = 25, 86 percent).
Overview identified economic evaluations of childhood pneumococcal vaccination strategies (n = 29)
Abbreviations: DKK, Danish krone; GSK, GlaxoSmithKline; HKD, Hong Kong dollar; JPY, Japanese yen; KW, Korean Republic won; NOK, Norwegian krone; NT$, New Taiwan dollar; NV, no vaccination; SEK, Swedish krona.
Table 1 shows there is a marked lack of consensus regarding which vaccine is optimally cost-effective. To illustrate, of four CEAs that included all strategies, two reported that PCV10 was most cost-effective (Reference Robberstad, Frostad, Akselsen, Kværner and Berstad29;Reference Castiglia, Pradelli and Castagna42), while another two concluded that PCV13 had more cost-effective outcomes (Reference Rozenbaum, Sanders and Van Hoek30;Reference Tyo, Rosen and Zeng35). The majority of identified optimal strategies were dominant in that they are estimated to be more effective and less costly than alternative strategies (n = 18, 62 percent). Of the twenty-two studies that reported industry sponsorship, twenty-one concluded that the sponsor’s product was the most cost-effective.
Transparency of reporting
Three studies reported incomplete estimates (Reference Newall, Creighton, Philp, Wood and MacIntyre27;Reference Rozenbaum, Sanders and Van Hoek30;Reference Tyo, Rosen and Zeng35). Of these, one failed to report the total net costs of immunization programs (or constituent cost components from which they could be derived) (Reference Rozenbaum, Sanders and Van Hoek30). The second study did not report the total net costs, and although it report the direct and indirect savings of interventions, it excluded the vaccination costs, again precluding the derivation of total net costs (Reference Tyo, Rosen and Zeng35). The third study did not clearly report all costs and health outcomes for each vaccination strategy (Reference Newall, Creighton, Philp, Wood and MacIntyre27). Consequently, we could not reproduce the complete CE plane and ICERs for these studies.
Inclusion of relevant comparators
Of the twenty-nine identified CEAs, only five considered all possible strategies (no-vaccination, PCV7, PCV10, and PCV13; Table 1) (Reference Newall, Creighton, Philp, Wood and MacIntyre27;Reference Robberstad, Frostad, Akselsen, Kværner and Berstad29;Reference Rozenbaum, Sanders and Van Hoek30;Reference Tyo, Rosen and Zeng35;Reference Castiglia, Pradelli and Castagna42). From the studies that omitted one or more strategies, twenty excluded either the no-vaccination or PCV7 strategy (Reference Ansaldi, Pugh and Amicizia20–Reference Lu, Chung and Huang26;Reference Pugh, Wasserman and Moffatt28;Reference Rubin, McGarry and Strutton31–Reference Shiragami, Mizukami and Leeuwenkamp33;Reference Vemer and Postma36–Reference Wilson, Wasserman and Jadavi38;Reference Zhang, Leeuwenkamp, Oh, Lee and Kim40;Reference Lee, Chia Wu, Topachevskyi, Delgleize and DeAntonio41;Reference Delgleize, Leeuwenkamp, Theodorou and Van De Velde43–Reference Gouveia, Fiorentino, Jesus, Costa and Borges46). We identified four PCV10 omissions (Reference Blank and Szucs21;Reference Wu, Rinaldi, Huang, Chang and Chang39;Reference Gouveia, Fiorentino, Jesus, Costa and Borges46;Reference Hoshi, Kondo and Okubo47) and one PCV13 omission that were not justified by a supporting rationale for exclusion (Reference Wu, Rinaldi, Huang, Chang and Chang39). Cases of justified exclusions included instances where vaccines were not licensed at the time the study was conducted; the exclusion of PCV10 and PCV13 was considered justified in two studies on this basis (Reference Sohn, Suh, Jang and Kwon34;Reference Hoshi, Kondo and Okubo48). More broadly, we interpreted the omission of the no-vaccination strategy as justified, as most of the identified studies showed that pneumococcal vaccines are cost-effective in HICs. Nevertheless, we interpreted the omission of PCV7 as unjustified, as there is limited evidence that this strategy is dominated by other PCVs. Moreover, there were five CEAs of either PCV10 or PCV13 that omitted the other type (Reference Blank and Szucs21;Reference Rubin, McGarry and Strutton31;Reference Díez-Domingo, Ridao-López and Gutiérrez-Gimeno44;Reference Gouveia, Fiorentino, Jesus, Costa and Borges46;Reference Hoshi, Kondo and Okubo47). As there was no consensus on the CE of PCV10 and PCV13, the exclusion of either one was considered not justified.
Appropriateness of incremental comparisons
There were eighteen studies (62 percent) that found one of pneumococcal vaccine types to dominate the other strategies and so had no ICERs to report (Reference Blank and Szucs21–Reference Knerer, Ismaila and Pearce24;Reference Lu, Chung and Huang26;Reference Pugh, Wasserman and Moffatt28;Reference Robberstad, Frostad, Akselsen, Kværner and Berstad29;Reference Rubin, McGarry and Strutton31–Reference Shiragami, Mizukami and Leeuwenkamp33;Reference Wilson, Wasserman and Jadavi38;Reference Zhang, Leeuwenkamp, Oh, Lee and Kim40–Reference Delgleize, Leeuwenkamp, Theodorou and Van De Velde43;Reference Earnshaw, McDade, Zanotti, Farkouh and Strutton45–Reference Hoshi, Kondo and Okubo47). Within the other eleven studies, the ICER estimates were questionable in four evaluations (36 percent) (Reference Kuhlmann and von der Schulenburg25;Reference Newall, Creighton, Philp, Wood and MacIntyre27;Reference Rozenbaum, Sanders and Van Hoek30;Reference Tyo, Rosen and Zeng35). Two compared candidate strategies to the no-vaccination strategy alone rather than making incremental comparisons to other vaccine strategies (Reference Rozenbaum, Sanders and Van Hoek30;Reference Tyo, Rosen and Zeng35). Similarly, another study compared all possible vaccination strategies to the no-vaccination and PCV7 strategies, the latter of which was current practice (Reference Newall, Creighton, Philp, Wood and MacIntyre27). Finally, one study compared both PCV10 and PCV13 to PCV7 rather than conducting a standard incremental analysis that would have shown PCV10 to be subject to extended dominance (Reference Kuhlmann and von der Schulenburg25).
Overall appropriateness of comparisons
Table 2 provides a summary of the three assessment criteria of clear reporting of costs and effects, relevant comparator inclusion and appropriate incremental comparisons as applied to the reviewed studies. It also provides our summary judgment regarding the appropriateness of the comparisons based on these three criteria. Of the twenty-nine CEAs, four (14 percent) presented analyses that were judged to be fully appropriate (Reference Robberstad, Frostad, Akselsen, Kværner and Berstad29;Reference Sohn, Suh, Jang and Kwon34;Reference Castiglia, Pradelli and Castagna42;Reference Hoshi, Kondo and Okubo48). The remaining evaluations were found to contain shortcomings, including twenty-one (72 percent) that omitted one or more relevant comparators without justification, leading our conclusion that their comparisons were not fully appropriate (Reference Ansaldi, Pugh and Amicizia20–Reference Knerer, Ismaila and Pearce24;Reference Lu, Chung and Huang26;Reference Pugh, Wasserman and Moffatt28;Reference Rubin, McGarry and Strutton31–Reference Shiragami, Mizukami and Leeuwenkamp33;Reference Vemer and Postma36–Reference Lee, Chia Wu, Topachevskyi, Delgleize and DeAntonio41;Reference Delgleize, Leeuwenkamp, Theodorou and Van De Velde43–Reference Hoshi, Kondo and Okubo47). In addition, four failed to report cost and effects adequately and/or make appropriate incremental analyses, leading to our conclusion that their comparisons were not appropriate (Reference Kuhlmann and von der Schulenburg25;Reference Newall, Creighton, Philp, Wood and MacIntyre27;Reference Rozenbaum, Sanders and Van Hoek30;Reference Tyo, Rosen and Zeng35).
Summary critical appraisal of cost-effectiveness analyses (n = 29)
Abbreviation: NA, not applicable.
Heterogeneity of results
Figure 2 presents the qualitative summary CE planes for each pair of vaccine strategies according to each study as numbered in Table 1. Each number qualitatively records the relative CE of each of the strategy comparisons as reported by the respective studies. For example, Figure 2A shows all the studies that compared PCV7 and no vaccination found PCV7 to be effective. Studies 1 and 3 found PCV7 to be net costly and cost-ineffective, 8, 12, 13 and 16 found it to be net costly and cost-effective, while 7 found it to be cost-saving and thus dominant. Figure 2F reveals largely contradictory findings of dominance among those studies comparing PCV10 to PCV13. Almost all studies funded by Pfizer comparing PCV13 to PCV10, concluded that PCV13 is dominant. Conversely, most studies funded by GSK found that PCV13 is dominated by PCV10. Not all studies favor their own vaccine, one study found the rival’s product is more cost-effective (Reference Vemer and Postma36). Nonetheless, the summary CE planes show considerable heterogeneity regarding which strategy is estimated the most cost-effective.
Overview cost-effectiveness comparison of pneumococcal conjugate vaccines. The qualitative summary planes are shown for each pair of vaccine strategies: PCV7 and no-vaccination (2A), PCV10 and no-vaccination (2B), PCV13 and no-vaccination (3C), PCV13 and PCV7 (3D), PCV10 and PCV7 (3E), PCV13 and PCV10 (3F). GSK, GlaxoSmithKline; PCV7, 7-valent pneumococcal conjugate vaccine; PCV10, 10-valent pneumococcal conjugate vaccine; PCV13, 13-valent pneumococcal conjugate vaccine.
Sponsorship association
Our results indicate that a greater proportion of industry-sponsored studies excluded PCV7 (n = 16/22, 73 percent) relative to independent studies (n = 2/7, 29 percent), however, this is not a statistically significant difference (p = .05). Nevertheless, we did find a statistically significant association between industry sponsorship and the exclusion of the no-vaccination strategy (p = .02), with sponsored studies excluding the no-vaccination strategy more often (n = 16/22; 73 percent) than independent research (n = 1/7; 14 percent). We found no association between sponsorship and the three assessment criteria (transparency of reporting, inclusion of relevant comparators, appropriateness of incremental comparisons; p = .58; p = .65; p = .24). We did find a highly statistically significant association between industry sponsorship and policy recommendations in the subset of studies that include both PCV10 and PCV13. All the economic evaluations sponsored by Pfizer concluded that PCV13 was the optimal strategy (n = 11/11; 100 percent), whereas nine out of ten studies (90 percent) sponsored by GSK found that PCV10 was cost-effective (p < .001).
Discussion
Our analysis appraises the reliability of the policy recommendations of CEAs of infant pneumococcal immunization programs by examining the specific methodological issue of the comparisons made between strategies. Ostensibly, there is a large literature on pneumococcal vaccination and one might expect clear policy guidance to emerge. In practice, the studies are heterogeneous regarding the optimal policy choice. This appears, in part, due to the way strategies have been compared. We find very few studies have included a complete set of strategies and compared them appropriately.
Our findings show that only four (14 percent) of the twenty-nine CEAs adhered to economic evaluation guidelines regarding strategy choice, incremental comparisons performed, and outcomes reporting (Reference Robberstad, Frostad, Akselsen, Kværner and Berstad29;Reference Sohn, Suh, Jang and Kwon34;Reference Castiglia, Pradelli and Castagna42;Reference Hoshi, Kondo and Okubo48). This indicates there are disconcerting shortcomings regarding the reliability of CEA evidence regarding infant pneumococcal vaccination. The omission of relevant comparators is a prevalent problem in the reviewed studies. Of the twenty-nine CEAs, only five (17 percent) included all relevant intervention strategies (Reference Newall, Creighton, Philp, Wood and MacIntyre27;Reference Robberstad, Frostad, Akselsen, Kværner and Berstad29;Reference Rozenbaum, Sanders and Van Hoek30;Reference Tyo, Rosen and Zeng35;Reference Castiglia, Pradelli and Castagna42). The omission of a given strategy might be justified if all prior analyses had shown it clearly to be dominated. Our study showed that there was no clear consensus within the existing literature that any strategy is always dominated, therefore the omission of any of the strategies examined is unjustified. Arguably, where the rationale for the exclusion of strategies is debatable, the presumption should be to include rather than omit strategies, thus avoiding concerns of bias stemming from incomplete comparator inclusion.
We also identified errors in the incremental comparisons of the identified CEAs. Eleven studies had relevant ICERs to report, of which four studies performed incorrect incremental comparisons. We also found simple failures to report outcomes completely, making it difficult to assess the appropriateness of comparisons in those cases, although this occurred only in a minority of studies. Additionally, some cases required additional calculations to determine the total costs of the intervention strategy. This finding is consistent with previous studies from Kauf (Reference Kauf49) who also found poor reporting in CEAs for meningococcal vaccination in developed countries. This indicates that poor reporting persists as a basic problem. This is notable as transparency and reproducibility are fundamental pillars of scientific research. Moreover, we identified shortcomings within infant pneumococcal vaccination CEAs in a HIC context. Earlier studies have identified similar problems in other contexts. Both Saokaew et al. (Reference Saokaew, Rayanakorn, Wu and Chaiyakunapruk8) and Zakiyah et al. (Reference Zakiyah, Insani, Suwantika, van der Schans and Postma9) found limitations in comparator inclusion in published CEAs for childhood vaccinations in LMICs. Furthermore, our findings are also consistent with Kauf (Reference Kauf49), which found that some CEAs failed to recognize the (in)efficiency of an intervention strategy.
Previous research on pneumococcal vaccination has also suggested that research sponsorship influences conclusions regarding the CE of immunization strategies (Reference Saokaew, Rayanakorn, Wu and Chaiyakunapruk8). Our research echoes this, as a clear association between the most optimal vaccination strategy and sponsorship was identified. We found a statistically significant association between funding and no-vaccination strategy omission, as well as conclusions regarding the most optimal strategy. These results are also confirmed by a recent study from Zakiyah et al. (Reference Zakiyah, Insani, Suwantika, van der Schans and Postma9), who reviewed pneumococcal evaluations in Asian contexts. They observed that many studies poorly described the role of the funder. As research sponsorship seems to have a significant impact on critical assumptions in CEA, and therefore CE conclusions, there is a need for more independent research.
Although methodological deviations within CEA is not a novel finding in general, the persistence of these inconsistencies in recent literature is concerning. Our analysis confirms that greater effort is still required to enhance methodological standards in CEA. Journal editors and reviewers need to familiarize themselves with the methods literature in order to adequately review submitted studies. Similarly, policy makers should be aware of the presence of inappropriate analyses and understand the correct method for health economic evaluations.
Better adherence to long-established methodological recommendations is required. To prevent overestimation of the efficiency of novel intervention strategies, all available strategies should be evaluated to recognize relevant comparators for evaluation. To this end, we recommend that all earlier developed strategies as well as the non-intervention strategy are considered for analysis. In addition, when an intervention is omitted due to the assumption it will be dominated, then this assumption should be made explicit to enhance the transparency and reliability of the research. Finally, in order to increase the transparency of reporting, the economic evaluation results should present the total estimated costs for each of the intervention options.
Our review has both limitations and strengths. There may be additional reasons for variation in costs and effects which have not been investigated in the present study including different disease prevalence, model types, cost assumptions, and other methodological considerations. Although we employed published methodological guidelines to inform our assessment of the reliability of CEAs, our categorization of the adequacy of comparisons naturally remains somewhat subjective. Additionally, the small number of observations formed a limitation for quantitative analysis. Nonetheless, our use of quantitative analyses has made the conclusions less subjective in nature.
Conclusion
We identified frequent basic methodological shortcomings in CEAs of infant strategies against pneumococcal disease in HICs. The omission of potentially relevant vaccination strategies and a failure to make the correct incremental comparisons between strategies are common. We also found considerable heterogeneity in the study findings. This is apparently closely associated with study sponsorship. It is hoped our work will renew awareness of established methodological guidelines to improve continued academic research and prevent adoption of inefficient health policies. Our findings also highlight the need for more independently funded research to provide policy clarity.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462323000351.
Acknowledgement
We thank Dr Tony Newall for his valuable comments that helped us shape the direction of our research.
Funding statement
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Competing interest
The authors declare none.
Open access
