Background
Health economic evaluations are valuable tools for guiding policymakers in allocating scarce healthcare resources. Quality assessment is important for maintaining methodological standards to obtain valid and reliable results (Reference Ben, van Dongen and El Alili1). It enhances study transparency and reproducibility and facilitates appropriate resource allocation within healthcare systems (Reference Husereau, Drummond and Augustovski2). Reviews conducted since 2015 on health economic evaluations for medical devices have highlighted concerns about the quality of these studies (Reference Martelli, Devaux and van den Brink3;Reference Fontenay, Catarino and Snoussi4) indicating that they are often insufficient to address the important features of medical devices (Reference Craig, Carr and Hutton5;6). The value, accessibility, and affordability of new medical devices are critical considerations for patients, healthcare providers, and health systems, alongside their effectiveness and safety. The cost-effectiveness of these technologies and the most appropriate ways to evaluate them are of increasing importance (Reference Craig, Carr and Hutton5).
The economic evaluation of medical devices differs from pharmaceuticals in several important ways (Reference Drummond, Griffin and Tarricone7–Reference Tarricone, Torbica and Drummond10). Key considerations include limited clinical and economic evidence (Reference Craig, Carr and Hutton5;Reference Sorenson, Tarricone, Siebert and Drummond11;Reference Zwanziger, Hall and Dick12), learning curve effects (Reference Craig, Carr and Hutton5;Reference Drummond, Griffin and Tarricone7), organizational impact (Reference Craig, Carr and Hutton5;Reference Drummond, Griffin and Tarricone7;Reference Sorenson, Tarricone, Siebert and Drummond11), incremental innovation (Reference Craig, Carr and Hutton5;Reference Drummond, Griffin and Tarricone7;Reference Tarricone, Torbica and Drummond10;Reference Basu, Eggington, Hallas and Strachan13;Reference Kirisits and Redekop14), dynamic pricing (Reference Daubner-Bendes, Kovacs and Niewada15;Reference Akpinar16), diversity in device types and applications (Reference Kirisits and Redekop14), and challenges with transferability of results (Reference Kirisits and Redekop14). ‘Insufficient evidence’ in evaluating medical devices refers to the limitations of randomized clinical trials, including lack of randomization, small sample sizes, and short follow-up periods. These limitations make it difficult to draw definitive conclusions about the effectiveness and cost-effectiveness of devices in real-world settings (Reference Craig, Carr and Hutton5;Reference Sorenson, Tarricone, Siebert and Drummond11;Reference Zwanziger, Hall and Dick12). The ‘learning curve’ describes the improvement in user proficiency over time (Reference Healey and Samanta17). The ‘organizational impact’ of a medical device includes various factors affecting its adoption, use, and integration within the healthcare system, with user education and organizational adjustments being essential for maximizing its benefits (Reference Craig, Carr and Hutton5). ‘Incremental innovation’ in medical devices refers to the continuous improvements and modifications made over the device’s lifecycle (Reference Drummond, Griffin and Tarricone7;Reference Tarricone, Torbica and Drummond10;Reference Basu, Eggington, Hallas and Strachan13;Reference Kirisits and Redekop14). ‘Dynamic pricing’ in the context of medical devices refers to the fluctuating costs associated with new devices and their consumables, influenced by factors such as market monopolies, manufacturer pricing strategies, and ongoing incremental innovations (Reference Drummond, Griffin and Tarricone7;Reference Sorenson, Tarricone, Siebert and Drummond11;Reference Kirisits and Redekop14). ‘Diversity’ in medical devices refers to the range of differences in complexity, features, usability, technological specifications, and clinical settings (Reference Kirisits and Redekop14). ‘Transferability’ in medical device economic evaluations refers to the challenge of applying cost-effectiveness results across different healthcare settings, often complicated by variations in device features, clinical usage, and additional cost components, all of which increase uncertainty (Reference Kirisits and Redekop14).
A more rigorous approach is necessary to explore the impact of these various aspects; however, to our knowledge, there is currently no methodological quality assessment instrument designed specifically for medical device economic evaluations. To qualify as ‘specifically tailored’ for medical devices, an instrument should incorporate criteria enabling the assessment of one or more of the seven defined features essential to their evaluation, or alternatively, contain items that adequately address these features.
In 2012, the Agency for Healthcare Research and Quality (AHRQ) in the United States conducted a systematic literature review to assess the best practices for conducting and reporting health economic evaluations (Reference Walker, Wilson and Sharma18). Ten quality assessment instruments (Reference Chiou, Hay and Wallace19–Reference Grutters, Seferina and Tjan-Heijnen28) published between 1992 and 2011 were identified. To identify additional instruments, including those published after 2012 and in grey literature sources, we conducted a systematic literature review of methodological quality assessment instruments for medical device economic evaluations. This review aims to capture recent advancements and address specific considerations for medical device economic evaluations, which were not thoroughly covered in the prior review.
Our primary aim was to identify, summarize, and assess the relevance of existing instruments for evaluating medical device economic evaluations, focusing on seven defined medical device-specific features. This review has two key objectives: (i) to determine whether any existing quality assessment instruments are specifically tailored for medical device economic evaluations, and (ii) in the absence of a suitable instrument, to evaluate each item within current methodological quality assessment instruments for its potential inclusion. These items will be assessed based on their relevance to the seven device-specific features and included in a Delphi pool for expert consensus in the next phase of this project.
Methods
The reporting of this systematic review was guided by the standards of the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) Statement (Reference Page, McKenzie and Bossuyt29). The PRISMA checklists are available in Supplementary Materials 2 and 3. Our protocol, “Health Economic Evaluation Methodological Quality Assessment Tools: A protocol for a systematic review,” was registered with the International Platform of Registered Systematic Review and Meta-analysis Protocols (INPLASY DOI: 10.37766/inplasy2023.7.0093).
Eligibility criteria
Eligible studies were full-text, peer-reviewed primary studies introducing original instruments designed to assess economic evaluations of medical devices. Updated versions of instruments offering a different perspective were also included.
We excluded studies that focus on frameworks or guidelines for conducting economic evaluations, as well as those that adopt an original tool or checklist for purposes other than medical device economic evaluation, or that describe or validate an existing instrument. Reviews (scoping, rapid, systematic, literature), editorials, commentary, conference abstracts, dissertations, and these were also excluded. Studies published in a language other than English were excluded.
Information sources
We searched electronic databases—Ovid MEDLINE, Ovid EMBASE, CINAHL (via EBSCOhost), EconLit (via EBSCOhost), and Web of Science (via its online interface)—for English-language literature published between January 1, 2012 and May 24, 2023, with the final search conducted on May 24, 2023. A grey literature search was performed between November 16 and November 30, 2023, using the CADTH Grey Matters tool (30), as well as the International Network of Agencies for Health Technology Assessment (INAHTA) database and the Professional Society for Health Economics and Outcomes Research (ISPOR) website, to locate relevant guidance documents and instruments.
Search strategy
A systematic review search strategy was designed in collaboration with a University of Alberta Health Sciences librarian experienced in systematic reviews. We used the previous systematic review (Reference Walker, Wilson and Sharma18) search strategy as a foundation and made several adjustments to improve its relevance to our study. While the previous review used terms like “cost–benefit analysis,” “cost of illness,” and “economic evaluation” to capture economic analyses, we expanded the scope to specifically identify studies on quality assessment tools. To achieve this, we introduced terms such as “checklist,” “tool,” “questionnaire,” and included names of widely used checklists. Search terms included a combination of controlled vocabulary (e.g., Medical Subject Headings and EMBASE Subject Headings) and relevant keywords related to medical device economic evaluations. Additionally, we reviewed reference lists of included articles to identify further studies. The full electronic search strategy, including all limits and filters applied, is provided in the Supplementary Material 1.
Selection process
The results of the initial searches were downloaded into EndNote (31) reference manager. Duplicate articles retrieved from multiple databases were removed, and the remaining articles were uploaded to Covidence (32), a web-based systematic review manager. Covidence was used to track the search results throughout the title and abstract review, article selection, and data extraction stages.
Titles and abstracts of all citations identified in the searches were screened in duplicate (IA, AU) to assess potential relevancy. The full text of any potentially relevant articles was also assessed in duplicate against the selection criteria. Discrepancies were resolved by consensus, with a third reviewer (MP) providing arbitration, as necessary. Tool or checklist eligibility was defined based on the definition by Zoratti et al. (Reference Zoratti, Pickard and Stalmeier33). Reporting checklists were defined as “instruments that are used to evaluate the presence or absence of components without value on that component’s use.” Critical appraisal tools were defined as “an extension of reporting checklists and include some interpretation or evaluation of the reported content.” (Reference Zoratti, Pickard and Stalmeier33)
While the relevance of methods such as sensitivity analyses, risk of bias assessment for missing results, and assessing certainty in evidence for many systematic reviews, these methods were not directly applicable to our synthesis of methodological quality assessment tools. Our review focuses on evaluating and synthesizing existing methodological quality assessment tools to assess their effectiveness and applicability in medical device economic evaluations, rather than synthesizing quantitative outcomes. As a result, methods such as sensitivity analyses and bias assessments for missing data were not relevant in this context. Additionally, assessing certainty in the evidence is more suited to clinical or outcome-based reviews, rather than methodological reviews. Instead, we focused on ensuring methodological rigor in study selection and maintaining transparency throughout the synthesis process.
Data collection process
One reviewer (IA) extracted the data from each article to a data extraction form developed by IA. A second reviewer (AU) cross-checked all extracted data for accuracy and consistency. Data discrepancies within articles were noted and it was established that data extraction was prioritized to come from the summary of tables, supplemented by the main text as needed.
Data items
The following data elements were extracted:
-
• Descriptive characteristics of the published instruments (e.g., name, first author, year of publication, author affiliation, journal, number of items, item response options, intended use, target audience, the methods of development, funding source, any validation data)
-
• Only from methodological quality assessment instruments:
-
◦ Each item and its appropriateness to assess medical device economic evaluations.
-
◦ Content review with respect to the seven medical device-specific features (insufficient evidence, learning curve effects, organizational impact, incremental innovation, dynamic pricing, diversity, and transferability of the results).
-
Results
Study selection
In the initial electronic literature search, 6,002 records were identified. After removing duplicates, a total of 4,280 records remained. Screening of 4,203 titles and abstracts and 77 grey literature sources led to the retrieval of 15 articles for full-text review. The 2012 review from Walker et al. (Reference Walker, Wilson and Sharma18) identified 10 instruments that we considered as well. Of these 25 articles, 15 were deemed eligible and included in the review (Reference Husereau, Drummond and Augustovski2;Reference Chiou, Hay and Wallace19–Reference Grutters, Seferina and Tjan-Heijnen28;Reference Grimm, Pouwels and Ramaekers34–Reference Kip, MJ and Henriksson37). The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) study flowchart detailing the process of study selection and exclusion is provided in Figure 1.
Figure 1. PRISMA flow diagram.
Source: Page MJ, et al. BMJ 2021;372:n71. doi: 10.1136/bmj.n71. This work is licensed under CC BY 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/
After a full-text review, two instruments (Reference Husereau, Drummond and Augustovski2;Reference Kip, MJ and Henriksson37) were excluded because they were specifically designed as reporting quality assessment instruments.
Study characteristics
Thirteen instruments (Reference Chiou, Hay and Wallace19–Reference Grutters, Seferina and Tjan-Heijnen28;Reference Grimm, Pouwels and Ramaekers34–Reference Kim, Do and Synnott36) were designed for the methodological quality assessment of health economic evaluations in general, rather than with a specific focus on medical device economic evaluations. Detailed characteristics of each instrument are summarized in Table 1.
Table 1. Included quality assessment instruments
AGREEDT: Alignment in the Reporting of Economic Evaluations of Diagnostic Tests and Biomarkers; BMJ: British Medical Journal; CEA: cost-effectiveness analysis; CHEERS 2022: Consolidated Health Economic Evaluation Reporting Standards 2022; CHEQUE: Criteria for Health Economic Quality Evaluation; CUA: cost-utility analysis; NA: not applicable; TRUST: Transparent Uncertainty Assessment.
The included studies encompass a variety of instruments designed to assess health economic evaluations. These instruments, developed between 1992 and 2023, vary in their number of items, with some containing as few as 16 (Reference Chiou, Hay and Wallace19) and others as many as 91 (Reference Inotai, Pekli and Jona35). The item response options also vary, including open-ended responses (Reference Gerard24–Reference Grutters, Seferina and Tjan-Heijnen28), yes/no options (Reference Chiou, Hay and Wallace19;Reference Inotai, Pekli and Jona35–Reference Kip, MJ and Henriksson37), and weighted scales (Reference Chiou, Hay and Wallace19). Intended uses range from providing guidance for economic analyses in clinical trials (Reference Adams, McCall, Gray, Orza and Chalmers23) to developing evaluation criteria for cost-utility analyses (Reference Gerard24) and assessing the quality of cost-effectiveness studies (Reference Chiou, Hay and Wallace19). The target audiences for quality assessment checklists include researchers (Reference Chiou, Hay and Wallace19;Reference Evers, Goossens, de Vet, van Tulder and Ament20;Reference Ungar and Santos22–Reference Sacristan, Soto and Galende25;Reference Grimm, Pouwels and Ramaekers34–Reference Kim, Do and Synnott36), decision-makers (Reference Chiou, Hay and Wallace19;Reference Ungar and Santos22;Reference Grutters, Seferina and Tjan-Heijnen28;Reference Grimm, Pouwels and Ramaekers34–Reference Kim, Do and Synnott36), policy-makers (Reference Evers, Goossens, de Vet, van Tulder and Ament20;Reference Ungar and Santos22;Reference Gerard24), journal editors (Reference Chiou, Hay and Wallace19;Reference Drummond and Jefferson21;Reference Sacristan, Soto and Galende25) and pharmaceutical industry professionals (Reference Clemens, Townsend and Luscombe26). One checklist’s authors (Reference Siegel, Weinstein, Russell and Gold27) did not explicitly state the target audience. The development methods of these instruments vary, including literature reviews, expert panel reviews, and collaboration with clinicians and policymakers.
Table 2 presents potentially relevant items extracted from various instruments, providing a comprehensive overview of key considerations in medical device economic evaluations. Out of 388 items from 13 methodological quality assessment instruments, only seven items, found in four instruments (Reference Adams, McCall, Gray, Orza and Chalmers23;Reference Gerard24;Reference Grutters, Seferina and Tjan-Heijnen28;Reference Inotai, Pekli and Jona35), were relevant to medical device economic evaluations. Relevance was assessed by comparing each checklist item against specific criteria developed for medical device economic evaluations, including the seven items outlined in the background section. These include considerations for dynamic pricing (Reference Adams, McCall, Gray, Orza and Chalmers23;Reference Inotai, Pekli and Jona35), low evidence cases (Reference Inotai, Pekli and Jona35), learning curve effects (Reference Grutters, Seferina and Tjan-Heijnen28), incremental innovation (Reference Gerard24), and organizational impact (Reference Inotai, Pekli and Jona35). This approach ensured a focus on aspects important to medical devices. None of the identified instruments in the review were deemed suitable for the standalone evaluation of medical devices.
Table 2. Medical device-specific features and relevant items by instrument
CUA, cost-utility analysis.
Consequently, we expanded our search to include grey literature sources and economic evaluation methodological guidelines. We reviewed 77 sources but found no existing instruments tailored specifically for medical devices. Relevant information for only five medical device-specific features was found in six guidelines. These features included low evidence in New Zealand, learning curve effects in Japan, the UK, the Netherlands, and New Zealand, incremental innovation in France, Ireland, Japan, the Netherlands, and New Zealand, diversity in the Netherlands, and dynamic pricing in France. None of the guidelines addressed the domains of organizational impact or transferability. Additional information on medical device economic evaluations from Canada, France, Ireland, New Zealand, and the Netherlands that could not be classified under the defined medical device-specific features was also found. These countries offer recommendations such as resource measurement and costing (Canada, France, New Zealand), adverse effects (Ireland), value components beyond health outcomes (the Netherlands), and outcome measures and evaluation methods (the Netherlands). These relevant items are presented in Table 3.
Table 3. Medical device-specific features and relevant guideline items by country
C2H: Center for Outcomes Research and Economic Evaluation for Health; CT:computed tomography; CEA: cost-effectiveness analysis; CUA: cost-utility analysis; DCE: discrete choice experiment; MCDA: Multi-Criteria Decision Analysis.
Discussion
Despite conducting a comprehensive search of peer-reviewed literature, we did not identify any methodological quality assessment instruments specifically tailored for the economic evaluation of medical devices. This gap highlights a critical area where current research is lacking. Even though Walker et al. (Reference Walker, Wilson and Sharma18) did not incorporate grey literature, to ensure comprehensiveness and avoid overlooking relevant tools, we expanded our search to include grey literature sources. This broader approach was crucial for several reasons: First, it enabled us to capture existing instruments that might not have been published in academic journals but could still be highly relevant for our review. Second, grey literature often contains supplementary information that supports or contextualizes the peer-reviewed studies (Reference Adams, Smart and Huff46–Reference Paez48). Guidelines and reports from Health Technology Assessment organizations can provide practical insights into the application and relevance of the methodological quality assessment items which is essential for developing new tools (38;40;41;43). Third, including grey literature helps mitigate publication bias, as not all valuable research is published in peer-reviewed journals (Reference Adams, Smart and Huff46;Reference Paez48). Finally, the lack of medical device-specific instruments identified through both peer-reviewed and grey literature searches underscores the need for targeted research and development in this area. Without instruments that account for the relevant aspects of medical devices – such as incremental innovation, learning curves, and dynamic pricing – it is challenging to conduct robust, high-quality economic evaluations that fully assess a device’s value over time. This gap limits policymakers’ ability to make informed decisions, potentially leading to inefficient resource allocation or delayed adoption of valuable innovations. Addressing this need is essential for establishing a framework that supports rigorous, relevant economic evaluations, ultimately enhancing healthcare quality and efficiency. Considering these gaps, we also examined the relevance of items within the assessment instruments. Remarkably, only four of the included assessment instruments contained seven relevant items, indicating a substantial gap in the comprehensiveness of tools available for the economic evaluation of medical devices. This finding underscores the need for further development of methodological quality assessment instruments that adequately capture the rigorous, relevant economic evaluations. Incorporating insights from grey literature into the development process can help ensure that new instruments are comprehensive and practically applicable.
Notably, this systematic review addresses a significant gap in the existing literature, as no prior reviews have specifically explored this research question. This systematic review adhered to rigorous standards, as outlined in the review protocol published in INPLASTY. We conducted a comprehensive literature search across multiple databases and grey literature. This approach ensured that we considered a wide range of sources and potential instruments. However, the review has limitations. Only English-language articles were included, which may introduce language bias. While prior studies, such as Morrison et al. (Reference Morrison, Polisena and Husereau49), found no systematic bias from language restrictions in conventional medicine reviews, further research is needed to understand the impact of such restrictions in specialized fields like health economics. Additionally, there was no specific tool for assessing the methodological quality of included studies. Instead, we evaluated instruments based on their development, validation, applicability, previous use, citations, and updates. This approach ensured credibility, but the lack of a standardized quality assessment tool highlights another gap in literature.
Conclusion
Existing instruments cover general items related to the conduct of health economic evaluation studies. However, there is currently a lack of a specific instrument to systematically assess the methodological quality of published economic evaluations for medical devices. To address this gap, future research should focus on developing methodological quality assessment instruments that adequately capture the complexities of medical devices.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0266462325000212.
Data availability statement
Data extracted from included studies will be made available upon request.
Funding statement
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Competing interests
The authors declare none.
Open access
