Introduction
As stated in the new definition of health technology assessment (HTA) presented in O’Rourke, Oortwijn, and Schuller’s 2020 article, the “dimensions of value” for HTA include “ethical, social and legal issues” (Reference O’Rourke, Oortwijn and Schuller1). Though there have been longstanding discussions on the value of ethics in HTA, less awareness exists on methods for retrieving ethics-specific information from published literature. Also addressed in the definition is the process of HTA as one that is “formal, systematic and transparent” to source and assess the best evidence (Reference O’Rourke, Oortwijn and Schuller1). Keeping in mind that various approaches to ethical analysis exist (normative, empirical, etc.) which may not require a formal literature search, this investigation highlights information-seeking activities in support of this important component of HTA when necessary.
Specific details of current practices for ethics information retrieval in HTA—including databases searched, preliminary analysis of search strategies, and grey literature consulted—are not well-established. Some guidance for information specialists exists in the ethical analysis chapter of the Summarized Research in Information Retrieval for HTA (SuRe Info) (sure-info.org) (Reference Bakke Lysdahl and Droste2). Currently no methodological guidance for ethical analysis is posted on the CADTH website, and internal information retrieval processes at CADTH include both bibliographic database and grey literature searches for selected reports. Formal and systematic searches for ethical analyses are published within these CADTH reports, typically available via appendices. In an effort to best address information retrieval requests for ethical analyses at their home HTA agency (CADTH), the authors sought to understand what general trends exist across other HTA agencies and within the published literature. The purpose of this study is to locate and analyze published ethics review guidance and ethical analysis in HTAs to draw conclusions on current information retrieval practices.
Background
Information retrieval methodology
Evidence regarding best practices for conducting ethics reviews is still lacking in the field of HTA, and resources on information retrieval methodology are even fewer. There is little mention of searching for ethics compared to information retrieval of clinical or economic aspects of HTA, though there are a number of studies that note the lack of guidance in this area (Reference Bakke Lysdahl and Droste2–Reference Mertz, Strech and Kahrass10). The recent PRISMA-Ethics reporting guideline also addresses poor reporting quality of ethics systematic review methods, including search strategies (Reference Kahrass, Borry and Gastmans11). These resources acknowledge and confirm the lack of information on a standard practice for retrieving ethics materials, calling for more research to be done in order to cultivate common methods for information retrieval. Such methods could include developing a list of relevant databases to run literature searches in, guidance on search strategies or filters to employ, and which grey literature resources to search for ethics materials. There are also other search approaches that may be used in ethical analysis that do not require a separate literature search, such as pulling references from a more general clinical search or an ethicist’s personal library, relying on expert opinion, and conducting primary research. However, these methods are also underreported, and it is unclear to what extent they are used in practice. The authors acknowledge these other methods, but will focus on studying the more traditional methods of information retrieval previously mentioned.
Literature on ethics information retrieval methods presents biomedical databases (MEDLINE/PubMed, EMBASE, CINAHL) as the most common resources to search (Reference Droste, Dintsios and Gerber4;Reference Mertz, Strech and Kahrass10). HTA databases such as those produced by the Centre for Reviews and Dissemination (CRD) are also recommended (Reference Dejean, Giacomini, Schwartz and Miller3).Footnote 1 Droste, Dintsios, and Gerber list other subject-specific database categories including those in ethics, social sciences, and law (Reference Droste, Dintsios and Gerber4;Reference Mertz, Strech and Kahrass10). Lysdahl and Droste’s chapter on ethical analysis notes that while including subject-specific databases is recommended, the choice of databases is highly dependent on the research topic as well as the region in question (Reference Bakke Lysdahl and Droste2). Though the article authors present no specific search filters, use of relevant MeSH terms and subheadings, database subset limits, and text words are discussed (Reference Dejean, Giacomini, Schwartz and Miller3;Reference Droste, Dintsios and Gerber4). Search strategies are highly dependent on the research topic—such as informed consent when receiving vaccines or the implications of gene editing. Thus, MeSH terms and text words that are specific to narrower ethics-related concepts may be added to a general ethics strategy. There is little discussion of grey literature sources aside from mention of grey literature’s existence, in which it is positioned as more akin to expert input or feedback than a key component of a literature search (Reference Polus, Mathes and Klingler8). The body of literature discussed here—both regarding ethics in HTA generally, and ethics information retrieval—demonstrates a need for further study and evidence in order to develop a standard methodology for information retrieval practice and analysis.
Methods
This study reviews published guidance and ethics-related reviews conducted by international HTA agencies. Literature searches were conducted in Ovid MEDLINE, LISTA, Scopus, and Google Scholar in September 2020. Once a list of relevant articles was determined, backward and forward citation tracking was conducted on these articles via Scopus. A narrative review of the literature was subsequently conducted using this body of research. HTA agency websites were searched for published guidance on ethics searching, as well as for any reports which included ethical analyses. The authors excluded any reports published by CADTH, as their focus was on scoping organizations outside of CADTH, but reports with sections outsourced to CADTH researchers (from Health Quality Ontario) were considered. Search methods sections of each report were analyzed to determine the databases used, MEDLINE search strategies employed, and grey literature consulted. To complement this review of reports, the authors also reached out to HTA information and ethics specialists via listservs and email for insight into current practice in HTA.
Scoping current practice in HTA
A group of international HTA agencies, excluding CADTH, was chosen for study based on the HTA section of the CADTH “Grey Matters” Checklist (12). The researchers browsed and searched the 57 HTA websites included in the checklist for ethics-related content, limited by publication type—methods guidance and HTA reports—where applicable. HTA reports with explicit mention of ethics analysis or methodology were collected. No methods documents were retrieved. Results were not limited by date or language, though English versions of foreign language documents were favored when available.
In addition, the research team called on information specialists in the HTA community for more information on current ethics search practices. A general callout was sent to the ExpertSearching (MLA), CanMedLib (CHLA), and Information Retrieval Group (HTAi) listservs. The researchers also compiled a list of HTA organizations identified from the above search and sent emails to contacts at these organizations for more information on their information retrieval methodology.
Analysis of reported search methodology
HTA reports with explicit ethics analysis or methodology were collected and analyzed to determine information retrieval practice. Results with relevant titles and abstracts were chosen for full-text analysis. Reports were screened by conducting a full-text search for the keyword stem “ethic”. Full-text search allowed the researchers to determine the extent of ethics analysis in each study. Reports with only a keyword or phrase on ethics mentioned were excluded. Reports that included ethics analysis or a dedicated ethics section—or addressed ethics in their methods section—were retained for analysis. Information was then extracted from these reports’ methods sections and appendices, including details such as the databases searched, if grey literature sources were consulted, ethics-related keywords and subject headings (if used), and published ethics-related search strategies.
The researchers then compiled a table with each database mentioned to determine the frequency with which different databases were searched. An analysis of search strategies was also carried out where reports included ethics-specific searches. The focus here was on MEDLINE/Pubmed strategies for ease of analysis and comparison across reports. All ethics-related MeSH terms, MEDLINE subheadings, PubMed subsets, and text words were collected and analyzed for frequency.
Results
Findings from our background literature search and analysis of HTA websites indicate that there is still little published guidance from HTA agencies on ethics analysis or information retrieval methods for the purpose of ethical analysis. There are also few HTA reviews that contain substantial ethics analysis, and even fewer that specify methodology for ethics information retrieval. Of the fifty-three HTA agencies reviewed, seven organizations had published a total of twenty-five reports with an ethics component. Thiteen reports out of twenty-five included ethics-specific methods (Table 1). Citations for these reports can be found in the Supplementary material. Ten of these reports did not utilize any subject-specific databases outside of the health sciences. Eight reports published ethics searches. Search strategies reported from these eight reports overlapped in their use of medical subject headings (MeSH) and text words related to ethics or bioethics, but often also included terms that were specific to the research topic in question (such as informed consent, autonomy, or patient rights). No relevant methods guidance documents were retrieved.
HTA reports including substantial ethics analysis, by agency
Employing an ethics-specific search strategy or filter is not always done when conducting ethical analyses. Thus, the research team analyzed all databases searched in all twenty-five reports. The aim was to determine which databases are used in reports with an ethics component—with a particular focus on whether subject-specific databases outside biomedicine are used. Reports were analyzed regardless of whether an ethics-specific strategy was mentioned. If an ethics strategy was given, it was taken for analysis. In other cases without an ethics strategy, the clinical strategy was used. Frequency of use of an ethics strategy across reports was also recorded (Table 2).
Database list and frequency
a indicates databases that are searchable but no longer updated.
b indicates databases that are no longer available.
The most frequently searched databases fall within the biomedical subject area, with some commonly used in HTA (HTA York, NHS EED, DARE). While there are subject databases listed which fall outside the realm of biomedicine (social science, legal, ethics) they are not utilized nearly as often. Other databases, such as multidisciplinary science citation indexes (Scopus, Web of Science) and dissertation or conference proceedings databases, are also included.
Of the 25 reports, 10 published ethics search strategies. MEDLINE/PubMed searches were analyzed to determine common MeSH terms and text words included in ethics-specific strategies (Table 3). Truncated text words encompass all variants of the word used across reports.
Frequency of MeSH terms and text words used
* Indicates truncation
MeSH terms and text words related to the basic concept of ethics (ethics, ethical, moral, etc.) were most frequently used. Other terms encompassed multiple facets of ethics—such as sociocultural, legal, and human rights—but were included less often. Truncations in Table 3 indicate either a direct search term, or a synthesis of search terms with varied word endings, that is, ethic* indicates all variants such as ethics, ethical, and ethically. The search strings listed also synthesize common adjacency strings found across the strategies studied. These strings utilize an adjacency operator (adjN), which commands Ovid MEDLINE to retrieve results that have the given keywords in a certain proximity to each other. The variable N indicates the level of proximity—adj3 retrieves phrases with keywords up to two words apart in the text, for example. Utilizing adjacency operators where available can help account for possible differences authors might use when phrasing concepts or ideas across database records.
Ten of the relevant reports analyzed did not discuss methods for retrieving ethics-specific information, so it is useful to note that these reports had substantial ethics analyses without clear strategies for ethics information retrieval. One might assume in some cases, during the screening process of a report’s general literature search in biomedical databases, articles related to ethics may have been set aside for an ethics specialist to analyze. Additionally, ethics researchers may have their own collection of literature they call upon when conducting an HTA. The authors base our assumptions on information from listserv responses, where not conducting a separate ethics search, but instead screening the larger body of results for relevant articles, which was anecdotally described by other HTA information specialists. Other reports listed expert input and primary research conducted as other sources of ethics information—methods that are not related to database searching. Though outside the scope of this study, it is important to note other ways ethics information is discovered.
Grey literature searching was not discussed in any of the studied reports. As such, the authors cannot provide insight into specific sources searched or strategies utilized for retrieving grey literature.
Response rates from querying listservs and individual contacts at HTA agencies were very low—fewer than five responses—and any responses could only be considered as anecdotal evidence to support the findings of our study.
Discussion
This analysis shows a heavy dependence on biomedical databases and HTA databases to identify information on ethics in HTA. However, it is unclear at this stage whether the common use of these databases is based on evidence or simply because these are the same resources used to retrieve clinical information and would therefore be logical, familiar, and accessible starting points for HTA information specialists. The long list of databases in Table 2 indicates that while there are numerous options for subject-specific searches in other disciplines, none stand out as being frequently searched resources in the way that biomedical databases do. This is consistent with previous research stating that database appropriateness is highly dependent on the research question (Reference Hofmann13–Reference Assasi, Tarride, O’Reilly and Schwartz15).
MeSH terms and text words used in strategies most often included the basic concept of ethics. However, there are numerous terms used less frequently that demonstrate how contextual ethics questions can be in relation to a study’s larger research topic. Terms like “informed choice”, “stigma”, “legal”, and “treatment refusal” may be highly relevant to certain research questions, and irrelevant to others. The frequency of search terms outside of general ethics is thus more variable but can nonetheless play an important role in a given strategy depending on the context. Additionally, in cases where ethics subject-specific databases are searched, terms for ethics can be excluded so as to not unnecessarily limit the search. Other concepts, such as population, intervention, or indication can be isolated from the ethics concept in such cases.
This study provides information on the current state of ethics information retrieval practice in HTA. Due to the limited number of ethics reports published in HTA, database and search strategy analysis were done on just a small sample of relevant reports. While guidance on conducting ethical analyses and searching for ethics information may exist within an HTA organization, the authors were not able to locate any publicly available handbooks. The authors identified studies in all languages, though English versions of HTA sites were searched, and English language versions of relevant reports were favored when available. Thus, the study was limited by what was available on English language versions of HTA sites and the search capabilities of those sites. In addition to there being a small sample size, there are differences in how methods are discussed across reports. Reports which did not include a separate ethics literature search—but included ethical analyses—do not disclose how information was retrieved or screened for ethics analysis. Another limitation of this study was the decision to analyze MEDLINE search strategies only. Different databases have their own controlled vocabularies and syntax that may provide more insight into how ethics searches are conducted. The researchers also received minimal feedback from reaching out to HTA information specialists, and thus firsthand knowledge of how ethics are approached at HTA agencies was limited. Lastly, this study focused specifically on ethics in HTA, and chose HTA reports for analysis. However, there are other sources of knowledge syntheses that may approach ethics analysis differently. A larger study of ethics information retrieval in ethics systematic reviews or knowledge synthesis more generally—both within the health sciences and in other disciplines—would be useful in gaining a more comprehensive understanding of the topic.
The lack of reported methods for information retrieval of ethics in HTA raises questions about what current practice for ethics literature searching looks like. The authors also question whether structured or systematic literature searching is necessary for ethical analyses. An apparent absence of search methods may indicate that information for ethical analysis comes from other sources, such as ethicists’ personal libraries or expertise. The clinical literature search, if sufficiently broad, may also provide enough context for an ethicist to conduct their analysis. The main takeaway is that it is unclear what people are doing to access the best informational support needed for an ethical analysis, because information retrieval methodologies are not well-described for these parts of HTAs. It is also important to note that ethical analysis methodologies may differ from other HTA methodologies, and therefore the information needs may also differ.
Information gleaned from this study provides a foundation on which to further study ethics information retrieval. Citations from the HTA reports included in the study are being compiled by the authors for further analysis. This analysis will include a journal title inventory, which will be used to evaluate the holdings of databases to understand which databases are the best options for retrieving ethics information. These citations can also be used as a test set for evaluating and developing an ethics search filter. Information on the current state of ethics information retrieval can now be employed to investigate evidence-based methodology to advance literature-searching efforts for ethics in HTA. Additionally, the questions raised by this study lead the authors outside of HTA, where further research is being conducted to determine search practices in other forms of medical knowledge synthesis.
Conclusion
This study of current practice in HTA ethics information retrieval further highlights the findings of previous studies—while ethics analysis can play a crucial role in HTA, methods for literature searching remain underreported and unclear. In order to bring the new definition of HTA into practice, further study and development of evidence-based methodology for information retrieval is important (Reference O’Rourke, Oortwijn and Schuller1). The findings from this preliminary study provide insight into the current state of ethics literature searching, and will help inform continued work on filter development, database selection, and grey literature searching.
Ethics analyses can be an important component of HTA. Though methods for analysis are varied, each method should be supported by reliable and up-to-date information resources. Thus, understanding information retrieval’s role within ethics HTA methodology is imperative. This study examined current practices for retrieving ethics resources across HTA by analyzing published HTAs with ethics components and the methods they report. There may not be much information on evidence-based methods, but some patterns across agencies can still be identified. These patterns provide a foundational understanding of how ethics searches are being conducted today and lay the groundwork for further study into ethics information retrieval.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462323000247.
Open access
