Health technology assessment (HTA) is a multidisciplinary process that provides information about the effectiveness and safety, economic, ethical, and social aspects of a health technology. Despite this comprehensive understanding shared by most HTA definitions, sociocultural aspects are rarely assessed in HTA (Reference Draborg, Gyrd-Hansen, Poulsen and Horder1;Reference Lehoux, Tailliez, Denis and Hivon2). However, they can have a crucial impact on the outcome of the implementation and use of a health technology. This can be illustrated by a comparison of home-based palliative care with and without additional care giver support (Reference Brereton, Wahlster and Lysdahl3): In this case, sociocultural aspects can be linked to the change of roles and status of family members, who often act as the informal carers. The decision to care for a relative can be influenced by social expectations such as the expectation that women have to care for their relatives at home. Becoming a caregiver in turn influences the relationship to the patients they care for (another social aspect of interest) and can be related to a risk of overburden and social isolation for the informal carer.
One possible hypothesis for the reason why sociocultural aspects are hardly considered in HTA is that we lack relevant assessment methods. Accordingly, this review aims at providing an overview of methodological approaches to address sociocultural aspects related to health technologies in HTA. The objective was to provide an overview of the identified methods as well as their potential strengths and limitations regarding their application in HTA.
Our understanding of social and cultural aspects follows the definition of Gerhardus and Stich (Reference Gerhardus and Stich4): “With the implementation of a technology, this technology interacts with the society and its different areas. These interactions concern values, attitudes, meanings, balance of power, behavior, and allocation of resources of a society. Different groups of a society might be affected by a technology in different ways. For the assessment of social aspects, this means that different perspectives should be taken into account.” As social and cultural aspects are interrelated, we use the term “sociocultural” as superordinate phrase. Our definition is based on the assumption that “technology assessment is socially shaped, and as a part of such shaping, […] technology assessment is involved in the social shaping of technology […]” (Reference Clausen and Yoshinaka5).
Methods
Literature Search
We developed a search strategy that was based on the definition above and combined MESH-terms as well as free text terms. Examples are “societal”, “social values”, “public opinion”, “cultural aspects”, “patient participation”, and “health technology assessment” (see Supplementary Materials, Part A).
We searched fourteen databases for publications addressing sociocultural aspects of health technologies that were published between 1970 and 2013. These were MEDLINE, EMBASE, BIOSIS Previews, CINAHL, PsychInfo, Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index, and the Databases of the Cochrane Library (Database of Abstracts of Reviews of Effects, National Health Service Economic Evaluation Database, Health Technology Assessment Database, Cochrane Methods Studies, Cochrane Reviews, and Cochrane Technology Assessment). The search was limited to publications in English, German, Dutch, French, or Spanish. We also conducted hand searches in two scientific journals – Health Policy and The International Journal of Technology Assessment in Health Care, and screened the included publications and their reference lists. In addition, we e-mailed all INAHTA member agencies in August 2007 and in September 2013, asking them for methodological approaches they use to assess sociocultural aspects in HTA.
Selection Criteria and Process
With the objective to identify a wide range of methods, we did not limit our search to the HTA-context and included publications that empirically studied sociocultural aspects of a health technology as well as literature reviews to investigate these aspects. We included publications focusing on ethics or morals combined with sociocultural aspects, and excluded those only focusing on conceptual frameworks or methodological guidances without application to a health technology. Abstracts were excluded where the full text was not available. Titles, abstracts, and full texts of identified publications were screened for inclusion by one of the authors. In case of uncertainty, a second author was consulted.
Data Extraction
A matrix was developed for data extraction (see Supplementary Materials, Part B). Aspects extracted were objectives, sociocultural aspects addressed, methods for the assessment of sociocultural aspects, disease/condition, type of technology assessed, country, study population, and time frame.
Data Analysis
The identified publications were assessed with regard to whether they fall under the definition of sociocultural aspects, their purpose and relevance for HTA as well as their strengths and limitations. The results were organized based on the classification of methods suggested by Gerhardus and Stich (Reference Gerhardus and Stich4). We organized the methodological approaches into five groups: (i) checklists, (ii) literature reviews, (iii) stakeholder participatory approaches, (iv) primary data collection methods, and (v) combinations of methodological approaches:
(i). Checklists offer several questions to be addressed by HTA-researchers. This can be done by “self-testing”, that is the HTA-researcher addresses the questions, or by asking other experts in the field of interest.
(ii). Literature reviews are tools to identify and synthesize scientific evidence from a range of studies. In that sense they encompass the results of the other approaches that involve primary research. Systematic reviews contain clear inclusion and exclusion criteria, an explicit search strategy, and an analysis of included studies.
(iii). Stakeholder participatory approaches comprise methods for the active integration of various stakeholders into the assessment process. They aim at ensuring legitimacy of decisions, transparency of perspectives and at improving the relevance of research, but differ in their procedures (Reference Abelson, Giacomini, Lehoux and Gauvin6–Reference Gauvin, Abelson, Giacomini, Eyles and Lavis8). Participants are involved as consultants rather than as research subjects.
(iv). Primary data collection methods comprise a wide range of quantitative and qualitative methods and summarize various empirical studies, according to their mode of data collection. This category also contains studies applying a mixed methods approach. Combinations of different primary data collection methods are also included in this group.
(iv). Combinations of methodological approaches include any combination of methods across the groups 1–4.
As the objective of the study was to identify methods for the assessment of sociocultural aspects, we did not rank the various methods according to their quality. We only investigated their strengths and limitations so that readers can apply (or modify) them according to their own context.
Results
As presented in Figure 1, the search yielded 15,276 publications after deduplication check. A total of 14,720 of the identified publications were excluded based on title and abstract screening, and 556 qualified for full text reading; 99 of the latter met the inclusion criteria and a further 26 were identified from the reference lists of the included publications as well as from hand searches, resulting in a total of 125 publications.
Flow chart of review process: number of publications.
We identified fifteen publications on checklists, sixteen publications on literature reviews, fourteen publications on stakeholder participatory approaches, and classified sixty-seven publications as primary data collection methods (see Supplementary Materials, Part B, Table 1–4e). The latter include focus groups, qualitative interviews, surveys, registries, databases and health records. Thirteen publications combined different methodological approaches (see Supplementary Materials, Part B, Table 1– 5); among them combinations of primary data collection methods with literature reviews, stakeholder participatory approaches, and checklists.
Thirty-three publications were explicitly developed for or used in HTA (see Supplementary Materials, Part B, Table 1–5), that is, they explicitly assessed a health technology or suggest methods for the assessment (see Table 1). Ninety-two publications did not have an explicit HTA-reference, but assessed a health technology in the context of a wider empirical study.
Technologies Addressed in the Included Studies Classified by the Methodological Approach Applied (not all publications assess a specific technology and some publications refer to more than one technology (see supplements part B for more detailed information))
Table 1 gives an overview of the technologies addressed in the included publications. The themes most frequently addressed were sociocultural aspects of genetic and reproductive technologies, followed by transplantation technologies.
Group 1: Checklists
We identified fifteen publications on checklists (Supplementary Materials, Part B, Table 1). An example is the HTA core model developed by the European Network of Health Technology Assessment (EUnetHTA) (9). Although it was originally constructed as a methodological guidance, the part for social aspects can be applied as a checklist. The questions focus on patient related aspects and less on societal aspects. The latter consists of two questions on access to available therapies. Two of the seven identified checklists were applied to health technologies. The other publications presented a checklist without applying it. The checklist applications addressed the sociocultural aspects of multi-slice computed tomography, drug eluting stents, surgical interventions, bariatric surgery, and stem cell transplantation. Only one checklist was applied in an expert consultation. All other applications combined checklists with other methodological approaches (see group 5).
Checklists were presented as lists of items covering sociocultural aspects. The authors of the included articles stated the following advantages of checklists: improved transparency and increased acceptance of HTA, the impact on public involvement in communicating characteristics of medical technologies, the support of decision makers, and the pragmatic and timesaving applicability. They considered the limited validity, difficulties in dealing with overlaps between checklists’ elements as well as challenges related to the diversity of health technologies (the application of a standardized list of questions can suggest the risk of missing important issues) as limitations of checklists.
Group 2: Literature Reviews
Five of the sixteen publications that we identified as literature reviews were systematic reviews (see Supplementary Materials, Part B, Table 2). The identified publications considered access to medical care, views about technologies, treatment burdens related to health technologies, as well as the social acceptability, social impact, and potential uptake of a technology.
Although literature reviews are common in HTA, reviews on sociocultural aspects seem to face specific challenges. Less developed review procedures, difficulties in performing the quality assessment especially for qualitative studies, a limited generalizability of results, and the fact that some articles do not present the complete data. Additionally, the authors of the included papers considered the identification of qualitative studies as difficult due to poorer archiving and indexing, and identified varying discipline-specific reporting cultures (i.e., the way of formal text organization and discussion, the way titles are used and abstracts are written) as factors that can also make the process more arduous. On the other hand, the authors saw literature reviews as a useful source of evidence and valued the integration of nontraditional sources such as weblogs as an advantage. Ethical concerns such as using data that were not intended to be part of scientific research were also acknowledged. As many of the sociocultural aspects are context-dependent they are less likely to be published in English and to be internationally indexed.
Group 3: Stakeholder Participatory Approaches
The fourteen publications that applied stakeholder participatory approaches referred to public views of a technology, patient preferences for treatment outcomes, societal concepts of diseases, as well as cultural and ethical dimensions of a technology (Supplementary Materials, Part B, Table 3). The included publications explored the opinions of citizens, patients, or experts, as well as of private or federal agencies. Participative methods ranged from single consultations based on 2-day workshops to several consultations, each lasting more than 1 day. Workshops were the approach most often used. The results of the identified studies were used as material to prepare reports and to validate recommendations.
The authors of the included articles stated the following advantages of participatory approaches: variability of perspectives, the inclusion of controversy, generating useful ideas and creative solutions, as well as the opportunity to involve different types of stakeholders and the public on the same discussion level. The lack of representativeness and validation of results, time and costs, adverse group effects such as dominant participants, and issues of participants’ independency were mentioned as limitations.
Group 4: Primary Data Collection Methods
Of the sixty-seven publications in this category, thirty-two publications were on surveys, eight publications on individual interview studies, ten publications were focus groups, and three used registries, databases, and health records. A further fourteen publications combined different primary data collection methods.
Surveys
The publications used surveys to assess sociocultural aspects (see Supplementary Materials, Part B, Table 4a) using interviewer administered and self-administered questionnaires. The focus was on investigating the acceptance of a technology, health perceptions and experiences of people living with health technologies, trust in care providers and the technology, users’ and providers’ attitudes toward health technologies, social diversity of users, access to and the distribution of health technologies. Authors of included publications reported the lack of validity and reliability, limited representativeness as well as social desirability as limitations of the specific surveys. Strengths were rarely emphasized by the authors.
Focus Groups
Ten publications used focus group interviews and discussions to explore treatment decisions or attitudes toward a health technology (see Supplementary Materials, Part B, 4b). People affected by a health technology, lay people, and relatives took part in the discussions. Most researchers used qualitative coding approaches for analysis. The authors listed the diversity of perspectives and the option to explore social influences on public consultations as strengths of focus groups. The limited generalizability of results and the risk of bias produced by the selection of participants were mentioned as limitations by the authors of the included papers.
Individual Qualitative Interviews
Interviews were conducted with patients, program participants and their relatives, and professional experts on a health technology (see Supplementary Materials, Part B, Table 4c). They explored illness experiences and health related aspects such as socio-economic status, access to health care, perceptions of a health technology or reasons for refusing the use of a health technology. The two publications that reported on strengths and limitations stated the obtained perspectives of key stakeholders as an advantage, and recruitment bias as well as the limited generalizability of results as disadvantages.
Registries, Data Bases, and Health Records
The publications that used registries, data bases, and health records as data sources (see Supplementary Materials, Part B, Table 4d) were one review assessing different databases for their potential use in HTA, one study on the disparity of pediatric cochlear implants, taking the ethnicity and sociodemographic variables of children into account, and one study on human papilloma viruses (HPV) testing, focusing on the association between physicians’ medical specialty and the uptake of new medical technologies. The approach was seen as a promising source for HTA. However, the authors of the included publications stated several limitations, for example, nonstandardized and rarely validated coding systems, a too narrow focus on a specific technology, slow response to the introduction of new technologies, a lack of patient characteristics, and uncertain funding.
Combinations of Primary Data Collection Methods
Fourteen publications combined different methods of data collection (Supplementary Materials, Part B, Table 4e). The included publications were on experiences of technology users’ and clinicians’ perceptions of and satisfaction with a health technology. The articles included combinations of surveys, interview studies, focus groups, media analysis, observations, different quantitative data, and the analysis of documents. The authors stated the option of systematic group comparisons as well as the potential to explore phenomena from various perspectives as advantages of the approach. Limitations cited include possible selection biases, challenges to integrate results from different methodological approaches, and the additional effort required in terms of labor and time.
Group 5: Combined Methodological Approaches
Among the thirteen included publications were combinations of literature reviews and primary data collection methods as well as of participatory approaches and primary data collection methods (see Supplementary Materials, Part B, Table 5). Of particular interest are studies using checklists as they are almost routinely being combined with other methods. Often checklists are used to prepare the use of other, more time-consuming methods. The topics covered implications of biomedical research, social aspects of xenotransplantation, medical needs and age, treatment decisions, public attitudes toward new genetic technologies, images of robots, and social aspects related to telecare. The authors of the included publications stated the possibility to compare the results of different methods as well as the potential to explore phenomena from various perspectives as advantages of the approach. Limitations were not stated.
Discussion
The starting point of our review was the hypothesis that a lack of available methods possibly prevents HTA-researchers from assessing sociocultural aspects of health technologies. This hypothesis was not supported by our findings. We identified 125 publications on methods for the assessment of sociocultural aspects of health technologies. In the following, we discuss the potential usefulness of the identified methods for HTA.
Checklists, the first group of methods, aim at providing a broad and quick overview thereby offering a pragmatic tool to capture sociocultural aspects (Reference Hofman10). All publications on checklists are related to HTA, which indicates that checklists are a known tool in this area. However, only two were applied and only one of them was applied in an expert consultation, as the category suggests. All checklist applications structured other approaches. Applying a checklist in HTA can be advantageous at the beginning of the assessment when relevant aspects need to be identified early on to guide the subsequent process or to indicate if more elaborated methods are necessary. However, the focus on standardized items might lead to important aspects being overlooked (Reference Mozygemba, Hofman, Lysdahl, Pfadenhauer, Van der Wilt, Gerhardus, Lysdahl, Mozygemba, Burns, Chilcott, Brönneke and Hofmann11;Reference Lysdahl, Mozygemba and Burns12). Therefore, when applying checklists, attention should be paid to the involvement of cultural diversity and openness to additional information should be maintained.
The second group, literature reviews, are a tool familiar to HTA-researchers and HTA-users. They provide an overview of the available evidence by encompassing the results of the methodological approaches used in primary research. The authors of the EUnetHTA-guidance on the assessment of social aspects recommend literature reviews as a first method of choice (9). However, often the number of publications on sociocultural aspects of a specific technology in peer-reviewed journals is very low. When gray literature is included, the process of searching and selecting the literature becomes time-consuming. Gray literature also comes with varying levels of quality and less standardized formats, which makes the assessment process more difficult. Especially the studies on cultural aspects are often published in the local language and are less frequently referenced in international databases. The context dependency of reviews poses a further challenge. This, however, should be incorporated as part of the quality assessment of the included studies. The GRADE-CERqual approach provides a method for assessing the confidence of evidence from reviews of qualitative research (Reference Lewin, Booth and Glenton13).
Participatory approaches, the third group, enhance the involvement of different perspectives related to a health technology and give stakeholders an active role in the assessment. Approximately half of the identified publications were related to HTA, which indicates familiarity with the approach. In HTA, stakeholder participatory approaches may improve the transparency and legitimacy of decisions (Reference Mozygemba, Hofman, Lysdahl, Pfadenhauer, Van der Wilt, Gerhardus, Lysdahl, Mozygemba, Burns, Chilcott, Brönneke and Hofmann11). The consideration of the values of potential users leads to a better understanding of issues around a technology's acceptance (Reference Bridges and Jones14;Reference Pizzo, Doyle, Matthews and Barlow15). Participatory approaches can support HTA at several stages of the HTA-process. They can also frame the whole HTA as an interpretative procedure, as it is the case for interactive HTA (Reference Reuzel16;Reference Reuzel, van der Wilt, ten Have and de Vries Robbé17). Participatory approaches introduce different perspectives as they are not a mere tool for data generation, but as Gauvin et al. (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis8) stated, can also induce a shift in control over the HTA. This means that the persons who are contacted can have an impact on the direction the HTA takes as well as on the content that is being investigated.
HTA agencies emphasize the inclusion of values as an advantage (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis8) and point out the impact of participatory involvement on decision-making processes. However, HTA-agencies hesitate to rely on participative approaches. Possible reasons for this reluctance are limited resources and fears of losing control. In addition, some HTA-agencies are unsure about the quality of evidence that is generated using participatory approaches (Reference Abelson, Giacomini, Lehoux and Gauvin6). The majority of participatory approaches are qualitative and their results are not representative, as is characteristic for qualitative methods. To handle adverse group effects in an adequate manner, the selection criteria should be transparent and well considered when involving different groups of participants with specific interests, knowledge, etc., about the investigated subjects.
The category “primary data collection methods” contains publications that systematically applied qualitative and/or quantitative methods to address sociocultural aspects of health technologies. Primary data collection methods allow a systematic data collection and analysis and usually result in a description of the phenomenon of interest. They may facilitate a better understanding of social structures and enable discussion of the results in a wider framework. Methodological tools and research questions can be tailored to the specific assessment situation. The tools and procedures have their own advantages and disadvantages; they differ in effort and objective, but also in their philosophy of science and methodology (Reference Gerhardus and Stich4). The latter shapes the assessment idea and determines whether independent and isolated measures need to be used or if the context of a health technology has to be taken into account.
Applying qualitative approaches such as focus groups means that statistical generalizability cannot be expected. Instead, qualitative studies should aim at theoretical generalizability, meaning that selection criteria take the distribution of variables in the target group into consideration. However, when applying primary data collection methods in HTA, it should be clarified beforehand if a larger research context should be chosen (Reference Mozygemba, Hofman, Lysdahl, Pfadenhauer, Van der Wilt, Gerhardus, Lysdahl, Mozygemba, Burns, Chilcott, Brönneke and Hofmann11). Researchers should also have experience in conducting empirical research and in the application of the chosen method. The application in HTA must be carefully considered, and it will often be restricted to cases with a scarce evidence base. The small proportion of publications applying primary data collection methods in relation to an HTA indicates limited importance of the approach in HTA.
The fifth group contains combinations of methodological approaches. Combinations of primary data collection methods and stakeholder participative approaches were used in relation to an HTA. Furthermore, all checklist applications (all had an HTA context) referred to combined methodological approaches. The checklists were applied to structure other approaches such as literature reviews or primary data collection methods. Only one application used a checklist in an expert consultation. This indicates that checklists are heuristic devices usually combined with other approaches and less often applied to explore the field before a more comprehensive method is used.
We did not recognize a specific pattern between the methodological approaches used and the technologies addressed (see Table 1). We found an accumulation of publications addressing genetic testing, reproductive medicine, and transplantation technologies. These technologies can be considered as technologies affecting “deep human meanings,” which Banta (Reference Banta, Banta and Luce18) identified to be the situation when social values have their major impact in a technology assessment. According to Banta, “technologies in these areas interfere, or seem to interfere, with natural processes, and may be very disturbing to moral feelings” (Reference Banta, Banta and Luce18). That these topics are most often addressed could be due to their embeddedness in controversial public discourses, which could lead to increased caution with regard to the technology's implementation.
In general, as shown by our review, there are various methods available for the assessment of sociocultural aspects of health technologies. Possible reasons for the poor assessment of these aspects could be limited resources (economy, time), lack of expert skills in social sciences, poor recognition of the significance of sociocultural aspects in HTA and assessment routines focusing on medical and economic aspects more than on others.
Strengths and Limitations
The results comprise a variety of methodological approaches and examples of how sociocultural aspects of health technologies can be assessed. Although, we did not screen HTA-reports to identify methods that were used in relation to specific technologies, we used the feedback on methodological approaches we got from INAHTA agencies to help minimize this gap. Other definitions of sociocultural aspects and search criteria may produce different results.
The classification of methods we used (Reference Gerhardus and Stich4) leans on approaches known by HTA-agencies and it enabled us to give a structured overview of the findings. However, the pragmatic approach to structure the methods presented in the included papers occasionally resulted in difficulties, for example, based on nondistinct categories.
Another limitation of this study is that it does not attempt to empirically qualify the strengths and weaknesses of the different methods as this would have been beyond the scope of this article. Instead, it reports them as they have been experienced by the authors of the included publications and does not rank the various methods according to their quality. While this is crucial for methods synthesis and development, this was beyond the scope of this study. Here, we have only investigated their strengths and limitations in order for readers to apply (or modify) them according to their own context.
We applied a very sensitive search strategy. The first search was run in 2007, followed by an update in 2013. An updated literature search using the same strategy in 2018 retrieved approximately 12,000 hits. Based on the title/abstract screening and an exploratory analysis of a sample of fulltexts, we estimated that in addition to the 125 articles already included, we might have to include roughly an additional 600 publications. A subsample of the additional publications indicated that the added value would be rather limited; hence, we decided against a further update. This might mean that we missed recent developments in the field. As our original search strategy yielded a large amount of literature, we recommend that a more specific search strategy should be used for future research.
In conclusion, contrary to the “lack of methods” hypothesis found in the literature (Reference Banta19;Reference Busse, Orvain and Velasco20), we identified a broad variety of methods to address sociocultural aspects of health technologies, some of which have already been applied in HTA. The results can help health care developers and innovators, providers, decision makers, and researchers involved in the assessment of health technologies to decide on the method(s) appropriate to the respective assessment situation.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462319000102.
Conflicts of interest
The authors declare that there are no conflicts of interest.
Open access
