Going Back to Basics

In this paper, we present an approach to HTA that incorporates reconstruction of stakeholders’ interpretive frames that was developed in the context of values in doing assessments of health technologies (VALIDATE), a project co-funded by the Erasmus+ program of the European Union. A key rationale for this approach is the insight that empirical evidence concerning a health technology should always be viewed in conjunction with the interpretive frame in which such evidence makes sense. With the VALIDATE approach, we wish to recall to memory that HTA from its original intent should be considered a type of policy analysis, clarifying to decision makers a range of options and how each of these options performs in light of a set of core values (Reference Banta and Luce9). As such, our approach may be viewed as a call to the HTA community to go back to its basics. In the remainder of this paper, the context of HTA is described and how HTA can be positioned in this context. It summarizes the how and why of the VALIDATE approach and identifies differences and commonalities with current approaches to HTA, using HTA of digital mental health care as an example. We also discuss what, exactly, we mean by “stakeholder”: Who may be qualified as such, and on what grounds? In a more practical vein, we will discuss how the VALIDATE approach might be incorporated in current practices of commissioning HTA, conducting HTA and decision making. Finally, the paper offers suggestions for the sort of knowledge and skills that HTA practitioners need to develop in order to put the VALIDATE approach into practice.

The Context of HTA

The context of HTA consists of communities that try to find, develop, and test ways of protecting, restoring, and strengthening the health of their population in a way that meets their core value commitments. This task is compounded by the highly dynamic nature of these settings, resulting from scientific and technological developments (e.g., artificial intelligence), new health threats (e.g., the COVID-19 pandemic), and cultural changes (e.g., changing conceptions of health and disease). It is in this context that HTA aims to provide insight into how and to what extent specific health technologies can help to protect, restore, or strengthen the health of specific patient populations in a given context, and how the development and use of such health technologies bring about a host of other concurrent changes. Thus, HTA aims to help various stakeholders in collaboratively arriving at a considered judgement regarding the value of specific health technologies (Reference O’Rourke, Oortwijn and Schuller10).

Judgements, Problem Definitions, Background Theories, Ethical Commitments, and Evidence

Drawing on basic insights from policy sciences (Reference Fischer11), the VALIDATE approach recognizes that judgements (here about the value of a health technology) are always closely tied to specific problem definitions of stakeholders and that, in turn, these two are intertwined with underlying background theory and ethical commitments. Jointly, these constitute so-called interpretive frames. Box 1 serves to briefly illustrate how judgements on the value of the health technology (i.e., proposed solutions), problem definitions, background theories, and ethical commitments are organized in interpretive frames, and how they relate to collecting empirical evidence for HTA.

The interpretive frame reflects that the increasing waiting lists and waiting times in mental health care are considered problematic (problem definition); the urgency of the problem is underscored because mental conditions such as depression are acknowledged to interfere with a wide range of daily activities and cause considerable suffering (background theory), and the alleviation of such suffering is considered to be an important collective responsibility (ethical commitment). Together with experiences with cognitive behavioral therapy and insights on its mechanism of action (e.g., schema activation; background theory), this leads to the conclusion that digital forms of mental health care could be a promising way of addressing current problems in mental health care (judgement of solution), provided that its (cost-)effectiveness is not compromised as the result of the digital format of provision (background theory and ethical commitment). Clearly, whether such is the case should be borne out by data from relevant empirical studies. The task for HTA would, then, be to systematically retrieve, critically appraise, and synthesize the results of such studies. A key question that should be raised, then, is whether such an HTA addresses all relevant concerns that different stakeholders may have with regard to digital health in the field of mental health.

Different Perspectives

To address that question, it is important to note that several competing, yet under-researched and underused views of digital forms of mental health care have been expressed. Coeckelbergh (Reference Coeckelbergh12), for instance, has argued that craftsmanship is an essential element of health care, and questioned whether digital forms of health care allow for such craftsmanship to materialize. In a somewhat similar vein, Scheepers (Reference Scheepers13, p. 182) claims that “What people [with mental health problems] need, most of all, is each other.” Denys (Reference Denys14) argues that the increased demand for mental health care is actually a result from economic prosperity and requires, as such, strategies for resolution other than the ones currently envisaged.

The question is how such rival interpretations should be understood. Specifically, what, if anything, should HTA practitioners do with them? To answer such questions, it is important to note that the rival interpretations do not necessarily refute all elements of the interpretive frame, presented above. They may conceive of mental conditions in a different way, but they would probably not deny that they can cause considerable suffering. Nor would they deny that attempts to alleviate such suffering is an important collective responsibility. And they would probably not deny either that the demand for mental health care has been on the rise, and that cognitive behavioral therapy has been shown to be effective in relieving symptoms of depression. Perhaps the best way to describe those rival interpretations, then, is by saying that they bring a different perspective to the issue. They draw attention to aspects that were not, or not sufficiently, addressed by others: What is the cause of this increasing demand for mental health care (Reference Denys14)? What is it that people in such situations really need (Reference Scheepers13)? What is it that happens in the clinical encounter, that makes it of such (therapeutic) value (Reference Coeckelbergh12)? This can be traced down to differences in their interpretive frames (along with commonalities), giving rise to a dissimilar judgement of digital forms of mental health care, a different interpretation of the evidence, and, importantly to HTA, a different body of knowledge and collection of empirical evidence that is deemed relevant to the issue at hand.

Doing HTA in an Integrative Way

Now, when commissioned to conduct a specific HTA (e.g., of digital forms of mental health care), the following items are available to the HTA practitioner:

• - the health technology, at some stage of its development, its variants, and the practices and contexts in which it is used;

• - a range of studies in which experiences with the technology are being reported, a variety of properties of the health technology have been examined, their design, their results, and the interpretation of the results; and

• - a series of judgements of the health technology, some favorable, others perhaps more critical, emphasizing different aspects of the health technology, each bringing a particular perspective on the health technology.

In the VALIDATE approach, the task for the HTA practitioner is to reconstruct the interpretive frames of which these judgements are part: What is the associated problem definition? What is the content of the background theories and ethical commitments that give rise to this judgement and problem definition? What sort of evidence would be considered plausible and relevant in the context of this frame? Such reconstruction can be done by conducting interviews with a variety of stakeholders, through document analysis, participatory observation, or a combination thereof. When the HTA practitioner has succeeded in reconstructing multiple interpretive frames and in validating them through respondent validation, a next step is to ask: What (technological) solutions and outcomes would it be most sensible to include in the analysis? Have these been enacted, have they been researched, and, if so, what experiences and results were obtained, and are those results the ones that would be collected when taking the different stakeholders’ perspectives into account? If it turns out that they have not been put in practice or put to the test, how might that be achieved?

Although current approaches to HTA are frequently confined to the systematic retrieval, critical appraisal, and synthesis of available data from relevant studies on the health technology of interest, the VALIDATE approach takes a step further. It aims to identify the interpretive frame of stakeholders in which the technology makes sense and that confers relevance to the collected data. It explores whether alternative interpretive frames can be developed, and, if so, what solutions and studies would be associated to them. Thus, it provides the HTA commissioning organization with a broader understanding of different perspectives on the issue, associated strategies for its resolution, and the quality and relevance of the supporting evidence.

Stakeholders in HTA: Who, Why, and How

The VALIDATE approach can also help to decide who should be considered as stakeholder in the context of a specific HTA: It is someone who in the HTA process can provide input to the development and articulation of a specific interpretive frame, explaining background theory and underlying ethical commitments, providing arguments, pointing to supportive evidence, and helping to articulate specific questions for HTA to be pursued. The reason why anyone would be able to fulfill such a role may vary, and result from experiential knowledge, formal knowledge, having a role in the decision-making process, or experiencing the impact of a decision.

The interest in getting involved as a stakeholder is, then, not merely related to the outcome of an HTA, but just as much to the assumptions underlying an HTA: Can this person identify with and subscribe to the assumptions that determine the options that will be considered in the HTA, the criteria that will be used to judge their value, and the studies that will be included? As such, the person who is involved as stakeholder in an HTA should be representative of a specific perspective (i.e., framing) of the issue, rather than of a specific group (e.g., patients, care givers, health care professionals, etc.). For, although perspectives may to some extent be associated with such group membership, different ways of framing the issue may still be found within them (as, e.g., in the case of mental health; see Box 1). In a similar vein, Sen (Reference Sen15) distinguishes between “membership entitlement” and “enlightenment relevance”: A person’s voice may be relevant because he or she is a member of the group that is involved in the negotiated policy, but it may also be relevant because of the enlightenment and the broadening of perspectives that such a voice might provide.

In practice, in order to involve stakeholders in HTA, it will usually be helpful to start consulting patients, their relatives and care givers, and the various healthcare professionals involved. The goal is to understand the issue from their perspective, by asking them to explain the sort of problems encountered or experienced, the sort of things that seem to work and those that do not, and incorporating such experiences in a wider framework of evidence and significance. In parallel, the literature should be consulted, also looking at opinion papers and position papers. Sometimes others have already done part of the work, for instance, the various frames that have been reported in the literature on obesity (Reference Baker, Brookes, Atanasova and Flint16). It is usually helpful to ask those who have been consulted whether they know of people who have a view of the subject that differs from theirs and continue from there (“snowballing”). Policy and decision makers should be included since they present a specific perspective on the issue, taking into account the use of resources for different purposes, giving rise to different opportunity costs. Likewise, it will usually be important to include manufacturers in order to reconstruct the framing that gave rise to the specific solutions (drugs, devices, and apps) that have been developed. Stakeholders may have organized themselves in interest groups (e.g., patients’ advocacy groups and healthcare professionals’ associations) that can usefully complement individual perspectives. Finally, the HTA practitioners and HTA agencies are themselves stakeholders that need to declare and reflect on their theoretical perspectives and normative frameworks.

The foregoing account can also help to determine whether stakeholder involvement has been sufficiently comprehensive. The key criterion would be whether wider involvement results in still different interpretive frames. The number of frames that can be developed is difficult to predict in advance but will rarely exceed five to seven (Reference Bächtiger and Parkinson17). It may, in fact, turn out to be only one, in which case we would be dealing with a well-structured policy problem (Reference Dunn18). If multiple frames could be reconstructed, it stands to reason that the decision as to which of those will be pursued further is made in accord with the commissioning organization. Relevant considerations in this regard will include feasibility, available budget and time, and anticipated policy implications and their legitimacy.

Finally, it needs to be acknowledged that stakeholder involvement merits a separate chapter in HTA reports. In addition to reporting the content of the interpretive frames that were developed, HTA practitioners would, in the interest of transparency, need to report who were involved as stakeholders, what they did in order to reconstruct their interpretive frames, to structure them and to obtain feedback, and how this feedback was used.

Incorporating the VALIDATE Approach into HTA Practices

Another question that needs to be addressed is how the activities that are associated with the VALIDATE approach can be integrated into practices of HTA. This may vary from country to country, depending on how such practices are organized. However, irrespective of how HTA activities are organized in specific jurisdictions, three generic tasks may be distinguished in relation to the conduct of HTA: commissioning an HTA, conducting an HTA itself, and making decisions on the basis of its outcomes. Although each of those tasks can in principle be performed by a single organization, more typically, the tasks will be divided among multiple ones. A commissioning organization acts on behalf of the organization that has political authority to make decisions (although in some cases, it has this political authority itself). Its main responsibility is to select topics (technologies that should be subjected to HTA), taking into account the context (e.g., major causes of morbidity and mortality, available resources and technological developments), policy objectives and policy instruments, and to make sure that high-quality, policy-relevant information becomes available in a timely fashion. Basically, it fulfills the function of a broker. The responsibility of the organization where HTAs are being conducted consists of the systematic and transparent retrieval, critical appraisal, and synthesis of the evidence that may be deemed relevant for the policy question. The results should enable the policy-making organization to decide and enact measures, justifying them on the results of the HTA on the one hand, and on an overall healthcare policy on the other (e.g., reducing burden of disease in the community in an equitable and responsible way). The communication between these different types of organizations (or tasks, when performed within a single organization) is precarious. The VALIDATE approach can actually be of help here, making explicit the interpretive frames that are operative in defining questions for HTA and revealing any differences in such frames that might otherwise remain unnoticed. The commissioning organization may fulfill an important role here. In its commissioning, it can stipulate that it expects not just the facts, but also the interpretive frames in which such facts are meaningful. It may also want to know whether multiple interpretive frames could be developed, on what grounds, and, if so, how that affects the HTA. The inquiry itself, that is, the gathering of a diversity of input and of structuring this input, can perhaps best be performed by the organization that conducts the HTA. This may require development of expertise in reconstructing and critically appraising interpretive frames (see below for a discussion of the kind of expertise that is needed). It stands to reason that the organization that conducts the HTA reports to the commissioning organization the findings of this scoping exercise, pointing out the implications for the HTA that is to be conducted, including an indication of the sort of evidence that is to be sought out, the likely availability of such evidence, the amount of the associated work, and a realistic timeline. At this stage, the commissioning organization will need to align with the policy-making organization in order to determine what is necessary and feasible. This process of HTA, with its various elements, and where and how the VALIDATE approach might fit in, are schematically presented in Figure 1.

Figure 1.

Values in doing assessments of health technologies (VALIDATE) and the process of HTA. Depicted are the various steps that can be distinguished in the HTA process. (1) At some point in its lifecycle, it is decided that a formal (re)assessment of a health technology would be appropriate. For the sake of transparency, it would be important to clarify who makes such decisions, on what grounds, and who are involved. It requires that some system is in place for identifying and selecting candidate emerging or evolving health technologies, and an adequately staffed and funded agency, preferably with a legally based remit to fulfil such task. Existing approaches range from ad hoc requests and nomination procedures to horizon scanning systems. (2) A next step would be that for selected technologies, the issue that those technologies are meant to help resolve is explicitly identified and explored. Specifically, what needs to be clarified at this stage is whether the issue can be framed in multiple, reasonable ways, giving rise to different questions to be explored. It is particularly here that the method of reconstructing interpretive frames, as part of the VALIDATE approach, can be usefully deployed. It should reveal to decision makers whether there are multiple perspectives on the issue that they might want to take into account, in what sense these perspectives differ from each other, and what implications they have for subsequent inquiry. We suggest that this stage of the process can best be characterized as one of scoping. (3) The outcome of the preceding stage requires some type of formal decision as to how to proceed with the inquiry: Can it stop here, or should it proceed and, if so, what research questions will be pursued and how? Likely considerations will include feasibility, available time and resources, perceived risks associated with nonpursuit, and so on. If it is decided that further inquiry is needed, this will typically consist of more in-depth analysis of the issues that have been identified during the scoping stage. Elements of the VALIDATE approach that can be used in this stage include methods for resolving conflicting requirements that result from different values. Such an inquiry may result in recommendations as to how a health technology is to be used in practice. (4) Whereas the former stages focus on decisions regarding the process of HTA, this stage focuses on decisions (or recommendations—depending on the organizations’ remit) regarding the health technology itself: Should it (continue to be) used (as then), should it be publicly funded, do certain restrictions apply, should its use and outcomes be monitored, and so on. Here, the relation with the VALIDATE approach is that it should enable decision makers to justify their decisions by reference to underlying values and background theories. Apart from what is being done, when, and how, the question may be asked as to who is acting at the various stages of the process. The answer may vary across different jurisdictions. Central governments may have outsourced various parts of the process to regional governments or implementing entities. The key question is, however, how the associated authority, responsibility, and accountability have been devolved.

A relevant question is, of course, whether such steps should always be taken, or in case of specific subjects only (e.g., where substantial controversy may be expected). To answer this question, we would suggest that the sort of activities that are advocated in the VALIDATE approach should be considered as a means to identify the nature of the policy problem for which the HTA is being conducted: Is it a well-structured, moderately structured, or ill-structured policy problem (Reference Hoppe19)? If substantial differences in background theories, normative commitments, or both could be retrieved, giving rise to different problem definitions and associated solutions, the problem is ill-structured or moderately structured. In such cases, some type of problem structuring is required as part of the policy process. Glossing over this fact incurs the risk of policy failure (Reference Bacchi20). The problem is that frequently, one cannot know in advance whether multiple interpretive frames can be developed with respect to a specific issue that deserve to be taken seriously and, if so, how these would translate in questions to be pursued in an HTA. We would therefore suggest incorporating exploration of the nature of the policy problem as a structural element of HTA. In fact, we consider VALIDATE as just one manifestation of a wider development in HTA and policy sciences more generally, rethinking the value of stakeholder involvement as an instance of strengthening the deliberative quality of healthcare policy and decision making. Advocacy for incorporating scoping in HTA and for setting up various forms of deliberative processes have developed in the same spirit, and a future task could be exploring whether the VALIDATE approach, or elements thereof, can be used to help structure such processes.

Skills and Knowledge That HTA Practitioners Need to Develop in Order to Start Adopting the VALIDATE Approach

There are, in our view, a how and a why to the question of requisite knowledge and skills. The how question is relatively straightforward. In addition to the knowledge and skills (both procedural ad attitudinal) that are currently taught in HTA curricula across the globe (Reference Mueller, Gutierrez-Ibarluzea, Chiumente and Oortwijn21) and that we consider fully and unreservedly relevant, the HTA practitioner would be well advised to develop the skill of identifying the relations between health technology and its associated primary studies on the one hand, and the underlying assumptions on the other hand. For this, learning to reconstruct stakeholders’ interpretive frames is imperative. The method has been developed in the context of policy sciences, and is well documented (e.g., Reference Grin, van de Graaf and Hoppe8;Reference Sorenson, Drummond and Kanavos2 Reference Sorenson, Drummond and Kanavos2;Reference Sorenson, Drummond and Kanavos2 Reference Durose and Richardson3). Semistructured interviews (qualitative research), document analysis, and participant observation can all serve as a means to produce material for this type of analysis. Key questions, such as who should be involved, and how stakeholders should be identified and included, were discussed above. Although we do think that it is important for students of HTA to familiarize themselves with the VALIDATE approach, HTA agencies may decide to enhance their current expertise in policy analysis, the field that inspired many of the ideas and methods of the VALIDATE approach (Reference Kazanjian and Green24).

The why of VALIDATE is a question referring to a different type of normativity. The VALIDATE approach explicitly holds that empirical evidence is not the sole, neutral arbiter of how things are and what matters. Current approaches to HTA would, perhaps, not make this claim either. Paradoxically, however, this is how it often works out in practice. A key difference between VALIDATE and current approaches to HTA is that the former acknowledges the contingency of empirical evidence (at least in the context of HTA) upon interpretive frames. Contingency, here, means that empirical evidence derives its relevance from the ethical commitments that are part of the interpretive frames. In a similar vein, plausibility of empirical evidence derives from an interpretive frame’s background theory. Therefore, in HTA, ethical choices are made upfront, and not, as suggested, for instance, in the assessment–appraisal model, after the empirical evidence has been collected. It also acknowledges that such choices are inherent to judgements of safety, effectiveness, and cost-effectiveness, challenging the separation of these issues from ethical, legal, and social/environmental issues, as is typically done in current approaches in HTA.

The Contingent Nature of Scientific Evidence

If, as we suggested, stakeholders may hold different interpretive frames, this means that for some, the sort of health technologies that have been developed to combat disease are not the sort of health technologies that they would have opted for in the first place. It also means that the sort of empirical studies that have been mounted in order to learn more about these health technologies are not the sort of studies that they would have liked to see. Moreover, finally, it means that the interpretive frames that have acted as backdrop to interpret those findings are not the frames that they would have preferred to see in this role. Such controversies cannot be resolved by merely conducting further empirical research while disregarding the differences in underlying interpretive frames. Instead, the sort of approach that would be required would address questions like:

• - In what ways is the problem (e.g., the increasing demand for mental health care) defined?

• - What sort of strategies, including those which are underemphasized in literature, practice, or debate, might be considered most promising, in case of the different problem definitions?

• - To what extent have such strategies been implemented?

• - What experience has been gained with those strategies, have they been subjected to critical examination, and, if so, what can be learned from those studies? Moreover, to the extent this has not been the case, how may such strategies be explored in future research and practice?

Current approaches to HTA do not naturally lead to such questions because evidence is considered to be given, independent from an interpretive frame. It is considered to serve as a neutral basis for reaching a judgement regarding the value of the technology in question. The reason why this approach runs into problems is that ethical choices have been made at the outset, bringing along their own informational requirements. That is the reason why in the VALIDATE approach, the analysis of interpretive frames, with their background theories and ethical commitments, is conducted as the start of the HTA.

Finally, doing and using HTA brings along certain responsibilities and raises issues of the role of scientific knowledge in public policy making. As noted by Fischer (Reference Daniels and van der Wilt25), the task is not just to provide technical information to problem solving, but also “to combine this with a new function of facilitating public deliberation and learning.” Awareness of such issues, and the ability to reflectively and accountably choose a role in the complex process of informing decisions by means of HTA, would, in our view, be appropriate subjects for any HTA curriculum and HTA-related capacity building activities.

An Issue of Accountability

For those involved in HTA, it should not come as a surprise that stakeholders can, and often do, have different views on how specific health problems can best be resolved, and what role specific health technologies can play. It raises an important question though: (When) should judgements of health technology be taken seriously, especially divergent judgements, and what does it mean, exactly, to “take them seriously”? It might be rejoined that HTA is about evidence, not about judgements. It is considered to be about producing or retrieving, critically appraising, and synthesizing evidence, in order to arrive at considered and evidence-based judgements. What, in our view, is being overlooked here is that empirical evidence is not self-contained; empirical evidence needs to be actively generated, and the nature of the empirical evidence that is being generated depends on the interpretive frames that assert themselves in this way. Therefore, exploring interpretive frames, thereby identifying the (technological) solutions that should be examined, the questions that should be the focus of the inquiry, and the means by which such questions can best be tackled, would be a fruitful and logical starting point for HTA.

This, in our view, is an issue of accountability: To whom is the HTA-practitioner accountable, and for what (Reference Kazanjian and Green26;Reference Daniels and van der Wilt27)? Many current approaches to HTA have little analytic resources to deal with different perspectives on a specific policy issue, other than concluding: “This is what the empirical evidence says, and then there are still a number of other issues.” It remains completely elusive, however, as to what the status of these “other issues” is. This results from the special epistemic status that current approaches to HTA ascribe to empirical evidence, trumping any claims or concerns that are considered largely subjective. In the VALIDATE approach, the importance of empirical evidence is acknowledged, but considered contingent upon the contents of interpretive frames.

Willingness of HTA commissioning organizations to incorporate analyses of stakeholders’ interpretive frames into HTA would reflect their ability to take seriously the diversity within their communities in prevailing interpretations of major health and healthcare issues. Ignoring such diversity is not only ethically questionable but also runs the risk of ending up with an HTA that is of marginal relevance to the policy-making process at best. In contrast, systematically positioning HTA in the context of stakeholders’ interpretive frames holds the promise of a less mechanistic, and more constructive type of policy analysis.