The process of health technology assessment (HTA) has become an important part of effective decision making in high-income countries. Since its beginnings in the early 1970s, HTA has expanded to become a global industry that addresses issues as specific as evaluating a new medical device or drug, to evaluating policies concerning major public health problems (Reference Banta1). Governmental interest in HTA has paralleled the growth in healthcare spending (Reference Jonsson and Banta14). The aims of HTA have been quoted as “globalize the evidence, localize the decision” (Reference Eisenberg6;Reference Garner, Meremikwu and Volmink9).
The focus of HTA reports can vary according to their purpose, from providing evidence of effectiveness and cost-effectiveness of a health technology, to the consideration of specific political, ethical, social, organizational, or legal perspective for a particular setting. The methodologies adopted by HTA reports are varied and include primary research, systematic reviews, and economic evaluations (Reference Draborg4).
The preparation of these reports requires a great deal of time and effort and, inevitably, there is a monetary cost associated with this. A process of adaptation could maximize the value of HTA reports by using the parts that can be adapted to inform policy in other countries or contexts, and in the country or context for which the report was initially prepared. This would both support agencies with limited funding and reduce the cost and time associated with developing new HTA reports. Adaptation would be even more important in contexts or countries where resources are scarce and the disease burden is high.
To consider the adaptation of a report, it is first necessary to understand the process of adaptation and how it might be applied. The extent to which adaptation is possible depends partly on the generalizability of the topic under consideration and the different contexts in which it is to be considered (Reference Hailey10;Reference Sculpher, Pang and Manca18). Clearly, variations in clinical practice at a local level are inevitable. These variations may be influenced by many factors, including cultural, professional, legal, political, and economic issues. Some of these factors may act as a barrier to adaptation. The importance of these issues to the health technology under consideration, and to the context in question, will influence the feasibility and complexity of possible adaptation.
Few published accounts describe how the process of adaptation has been undertaken in the past (Reference Wang, Moss and Hiller22). The usefulness of a decision support tool to help local decision makers incorporate HTA into daily practice had been demonstrated previously in Denmark (Reference Ehlers5). However, we found no examples of an analogous tool for general use in the adaptation of HTA reports. The EUnetHTA framework provided a rich pool of experience and knowledge in which to canvas opinion concerning the need for such a tool, the form it might take, and how it might be used in practice.
The issue of linking globally available evidence to local contextual information means that HTA reports can rarely be simply taken from one context and applied to another (Reference Eisenberg6). In practice, the assessment in the HTA must be extracted, updated, and adapted. Resources, such as the International Network of Agencies for Health Technology Assessment (INAHTA) checklist (12) and the Equity-Oriented Toolkit for Health Technology (23) are available to guide those using and producing HTA reports. Other relevant documents include the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument, designed to provide a framework for assessing the quality of clinical practice guidelines (20). However, none of these resources has been designed with the specific purpose of supporting adaptation of existing HTA reports for a different context.
This study describes the work of EUnetHTA (8) Work Package 5 (WP5) (7). It aims to provide insight into the need for adaptation of HTA reports as viewed by fellow Partners of EUnetHTA; to review collective previous experiences of adaptation, examine how the process is best achieved; and to describe the process by which a consensus was reached as to how to facilitate the process of adaptation in the future, and a toolkit developed to facilitate the process of adaptation.
METHODS
An iterative process involving several methods was used to address our aims. These methods are described below, listed under the stages of development of the toolkit.
Stage 1: Previous Experience of Adaptation
Literature Review. Initial searches: two electronic databases were searched for published papers on adaptation: MEDLINE and Health Management Information Consortium (HMIC) from September 1996 to February 2006. These searches were limited to English language publications.
A list of identified published papers was made available to WP5 EUnetHTA Partners. Partners were asked in February/March 2006 if any papers were missing from the list and if they were aware of any gray literature in this area. In October 2007, members of the International Network of Agencies for Health Technology Assessment (INAHTA) (11) were also asked by email if they were aware of any gray literature in this area.
To update and broaden the literature review, subsequent literature searches were undertaken in January 2008. Five databases were searched: MEDLINE and EMBASE from 1996 to January 2008, HMIC to 2007, CINAHL (Cumulative Index to Nursing & Allied Health Literature) 1982 to December 2007, and CAB Abstracts (for nutrition and public health) from 2000 to December 2007. Searches were performed without limitation on language.
Preliminary Survey of Previous Experience. A preliminary survey was conducted to gain an understanding of the previous experiences of fellow EUnetHTA Partners in adapting HTA reports from other countries. In April 2006, a total of twenty-nine European HTA organizations/networks (twenty-seven EUnetHTA partners and two further interested organizations), were asked to complete this survey consisting of six questions (see Box 1).
Preliminary survey questions on adaptation
Questions 2 through 6 involved some aspect of quantitative response. We used SPSS (19) to analyze this data, looking at frequency and percentage data. Questions 1, 5, and 6 involved some aspect of qualitative response. The comments were assessed using a thematic analysis that focused on identifying themes. Themes were defined as patterns that appeared across participants’ comments and were identified by the careful consideration of each individual response. Quotations were chosen from the comments and were used to further elucidate each theme.
Stage 2: Initial Ideas on Toolkit Structure and Content
Delphi Survey Round 1. The results of the preliminary survey indicated a need to adapt HTA reports and for a tool to facilitate this process. In May 2006, a description of the proposed tool was circulated to the same twenty-nine European HTA organizations/networks the main objective of this Delphi survey (Reference Jones and Hunter13) was to collect comments and thoughts on the proposals put forward.
Full details of the survey questions and responses are beyond the scope of this paper, but are published in the HTA monograph series (Reference Chase, Rosten, Turner, Hicks and Milne2). The results briefly described here address ideas that were agreed on concerning the structure and function of such a tool.
Partners’ Meeting, London, June 2006. The comments from the first round of the Delphi survey were discussed at a face-to-face meeting of twenty-four WP5 EUnetHTA Partners in London, June 2006. Participants in the meeting considered the distinction between adaptation and adoption, the need for adaptation, the spectrum of adaptation, stages of adaptation, and the structure and function of a tool to facilitate the process of adaptation, henceforth referred to as the adaptation toolkit. Postmeeting, these ideas were circulated by email to WP5 EUnetHTA Partners to achieve overall consensus.
Stage 3: Toolkit Content Development
Delphi Survey Round 2. The results from the first Delphi round and the partners meeting were used to further develop the toolkit. A second round of the Delphi survey was conducted to obtain Partners’ views regarding these developments, and to consider further development of the toolkit. The Delphi round 2 survey consisted of four questions, each pertaining to a specific part of the toolkit. These are listed below:
Question 1: This question comprised a description of the adaptation process. It asked partners to consider at which stage of adaptation, the toolkit would help.
Question 2. This question comprised a description of the speedy sifting section of the toolkit. Partners were asked whether there any questions regarding this section which were missing.
Question 3. This question comprised a description of the main section of the toolkit and some of the issues raised by partners. Partners were asked for their thoughts on content.
Question 4: Any further comments.
Responses to these questions were assessed using a thematic analysis. A more detailed description of this survey is available in the HTA monograph (Reference Chase, Rosten, Turner, Hicks and Milne2).
Stage 4: Toolkit Content Development
Partners’ Commentary Work on Toolkit ‘Domains’. From initial work, a series of domains were identified for inclusion in the adaptation toolkit. Each domain was designed to addresses a specific element of an HTA report (technology use, safety, effectiveness, economic evaluation, and organizational aspects). The Partners were then asked to produce commentaries on the content of these domains, considering checklists, questions, etc., and issues for inclusion in the toolkit.
E-meetings with Partners. Once received, commentaries were collated and e-meetings for each toolkit domain were held to discuss which of the checklists, questions, and issues should be incorporated in the toolkit. As a result of e-meeting discussions, the Lead Partner collated the finalized checklists for each domain.
Stage 5: Review and Collation
As a result of e-meeting discussions, the Lead Partner collated the finalized checklists for each domain. The review process included two stages: to review each toolkit domain, and to review the draft toolkit in its entirety.
In the first stage of the review, Partners who did not undertake commentary work on a specific domain were randomly allocated the finalized checklists for one of the other four domains. In addition, all Partners were asked to provide final agreement on the first section of the toolkit (known as the speedy sifting section). This was undertaken in October 2006. As Lead Partner, NCCHTA reviewed the checklists, questions, and issues for each domain.
In the second phase of review, the entire toolkit was made available for review by twenty-eight Partners. This was undertaken in November 2006. The Lead Partner of WP5 made the appropriate changes to the toolkit as a result of review.
Stage 6: Applicability Testing
After these five stages of development and review, an initial version of the toolkit was finalized and then subjected to a process of quality assurance testing, termed applicability testing. This involved HTA partner organizations trying out the toolkit, gaining experience from its use and subsequently making improvements to the toolkit.
In the first round of applicability testing, twenty-eight EUnetHTA agencies were contacted to participate in testing, and sixteen agreed to participate. They selected one or more HTA reports from a different country and tested the toolkit as an aid to adapting the report to meet the needs of their own health service. They completed a questionnaire on their experience of using the toolkit and how it might be improved. Responses were submitted in June 2007. Three of these evaluators also underwent a 1-hour, face-to-face or telephone interview to further explore their experience. The aim of round 1 of applicability testing was to gain knowledge and feedback on using the toolkit. Subsequently, the WP5 Lead Partner made changes to the toolkit as a result of the testing.
The second round of applicability testing was launched at a face-to-face meeting held in Venice in September 2007. This round of applicability testing aimed to address what further aspects could be added to the toolkit to improve it after its production. Attendees of this meeting were invited to select one of five working groups and spend time examining the toolkit in the context of a selected topic. The topics for the work groups were drawn from the feedback in the first round of applicability testing.
Each group was given a set of questions to address, but was also invited to develop its own areas of investigation. The groups worked collaboratively by email over the following 5 months to produce a group report. In addition to the five groups, EUnetHTA Partners who had not worked on developing the toolkit were invited to adapt an HTA report, similar to the round 1 testing. Three groups agreed to undertake this task and were asked to complete a simplified evaluation questionnaire.
RESULTS
Stage 1: Previous Experience of Adaptation
Literature Review. No published accounts or examples of adaptation of HTA reports were identified. A paper on the generalizability of economic evaluations was identified (Reference Sculpher, Pang and Manca18), which provided guidance on adapting economic evaluations. Widening the search to include languages other than English produced no additional relevant results.
In gray literature, the partnership identified one German language paper (personal communication from member in Germany) on the development of a decision analytic model to facilitate adaptation (3). It described the parameters that should be taken into account in the transfer of evidence in decision analytical models. This paper was translated into English, and provided guidance on important factors to consider when adapting HTA reports. No further reports were identified.
Preliminary Survey of Previous Experience. Of the twenty-nine agencies/networks contacted, twenty-one chose to participate in the survey (72 percent response rate). It is important to note that, of the eight who did not participate, four did not have a formalized HTA agency in their countries and, therefore, believed that they had insufficient experience to complete this survey. In this respect, a more representative response rate of HTA agency experience would be 21/25 agencies (84 percent response rate).
Question 1 concerned the remit of the agency providing their response. The various European HTA organizations/networks had slightly different remits. However, despite these differences, all agencies had the central aim to research, or commission research into, the relevant aspects of new and existing health technologies.
In relation to question 2, policy makers were identified as the most important target audience for European HTA agencies.
Nineteen participants answered question 3, which related to adaptation; eleven (58 percent) indicating that they had adapted an HTA report from another country, and eight (42 percent) indicating that they had not.
Eighteen participants answered question 4; eight (44 percent) indicating that they knew of one or more of their HTA reports that had been used by another country and ten (56 percent) indicating that they did not know of any. These results indicated that adaptation is widespread, but not universal.
Question 5 was: “How useful is it for your HTA agency to make use of reports from other countries?” Seventeen of the twenty-one participants answered this question; with fourteen (82 percent) responding that it would be very useful and the remaining three (18 percent) responding that it would be quite useful. None of the respondents believed that it would not be useful.
Participants were also asked to elaborate on why they thought it would be useful to use HTA reports from other countries. Seventeen of the twenty-one participants chose to elaborate. The following themes and pertinent quotations were identified from their comments and shown in Box 2.
Themes identified from respondents regarding the use of HTA reports from other countries
Based on the responses and ideas from the preliminary survey, it became clear that the adaptation of HTA reports was considered desirable and that a tool to facilitate this process was needed.
Question 6 asked respondents to indicate the elements (or domains) of HTAs that should be focused on for adaptation, that is, which domains provide data and information that are most readily adaptable? The ten elements put forward were taken from previous work; the EUR-ASSESS framework (Reference Liberati, Sheldon and Banta16). Eighteen of the twenty-one participants answered this question. Table 1 below sets out each of the ten elements and indicates the number of participants who thought each should be focused on.
EUR-ASSESS Framework Elements: HTA Domains to Focus on for Adaptation
Responses to question 6, supported by more than 60 percent of those responding, indicated that information in the following five domains was believed to be more applicable and adaptable across different countries and settings:
• safety
• efficacy and effectiveness
• economic evaluation
• technology use and development
• organizational aspects
Participants were asked to elaborate on why they thought the elements they had highlighted were important.
Sixteen of the twenty-one participants chose to elaborate. A common theme that emerged in responses to this question was that the important parts of HTA reports are those concerning clinical effectiveness and efficacy, that is, the information that can be separated from the setting of the original HTA report. Box 3 presents quotations in response to this question.
Quotations about which domains to focus on in adapting HTAs
Following from this, domains such as ethical impact, legal aspects, and social and organizational aspects were rated less highly. Considering the spectrum of adaptation, these domains typically would be difficult to adapt into different contexts as they are more context specific. Specific information from the target setting would be required in the relevant section of the adapted HTA report Therefore, these domains were not incorporated into the toolkit.
Stage 2: Initial Ideas on Toolkit Structure and Content
Delphi Survey Round 1. Of the twenty-nine organizations/networks invited to participate, nineteen responded (66 percent response rate). The responses related to various aspects, which it was believed required further clarification or discussion. These aspects included the overall approach to adaptation, the adaptation process, stages of adaptation, and construction of the toolkit. These comments were pooled for discussion at a later date in the face-to-face Partners’ meeting in London.
Pooling the ideas and comments from the Delphi survey, it was proposed that construction of the toolkit should be as follows:
The initial section would comprise a screening tool. The aim of this tool would be to help users to determine whether the HTA report should be considered further for adaptation. The second section of the toolkit would comprise a critical appraisal tool. The aim of this tool would be to help users assess the relevance and reliability of a report from another setting and decide how to use it in their own setting.
The product of the adaptation process was viewed to be the information extracted from the report that is (i) relevant to the needs of the user, (ii) quality assessed, (iii) critically appraised, and (iv) ready to be incorporated into a new framework for an HTA report in the new setting or country.
The Delphi exercise indicated that the process of adaptation involves the following steps: (i) checking the relevance of the question(s) addressed in the original report to the question currently being addressed, (ii) identifying the information in the report that is relevant and most likely to be transferable to the setting currently being considered; (iii) assessing the reliability of the information under various domains (benefits, harms, cost-effectiveness, organizational impact, social and legal issues, etc.), and (iv) identifying and setting out the problems that may occur when the extracted, relevant, quality assessed information is transferred into a local HTA report; and deciding how to deal with them.
Partners’ Meeting, London June 2006. The output from this meeting was a description of the group's discussions. The partnership agreed that making use of all or part of a report from elsewhere could be done in a wide range of ways. It was agreed that: “There is a spectrum ‘of adaptation’; (Figure 1), with progressively more of the original report being used, affording the possibility of saving time and money through reduced duplication.” Additionally, the group described a distinction between adaptation and adoption, which is illustrated as follows: (i) Summarizing: translation of the summary and use of this for background information; (ii) Updating searches: use of the original search strategy to identify any more recent evidence or adding to the search strategy and extending it; (iii) Adapting: the systematic extraction of relevant HTA information from an existing report (from a whole report or from part of a report); and (iv) Adopting: making use of the report without making any changes (except perhaps translation into users’ own language).
The spectrum of adaptation.
Items 1 to 3 above require further work beyond the use of information from the original report to develop an adapted report.
Consensus. The EUnetHTA Partners involved in this work agreed on the concepts reported in these results. They also agreed on the need to develop a toolkit to aid the process of adaptation of HTA reports in the future. This toolkit should contain the highest ranking “domains” as determined from the preliminary survey, that is, safety, efficacy and effectiveness, cost effectiveness, technology use and development, and organizational aspects (see Table 1).
Stage 3: Toolkit Content Development
Delphi Survey Round 2. Twenty-one of the twenty-nine partners responded (72 percent response rate). Respondents provided comments on how to improve the questions in the toolkit, and requested the inclusion of examples of how the toolkit will actually work and what it should produce. The comments, examples and suggestions received in response to the second round of the Delphi survey were used to further develop the description of the toolkit.
Stage 4: Toolkit Content Development
Partners’ Commentary Work on Toolkit ‘Domains’ and E-meetings. The responses to the second round of the Delphi survey were used to further develop the description of the toolkit. Table 2 shows the number of partners allocated commentary work, producing commentaries and participating in e-meetings. Decisions were made within each e-meeting on which checklists to be included within relevant toolkit domains.
Number of Partners Working on and Deciding on Toolkit Domain Content
Stage 5: Review and Collation
Twenty-one of the partners reviewed a domain of the toolkit. Subsequently, twenty-three of the partners reviewed the entire toolkit. Suggestions for improvements were taken forward by the Lead Partner.
Stage 6: Applicability Testing
Round 1 Applicability Testing. The first round of testing resulted in positive comments on usability, usefulness, and content; in some areas the meaning of words needed to be clarified, and some made suggestions to improve the flow of the document. Participants believed that additional sifting questions should be added. Overall, the response was that the toolkit would be used and recommended to others to use. Following the first round of applicability, testing an updated version of the toolkit was produced.
Round 2 Applicability Testing. Three EUnetHTA Partners undertook a process of using the toolkit for adaptation of an HTA report. Two of the respondents used the toolkit to actively adapt a report and commented on all sections of the toolkit. One group adapted a retrospective approach to assess usefulness and focused on the economic evaluation domain.
Following the second round of applicability testing, minor amendments were made to the toolkit to produce a fourth version in October 2008. This represents the final version of the document in terms of the current project.
DISCUSSION
The aims of this study were to examine the processes of adaptation, to review previous experiences, and to investigate the possible need for adaptation of HTA reports: and subsequently to develop a tool to aid in the process of adaptation. Several methods were used. The varied methodology was one of the strengths of the study and enabled the issues surrounding adaptation, and the process itself, to be explored thoroughly, giving the opportunity for participants to contribute at all levels. An iterative process was used throughout; this facilitated progressive thinking among partnership members and enabled consensus and understanding to be reached on a variety of ideas and concepts, and was invaluable in the development of the toolkit instrument.
In undertaking our literature search, no published or gray literature accounts or examples of adaptation of HTA reports were identified. A limitation of the initial work was the use of an English-only search for papers. Subsequently, searches were repeated with no language restrictions, but this produced no additional relevant results. Other checklists and toolkits are available, for example the INAHTA checklist (12) and the Equity-Oriented Toolkit for Health Technology (23). However, these tools were designed as an aid to writing new HTA reports, not for adaptation of HTA reports from another context. The toolkit structure described would be the first tool to be specially designed for this purpose. This is the first published study to explore previous experiences of adaptation, the process, and benefits.
Our preliminary survey of European agencies yielded information concerning the process and need for adaptation. A consensus of views showed a need to reduce the workload on HTA agencies, speed up the process of preparing HTA reports, and avoid duplication of effort and use of resource. These views endorsed strongly the benefits of being able to adapt reports from other contexts or agencies to achieve these aims. Agreement was reached between participants that a tool is needed to facilitate this process, and that the tool should include guidance on adapting information and data in five important domains: safety, efficacy and effectiveness, cost-effectiveness, technology use and development, and organizational aspects.
Having achieved the initial objectives and having established that there is a need for adaptation, the next stage was to arrive at a consensus on how to facilitate the process of adaptation in the future. The face-to-face meeting in London enabled the partnership to reach a consensus through discussion and to agree on what we mean by adaptation. Participants initially agreed on what they understood to be the adaptation process, “the spectrum of adaptation,” and where they believed a toolkit would provide most support in this process. The Delphi surveys were instrumental in determining the possible structure and function of this toolkit. (The HTA monograph (Reference Chase, Rosten, Turner, Hicks and Milne2) covers these surveys in greater detail.)
The toolkit underwent two different rounds of applicability testing. Although the response rate from the partners was modest, the reports produced from the applicability testing indicted that the toolkit was useable, and that those who used it to adapt an HTA report would recommend it to another group. However, due to the Project's time scale, the toolkit did not undergo a review or quality assurance testing with any agency outside of our partnership. Further work to address quality assurance testing of the toolkit outside of the partnership will be undertaken in the near future.
All methods described above involved input from the partnership of HTA agencies, drawing on the strength of the partnership itself, which consisted of EUnetHTA Partners from across Europe, each with different systems and a wealth of HTA experience. This breadth of knowledge further strengthened the study.
CONCLUSION
The conclusion drawn from this work is that a need exists among EUnetHTA Partners for a tool to facilitate adaptation. This tool would be an aid to assess the relevance, reliability, and transferability of data and information from existing HTA reports originally written in a different context from the user's own. As a result of this work, the EUnetHTA Partners in WP5 developed such a tool, the adaptation toolkit. The toolkit, its possible uses and the accompanying glossary of terms used in adaptation have been described in more detail in separate publications (Reference Chase, Rosten, Turner, Hicks and Milne2;Reference Rosten, Chase and Hicks17;Reference Turner, Chase and Milne21).
CONTACT INFORMATION
Sheila Turner, BSc, PhD (s.turner@soton.ac.uk), Senior Researcher, Deborah L. Chase, BSc, PhD (dla1@soton.ac.uk), Senior Researcher, Ruairidh Milne, MB, BS, FFPH (rm2@soton.ac.uk), Senior Lecturer in Public Health, Andrew Cook, MBBS, BSc (Hons), MPH (andrewc@soton.ac.uk), Consultant in Public Health Medicine, Nicholas J. Hicks, MBBS, BA (Hons), FFPH (hicks@soton.ac.uk), Consultant in Public Health Medicine, National Coordinating Centre for Health Technology Assessment, University of Southampton, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK
Claire Rosten, BA, MA, PhD (c.e.rosten@brighton.ac.uk), Research Fellow, Research Design Service South East, University of Brighton, Mayfield House, Brighton, BN1 9PH, UK
Liz Payne, BA (Hons), PG Dip Lib, MCLIP (eapayne@f2s.com), Independent Information Specialist, Suzanne Coles, MBBS, BSc (Hons) (s.e.coles@soton.ac.uk), Specialist Registrar in Public Health Medicine, Eleanor Bell, MA (Hons) (ebell@soton.ac.uk), Project Manager, National Coordinating Centre for Health Technology Assessment, University of Southampton, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK
