Background and Introduction
The European Network for Health Technology Assessment (EUnetHTA) supports international cooperation in health technology assessment (HTA) in Europe. EUnetHTA started in 2006 with a project and has since progressed through three joint actions, each part funded by the European Union (EU). The overarching aim of the joint actions has been to progressively strengthen cooperation in HTA in Europe, initially demonstrating the proof of concept, before developing shared methodologies and tools and finally embedding cooperation into routine practice.
EUnetHTA Joint Action 3 (JA3) started in June 2016 and ran until June 2021. One objective of JA3 was to support voluntary cooperation at the scientific and technical level between HTA agencies by providing the scientific and technical mechanism of a permanent cooperation on HTA. To meet this objective, an activity to develop the underlying principles for a future model of HTA cooperation was carried out.
This activity was undertaken against a background of ongoing discussions about a future legislative basis for HTA in the EU. In January 2018, the European Commission (EC) published a proposal for an HTA Regulation (1), the European Parliament adopted its amendments at first reading in February 2019 (2), and the European Council published its proposals in April 2021 (3). Discussions between the three groups have now reached a final text published in December 2021 (4). The regulation will provide a legal framework for future EU cooperation on HTA.
Objective
The learnings from JA3 that were identified as part of the activity to develop the scientific and technical mechanism of a permanent cooperation on HTA have a broader applicability to people, groups, and organizations involved in HTA who are developing HTA systems and collaborations. This policy article presents these learnings.
Methods
The activity was undertaken in two phases:
1. A descriptive phase that aimed to identify the elements of a model of HTA cooperation that were already in place, which elements could be improved or were missing, and to make recommendations to fill the weaknesses and gaps.
2. An analytic phase that aimed to bring together the learnings and recommendations from the documents identified in the descriptive phase to identify a set of underlying scientific and technical principles for a future model of HTA cooperation.
The Descriptive Phase
This phase of the activity was supported by a seventeen-member task group recruited from JA3 partners. There were three parts to the descriptive phase: (i) audit, (ii) discussion, and (iii) consultation.
Audit
A template was used to collect information about the elements of a model of HTA cooperation that were already in place. The template included standardized headings to support the process of categorizing the existing elements of a model of HTA cooperation and to help reveal the gaps where elements were missing.
Discussion
A series of briefing papers were prepared using the information collected in the template. These briefing papers were used to support discussions among JA3 partners to develop a consensus on the most important areas that were missing or needing improvement. To ensure a variety of perspectives, discussion groups included JA3 partners who were involved in governance structures (e.g., members of the Executive Board and Project Managers Group) and JA3 partners who were not.
During each discussion, JA3 partners were asked to:
1. come to a consensus about the important issues remaining for developing a model of HTA cooperation;
2. prioritize the issues identified; and
3. propose next steps to address the priority issues.
Consultation
A consultation with JA3 partners was used to validate the issues identified in the audit and discussion and gather comments about the proposed next steps.
The Analytic Phase
This phase of the activity was led by the JA3 Executive Board. This phase of the activity synthesized data from (i) the report from the descriptive phase of the activity and (ii) the formal milestones and deliverables for JA3. Particular attention was paid to compiling the experiences from across the different joint HTA activities carried out in JA3, identifying the lessons learned, and making recommendations for future working and further development.
Conceptual Framework
A conceptual framework helped organize the data for the analysis. The framework was developed as part of work carried out by the JA3 implementation and impact work package and was used flexibly and adjusted as the data analysis developed. The framework includes six broad categories:
1. Concept and strategy, for example, the vision, purpose, and aim of HTA cooperation;
2. Participation of individuals, groups, or organizations in HTA cooperation, for example, to include engagement of HTA agencies and also experts, stakeholders, and related organizations;
3. Governance and decision-making, for example, the leadership and decision-making processes to support HTA cooperation;
4. Science and procedures, for example, methodological guidance, procedures, and templates that support HTA cooperation or development of joint HTA activities;
5. Infrastructure, for example, services, tools, and databases needed to coordinate and meet the aims of HTA cooperation;
6. Evaluation, for example, feedback mechanisms to ensure that HTA cooperation can respond to changes and remain relevant to those involved.
Results
The Cooperation Framework
JA3 was a voluntary network with time-limited project funding, joint HTA activities were undertaken by a work package made up of a group of agencies and each activity had an agency that was the activity lead and who was responsible for the final output from the activity. Five learnings related to the framework needed to underpin HTA cooperation were identified from the JA3 experience.
1. An ability to act as its own entity. To function optimally the cooperation must be able to approve its outputs, be liable for its outcomes, own its infrastructure, and create official agreements with other organizations.
2. A resourcing and commissioning model that guarantees projects and project teams. Voluntary participation provides flexibility around involvement but can affect the ability to constitute project teams, the ability to undertake prioritized work, the clarity about uptake of outputs, and the ability to implement templates and procedures.
3. A set of guiding principles and processes applied across all activities. The work package structure provides flexibility in how joint HTA activities are undertaken but can allow differences in approach across different types of joint HTA activity to grow. Where differences have no clear rationale, this can affect the ease of engagement and perceived coherence of the cooperation.
4. Ongoing stability. Time-limited project funding can lead to gaps between projects without resourcing. The absence of continuity creates staffing issues since staff leave toward the end of each project when their contracts expire. It can also create issues with stability of infrastructure hosting and maintenance when project responsibilities end and can influence planning and usability of outputs as all activities must be finished at the end of the project.
5. Flexibility to evolve. In JA3, changes needed to be made to working practices to respond to demands arising from the COVID-19 pandemic (5). HTA cooperation must have the flexibility to respond to feedback from participants and changes in the external environment that affect the outputs and activities that will be valued by users.
Valued HTA Activities
Three main joint HTA activities were carried out in JA3.
1. Early Dialogues (ED): that is, advice provided by multiple HTA agencies to technology developers about their evidence generation plans.
2. Joint and Collaborative Assessments (JA/CA): that is, relative effectiveness assessments carried out jointly by multiple HTA agencies.
3. Post-Launch Evidence Generation (PLEG): that is, HTA activities undertaken jointly after a health technology is launched to support further evidence generation to address uncertainties in the evidence at the time of initial launch.
The experience of JA3 is that each of these activities are valued activities that should be part of ongoing HTA cooperation. PLEG is a less established joint HTA activity compared with ED and JA/CA, but PLEG activities are becoming ever more visible and the profile of PLEG will continue to grow (Reference Moseley, Vamvakas, Berntgen, Cave, Kurz and Arlett6).
In addition to the activities above, JA3 partners also identified a need to expand the joint HTA activities undertaken to include horizon scanning and re-assessment.
1. Horizon scanning systems identify emerging health technologies and future technological and methodological challenges. Horizon scanning helps HTA processes be prepared for and respond to methodological and technological innovation.
2. Processes to update the outputs of joint HTA activities are needed to identify when an output might be out of date and need updating and to ensure that an HTA remains of value to decision makers after it is published.
Finally, JA3 identified that there should be explicit and transparent links between joint HTA activities on the same topic, so that activities can inform each other and capitalize on the joint HTA activities already undertaken.
The valued joint HTA activities are summarized in Figure 1.
Joint HTA activities and the links between activities along the life cycle of a health technology.
Principles of Involvement
In JA3, agencies who were users or producers of HTA were engaged as partners. Joint HTA activities were also supported by stakeholders, including industry, patient organizations, providers, and payers. Emerging from JA3 are a set of involvement principles that support engagement in HTA cooperation.
1. Participation should be guided by underlying principles applied across all activities and a framework for engagement. Differences in approach to participation in different HTA activities should have a clear and justified rationale.
2. Participants should be engaged in specific joint HTA activities but also need to be provided with general information updates and consultation in decisions that will affect them.
3. The structures for collaboration need to be as simple as possible. Complicated and burdensome structures can act to decrease the possibilities of cooperation because of the challenges engaging with them.
4. Anybody directly participating in a joint HTA activity (e.g., in the project team or as an expert) should provide up-to-date information about their interests. The declaration of interest for each participant should be assessed before any involvement starts. Normally, conflicts of interest should be avoided.
5. When designing joint HTA activities and preparing supporting resources:
• Cooperation structures, methods, and tools across activities should be aligned where possible to make participation easier.
• All participants should have an overview of the role, responsibilities, and expectations of participation.
• Participant documents should be harmonized so that they are easier for participants to become familiar with, encouraging ease of use, understanding of activities, and avoidance of procedural errors.
• There should be publicly available guidance on involvement.
6. Relying on a single method of communication such as an intranet may not reach all relevant participants. Multiple methods of communication should be used.
HTA Agency Participation in HTA Cooperation
HTA cooperation relies on the agencies in the cooperation to undertake activities and to use the outputs created. The number of HTA agencies involved in EUnetHTA has grown over time, starting with thirty-five in the EUnetHTA Project 2006–8 (Reference Kristensen, Mäkela, Neikter, Rehnqvist, Håheim and Mørland7) to eighty-one agencies in JA3. Over time, EUnetHTA has experienced how HTA responsibilities in a country change, new agencies are set up, and existing agencies change and move organizations as the health care and political environment changes. Having experienced the growth and change in EUnetHTA, there are four learnings from JA3:
1. HTA cooperation needs to be able to allow more than one agency per country to participate. This is to reflect the variations in how HTA is undertaken in different countries and the need to ensure inclusivity and that all relevant perspectives are captured.
2. A large cooperation will require significant time and resources to manage and coordinate it to ensure ongoing engagement of all participants.
3. Participation in the cooperation needs to be underpinned by transparent criteria about who needs to be engaged, based on their responsibilities and remit rather than their organization title.
4. A simple mechanism to add, remove, and change participants in “real-time” is necessary so that the cooperation continues to reach out, engage, and meet the needs of the agencies it serves.
Governance of the Cooperation
In JA3, operational decisions were made by an Executive Board made up of HTA agencies who were leads and co-leads of the work packages and elected JA3 partners. Part-way through JA3 changes to the Executive Board were needed to improve its ability to make decisions.
Arising from this JA3 experience are the following learnings:
1. Bodies making decisions on behalf of the network, need to:
• follow a set of shared and transparent principles,
• have flexible membership that can be configured differently for different types of health technologies,
• have a membership that is representative of the members HTA approaches,
• have a process for assembling the decision-making body that is transparent and inclusive and open to all HTA bodies,
• have clear communication structures with information cascade to participants that are not part of the governance and decision-making group.
2. The decision-making body cannot function alone and need to have access a range of resources and personnel to support decision making, including:
• a Secretariat to provide administrative and coordination support;
• a chair and vice chair to work with the Secretariat and to lead meetings;
• expert groups to work and advise the decision-making body on scientific and procedural issues;
• a comprehensive set of corporate governance policies with the rules and procedures guiding the cooperation.
3. Governing and decision-making structures need to be subject to evaluation and have sufficient flexibility to change to ensure that they continue to function optimally as the cooperation and its work evolves.
A summary of needed elements of a governance structure is shown in Figure 2.
The support needed by the decision-making body to function optimally.
Project Teams
Different configurations of project teams have been trialed in different EUnetHTA joint actions and activities. At the beginning of the project, there were large teams of partners involved in each activity, by the end of JA3, project teams had become smaller and now have greater similarity across different HTA activities. Based on this evolution in project teams, the following learnings emerged.
1. Each joint HTA activity should be guided by the following recruitment principles:
• The project team must have the relevant skills and experience for the activity.
• Selection criteria in terms of skills and experience should be transparent and available.
• The procedure for selecting the team should be transparent, so it is understood how the skills of team members are evaluated and by whom.
2. Joint HTA activities do not need to be undertaken by a large project team. It is normally sufficient for a project team to include:
• an HTA agency that leads the activity;
• a second HTA agency that supports them;
• additional project team members who provide input and review;
• dedicated project management to manage the activity and the project team.
3. Adequate activity management is critical when managing different teams of people undertaking different and multiple activities. Each joint HTA activity needs to have a project manager who is part of the project team and who acts as a dedicated point of contact. The project manager must have access to scientific experts to ensure that the scientific elements of activity management, for example, judging eligibility of requests, quality checking of the output, and managing differences of opinion are carried out appropriately.
4. To support a high-quality joint HTA output, it is necessary that project teams are not working in isolation and are able to access:
• scientific and methodological expert advisory groups, for example, information retrieval experts and statistical specialists;
• a common set of scientific tools to complete the activity;
• a range of centralized support services, for example, publishing and editing, IT, and stakeholder engagement;
• scientific oversight provided by a standing group of experts to ensure consistency in approach within a joint HTA activity;
• independent conflict resolution;
• corporate governance support, for example, to manage conflicts of interest.
• input into the project plan from the target users.
Experts and services providing guidance to project teams should be involved early in the process from the planning stages so that issues that could affect approval are identified in a timely manner and can be resolved without creating process delays.
A summary of the needed elements to support project teams is shown in Figure 3.
The support needed for project teams to produce joint HTA outputs.
Engagement of External Actors
JA3 involved a variety of stakeholder groups but had a particular aim to improve the involvement of experts in joint HTA activities and the collaboration with the regulatory authority the European Medicines Agency (EMA).
The following are key lessons from JA3:
1. HTA agencies vary in their approach to and understanding of stakeholder engagement and this is an area with divergent opinions. The opinions are particularly divergent in regard to the involvement of industry and payers and the involvement of stakeholders once the outputs from joint HTA activities have been drafted but are not yet finalized.
2. Patients and healthcare professional input should be routinely included in an HTA activity.
3. Stakeholder engagement in topic identification and project planning is important and appropriate.
4. Stakeholder and expert contributions should be supported and recognized:
• The contribution that stakeholders and experts provide in a joint HTA activity must be transparent and documented in the output.
• Guidance must be given to stakeholders about the input expected.
• Guidance must be given to project teams about using and reporting the input.
5. Collaboration with regulators such as the EMA adds value and is able to respect the differences in the remits and responsibilities of HTA agencies and the regulators.
6. HTA cooperation must link to the environments with which it interacts and pro-actively and strategically work to collaborate with related organizations and networks.
Support Services
In JA3, some support services (e.g., stakeholder management, information technology (IT), editing, communications) required to carry out joint HTA activities and coordinate cooperation were provided centrally and others were decentralized and carried out by work packages or individual HTA agencies.
A learning from JA3 is that support services are fundamental to maximizing cooperation and producing high-quality outputs and that these are more effectively, efficiently, and consistently delivered if centralized. This is because of the need for a harmonized approach and/or specialized skills. Particular areas where greater centralization is beneficial include:
• Stakeholder management and expert engagement with a particular focus on patient engagement.
• Conflict of interest: a centralized committee to assess potential conflicts of interest and a central support function to manage declarations of interest made by project teams and experts involved in joint HTA activities.
• Information services: the foundation of HTA is scientific literature, a centralized service to identify, procure, and share literature in accordance with copyright law supports scientific rigour.
• Editing and publishing: centralized editing and publishing supports a consistent “house style” which is important for users of reports.
• Activity coordination: for example, organization and planning of meetings, liaison with collaborating organizations, and submission and exchange of documents;
• Scientific software: for example, software to support information retrieval, evidence synthesis, and statistical analysis. Tools are not necessarily interoperable and so where participants have to use their own software this can create a challenge to collaborate.
• Quality Management: a consistent approach to quality management across activities improves engagement and perceived coherence (Reference Luhnen, Ormstad, Willemsen, Schreuder-Morel, Helmink and Ettinger8).
A summary of the support services required to support HTA cooperation is shown in Figure 4.
The support services needed for an optimal cooperation. IT, information technology.
Training and Development
The main aim of JA3 was to increase production of joint HTA outputs. This focus on increased production meant that less resource was given to methodological development and training compared with previous joint actions.
The experience of JA3 has been that there remains a need for ongoing support in three areas: training, science development, and capacity development and implementation support.
1. Training: JA3 has shown that we are not yet at a point where HTA agencies have sufficient staffing for joint HTA activities to be fully sustainable. HTA cooperation must always ensure that the pool of staff available to be in project teams is sufficiently large. To support sustainability, there needs to be
• ongoing training to ensure that pools of expertise are expanded and maintained; and
• opportunities for observer status in joint HTA activities and on advisory bodies to support up-skilling of participants to take on greater responsibilities.
2. Science development: An ongoing program of scientific development activity is needed to ensure that new areas requiring technical guidance are identified and guidance developed before they become an issue for joint HTA activities. Existing templates and guidance need to be kept up to date as methods change.
3. Capacity development and implementation support: HTA is still establishing in some countries and these HTA agencies should be able to access support for capacity development and implementation to maximize the benefits they obtain from HTA cooperation.
Information Technology
The project nature of EUnetHTA has meant that IT has grown in a piecemeal fashion with IT developed in different project phases, by different agencies and using different software.
At the end of JA3, partners recognized that the piecemeal development of IT over the joint actions has affected usability and ease of engagement for participants and the ability of the HTA cooperation to maintain and develop its IT efficiently. IT is fundamental to supporting efficient and effective HTA cooperation. More specific learnings from JA3 were:
• IT should be managed as a single centralized function that supports all IT needs of the cooperation.
• Different IT tools should be integrated to support the smallest number of different tools required for necessary function. For example, a single tool used to manage all joint HTA activities.
• To support efficient maintenance and development, IT tools need to be developed consistently using the same software, programming, and hosting.
• IT tools should ideally all use the same access credentials so that participants do not need to maintain multiple user identities and passwords.
• IT requires ongoing and stable maintenance and hosting to maintain participant engagement.
• IT tools and infrastructure must be subject to a regular review and evaluation to help ensure that they continue to meet needs.
Discussion
From its inception, EUnetHTA aimed to facilitate the establishment of the technical and scientific elements of a sustainable European HTA network. The EU HTA regulation is a decisive milestone on the path toward sustainable HTA cooperation, but it must necessarily be accompanied by the technical elements necessary for its implementation.
The purpose of the activity that underpins this article was to define the constituent elements of the future model of HTA cooperation in Europe and a technical report (White Paper) is available on the EUnetHTA Website (9). Since discussions about the regulation were ongoing at the time the activity was completed, it was not possible to define exactly the structures and processes applicable to any specific legislative text. Instead, JA3 partners worked toward developing a consensual scientific and technical framework on which to build the future. From this framework, emerged learnings and principles that are specific enough to be relevant to the development of EUnetHTA and the implementation of the regulation, but also that have a generic quality that can support other groups or organizations setting up their own HTA services or looking to initiate their own regional collaborations.
With the final regulation text now available, the results in this article coming from the experiences of the EUnetHTA Joint Actions remain relevant to the establishment of the new EU HTA cooperation. Many of the lessons learnt in JA3 have come about from the experience of undertaking a scaling-up process to increase the number of joint HTA outputs, and from the experience of having an increasing number of participants in the network. This increase in the number of joint HTA outputs and in network participants reflects the reality of HTA within which the new EU HTA cooperation will have to operate.
This article focuses on the lessons learned for implementing network cooperation. However, sustainable HTA cooperation will require changes from within HTA agencies and from stakeholders, as well as other agencies involved in or affected by the HTA process. While future HTA cooperation should respect existing HTA procedures within countries, the differences in healthcare systems across the EU mean that to maximize the efficiency of HTA cooperation changes at a country level will be required so that HTA cooperation becomes part of routine working practice replacing steps of agency procedures.
Funding
The contents of this paper arise from the project “724130 / EUnetHTA JA3,” which has received funding from the EU, in the framework of the Health Programme (2014–2020). Sole responsibility for its contents lies with the author(s), and neither the EUnetHTA Coordinator, the European Commission nor any other body of the European Union is responsible for any use that may be made of the information contained therein.
Conflict of Interests
The authors declare to have no conflict of interests.
Open access
