The introduction of more and more digital health technologies (DHTs) places demands on the development and implementation of new health technology assessment (HTA) methods as well (Reference von Huben, Howell, Howard, Carrello and Norris1–Reference Kidholm, Ekeland and Jensen4). At present, mobile health (mHealth) applications form the most significant part of the technological solutions in the digital health field (5). In 2018, Moshi et al. (Reference Moshi, Tooher and Merlin6) showed in their study that the HTA assessment frameworks, which were in place at the time, had shortcomings in performing comprehensive assessments for mHealth in a medical context. Since then, there have been many developments due to the increasing number of HTA frameworks that are available for use in performing assessments for DHTs and especially for mHealth (Reference Vis, Bührmann, Riper and Ossebaard3;Reference Kidholm, Ekeland and Jensen4;7–9). In some countries, such as Germany, the reimbursement process is available for mHealth products (10).
However, a mHealth application can only be part of different DHT services. For example, a solution can be a digital service that includes sensors or communication devices to monitor the activities of daily living (ADL) of older adults living at home, and, in such situations, the mHealth application can inform healthcare professionals or relatives about the health status of older adults (Reference Camp, Lewis and Hunter11;Reference Anttila, Niemelä, Anttila, Koskinen, Rantanen, Tuikka and Knaapi-Junnila12). There is a growing trend of introducing artificial intelligence (AI) and robotics solutions as well as combinations of different DHTs to the healthcare sector (Reference Camp, Lewis and Hunter11–Reference He, Baxter and Xu14). These new products—especially AI—increase the complexity of HTA methods (Reference Alami, Lehoux and Auclair15). These new technologies require new considerations about implementing them in the health care environment (Reference He, Baxter and Xu14).
In Finland, the Ministry of Social Affairs and Health has published the Hyteairo and KATI programs, which aim to develop Finnish healthcare further by using novel DHTs (Reference Anttila, Niemelä, Anttila, Koskinen, Rantanen, Tuikka and Knaapi-Junnila12;13). Those programs aim to support the utilization of AI and robotics in Finnish healthcare and provide technologies to support smart aging and care at home (Reference Anttila, Niemelä, Anttila, Koskinen, Rantanen, Tuikka and Knaapi-Junnila12;13). Based on the aims of those programs, more evidence-based information is needed for AI, robotics, and DHTs of all kinds, for example, intelligent remote monitoring platforms, which support independent and safe living at home (Reference Camp, Lewis and Hunter11–13).
Based on these needs, a new general-purpose HTA framework and an associated process called Digi-HTA were developed through the cooperation of the Finnish Coordinating Center for Health Technology Assessment (FinCCHTA) and the University of Oulu’s Faculty of Medicine (Reference Haverinen, Keränen and Falkenbach2). The goal of the Digi-HTA framework is to cover all new DHTs such as mHealth, AI, and robotics solutions as well as combinations of different DHTs (Reference Haverinen, Keränen and Falkenbach2). Digi-HTA considerations include effectiveness, safety, and cost as well as patient and organizational considerations; but it does not cover ethical, social, and legal issues (Reference Haverinen, Keränen and Falkenbach2). The Digi-HTA also covers the technical characteristics of DHTs, such as interoperability, data security, and protection as well as usability and accessibility (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16). In addition to these common requirements, the Digi-HTA includes its own domains in AI and robotics (Reference Haverinen, Keränen and Falkenbach2). The Digi-HTA process uses a list of information security and protection requirements for social and health care sector procurements developed by Cyber Health Project as the criteria to perform assessments for data security and protection (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16).
The Digi-HTA framework and its associated process have been implemented since the end of 2019 to perform assessments for DHTs, and the first Digi-HTA recommendations are available on the FinCCHTA webpage (17). In the Digi-HTA assessment process, technology companies provide information about their DHT products using the Digi-HTA framework and Cyber Health Project’s criteria for data security and protection (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16). Companies also provide all the necessary documentation, such as research results (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16). The information on the products under assessment is supplemented by literature reviews carried out by HTA experts and cyber security specialists (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16). The Digi-HTA process provides informative recommendations for healthcare decision-makers (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16;17). Currently, due to the decentralized healthcare system in Finland, municipalities can decide for themselves whether to implement certain DHT products (18). From the beginning of 2023, the responsibility for decisions will be shifted to bigger entities—twenty-one new well-being services counties, which will be funded by the state (18). Presently, there is no such reimbursement process for DHTs implemented in Finland as, for example, in Germany (10;17). A previous study has already noted that despite good collaborative HTA reports, the impact of HTA information on hospital decision making appears to remain low in Finland (Reference Sihvo, Ikonen and Mäkelä19). This emphasizes that HTA should have an impact on practical implementation (Reference Rourke, Werkö, Merlin, Huang and Schuller20). Thus, it would be paramount to understand the healthcare decision-maker’s information needs about DHTs and the barriers to utilize the information provided by Digi-HTA recommendations when this new and still relatively unknown Digi-HTA process was introduced.
By implementing a general-purpose HTA framework, we would not exclude certain technologies from our assessments (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16). This way we could better support decision-makers and enable the introduction of new and innovative DHTs in healthcare (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16). However, the goal of evaluating very different types of DHTs with the same HTA framework places great demands on the comprehensiveness and clarity of it (Reference Haverinen, Keränen and Falkenbach2;Reference Vis, Bührmann, Riper and Ossebaard3).
Aims of the study
Clarify whether the information contents of the Digi-HTA recommendations are helpful for healthcare decision-makers?
Evaluate in which phases of the decision-making process the Digi-HTA recommendations are the most useful?
Identify what the major barriers preventing the use of Digi-HTA recommendations are in healthcare decision making?
Evaluate if DHT companies consider the Digi-HTA framework a suitable tool for performing assessments for different kinds of DHTs?
Material and methods
Exclusion Criteria
Evaluating the suitability of a list of information security and data protection requirements developed in the Cyber Health Project to perform HTA for different kinds of DHTs is not within the scope of this study (Reference Jääskelä, Haverinen and Kaksonen16).
Recruitment
The selection criterion for healthcare decision-makers was that as part of their work, they made decisions related to DHTs, and they represented different healthcare service areas. For healthcare professionals to be sufficiently familiar with the subject, the Digi-HTA recommendations and the assessment process were presented in detail to them at remote meetings. After the sessions, a questionnaire was delivered to the participants by email and they were asked to participate in the study (between March 2020 and March 2021).
The DHT companies were recruited when participating in the Digi-HTA assessment process (between January 2019 and October 2021). Potential companies were also recruited by email (between March 2020 and April 2020). The selection criterion was that the companies offer DHT products.
Data Collection
Data from healthcare professionals and DHT companies were collected through a web-based survey that included both structured and open-ended questions. The Webropol survey and reporting tool was used to collect data.
We received responses to our survey from twenty-four healthcare professionals. Their area of healthcare services and job roles are presented in Table 1. Some of the study participants represented more than one area of healthcare services.
Healthcare professionals (n = 24)
A total of eight DHT companies responded to our survey. Six of them participated in our Digi-HTA assessment process. Two of them did not participate in the assessment process, but they comprehensively reviewed the Digi-HTA assessment framework and responded to our survey (Table 2).
Digital Health Technology Products Offered by the Companies Involved in the Study (n = 8)
Data Analysis
Qualitative methods were used in this study. To form a comprehensive understanding of the material collected, it was first analyzed using an inductive content analysis method (Reference Elos and Kyngäs21). Initially, the healthcare professionals’ and technology companies’ responses to the open-ended questions were open coded. After that, the analyzed data were grouped into subcategories, and then similar findings were combined into the main categories to enable the final analysis. Finally, both textual and numerical data were analyzed by using the quantification method (Reference Sarja22).
Ethical Considerations
Before this study, the potential participants got the participant information in Finnish by email before agreeing to participate in the survey. All participants joined the study voluntarily. The results have been processed so that no healthcare professional participants can be identified in the results or quotations of this study. The personal information of DHT company representatives will not be presented in the results.
Results
The Information Content of the Digi-HTA Recommendations
Healthcare professionals answered whether the Digi-HTA recommendations provide clear and beneficial information to help them make decisions about DHT products (Table 3). They gave their opinions on the domains of Digi-HTA, which from their point of view, they considered the most important (Table 3). The healthcare professionals responded about the kind of information about effectiveness, which from their perspective, should be available in the Digi-HTA recommendations (Table 3).
The Information Content of Digi-HTA Recommendations (n = 24)
All respondents found that the Digi-HTA recommendation, as well as its recommendation scale, is clear and informative. According to the responses, four healthcare professionals would have liked the Digi-HTA recommendations to contain more information. Specifically, they wanted more information about the suitability of the product for different user and patient groups, customer experiences to achieve benefits, the necessary changes in the care process to gain benefits, and the cost benefits of long-term use of the product. There was also a need for more information on which municipalities or health care organizations are already using the product. One respondent wanted more information about how companies have considered sustainability and environmental issues in their business.
Applicability of Digi-HTA Recommendations in the Decision-Making Processes
Decision-makers responded about whether they saw the Digi-HTA recommendations as being beneficial for their own work and also about which phase in the decision-making process they were the most useful to them.
Of the respondents, 58 percent had previous knowledge about HTA. Most of the respondents (n = 21) agreed or strongly agreed that Digi-HTA recommendations are beneficial for their work. Digi-HTA recommendations would be the most useful for respondents before a procurement phase (n = 8), before product piloting or testing processes (n = 8) and in the procurement phase (n = 7). They would also be beneficial in market surveys (n = 6) and in a competitive tender process (n = 5). Some respondents (n = 9) indicated that they could solely rely on the Digi-HTA recommendations while making their decisions. They stated that this would be the case when, for example, the recommendation score of the product was high or when choosing between two equivalent suppliers. They could also solely rely on the Digi-HTA recommendations when identifying suitable products for a piloting or cocreation process or if the subject of the procurement was precisely defined.
What Factors Are Preventing the Use of Digi-HTA Recommendations as Part of Decision-Making Processes
Healthcare professionals expressed their views on the main barriers to the use of the Digi-HTA recommendations in decision-making process and what should be done to overcome these barriers.
Respondents thought the problem was that this new HTA process is not known well enough (n = 4). Thus, most of the respondents (n = 16) found that there is a need to raise awareness by arranging an extensive information campaign about the Digi-HTA recommendation process and its possibilities to support decision-makers. Healthcare professionals noted that the lack of integration between Digi-HTA recommendations and the decision-making process is an issue (n = 3), so they hoped that the Digi-HTA recommendations would be better integrated into the decision-making processes (n = 10).
According to the responses from healthcare professionals, they found that a low number of available Digi-HTA-recommendations acts as a barrier for utilizing them in their decision-making process (n = 5). Therefore, they thought that increasing the number of Digi-HTA recommendations would be an important step toward better integrating them into the decision-making processes (n = 3). Some of the respondents mentioned that in a rapidly evolving digital world, Digi-HTA recommendations should also be available quickly so that decision-makers can be supported in the desired timeframe (n = 5).
Suitability of the Digi-HTA Framework to Perform Assessments for Different Kinds of DHTs
DHT companies responded about their experiences in responding to the Digi-HTA-framework (Table 4). For each HTA-domain, they answered how applicable (Likert scale 1–5), from the perspective of their own product, they perceived the domain’s questions (Table 4). DHT companies also responded about whether the Digi-HTA framework could have a guiding effect on the product development of digital products in the field of healthcare (Table 4). Furthermore, where needed, they provided feedback and suggestions for improvements in each HTA-domain.
DHT companies’ feedback from the Digi-HTA framework (n = 8)
a Responses from three companies which have AI / a machine-learning algorithm included in their products
b Response from one company whose product falls within the robotics category
According to the health technology companies, most of them agreed or strongly agreed that the number of questions in the Digi-HTA assessment framework is reasonable (n = 5), and that answering the questions was easy (n = 5). One company stated that the number of questions in the Digi-HTA framework was too large, and another company noted that it was difficult to answer the questions about it. According to five companies, the information requested for the Digi-HTA was easy to find within their company or organization. However, one company mentioned that it was difficult to find the necessary information. Six out of eight companies thought that the Digi-HTA framework could steer the product development of digital health and well-being products.
Health technology companies stated that most of the questions in every HTA domain apply to their product. The average score was less than four only in the usability and accessibility domain. For the AI domain, we received responses from three companies that have AI or machine learning (ML) algorithms included in their products, and only one company’s product fell into the robotics category. The most feedback and suggestions for improvement were received in the following domains: usability and accessibility (n = 14), effectiveness (n = 9), cost (n = 4), technical stability (n = 2), and interoperability (n = 1).
As a general development proposal, it was suggested that the clarity of some questions would be improved so that they would be as understandable as possible for all health technology companies (n = 10). It was also hoped that the clarity of the Digi-HTA questionnaire itself would be improved, particularly regarding the instructions it contains and that the visual appearance, and the key questions would be highlighted (n = 8). Because DHT companies provide company confidential information about their product for assessment, they wanted the data handling process for that information described in more detail in the Digi-HTA assessment framework (n = 2). One company pointed out that HTA requirements should be harmonized with medical device regulation requirements.
During the first Digi-HTA assessments performed for rehabilitation and medicine dispensing-robots, we realized that from the viewpoints of the end-user (patient or citizen) and healthcare professional two separate domains for usability and accessibility are needed. For example, the evaluated exoskeleton rehabilitation robot includes an mHealth application for the rehabilitator, so end-user device (robot) and an mHealth application for the professional should be evaluated separately. For this reason, separate usability and accessibility domains for end-users and healthcare professionals were implemented in the subsequent development version of the Digi-HTA framework.
Despite this change, companies still found answering the questions about usability and accessibility difficult because we received most comments in this domain. However, we did not receive precise feedback from companies on what should be developed in the usability and accessibility section. One important aspect based on the responses was that digital platforms also pose new challenges to evaluate usability and accessibility. In some cases, a digital platform can support different peripheral devices, such as a video connection and remote monitoring or remote measuring devices, provided by other vendors.
Health technology companies gave feedback that, they are not all knowledgeable as to what information about effectiveness is required to prove the benefits of their products. Also, not all companies had a previous understanding of what would be the most substantial research evidence on product effectiveness. Some companies would have liked that the experiences of clients and their feedback about a product could be emphasized instead of research evidence. One company suggested that a separate HTA domain be created for customer feedback.
One company also said that rather than focusing only on the studies of a particular product, studies of similar products would be considered when assessing the effectiveness of a DHT product. This was found to be beneficial for new products when there is no evidence on the effectiveness of that particular product. One company gave a notable response: it suggested that a new HTA domain should be created for environmental aspects in order to consider them in the assessments.
In the cost domain, determining the costs of a scalable digital platform solution was considered challenging because the product would be tailored to every customer with different types of peripheral devices. Furthermore, defining the costs for a digital service provided by several public organizations was considered challenging because organizations may have very different cost structures.
Discussion
A study conducted in 2017 using a web-based questionnaire found that the impact of HTA recommendations on Finnish healthcare decision making appeared to remain low (Reference Sihvo, Ikonen and Mäkelä19). The aforementioned study and its methods were the starting point for our study. However, its themes were modified to make a better fit for DHTs and to better reflect the situation where the new HTA method had just been introduced. The healthcare professionals that participated in our study find the information provided by the Digi-HTA recommendations clear and informative as well as useful in their decision-making processes related to DHTs. Typically, in Finland HTA recommendations have been utilized in a procurement phase, but according to the respondents they could also use Digi-HTA recommendations before product piloting and testing processes and in market surveys (Reference Sihvo, Ikonen and Mäkelä19).
Despite the beneficial information content of Digi-HTA recommendations, it is challenging to integrate them into the healthcare decision-making processes. Fox example, in Finland, there is currently no formal process for such an integration. However, at the national level, the better use of the Digi-HTA recommendations is already being promoted as part of the implementation of the Hyteairo and Kati projects (Reference Anttila, Niemelä, Anttila, Koskinen, Rantanen, Tuikka and Knaapi-Junnila12;13). One possibility to increase utilization in the future could be to combine assessments for a certain type of DHT products with a reimbursement process, as is currently the case in Germany, this would presumably also increase the willingness of companies to participate in the process (10).
The decision-makers that participated in the study pointed out that more communication is needed about this new Digi-HTA process and its benefits for healthcare professionals so it could be exploited better in the future. The small number of available Digi-HTA recommendations was also perceived as an obstacle to their use in the decision-making process. Based on that, it is obvious that a sufficient number of Digi-HTA recommendations for different DHT product categories are needed to facilitate product comparisons.
Traditionally, the key domains in HTA recommendations have been effectiveness, cost, and safety, as recommended by the MUMM (Managed Uptake of Medical Methods) program implemented in the past in Finland (Reference Sihvo, Ikonen and Mäkelä19). During our Digi-HTA development work, we found that data security and protection, as well as usability and accessibility, must also be adequately taken into account in DHTs. Therefore, we added these to the list of key domains to be assessed (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16). The results of our survey also indicate the same thing. Based on the responses, information on the effectiveness of products was seen as the most important HTA domain in the Digi-HTA recommendations, but the other most important domains for which information was needed were information about the product and its maturity level, usability and accessibility, and data security and protection, and safety. Surprisingly, only two respondents saw the cost of the product as one of the most important domains. This could be explained by the fact that DHT companies typically provide information on the price of products for decision makers, so they feel this information is readily available, but in other areas they would need more detailed information, which is not easily available.
As can be seen from the product categories of the companies that responded to our survey, the term DHT can encompass very different products. Most of these products include mHealth or web-based applications as part of the service. These applications can then be connected to a robot, a virtual reality headset, a digital platform, or measurement and monitoring sensors. Products may also contain ML or AI algorithms integrated into them. This poses the challenge of how to conduct HTA assessments reliably and comprehensively for these very specific types of products. Due to the heterogeneity of DHTs, their effectiveness is also critically influenced by their implementation in a given context (Reference Haverinen, Keränen and Falkenbach2;Reference He, Baxter and Xu14;Reference Rehfuess and Gerhardus23). For example, AI solutions may require changes to care processes and integration into existing hospital ICT systems before deployment to get the most out of them (Reference Haverinen, Keränen and Falkenbach2;Reference He, Baxter and Xu14). Some DHTs are aimed directly at citizens, so they must be user-friendly and accessible to all targeted users (Reference Haverinen, Keränen and Falkenbach2). This requires that assessment frameworks should include all key domains related to deployment of DHTs.
From an international perspective, there are differences in the assessment frameworks as to how widely they support different DHTs or what different domains they contain (Reference Vis, Bührmann, Riper and Ossebaard3:Reference Kidholm, Ekeland and Jensen4;Reference Moshi, Tooher and Merlin6–9). For example, the aim of the National Institute for Health and Care Excellence’s evidence standards framework (ESF) is to cover a wide spectrum of DHTs and to ensure that they are clinically effective and offer economic value (7). ESF may also be used with DHTs that incorporate AI using fixed algorithms (7). To assess the interoperability, data security, usability, and accessibility as well as other technical issues of DTH, ESF should be used alongside digital technology assessment criteria (DTAC), which includes these aspects (7;8). Some frameworks focus on solely assessing mHealth solutions (9). In 2020, the European mHealth Hub (9) evaluated the 24 mHealth assessment frameworks developed in Europe, such as the German DiGA model. According to the study, there was great variation in how comprehensively the 12 key assessment domains included in this study were handled in evaluated frameworks (9). Vis et al. (Reference Vis, Bührmann, Riper and Ossebaard3) evaluated the available eHealth frameworks based on the scientific literature, and their study found that frameworks mostly included technical performance and functionalities as well as cost and clinical considerations. The comprehensive model for the assessment of telemedicine applications (MAST) also emphasized that for the HTA process to be appropriate, the product itself must be sufficiently mature (Reference Kidholm, Ekeland and Jensen4).
Compared to other existing assessment frameworks for DHTs, the Digi-HTA assessment framework includes a comprehensive set of domains, identified as essential for assessing DHTs (Reference von Huben, Howell, Howard, Carrello and Norris1–Reference Kidholm, Ekeland and Jensen4;Reference Moshi, Tooher and Merlin6–9;10;Reference He, Baxter and Xu14;Reference Jääskelä, Haverinen and Kaksonen16). The only domains not covered are ethical, social and legal issues. The framework includes effectiveness, safety, and cost as well as patient and organizational considerations (Reference Haverinen, Keränen and Falkenbach2). Also, the technical characteristics of DHTs, such as the maturity of a product, interoperability, data security, and protection as well as usability and accessibility, are handled in the Digi-HTA assessment process (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16). The Digi-HTA framework is also designed to enable HTA activities for a wide range of DHTs, not just a particular technology (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16;17). The study performed by Vis et al. (Reference Vis, Bührmann, Riper and Ossebaard3) highlighted the fact that the development of a framework to support all eHealth services may present challenges to making it sufficiently comprehensive and accurate for a particular eHealth service. This was a key concern when the Digi-HTA framework, with its support for a wide range of DHTs and coverage of a wide range of domains, was developed and was also one of the aspects of this study (Reference Haverinen, Keränen and Falkenbach2;Reference Jääskelä, Haverinen and Kaksonen16).
However, based on the responses to our survey from health technology companies, it can be concluded that the general-purpose Digi-HTA framework would be a suitable tool for collecting data from their DHTs and performing HTA their products. In practice, the Digi-HTA framework has already made it possible to collect data on different types of DHTs, such as three different types of healthcare robotics, digital platforms, and mHealth solutions besides enabling the carrying out of assessments for them (17). Health technology companies stated that most of the questions in every HTA domain apply to their product. Companies reported the biggest challenges in answering the questions in the category of usability and accessibility. During performed Digi-HTA assessments, we found also this area challenging to implement to make it as comprehensive as possible for mHealth applications, robotics solutions as well as different types of digital platform solutions.
Another key issue that emerged from the responses was that a large proportion of companies was not used to showing the effectiveness of their products through research. Thus, a new approach and guidance would be needed to enable the manufacturers of DHT to increasingly produce research data on the effectiveness of their products in the future. In Germany and UK, for example, companies are well instructed as to what kind of evidence on the effectiveness of products is required to obtain reimbursement for the product (7;10).
Though the general and primary purpose of the Digi-HTA framework is to support HTA activities for DHTs, it could potentially guide the development of a product and anticipate its further development and market access, for example, by using it as a tool for the early HTA approach (Reference Giunti, Haverinen and Reponen24). This was also confirmed by the companies that replied to the questionnaire because six out of eight companies think that the Digi-HTA framework could guide the product development of digital health and well-being products. This could help new companies entering the industry to understand what key issues need to be considered for DHT products before entering the market and during upcoming HTA processes.
This study has been carried out in the Finnish healthcare environment, but we believe that the results would be applicable to other countries where healthcare decision-makers have had experience with DHT products. Presumably, these decision-makers could have a similar idea of what essential information they would need about these new technologies. Due to the differences in the structure and decision-making processes of Finnish healthcare and the arrangements in other countries, not all the considerations in this study related to decision making are applicable internationally. We also think that in those countries where the digitalization of healthcare has already progressed, DHTs offered for assessments could be similar to those included in this study, so similar kinds of HTA methods such as Digi-HTA would be needed in these countries as well. One notable aspect which arose from the responses in this study was how sustainability and environmental issues would be taken into account in HTA in the future, so this perspective would be interesting to study internationally (25).
Conclusion
According to the healthcare professionals that participated in the study, the Digi-HTA recommendations contain clear and beneficial information. Better integration into decision-making processes would help them to make better use of these recommendations. More awareness about the Digi-HTA recommendation process and its possibilities to support decision-makers is needed according to the responses to our survey. Also, according to the study participants, a low number of available Digi-HTA-recommendations is seen as a barrier to utilizing the recommendations in the decision-making process.
The field of DHT covers a wide range of technology solutions such as mHealth, digital platforms, AI, and robotics. Responses of DHT companies to our survey indicate that the general-purpose Digi-HTA framework would be an applicable tool for performing assessments for DHT products that correspond to those surveyed. Further development is still needed based on results and the biggest improvement is needed for the usability and accessibility domain to make it comprehensive for all kinds of DHTs. Generalization of the results of this study requires further studies with more samples or more defined study protocols.
Limitations
This study was conducted using a web-based survey, and this may have an impact on the versatility of the qualitative analysis as the survey may steer respondents’ opinions in a particular direction. This study describes a situation where the Digi-HTA process has been introduced in Finnish healthcare. We think this study could open up a debate on this topic that should be studied even more deeply in the future as Digi-HTA recommendations are increasingly utilized in decision making.
The verification results of the Digi-HTA framework cannot be considered fully generalizable, because we received a low number of survey responses from DHT companies in each product category. For example, only one robotic company responded to our survey. However, the answers of the representatives of different types of DHT products were consistent; for the most part, they thought that most of the questions in the Digi-HTA assessment framework were generally applicable to their product.
Although the Digi-HTA assessment framework covers a comprehensive set of domains that are identified as essential for assessing DHTs, it does not include legal, social, and ethical aspects. From the legal perspective, the Digi-HTA process requires only that a manufacturer should provide the declaration of conformity document for products classified as medical devices as well as CE-marked products. However, when assessing AI solutions, for example, ethical considerations may be crucial. Therefore, the key aspects of those domains should also be considered in the subsequent development version of the Digi-HTA framework.
Funding Statement
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Conflicts of Interest
The authors declare that they have no conflict of interest.
Open access
