The assessment of medical devices is an important function to support the appropriate introduction and use of health technology (1). Policy makers increasingly adapt health technology assessments (HTAs) (Reference Wilkinson and Drummond2–4) to evaluate high-risk medical devices (HRMDs). This serves to either assess different types of health technologies (e.g., anatomic cementless knee prosthesis of different bearing surfaces indicated for a knee joint replacement) or to assess different medical device brands grouped into the same class of health technology (e.g., different anatomic cementless knee prostheses indicated for a knee joint replacement). HRMDs remain in the patients’ body and are grouped into the risk class III of medical device regulation, which includes knee joint implants (5). HTA contributes decision making at the macro level (e.g., decisions about the listing of technologies in the National Formulary) as well as at the meso level (e.g., decisions about the purchasing of medical devices). The latter evaluation type is described in the literature as a hospital-based HTA or mini-HTA. Studies show that these can have a major impact on decision making and decision outcomes (Reference Sampietro-Colom, Morilla-Bachs, Gutierrez-Moreno and Gallo6–Reference McGregor and Brophy8), for example, preventing intra- and postoperative failures. High quality information is important for conducting HTAs (Reference Kidholm, Ehlers, Korsbek, Kjaerby and Beck9).
Mexico lacks regional and national initiatives for the advancement of high quality of information on orthopedic HRMDs and HTAs at the level of purchasing decisions (Reference Lingg, Wyss and Duran-Arenas10;Reference Lingg, Wyss and Durán-Arenas11). Between 2014 and 2015, we qualitatively analyzed the regulation, assessment, and management of orthopedic HRMDs to understand how they influenced clinical procedures. This analysis provided the starting point for the present article, and the findings suggest that (i) decision-making processes focus on mechanical and technical specifications and fail to integrate a broader spectrum of decision criteria, such as clinical long-term outcomes of medical device brands; and (ii) orthopedic specialists are not adequately integrated into decision making because their main role is to physically evaluate medical device brands that have been taken into consideration for decision making by the purchasing entity.
In Mexico, HTAs are conducted by the Centre for Excellence in Health Technology (Centro Nacional de Excelencia Tecnológica en Salud, CENETEC), which is a specialized agency of the Ministry of Health to support policy decisions. Their reports are used by the Inter-institutional Commission of the National Formulary for Health Supplies to decide whether a new health technology will be listed or not in the National Formulary. To date, the literature lacks studies about the relevance of HTA activities at the level of purchasing in Mexico. The Mexican health system is highly segmented and fragmented (Reference Gómez-Dantés, Sesma, Becerril, Knaul, Arreola and Frenk12) and characterized by a public/private mix of hospital providers. The public sector represents the majority of treatments and encompasses various social security institutions and decentralized state healthcare systems.
Studies about purchasing are of interest because tenders are frequently selected as a purchasing scheme and the organization of purchasing decisions (e.g., at hospital level or regional level) is based on the demand for a medical device of a single hospital, or a region covering several hospitals, or the entire social security institution. This determines how much decision autonomy is left to the hospital level. Reviewing the information provided by the governmental Web site Compranet (13), which was established to make public procurement activities in Mexico transparent, shows that the scope of tenders among the different social security institutions has varied considerably over the past years and that financial aspects dominated decision making.
This cross-sectional study assessed the attitudes and thoughts of orthopedic specialists regarding their role in purchasing decision making of HRMDs, their experience with purchasing processes, and the relationship to clinical results, as well as potential areas for improvement. Based on the above criteria, we discuss the relevance of adequate technology assessment and potential areas for improvement of purchasing processes.
MATERIALS AND METHODS
The study is based on primary data collected through a survey, representing end-users of orthopedic HRMDs. Our research approach is based on a working framework and presented in Figure 1. The framework is guided by two considerations: (i) end-users have low purchasing decision autonomy, and (ii) purchasing fails to integrate a broader spectrum of decision criteria such as clinical long-term outcome of medical device brands. The framework influenced the data collection and analysis in three ways. First, the framework helped to describe the role of orthopedic specialists in purchasing decision making. Second, it was used to assess their experience with purchasing processes and the relationship to clinical results. Third, ratings of them on areas for improvement of outcomes of purchasing processes were obtained.
Research framework.
Study Population
The study population was defined as orthopedic specialists in Mexico. Data or data sources listing orthopedic specialists and orthopedic procedures in Mexico are not publicly available, and the associations of orthopedic specialists or national academies do not provide an accessible database on which this study would have been able to draw from. Based on information received from the associations of orthopedic specialists, suppliers of the medical device industry, and pharmaceutical companies, we estimated a total of approximately 1,700 orthopedic specialists in Mexico. Furthermore, we estimated a total of 26,000 orthopedic procedures (hip and knee joint replacement), of which 45 percent are concentrated in the State of Mexico and the Federal District. We used the estimated data on orthopedic specialists and procedures as orientation for the data collection.
Sampling
We used a non-probability (purposive) and maximum variation sampling technique to reach a heterogeneous group of participants with a sufficient number of orthopedic specialists per subgroup, defined by four participants’ characteristics: state, sector, care level, and work experience. The characteristic “sector” is important because the public sector is divided into services provided by social security institutions for the employees, and state healthcare systems, which cover the population that is self-employed or without an insurance cover. For the Web-based survey, we searched listings of public hospitals, private hospitals, and associations of orthopedic specialists to identify potential participants. For the paper-based survey, we selected the sample from attendees of a national conference of orthopedic s in Mexico.
Data Collection
We selected two routes of data collection (Web- and paper-based questionnaire) and used a structured self-administered questionnaire (Supplementary Table 1). The questions were developed based on the results of two preceding studies (Reference Lingg, Wyss and Duran-Arenas10;Reference Lingg, Wyss and Durán-Arenas11). The questionnaire was pre-tested on five orthopedic specialists in Mexico. We captured responses for questions regarding purchasing using a 5-point Likert scale. The questionnaire used to collect the data was administered on an online platform, SurveyMonkey®, and at a scientific conference.
Characteristics of Respondents by Recruitment Route
The Web-based data collection took place between June and October 2015 using SurveyMonkey® to inform orthopedic specialists about the survey and to collect their answers to the questionnaire. We made additional contact through email to identify orthopedic specialists to increase the participation rate. The paper-based data collection took place on April 29–30, 2015, and was conducted by two research assistants that were trained in the study procedures for administering questionnaires to study participants. We collected data from attendees of the “XXVIII National Mexican Conference of Orthopedic s and Traumatology 2015”. The two research assistants presented our study at this conference during registration and coffee break, informed attendees about the survey, and asked them if they were interested to participate in the survey.
Data Analysis
For the data analysis, we merged both sets of data and applied descriptive statistics. Ordinal scaled variables were transformed into a binary outcome by grouping the first two categories (e.g., “fully agree” and “agree”) into “1” and all other categories into “0.” Descriptive analyses were conducted using STATA software version 14 (STATA/IC 14.1 StataCorp). The data were tabulated, and proportions were compared between categories using the chi-squared test. We used Fisher's exact test to examine the significance of the association between specific variables of interest, and assessed the level of agreement between responses to different items using the kappa statistic (Reference McHugh14).
Ethics
This project was approved by the ethical committees from Autonomous University of Mexico (Mexico) (Date of approval: November 4th 2014, FMED/CI/SPLR/188/2014) and by the Ethical committee from northwest and central Switzerland (Switzerland) exempted it from ethical review according to Swiss law (Date of exemption: June 24th 2014). All interviewees gave written informed consent before the interview.
RESULTS
Sample Description
We identified a total of 600 eligible orthopedic specialists from the Web-based search and 215 orthopedic specialists during the conference (Figure 2). Overall, 187 orthopedic specialists agreed to participate in the survey, representing a 23 percent response rate based on the number of orthopedic specialists contacted directly. Approximately half of the respondents had more than 15 years of work experience, and a fifth had less than 5 years (Table 1). Most respondents were working in the public sector (68 percent, n = 127), and located in the Federal District and State of Mexico (58 percent, n = 108).
Survey process.
There were some differences in the characteristics between those recruited through the Web-search and those recruited by means of the conference. Those attending the conference mostly worked in the public sector, had less seniority, and were located in other states. However, there were no indications that these differences affected the attitudes of the participants.
Our study was concerned with purchasing processes in the public sector. Nevertheless, 60 participants reported working in the private sector, and of these, 58 percent had more than 15 years of work experience and only 10 percent had up to 5 years of work experience. In Mexico, orthopedic specialists often work in both the public and private sector, and after their retirement from the public sector, continue working in the private sector. These participants reported on their experience when they previously worked in a public sector or opinions about purchasing in public sectors. We present the data as a whole and describe any relevant differences between orthopedic specialists from the public and private sector. Table 2 summarizes the responses of the participants by key themes.
Questionnaire Responses on Role of Involvement, Problems in Clinical Practice, Effects of Procurement, and Aspects Improving Clinical Practice
Orthopedic Specialists’ Role in Purchasing Decision Making
Of the 187 surveyed orthopedic specialists, 86 percent agreed with at least one statement that they had some form of involvement in decision making: “I can influence” (81 percent), “My opinion is considered” (72 percent), and “It contributes to my clinical needs” (69 percent). There was no evidence of a difference for the area of purchasing inputs: “Planning of material and quantity” (88 percent), “Effectiveness of medical device” (82 percent), and “Medical device technology” (74 percent). Fisher's exact test for the three areas of purchasing inputs always resulted in 0.00. Interrater reliability provided a kappa between 0.63 (substantial agreement) and 0.84 (almost perfect agreement) (Reference McHugh14). Those who thought they had some form of involvement in decision making had good agreement in the area of purchasing inputs.
We observed a slightly lower proportion of participants indicating to have some form of involvement in decision making concerning orthopedic specialists employed by the social security institution, Mexican Institute of Social Security (Instituto Mexicano de Seguro Social, IMSS). The answers of participants in respect to other characteristics had little variation in terms of their care level (“Secondary care: 81 percent; “Specialist care”: 88 percent; “Private”: 87 percent), their location in the country (“Mexico & DF”: 87 percent; “Other states”: 85 percent), or their work experience (“Up to 5 years”: 88 percent; “6 to 15 years”: 84 percent; “More than 15 years”: 87 percent).
Problems in Clinical Practice and Associated with Purchasing
Of the 187 surveyed orthopedic specialists, 161 (86 percent) reported that their clinical practice has been exposed to one of the following situations: “Material failure” (64 percent), “Low effectiveness of medical device” (63 percent), “Available medical device does not meet patients’ needs” (51 percent), “Obsolete medical device technology” (47 percent), “Incomplete provision of sets” (68 percent), and “Delayed provision of implants and instruments” (72 percent). However, 172 (92 percent) reported effects of purchasing on clinical practice (purchasing output attributes), including: “Wrong material planning” (89 percent), “Low clinical effectiveness of product” (83 percent), “Inferior medical device technology” (79 percent), “Limited product portfolio” (83 percent), “Low quality of instruments” (85 percent), and “Sub-standard services received from medical device supplier” (87 percent).
Participants agreed slightly more to have an affected clinical practice based on “Quality of services received” and “Material planning.” Fisher's exact test for problems in clinical practice always resulted in 0.00. Inter-rater reliability resulted in a kappa between 0.26 (fair agreement) and 0.62 (substantial agreement) (Reference McHugh14). Those who reported that they had an affected clinical practice or saw effects of purchasing on clinical practice had a moderate agreement in the area of purchasing output attributes.
We observed a slightly lower proportion for experiencing problems in their clinical practice in terms of care level “Private” (78 percent), and work experience “More than 15 years” (81 percent). The answers of participants in terms of other characteristics had little variation: care levels (“Secondary care”: 93 percent; “Specialist care”: 88 percent), sectors (“IMSS”: 94 percent; “ISSSTE” [Instituto de Seguridad y Servicios Sociales de los Trabajadores del Estado; Institute of Social Security and Services for State Workers]: 100 percent; “SSA”: 84 percent; “Other”: 83 percent), and work experiences (“Up to 5 years”: 95 percent; “6 to 15 years”: 87 percent). We did not observe a difference for participants looking at the characteristic states (“Mexico & DF”: 85 percent; “Other states”: 87 percent).
Relationship between Orthopedic Specialists being Involved in Purchasing and Orthopedic Specialists Experiencing Problems in Clinical Practice
Among the respondents who had some type of involvement in decision making, many stated they had problems in their clinical practice in general (n = 137), which was related to the availability of materials and effectiveness or technology of medical device, or reported seeing effects of purchasing on clinical practice (n = 147), which was specified as low effectiveness of medical device, lack of meeting patients’ needs, obsolete medical device technology, lack of complete and adequate services provided through supplier. Fisher's exact test to examine the significance of the association between participants reporting that they have some form of involvement and problems in clinical practice resulted in 1.00. Fisher's exact test between participants reporting that they have some form of involvement and their clinical practice has been exposed to problems associated to purchasing resulted in 0.722. The Kappa statistic was below 0.2 for affected clinical practice in general and for effects of purchasing on clinical practice.
Aspects That May Improve Clinical Practice
Overall, 93 percent reported at least one aspect by which clinical practice could be improved: “Sustainable clinical long-term data of medical devices,” “Access to electronically stored clinical data,” “Report system on clinical effectiveness of a medical device,” “More clinical research,” “Cooperation between medical associations,” “Academic representations and federal agencies,” and “Better monitoring of quality in clinical practice.”
We observed a slightly lower proportion of participants working in the private sector reporting that there are aspects that may improve clinical practice for two groups of characteristics: care level (“Private”: 82 percent against “Secondary care”: 98 percent; “Specialist care”: 98 percent) and sector (“Private”: 82 percent against “IMSS”: 100 percent; “ISSSTE”: 91 percent; “SSA”: 97 percent; “Other”: 97 percent). Overall, 65 percent responded that most of these aspects were not or almost not considered in their public sector.
DISCUSSION
In this cross-sectional study among orthopedic specialists we assessed viewpoints of end-users concerning the purchasing process of HRMDs. We identified two important findings: (i) the majority of participants agreed with at least one statement that they had some form of involvement in purchasing, (ii) many participants reported experiencing problems in their clinical practice and they associated several of them with purchasing. In Mexico, agents of the purchasing process are responsible for preparing or realizing the needs assessment; this depends on whether the tender covers the demand for a single hospital or several hospitals. The heads of orthopedic departments and end-users provide information on the expected demand for new procedures and suggests the type of technologies needed for the expected surgeries. The orthopedic specialist who is working within the team of the orthopedic department is generally integrated in the purchasing process based on physical conformity checks of medical device specifications (Reference Lingg, Wyss and Duran-Arenas10;Reference Lingg, Wyss and Durán-Arenas11). This is reflected by the answers of participants concerning their influence on and the consideration of their opinion for a medical device technology.
However, the answers on decision making suggest that the way in which decisions are made is not always well linked to end-users. In our previous studies, we found that the decision autonomy of the end-user is relatively low, and that medical devices are not always evaluated based on post-market data once they have been listed in the standard list (Reference Lingg, Wyss and Duran-Arenas10;Reference Lingg, Wyss and Durán-Arenas11).
Many of the identified areas of problems that participants reported may originate from the current practices of purchasing processes for orthopedic HRMDs in Mexico. A previous study on purchasing identified similar findings for outcomes of purchasing practices exemplary for the social security institution IMSS (15). In the function of healthcare delivery, the main process occurs at the clinical level between clinicians and patients (Reference Ferlie and Shortell16). For the production of direct services to the patient, the support of administrative processes such as purchasing is essential and it increasingly influences clinical practice and outcomes (15;Reference Theodorou, Georgiou, Nikolentzos and Bellali17–Reference Sorenson and Kanavos19).
Understanding the performance of these processes in healthcare systems is important because they support clinicians with products such as medical devices that are critical for patient treatment (Reference Kakwezi and Nyeko20–Reference Bonaccorsi, Lyon, Pammolli and Turchetti24). Evaluating orthopedic HRMDs at the meso level based on a systematic assessment scheme such as mini-HTA may bridge the gap between purchasing inputs and outputs, and thus improve purchasing decision outcomes. HTA in Mexico has improved over recent years (Reference Gómez-Dantés and Frenk25) but requires still more advancement.
Few of the identified areas of problems that participants reported may originate from post-market regulations for HRMDs in Mexico at the meso and macro level. This may be attributed to the experience of end-users that their complaint systems do not sufficiently contribute to solving their problems in clinical practice (Reference Lingg, Wyss and Duran-Arenas10;Reference Lingg, Wyss and Durán-Arenas11). However, in our previous studies, we also found that end-users often tend to neglect reporting these problems. When these problems are rarely discussed or addressed by the end-users, decentralized organs such as the Federal Commission for the Protection against Sanitary Risks (26) that encompasses the Mexican Technovigilance department cannot exert their influence on post-market surveillance.
This study provides insight into viewpoints of end-users for purchasing of orthopedic HRMDs and into the potential contribution of HTAs at the level of purchasing decisions. Given the known needs for improvement of purchasing outcomes and aspects that may improve clinical practice, critical analyses of purchasing processes will be necessary to determine real practice relevance and to define improvement plans. Those analyses should integrate how purchasing decisions influence the type of problem that end-users experience in their clinical procedures.
LIMITATIONS
For this study, we used nonrandom sampling, which does not necessarily guarantee the sample being representative for the population of orthopedic specialists in Mexico. However, this study includes 187 end-users working as orthopedic specialist and is based on a purposive sampling. Thus, it covers different groups of surgeons in terms of work experience, state within Mexico, and at the health service attention level. The sample represents approximately more than 10 percent of orthopedic specialists who practice orthopedic surgery. Furthermore, in this study, we collected data from end-users employed by different public institutions and self-employed orthopedic specialists. Tender processes vary among institutions and over time and, thus, influence the response behavior of participants. Therefore, this study does not provide insight into purchasing inputs related to a specific tender scheme. Instead, this study provides insight into viewpoints of end-users on purchasing outcomes in general. These insights suggest that purchasing processes are in need of more detailed investigations.
CONCLUSIONS
The findings of this study suggest that purchasing processes need to be further investigated in more detail. The outcomes of purchasing decisions have been little investigated, and a better understanding of these will contribute to the strengthening of the purchasing of HRMDs. Neglecting problems that end-users experience during their clinical practice leads to the continuation of sub-standard provision of purchasing outputs and increased burden of risk for both the orthopedic specialist and the patient.
To prevent sub-standard outcomes of purchasing decisions, this study and the current literature suggest that technologies should be assessed during the purchasing process, end-users should be adequately involved, and decisions should be based on multiple criteria including clinical impact in the short-term (tissue trauma, rehabilitation duration, primary stability of implant, etc.) and long-term (survival of implant, material performance, etc.).
The focus on Mexico is particularly novel and provides insight into a health system where HTA activities are mainly realized at the macro level. However, this does not solve the need for adequate evidence-based evaluation of health technologies at the hospital level. This study concludes that Mexican stakeholders of the purchasing process underestimate the contribution of HTAs at the level of purchasing decisions. HTA in Mexico has improved in recent years but needs more advancement.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: https://doi.org/10.1017/S0266462317000101
CONFLICTS OF INTEREST
The authors declare that they do not have conflicts of interest.
Open access
