To the Editor:
The comments by Spada et al. (which are similar in style and content to the ones already made by the authors on the Italian Ministry of Health Web site) are based on misunderstandings and manipulations. The essence of systematic reviews is to sum up available knowledge and minimize bias. The latter is done by a priori stating study inclusion criteria both in the protocol and in the full HTA report (see Fig. 1, p. 299) and assessing methodological quality of included studies using an instrument which was specified in the protocol text and never changed (the Quality Assessment Diagnostic Accuracy Studies checklist or QUADAS). The purpose of inclusion criteria and bias minimization efforts is to ensure that what is included in a review is both relevant and contributes evidence weighted by its reliability to answer the study question (in our case the diagnostic performance of WCE in the small bowel). The high number of excluded studies is thus irrelevant, although it is a common feature of systematic reviews. What matters is what the included studies tell us.
We did not “dismiss” sequential study designs. We included them but judged them to be unreliable because of the likelihood of time bias occurring in the interval time (which was incompletely reported in a proportion of included studies) between WCE and comparator examinations. Lack of randomization makes the likelihood of systematic error in the morphology of intestinal lesions high, with a consequent impossibility to compare like with like in a fair way. To put it another way, each subject enrolled in studies comparing the diagnostic accuracy of WCE for obscure small bowel bleeding should be given an equal chance of being allocated to one or the other diagnostic procedure. If sufficient numbers are enrolled, there will be a fair chance that participants with similar looking lesions are assigned to either procedure. We disagree that the morphology of most vascular lesion morphology change slowly and the reference cited by Spada et al. does not support their statement (Reference Gordon, Watkinson and Hodgson2). As we have shown in our report, there is substantial uncertainty on this very point. In the absence of evidence, the combined opinion of several experts is better than that of one. Because of equipoise and the need to ensure fairness of tests, a large, independent randomized controlled trial is the only way to have a definitive answer. Efforts to evaluate new devices or procedures by any other study designs (such as interrupted time series, expertise based randomized trials, or parallel group nonrandomized studies) can be affected by confounding, making the results difficult to interpret (Reference McCulloch, Altman and Campbell3).
The ignorance of these basic principles of epidemiology and ethics is manifest in our survey of the first authors of the included studies, most of whom had not even taken randomization into consideration. While the “hands on” approach is understandable, unregulated introduction of untested new technology, led by a few key opinion leaders is fraught with dangers for any national health service. Our context analysis shows a high level of inappropriateness of use of the WCE, the diagnostic performance of which was and remains unknown.
As explained in our report, one of the tenets of conducting meaningful economic evaluations is the use of credible estimates of comparative effectiveness. We did not feel confident that our review had identified such estimates and decided to avoid further polluting the WCE literature with a meaningless set of ICERs.
About the national survey, this was based on a comprehensive population denominator (all the Italian centers delivering WCE) and response rate was 48 percent. We obtained data on 2,457 WCE procedures, this is one of the broadest data pools a policy maker has ever had as a basis for decisions.
Finally, we note that Spada et al. do not appear to be free of conflicts of interest. Their links to the Italian distributor of the device must be close, as the postage envelope in which their comments (to the Health Minister as well as to us) were sent bears the logo of the distributor.
Regardless of the basic points of study design and transparency, the introduction and widespread use of invasive devices should be regulated in a stricter way than at present. The most recent European commission guide (1) generically refers to the need for a “Clinical evaluation” (which it defines as “the assessment and analysis of clinical data pertaining to a medical device in order to verify the clinical safety and performance of the device”) before marketing a device.
In such an unsatisfactory regulatory environment, apparently authoritative groups with links to industry may have an easy time in convincing health services to buy the latest untested gizmo. At what cost to the taxpayer?
CONFLICT OF INTEREST
All authors report having no potential conflicts of interest.
