INTRODUCTION
Pneumonia remains a major cause of infectious disease mortality and morbidity worldwide [Reference Lozano1]. The incidence of pneumonia confirmed by chest radiograph (CXR) is considered an important, although conservative, indicator for pneumonia disease burden [Reference Madhi and Klugman2]. Ensuring the best possible estimates of radiographically confirmed pneumonia prevalence and incidence is essential in guiding decision making for treatment and prevention. Since 2002, active, population-based pneumonia surveillance has been conducted in two rural provinces in Thailand as a collaboration between the Thailand Ministry of Public Health (MOPH) and the US Centers for Disease Control and Prevention (CDC) [Reference Dowell, Scheld, Murray and Hughes3].
Prevalence and incidence for radiographically confirmed pneumonia in this surveillance system have been estimated previously [Reference Olsen4, Reference Hasan5] using a consensus diagnosis by a panel of radiologists as the ‘gold standard’. Although not typically accounted for in the analysis, this approach inherently assumes 100% sensitivity and 100% specificity for the radiology panel. However, studies have indicated that false-positive and false-negative results can occur [Reference Kramer, Roberts-Brauer and Williams6–Reference Williams9] leading to potentially biased estimates of incidence and prevalence. Therefore, there is in fact no true reference standard.
Latent class analysis (LCA) has been used in pneumonia research to estimate the pneumococcal pneumonia prevalence in the United States and Kenya given the absence of a reference diagnostic method with high specificity and sensitivity [Reference Butler10, Reference Jokinen and Scott11]. However, these estimates used more than three separate diagnostic tests, including blood culture, sputum Gram stain, sputum polymerase chain reaction, urine antigen testing, nasopharyngeal swab culture, pneumococcal surface adhesin A, and lung-aspirate culture to generate their results; in typical non-research settings, this number of separate tests is infrequently available to inform diagnosis. In the MOPH-CDC pneumonia surveillance system in Thailand, two test results are usually available to diagnose radiographically confirmed pneumonia: a reading of the CXR by the attending clinician and a reading of the same CXR by a radiology panel.
A 2 × 2 table generated from the results of two diagnostic tests contains four cells, but only 3 degrees of freedom (d.f.); because the sample size is fixed, knowing the values of any three cells also fixes the value of the fourth cell. As there are five unknown parameters (radiology panel sensitivity and specificity, clinician sensitivity and specificity, and disease prevalence), but only 3 d.f., only three parameters can be estimated using conventional statistical approaches. In these approaches, two parameters are assumed to be known (usually sensitivity and specificity of the radiology panel) and given fixed values (100% for both sensitivity and specificity), enabling the estimation of the other three parameters from the data. However, this approach incorrectly assumes these fixed values are exactly known, leading to confidence intervals that are too narrow when estimating the remaining parameters. We used a Bayesian latent class model (BLCM) to address the problem of the lack of a gold standard when results from two (imperfect) tests are available. The Bayesian approach incorporates previous knowledge or beliefs about all unknown parameters in the form of a prior distribution, which is then updated by information in the data, in the form of a likelihood function. Multiplying the prior distribution by the likelihood function generates the posterior distribution across all parameters, which contains updated knowledge taking into account the information added by the data.
Bayesian models for this two-diagnostic-test problem have been developed [Reference Joseph, Gyorkos and Coupal12, Reference Dendukuri and Joseph13]. We utilized these models to estimate the prevalence of radiographically confirmed pneumonia in patients hospitalized with acute lower respiratory illness (ALRI). This prevalence estimate was used to estimate the incidence of radiographically confirmed pneumonia hospitalizations in rural Thailand from 2005 to 2010, which we compared to incidence estimates from conventional analysis. The model also provides estimates of the sensitivity and specificity of radiographically confirmed pneumonia diagnosis by a radiology panel and a clinician.
METHODS
Setting
Since 2002, Thailand MOPH, in collaboration with the US CDC, has conducted surveillance for community-acquired pneumonia requiring hospitalization in Sa Kaeo and Nakhon Phanom provinces. Sa Kaeo, located in eastern Thailand on the Cambodia border, and Nakhon Phanom, in northeastern Thailand along the Laos border, are rural provinces solely served by public hospitals. This surveillance system encompasses all 20 acute-care hospitals in the provinces. In 2005, access to automated blood culture systems was established for all hospitals to support laboratory-based surveillance for pathogens causing bloodstream infections, including patients with pneumonia and sepsis.
Study population
Details of the pneumonia surveillance system (CDC IRB protocol no. 6076) have been previously published [Reference Olsen4, Reference Simmerman14, Reference Fry15]; briefly, surveillance officers reviewed admission logbooks daily to identify patients with diagnoses associated with possible ALRI for further data collection. An ALRI case was defined as a patient with evidence of acute infection (reported fever, documented temperature >38·2 °C, elevated age-specific white blood cell count or abnormal differential) and symptoms/signs of respiratory illness (cough, sputum production, haemoptysis, chest pain, dyspnoea, tachypnoea, abnormal breath sounds, rhonchi, wheezing, or rales/crepitations) at admission. CXRs were requested at the clinician's discretion. All ALRI cases who had a CXR ordered by their attending clinician and who had the radiograph interpreted by both the radiology panel and clinician were included in the analysis.
Data collection for study population
Hospital clinicians read analogue hard copies of the CXR films and classified the case as having either evidence of pneumonia or no evidence of pneumonia. Surveillance officers then recorded the findings in the pneumonia surveillance database. Surveillance officers used digital cameras to capture digital images of CXR films [Reference Javadi7] and transmitted them to the radiology panel from the Chest Diseases Institute, Thailand MOPH. The radiology panel, aware of the source of the images, but blinded to the clinical details except age and gender, then interpreted the images using standardized criteria [Reference Javadi7]. In brief, a patient was considered to have radiographically confirmed pneumonia by the radiology panel if two of three independent radiologists interpreted the CXR as consistent with probable or definite pneumonia.
Statistical analysis
Prior distributions
A uniform prior distribution ranging from 0 to 1 was used for radiographically confirmed pneumonia prevalence in hospitalized ALRI patients. Informative prior distributions (Table 1) for sensitivities and specificities of the CXR interpretation for both radiology panel and clinicians were derived for children aged <5 years from multiple sources (Supplementary Appendix). We performed a sensitivity analysis by extending the range by 0·10 for all informative prior distributions in Table 1. We used the same prior distributions and sensitivity analysis described above for estimating the parameters in patients aged ⩾5 years.
95% probability ranges and Beta prior distributions for sensitivities and specificities of radiology panel and clinicians for diagnosing pneumonia by chest radiograph
* Beta distribution coefficient, density function denoted Beta(α,β) for each parameter was derived by matching the centre of the range with the mean of the beta distribution given by α/(α+β), and matching the standard deviation of the beta distribution, given by
$\sqrt {\displaystyle{{{\alpha \beta}} \over {\left( {\alpha + \beta} \right)2^{\ast}\left( {\alpha + \beta + 1} \right)}}} $
, with one quarter of the total range [Reference Joseph, Gyorkos and Coupal12].
BLCMs used
Clinicians read the original CXR film and the radiology panel read a digital copy of the same CXR, which may have resulted in dependence between the clinicians’ and radiology panel readings. To check the robustness of our prevalence estimates, we generated posterior estimates for all parameters using models assuming both conditional independence and dependence of the readers. We used the model of Joseph et al. [Reference Joseph, Gyorkos and Coupal12] when assuming conditional independence of two tests, and the fixed-effects model of Dendukuri & Joseph [Reference Dendukuri and Joseph13] when allowing for dependence. In the conditional dependence model, non-informative prior distributions were used for the covariance of sensitivities and specificities of the radiology panel and clinician. We calculated Deviance Information Criterion (DIC) in all the models for checking goodness of fit.
We also estimated the age group-specific population incidence of radiographically confirmed pneumonia hospitalizations by two different methods: Bayesian and conventional statistical analysis. For the Bayesian approach, we estimated the number of radiographically confirmed pneumonia cases by applying the age group-specific median posterior prevalence from the dependent model to all ALRI cases with a CXR performed in each age group. In the conventional approach, the estimated number of radiographically confirmed pneumonia cases was calculated by applying the percent of CXRs within each age group read as pneumonia by the radiology panel to all cases in corresponding age groups with a CXR performed. We used population projections of Thailand 2000–2030 from the Thailand National Economic and Social Development Board (http://www.nesdb.go.th/temp_social/pop.zip) to calculate the person-years in 2005–2010. Incidence rate estimates were calculated using the estimated number of cases divided by total person-years. The 95% confidence interval (CI) was calculated based on binomial proportion [Reference Hasan5] in conventional analysis, while the Bayesian credible interval (CrI) of posterior prevalence from the dependent model was used to calculate the 95% CrI of incidence rate in the BLCM. Analysis was performed using WinBUGS v. 1·4·3 (Imperial College and MRC, UK). User-friendly software is freely available for the conditional independent model (http://www.medicine.mcgill.ca/epidemiology/Joseph/software/Diagnostic-Testing.html) and for the conditional dependent model (http://www.nandinidendukuri.com/winbugs-programs/latent-class-models-for-conditionally-dependent-tests).
RESULTS
During 2005–2010, 44369 patients were hospitalized with ALRI and had a CXR performed. Of these patients, 36 075 had a radiologist interpretation available and 32 640 had an interpretation available from a clinician; 26 236 (59%) had both available and were included in this analysis. This included 9639 cases of children aged <5 years and 16 597 cases of persons aged ⩾5 years. Compared to ALRI patients included in the analysis, those who were excluded due to missing CXR interpretations had similar distributions of age, sex, temperature, respiratory rate and white blood cell count for patients aged <5 years and persons aged ⩾5 years (Supplementary Table S2). Children aged <5 years included in the final analysis did not have substantially lower rates of intubation (2·3% vs. 2·8%), blood culture positivity (2·2% vs. 3%) and death (0·5% vs. 0·7%) than those who were not included. A similar trend was apparent in older patients, with lower rates of intubation (10·3% vs. 12·9%), blood culture positivity (9·6% vs. 11·4%) and death (5·5% vs. 7·3%) in included patients. The overall agreement for presence of pneumonia on CXR between the radiology panel and clinician was 62% (5933/9639) in children aged <5 years and 66% (11 030/16 597) in persons aged ⩾5 years (Table 2).
Clinicians' and radiology panel diagnoses of pneumonia by chest radiograph, Thailand, 2005–2010
Prevalence, sensitivities and specificities
Assuming conditional independence of pneumonia diagnosis between the radiology panel and clinicians, the estimated prevalence of radiographically confirmed pneumonia by BLCM was 0·76 (95% CrI 0·71–0·81) in children aged <5 years (DIC 33·014) and 0·56 (95% CrI 0·50–0·61) in persons aged ⩾5 years (DIC 34·890).
Assuming conditional dependence for pneumonia diagnosis between the two readings, estimated pneumonia prevalence was slightly higher at 0·80 (95% CrI 0·73–0·86; DIC 32·729) in children aged <5 years and 0·57 (95% CrI 0·5–0·64; DIC 34·392) in persons aged ⩾5 years (Fig. 1a , b ). According to DIC, the data favour the conditional dependent model compared to the conditional independent model. This also coincides with our clinical expectations, and thus the full results from the conditional dependent model will be reported. For children aged <5 years, the sensitivities were minimally lower than those estimated with the conditional independence model at 0·69 (95% CrI 0·65–0·73) for the radiology panel and 0·70 (95% CrI 0·67–0·73) for clinicians; specificities for radiographically confirmed pneumonia were 0·83 (95% CrI 0·76–0·89) for the radiology panel and 0·77 (95% CrI 0·67–0·86) for clinicians. For persons aged ⩾5 years, estimated sensitivities were 0·76 (95% CrI 0·71–0·80) for the radiology panel and 0·66 (95% CrI 0·60–0·72) for clinicians; specificities were 0·79 (95% CrI 0·71–0·86) for the radiology panel and 0·84 (95% CrI 0·78–0·89) for clinicians.
Chest radiograph-confirmed pneumonia prevalence and sensitivity and specificity of readings by radiography panels and clinicians from conditional dependence models. Bayesian latent class model vs. conventional analysis. (a) Age <5 years, (b) age ⩾5 years. CrI, Credibility interval. Red shows the estimates from conventional analysis, assuming radiology panel reading as a gold standard with 100% sensitivity and 100% specificity. Blue shows the posterior estimates from BLCM. The statistics in the right panel correspond to the BLCM estimates.
Sensitivity analysis
Extending the range by 0·10 for all informative prior distributions (Table 1) on the dependent model generated almost no change to the prevalence point estimates of the dependent model but widened the CrIs: 0·82 (95% CrI 0·70–0·93; DIC 31·145) in children aged <5 years and 0·58 (95% CrI 0·43–0·70; DIC 23·319) in individuals aged ⩾5 years. The same pattern was observed for sensitivity and specificity estimates. The DIC preferred the models with wider priors compared to those initially selected for all age groups.
Radiographically confirmed pneumonia incidence
The results from the dependent model from Figure 1 were used to estimate incidence by BLCM. For children aged <5 years, the incidence of radiographically confirmed pneumonia hospitalizations was 2394/100 000 person-years (95% CrI 2185–2574) from BLCM compared to 1736/100 000 person-years (95% CI 1706–1766) from conventional analysis. For persons aged ⩾5 years, the incidence was 235/100 000 person-years (95% CrI 206–264) from BLCM and 215/100 000 person-years (95% CI 211–219) from conventional analysis (Table 3).
Estimated incidence of chest radiograph (CXR)-confirmed pneumonia hospitalizations, comparing results of conventional analysis using the radiology panel interpretation as gold standard and Bayesian latent class model (BLCM), Thailand, 2005–2010
† Confidence interval calculation from a binomial proportion:
$0\!\cdot\!58 \!\pm \!1\!\cdot96 \ast\sqrt {0\!\cdot\!58^{\ast}\left( {1 - 0\!\cdot\!58} \right)/9639} $
, sample size is cases read by radiologists, which is 9639.
‡ Confidence interval calculation from a binomial proportion:
$0\!\cdot\!52 \!\pm \!1\!\cdot96 \ast\sqrt {0\!\cdot\!52^{\ast}\left( {1 - 0\!\cdot\!52} \right)/16597} $
, sample size is cases read by radiologists, which is 16597.
DISCUSSION
Using BLCM, we estimated a 38% higher incidence of hospitalized radiographically confirmed pneumonia in children aged <5 years and a 9% higher incidence in persons aged ⩾5 years with a wider range of potential disease incidence than estimates from conventional analysis. The estimates showed that the radiology panel's diagnosis of radiographic pneumonia, the current gold standard used in pneumonia diagnostics, was not necessarily more accurate than that of a clinician in this population in rural Thailand.
Our conventional estimate of the incidence of radiographically confirmed pneumonia hospitalizations in children aged <5 years was similar to that of a previous study in the same population [Reference Hasan5], which estimated incidence at 1800/100 000 person-years, but higher than estimates from a study in the United States [Reference Jain16], which found rates of 622/100 000 person-years for children aged <2 years and 238/100 000 person-years for children aged 2–4 years. The incidence estimate of radiographically confirmed pneumonia in children aged <5 years from BLCM, which accounts for the fact that radiology panel readings are not 100% sensitive and specific, were substantially higher than estimates from conventional analysis. Adjusting for the dependence between the radiology panel and clinicians’ readings slightly increased the estimated pneumonia prevalence, but led to a slight decrease in the estimated sensitivity and specificity of pneumonia diagnosis for radiologists and clinicians in both age groups.
The 95% CrIs for incidence estimates from BLCM, given in Table 3 are approximately seven-fold wider in both age groups than those estimated by conventional analysis. This is because the BLCM estimates fully take into account all inherent uncertainty in the problem, including the uncertainty of the sensitivity and specificity of the radiology panel. Although the conventional estimates appear more precise, they may not necessarily be accurate as they do not incorporate any inherent uncertainty around the sensitivity and specificity of the reference test; the point estimate from conventional analysis for children aged <5 years is not included in the wider CrIs from BLCM. The wider range of incidence estimates from BLCM should not be ignored as it provides evidence for public health officials that the true burden of pneumonia may be considerably greater than conventional estimates, once all inherent uncertainty is taken into account.
Although the same prior distribution of sensitivity estimates were used in both age groups, the posterior sensitivity estimates in children aged <5 years were similar for clinicians and radiologists, but lower for clinicians in persons aged ⩾5 years. Specificity of pneumonia diagnosis was higher for the radiology panel than for clinicians in children aged <5 years, but this pattern was reversed for patients aged ⩾5 years. Overall, the performance of the radiology panel was not clearly better than that of the clinicians despite the radiology panel consisting of board-certified specialists. The radiology panel was blinded to the patient's clinical information, except age and gender, and read the digitized versions of the original film, while the attending clinicians knew the clinical background of each case, which might have increased the clinicians’ accuracy of diagnosing pneumonia. A beneficial effect of clinical history in interpreting CXRs has been previously demonstrated [Reference Doubilet and Herman17].
Failure to account for the imperfect accuracy of a reference test, can result in diagnostic accuracy of other tests being under- or over-estimated [Reference Whiting18]. In our study, if the radiology panel reading had been considered the gold standard, the specificity of the clinicians’ CXR reading in children aged <5 years would have been estimated at 51%, significantly lower than the BLCM estimates of 70% in children aged <5 years. BLCM offers an approach to estimate the performance of the multiple tests simultaneously with absence of a reference standard with high specificity and sensitivity.
The major limitation of this analysis was the limited estimates available for prior distributions in the adult population. Plausible ranges for prior distributions of unknown parameters were mostly derived from data from children aged <5 years (see Supplementary Appendix). Consequently these same ranges were used for both age groups, which limit inferences for patients aged ⩾5 years. In addition, prior estimates of sensitivity and specificity came from a variety of different comparators, none of which was a perfect gold standard for pneumonia diagnosis. As the prior distributions utilized can substantially change the obtained posterior estimates, we were careful to include a range of plausible values as outlined in the Supplementary Appendix. As a further safeguard, we conducted a sensitivity analysis by extending the range of plausible estimates for sensitivities and specificities of the radiology panel and clinicians. This resulted in almost identical point prevalence estimates, but widened the 95% CrIs. The wider CrIs from this sensitivity analysis suggest that in the face of uncertainty of the true sensitivities and specificities for both radiology panel and clinicians, the true burden of pneumonia may be even higher than estimated here. Another potential limitation was that one or both readings were missing for 41% of patients with a CXR performed. However, patients included in the analysis were similar in demographics and clinical features as those who were not included. Patients not included in the analysis were slightly more likely to require intubation, have positive blood cultures or die than those who were included, suggesting that this group might contain more patients with severe illness. However, the effect of this bias, if any, would be minimal and result in an underestimation of pneumonia incidence.
BLCM is a useful approach to estimate prevalence and incidence of a disease diagnosed by multiple imperfect diagnostic tests or to evaluate these tests in the absence of a reference standard with high specificity and sensitivity. The estimates from a Bayesian approach provided an improved understanding of the incidence of radiographically confirmed pneumonia in rural Thailand, which could be important to inform public health decision making and this approach should be considered for replication elsewhere. Utilizing readings from a panel of radiologists as a reference standard without adjustment for uncertainty of test performance may result in significantly underestimated CXR-confirmed pneumonia incidence and burden calculations.
SUPPLEMENTARY MATERIAL
For supplementary material accompanying this paper visit http://dx.doi.org/10.1017/S0950268816000455.
ACKNOWLEDGEMENTS
We thank Pornpak Khunatorn, Patranuch Sapchookul, and Prasong Srisaengchai for data management support, and the PERCH Chest X-ray Reading Panel for their contributions. We also thank Dorothy L. Southern for providing guidance in scientific writing.
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the U.S. Centers for Disease Control and Prevention.
DECLARATION OF INTEREST
None.
