Introduction
Stroke is the third leading cause of death 1 and a leading cause of hospitalization in Canada, accounting for 16.9% of total hospitalizations. 2 Approximately one third of strokes are preceded by one or more transient ischemic attack (TIA). 3 However, unlike acute stroke, many individuals with a TIA initially seek medical attention from their primary care provider,Reference Goldstein, Bian, Samsa, Bonito, Lux and Matchar 4 - Reference Roebers, Wagner, Ritter, Dornbach, Wahle and Heuschmann 7 and the majority are not hospitalized.Reference Goldstein, Bian, Samsa, Bonito, Lux and Matchar 4 Population-based studies have reported that <10% of individuals with TIA present to the emergency department,Reference Giles, Flossman and Rothwell 8 and a recent study in OntarioReference Hall, O’Callaghan and Bayley 9 showed that only 17% of TIA patients were admitted to hospital. As a result, hospitalization data underestimate TIA occurrence, with previous work suggesting that population-based occurrence is underestimated by >66%.Reference Tu, Wang and Young 10
Although the Canadian Institute of Health Information (CIHI) discharge abstract database has previously been validated for the identification of stroke and TIAs.Reference Andrade and Harrold 11 - Reference Benesch, Witter and Wilder 15 few studies have examined the accuracy of physician billing records for identifying cases of TIA. As primary care diagnoses of TIA are subject to poor diagnostic accuracy,Reference Benesch, Witter and Wilder 15 - Reference Piriyawat, Smajsová and Smith 17 prior studies have shown reduced sensitivity for TIA case definitions involving physician billing data compared to those using hospitalization data alone.Reference Tu, Wang and Young 10 , Reference Newton, Wagner and Ramsey 18 , Reference Agrawal, Johnston, Wu, Sidney and Fullerton 19 However, the majority of these studies evaluated single-case definitions for TIA and did not involve population-based samples. Specifically, two previous studies used outpatient data to identify TIA as a comorbid diagnosis to diabetesReference Newton, Wagner and Ramsey 18 and atrial fibrillation,Reference Brophy, Snyder, Gaehde, Ives, Gagnon and Fiore 20 and two others assessed the ascertainment of TIA in paediatric patient samples.Reference Agrawal, Johnston, Wu, Sidney and Fullerton 19 , Reference Golomb, Garg, Saha and Williams 21 One recent studyReference Tu, Wang and Young 10 evaluated multiple case definitions for TIA from a sample of the general adult population and, in contrast to other work, reported improved sensitivity with the addition of physician billing data, but this study was cross-sectional and limited to billing data from volunteering family practice physicians.
The purpose of the present study was to evaluate the diagnostic accuracy of population-based physician billing administrative data for the identification of TIA cases. Using a validated algorithm for identifying ischemic stroke from hospital discharge data, we identified a cohort of individuals with stroke with a recent TIA as the reference standard and compared multiple algorithms for TIA case ascertainment from physician billing data in this cohort. Given the number of individuals evaluated and diagnosed with TIA in primary care settings, improving the ability to accurately identify these individuals has important implications for estimating the population-based occurrence of TIA, evaluating the impact of TIA on health outcomes, and for studies of health service utilization and health systems planning in this population.
Materials and Methods
Data Sources
The data for this study were obtained from the British Columbia (BC) Ministry of Health, via Population Data BC, and included linked records from: (1) the CIHI Hospital Discharge file, containing data on all hospital discharges, transfers and deaths from acute care hospitals in BC, excluding emergency department codes; and (2) the Medical Service Plan (MSP) Master Payment Information file, containing data on all medically necessary services provided on a fee-for-service basis by practitioners to individuals covered by MSP from 1992 to 2007. 22 During the period of this data extract, approximately 90% of physicians in BC operated on a fee-for-service payment basis. 23 The data in this extract were linkable across databases using individual‐specific personal health numbers assigned to all permanent residents of BC, with the exception of individuals covered by federal healthcare funding, including First Nations, veterans and those residing in federal detention institutions.
TIA Billing Practices
Prior to defining the base population and test algorithms for the identification of TIA, interviews with clinicians practicing in BC—including a stroke neurologist and a general practitioner (GP)—and administrative personnel—including billing clerks from a random sample of 20 general practice offices across BC and billing administrators from the Vancouver Coastal Health, Northern Health and Interior Health Authorities—were conducted to characterize practices used in the generation of diagnostic records within these administrative databases and identify common clinical trajectories for TIA patients across different care settings.
Based on these interviews, it was determined that, across health authorities, the use of International Classification of Disease (ICD) versions 9 and 10 codes (ICD–9 435.x or ICD–10 G45.x) in the primary diagnosis position was the primary method of coding hospital admissions for TIA. Primary care providers reported the use of ICD–9 code 435 or ICD–10 code G45 to capture physician encounters with TIA, and the majority also indicated that suspected events would be coded as TIA at the initial encounter and then, if possible, confirmed at the time of a follow-up encounter. The main clinical trajectories for individuals with TIA presenting to their primary care provider were identified as: (1) follow-up with the same GP within 60 to 90 days of the initial encounter, (2) referral to a specialist neurologist or cardiologist and (3) referral to a hospital. This information was used to inform the development of algorithms for TIA case ascertainment evaluated in the present study.
Base Population
The base population included all residents of BC eligible for health services with a diagnosis of TIA during the study period. TIA diagnoses were identified using either: (1) ICD–9 code 435.x or ICD–10 code G45.x in the primary diagnosis position from the hospital discharge abstract database, or (2) ICD–9 code 435 or ICD–10 code G45, with a corresponding specialist code for general practice [00], neurology [02] or cardiology [26] from the physician billing database (Table 1).
Description of cohorts for base population, reference standard and administrative data algorithms for the identification of TIA diagnoses
GP=general practitioner; SP=specialist practitioner; CIHI=Canadian Institute of Health Information; ICD–9/10=International Classification of Diseases, versions 9 and 10.
Reference Standard
The reference standard cohort consisted of all individuals with a hospitalization for ischemic stroke and either a hospitalization or primary care diagnosis of TIA within 90 days prior to the stroke. Although rates of stroke after TIA are highly variable,Reference Giles and Rothwell 24 as recurrent stroke is more likely to occur after definite TIA than in individuals with transient neurological episodes that mimic TIA,Reference Amort, Fluri and Schäfer 25 short-term (<90 day) stroke recurrence was used as a criterion to retrospectively identify individuals with “true” TIA for the reference standard. Hospital admissions for stroke were captured using a previously validated algorithm for ischemic stroke from the discharge abstract database, involving ICD–9 codes 434.x and 436.x or ICD–10 codes I63.x and I64.x in the primary diagnosis position.Reference Mayo, Danys, Carlton and Scott 13 , Reference Kokotailo and Hill 14 This cohort was then restricted to cases that had a diagnosis of TIA within 90 days prior to the stroke diagnosis, using the above-described coding for TIA from either the physician billing (single code) or discharge abstract databases (inpatient codes in the primary diagnosis position) to form the reference standard cohort (Table 1).
TIA Algorithms
Several different administrative data algorithms for the identification of TIA were evaluated. For comparability to prior studies, a previously validated algorithm using ICD–9 code 435.x or ICD–10 code G45.x as the primary or most responsible diagnosis was included to capture TIA cases from CIHI hospital discharge records.Reference Kokotailo and Hill 14 , Reference Heckbert, Kooperberg and Safford 26 Based on the potential for bias associated with TIA diagnoses from single primary care encounters, all algorithms for the ascertainment of TIA cases from physician billing data consisted of diagnostic records from two consecutive physician encounters within 90 days and included the following combinations of potential encounters: (1) a GP encounter with a GP follow-up, (2) a GP encounter and specialist referral follow-up and (3) a GP encounter with a hospitalization (Table 1).
Analyses
Measures of diagnostic accuracy, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive and negative likelihood ratios, overall accuracy and associated 95% confidence intervals (CI 95%) were calculated to compare each algorithm to the reference standard.Reference Akobeng 27 These measures were obtained for each individual algorithm, as well as for pre-specified combinations of algorithms, where cases captured from the different physician billing algorithms were added to the hospital discharge algorithm. Due to the potential for overlap in cases between the hospital discharge algorithm and the algorithm capturing TIA cases from an initial primary care encounter with a subsequent hospital admission, these two algorithms were not assessed in combination. Receiver-operating characteristic (ROC) analyses were conducted to compare area under the curve (AUC) values for each physician billing algorithm.Reference DeLong, DeLong and Clarke-Pearson 28 To compare the effect of the different case definitions on population-based estimates of TIA occurrence for the 3 million residents of BC over the study period, 22 we also estimated the annual TIA occurrence associated with each algorithm.
Results
A total of 102,492 individuals with a diagnostic record of TIA from the physicians’ billing or discharge abstract databases were identified for the base population. The reference cohort consisted of 3,110 individuals with an admission for ischemic stroke with a diagnosis of TIA within 90 days prior to stroke admission. A total of 21,284 cases were identified from the CIHI hospital discharge abstracts, 23,300 were identified from the algorithm capturing diagnoses from consecutive GP encounters, 3,931 were identified using the algorithm for a GP encounter with specialist referral, and 1,384 were identified using the algorithm with a GP visit with subsequent hospitalization for TIA (Table 1).
Results indicated that hospitalization records demonstrated high sensitivity for the identification of TIA cases 69.4% (CI 95%=69.1–69.8). By contrast, algorithms for TIA case ascertainment from physician billing data demonstrated reduced sensitivity, ranging from a very low sensitivity of 5.8% (CI 95%=5.7–6.1) for the algorithm capturing TIA from a GP encounter with specialist referral follow-up, to moderate sensitivity at 37.7% (CI 95%=37. 4–38.1) for the algorithm capturing TIA via two consecutive GP encounters. Sensitivity for algorithms combining physician billing data with cases identified via hospital discharge records was also poor, with these algorithms validating only 4% (CI 95%=4.3–4.6) and 12% (CI 95%=12.2–12.8) of cases, respectively (Table 2).
Validation of administrative data algorithms for the ascertainment of TIA cases using hospital discharge records for individuals with ischemic stroke with prior TIA as a reference standard
GP=general practitioner; SP=specialist practitioner; CIHI=Canadian Institute of Health Information; CI 95%=95% confidence interval; PPV=positive predictive value; NPV=negative predictive value.
All physician billing data algorithms demonstrated high specificity for TIA case ascertainment, ranging from 76.6% (CI 95%=76.3–77.0) for the GP with specialist referral definition to 96.3% (CI 95%=96.1–96.5) for cases identified from an initial GP encounter with subsequent hospitalization (Table 2). As a result of the low to moderate sensitivity and high specificity observed across algorithms, the PPVs and likelihood ratios associated with each algorithm were also very low, with the algorithm for TIA cases ascertained from an initial GP diagnosis with a subsequent hospital admission demonstrating the highest PPV at 5.8% (CI 95%=4.9–5.3) (Table 2). False positive rates (1 – specificity) were comparable across physician billing algorithms and included approximately 20% of identified cases, with the exception of the algorithm capturing an initial GP diagnosis with a subsequent hospitalization for TIA, which had the highest specificity and identified few false positives (4%).
ROC analyses demonstrated that physician billing algorithms showed very poor discriminability, with AUC values ranging from 0.53 to 0.57 (Table 3). Although all algorithms tested in the present study showed similarly low classification accuracy, comparisons of observed AUC values revealed significant differences between algorithms for the classification of TIA (Table 3).
Receiver-operating characteristic analysis of the discriminability of physician billing algorithms for the ascertainment of TIA
AUC=area under the curve; SE=standard error; CI 95%=95% confidence interval.
† Pairwise comparisons of AUC values for all physician billing algorithms compared to hospital-only definition, p<0.01.
For all algorithms, the population-based occurrence of TIA declined over time across the study period (1992–2007) (Figure 1). The average population-based occurrence of TIA associated with the reference standard was 0.08%. The average TIA occurrence using data from the CIHI discharge database alone was estimated at 0.5%. When algorithms for the identification of individuals with TIA from physician billing data were applied to the entire BC population across years, the algorithm involving two consecutive GP encounters showed an average estimated occurrence similar to the hospital discharge algorithm (0.6%), while algorithms capturing TIA cases from a GP encounter with specialist referral follow-up and a GP encounter with a subsequent hospitalization produced estimates similar to the reference standard, at 0.1 and 0.03%, respectively. When the algorithms for diagnoses based on a hospital discharge alone and two GP encounters were assessed in combination, the average population-based occurrence of TIA was estimated at 1.2% (Figure 1).
Impact of administrative data algorithm definitions on population-based estimates of TIA occurrence for the population of British Columbia from 1992–2007.
Discussion
The purpose of this study was to assess the diagnostic accuracy of multiple algorithms for the ascertainment of TIA from a longitudinal population-based physician billing administrative database. The observed accuracy of population-based physician billing data for the identification of TIA cases was poor, with algorithms demonstrating low to moderate sensitivity, low positive predictive values and area under the curve discriminability, and high false positive rates. However, when combined with a previously validated algorithm for TIA case ascertainment from hospital discharge data, this algorithm produced population-based estimates of annual TIA occurrence comparable to previous validation studies. These findings indicate that population-based physician billing data have insufficient sensitivity for identification of individuals with TIA, but algorithms capturing multiple GP encounters may be a useful addition to hospital discharge data for estimating population-based burden of TIA that includes patients diagnosed in primary care.
The results of this study are consistent with prior studies evaluating the use of outpatient physician billing records to ascertain TIA, where measures of sensitivity ranged from 33 to 53%.Reference Tu, Wang and Young 10 , Reference Newton, Wagner and Ramsey 18 , Reference Agrawal, Johnston, Wu, Sidney and Fullerton 19 The optimal algorithm for identification of TIA evaluated in the present study involved a TIA diagnosis from an initial GP encounter with a GP follow-up within 90 days. Similar to a prior study assessing the accuracy of identifying patients with stroke and TIA from participating family practices in Ontario,Reference Tu, Wang and Young 10 the algorithm of consecutive GP visits tested in the present study accurately validated 38% of TIA cases compared to the reference standard. However, these values are substantially lower than the reported discriminative utility for triaging TIA using clinical tools, such as the ABCD 3 and particularly the ABCD 3 –I score.Reference Merwick, Albers and Amarenco 29 Unlike prior work, the addition of physician billing records to the hospital discharge algorithm also did not improve the sensitivity of this case definition.Reference Tu, Wang and Young 10 Algorithms involving either specialist referral or hospital admission subsequent to the initial GP encounter assessed in the present study also showed low sensitivity (≤11%) for the identification of TIA. These findings may reflect management strategies for TIA in the primary care setting. Specifically, prior audits of primary care practices in the United States and abroad have reported low specialist and hospital referral rates associated with the primary care evaluation of TIA and a preference among GPs for the short-term outpatient management of TIA,Reference Goldstein, Bian, Samsa, Bonito, Lux and Matchar 4 , Reference Roebers, Wagner, Ritter, Dornbach, Wahle and Heuschmann 7 suggesting that TIA diagnoses are most likely to be accurately captured from algorithms for outpatient administrative data that include records from multiple visits within a short follow-up period, as observed in the present study.
Comparable to prior hospital-based studies, the sensitivity associated with an algorithm using only hospital discharge records to ascertain TIA in the present study was moderately high (69%).Reference Heckbert, Kooperberg and Safford 26 This finding confirmed the internal validity of using the criteria of stroke subsequent to TIA to define the reference standard. However, the PPVs associated with the physician billing algorithms evaluated in the present study were substantially lower than previously reported,Reference Newton, Wagner and Ramsey 18 , Reference Agrawal, Johnston, Wu, Sidney and Fullerton 19 and all algorithms showed poor AUC discriminability. A potential explanation for this finding relates to the definition for the reference standard employed in the present study. The occurrence of stroke after TIA in the base population used in our study was low (2.4%). As both PPV and AUC are dependent on the prevalence of the disease in the population being tested,Reference Altman and Bland 30 the use of a reference standard definition based on the criterion of stroke recurrent to TIA may have resulted in overestimation of cases detected by each of the physician billing algorithms, producing high false positive rates and lower estimates of PPV and AUC. For instance, although the algorithm that selected TIA cases using diagnostic records from two consecutive GP encounters showed the highest sensitivity (38%), the false positive rate for this algorithm was high (22%), resulting in a low PPV (5%) for this case definition. In addition, due to the high rates of false positivity across algorithms, significant differences in AUC observed between algorithms should be interpreted with caution.Reference Berrar and Flach 31 Despite these findings, as many individuals with TIA are diagnosed and managed by their primary care provider,Reference Goldstein, Bian, Samsa, Bonito, Lux and Matchar 4 , Reference Edlow, Kim, Pelletier and Camargo 32 case definitions that include individuals diagnosed with TIA in both hospital and outpatient settings are required for population-based research on health service utilization and resource planning. Thus, the use of clinical outcomes to decrease the potential for misclassification in the retrospective identification of TIA from administrative data records merits further study.
Notably, when the optimal physician billing algorithm from the present study was combined with an algorithm for identifying TIA from hospital discharge data, the average estimated annual occurrence of TIA over the study period was 1.2%—the same as that reported for a similar algorithm by Tu et al. (2013)Reference Tu, Wang and Young 10 (Figure 1). This finding suggests that the addition of cases ascertained from population-based physician billing data to hospital discharge records may be used to estimate TIA burden in the general population over time. One potential advantage of the algorithm assessed in the present study compared to the algorithm assessed by Tu et al. (2013) is the shorter follow-up window between family physician encounters (90 days vs. 1 year), decreasing the likelihood of capturing recurrent TIA events.
The strengths of this study included the use of longitudinal population-based data and the development of multiple case definitions to assess the accuracy of physician billing records for the identification of TIA. The use of information on billing practices and clinical trajectories from health professionals and billing administrators in different care settings for the development of comparison algorithms also increased the validity of these algorithms. However, the present study had several limitations. The use of conservative case definitions requiring two consecutive physician encounters for the identification of individuals with TIA may have contributed to the low observed sensitivities for physician billing algorithms in our study. In addition, the reference standard we used represented a high-risk subgroup of the TIA population (i.e., those who had a stroke recurrent to TIA). As rates of stroke recurrent to TIA vary substantially across studies, this narrow definition for the reference standard may have misrepresented the rates of TIA associated with each physician billing algorithm and limited our ability to validate their accuracy. Indeed, it is possible, given that the physician billing algorithms tested in the present study reflect measurement of the burden of TIA across different and changing clinical pathways, that no single method may be able to yield a “true” measurement of TIA.
Conclusions
Although many individuals with TIA are diagnosed and managed in the primary care setting and do not undergo hospitalization, there are many challenges for accurately identifying TIA cases from physician billing data. Our study was novel in the use of longitudinal population-based data from physician encounters to develop and test multiple case definitions based on clinical trajectories of patients presenting with TIA. The findings confirmed that physician billing records have insufficient sensitivity for the classification of TIA. However, the evaluation of these algorithms also revealed that these data may be used in combination with hospital discharge data to improve the estimation of population-based trends in TIA occurrence. Although there may be no single method for capturing “true” cases of TIA, given the importance of identifying diagnoses from different care settings for population-based health services research, alternative approaches for the identification of outpatient cases of TIA are required.
Funding
This research was supported by a doctoral award from the Canadian Institutes of Health Research to JDE.
Disclosures
Jodi Edwards has the following disclosure. Canadian Institutes of Health Research to JDE: award recipient, financial support. Mieke Koehoorn, Lara Boyd, Boris Sobolev and Adrian Levy do not have anything to disclose.
