Traumatic brain injury (TBI) is the leading cause of death and disability in children and youth in the United States, with a combined estimate of 511,257 emergency room (ER) visits, hospitalizations, and deaths per year, and more than $1 billion in hospital charges annually.Reference Faul, Xu LWald and Coronado 1 , Reference Schneier, Shields, Hostetler, Xiang and Smith 2 One Canadian report showed that children (<18 years of age) with head injury represented almost 45% of those visiting ERs and urgent care centers, and almost 25% of those were hospitalized in acute care. 3 In addition, a recent systematic review and meta-analysis of the incidence of TBI from international studies identified that the pediatric (<15 years of age) incidence rate of traumatic brain injury was 134/100,000 person-years.Reference Nguyen, Fiest and McChesney 4 Two Canadian pediatric studies were included in this review and reported incidence rates for mild TBI of 110/100,000Reference Gordon, Dooley and Wood 5 and 0.81/100,Reference Willer, Dumas, Hutson and Leddy 6 respectively. TBI thus contributes substantially to the health care burden. Pediatric TBI is of particular concern because it can lead to significant and lifelong deficits,Reference Toga, Thompson and Sowell 7 with a substantial burden on the patient, their caregivers, and society. Unfortunately, ongoing surveillance for pediatric TBI is lacking in Canada, resulting in gaps in the knowledge needed to adequately plan for the resources required for those affected by pediatric TBI.
Administrative data are valuable sources for disease surveillance because they are collected routinely, are cost-effective, cover wide geographic areas, and have a relatively complete capture of all patient encounters with the health care system.Reference Quan, Khan and Hemmelgarn 8 This is especially relevant to the Canadian system because there are comprehensive population-based administrative databases at regional, provincial, and national levels.Reference Jette, Reid, Quan, Hill and Wiebe 9 In the majority of administrative databases, conditions are coded using the World Health Organization’s International Classification of Disease (ICD) system. ICD-coded databases are used internationally to capture causes of morbidity and mortality, and for epidemiological, health management, and reimbursement purposes. 10
To effectively capture cases of pediatric TBI, it is important to develop a validated case definition. A systematic review looking at validated case definitions for TBI identified three studies, all from the United States that met eligibility criteria (Table 1).Reference St Germaine-Smith, Metcalfe and Pringsheim 11 Only one of these studies reported more than one test of diagnostic accuracy, comparing coding of an inpatient database to real-time assessment in the ER.Reference Bazarian, Veazie, Mookerjee and Lerner 12 They found low sensitivity (45.9%), but high specificity (97.8%) and negative predictive value (NPV) (99.2%).Reference Bazarian, Veazie, Mookerjee and Lerner 12 In addition, only one study validated ICD-10 coding for TBI in administrative data,Reference Rodriguez, Mallonee, Archer and Gofton 13 whereas the others validated ICD, 9th revision (ICD-9); Table 1.Reference Bazarian, Veazie, Mookerjee and Lerner 12 , Reference Shore, McCarthy, Serpi and Gertner 14 A validated case definition to study TBI outside of the United States has not, to our knowledge, been published. It is important to validate ICD coding specifically in other non-US databases for several reasons. Although the underlying coding framework is the same internationally, different countries may use slightly modified versions of ICD.Reference Jette, Quan and Hemmelgarn 15 For example, the United States still uses the ICD-9 Clinical Modification edition, 16 which contains fewer codes than the ICD-10 edition currently being used in Canada. In addition, Canada’s health care system differs from that in the United States, where coding standards may vary.Reference Jette, Quan and Hemmelgarn 15 Finally, Canadian coders undergo national standardized training, unlike coders in other countries, potentially leading to variations in coding standards between countries.Reference Jette, Quan and Hemmelgarn 15 , Reference Reid, St Germaine-Smith and Liu 17
Summary of sensitivity, specificity, PPV, and NPV reported by validation studies for TBI
CM=clinical modification; ER=emergency room; ICD=International Classification of Disease; NPV=negative predictive value; PPV=positive predictive value; Sn=sensitivity; Sp=specificity; TBI=traumatic brain injury.
The goal of the present study was to validate ICD-10 coding for TBI in two administrative databases in a large health region: the Discharge Abstract Database (DAD), which captures all hospitalizations, and the Ambulatory Care Classification System (ACCS), which captures all ER/urgent care visits and hospital clinic visits in the province of Alberta. Because Canada has a universal health care system, these two population-based databases capture all patients who access these services. The two objectives were to: (1) assess the validity of ICD-10 coding for TBI and (2) develop a validated case definition for pediatric TBI. It was our hypothesis that it would be possible to develop an accurate sensitive case definition to identify pediatric TBI using ICD-10 coded administrative health data.
METHODS
Study Site
Alberta Children’s Hospital (ACH) is the primary pediatric hospital for the city of Calgary and surrounding semiurban and rural areas. ACH has a total of 133 inpatient beds, and its ER has 60,000 visits per year for those aged 0 to 18 years. The estimated total population of Calgary was 1,079,310 per the 2006 Canadian census, with 272,275 aged 0 to 19 years. 18
Defining Study Population
The study population includes 7101 children aged 0 to 18 years, with primarily trauma, musculoskeletal, or central nervous system complaints captured between October 4, 2005, and June 6, 2007, at ACH through the Regional Emergency Department Information System (REDIS). These conditions were selected to increase the likelihood of capturing children who may have experienced a TBI.
REDIS is an electronic information system used to collect and track information on all children who visit the ACH ER. The “free text” diagnosis and clinical history fields of the REDIS charts were first reviewed by a trained research assistant to identify cases by looking for key words in the chart that would suggest a neurological problem, such as altered mental state, dizziness, confusion, disorientation, headache, balance problems, nausea and/or vomiting, blurred vision, seizures, concussion, head injury, and/or TBI. These charts were then reviewed again by a pediatric neurologist (KB) with expertise in brain injury to confirm diagnosis. Currently, there is not a single agreed-upon case definition for TBI. For this study, TBI was defined as a force to the body or head that results in disruption of normal function of the brain.Reference Menon, Schwab, Wright and Maas 19 TBI was confirmed if the injury was associated with any of the following neurological symptoms: loss of consciousness, altered mental state, posttraumatic amnesia, headache, dizziness, vomiting, confusion, and/or behavioral change. The definition of mild TBI is somewhat controversial in younger children (<5 years of age), in whom it can be difficult to assess an altered mental state and period of amnesia. Therefore, any young child presenting to the ER with a mild head injury without clear evidence contradicting the these criteria was included in the study. Children with simple scalp lacerations, facial injuries/fractures without symptoms of head injury or concussion described previously, or superficial injuries were excluded. Finally, the date the child was first seen in REDIS became the index date for all children.
The quality of the REDIS chart review was further verified among a subset of patients (n=715) through a follow-up telephone call to confirm TBI diagnosis (n=670) or who visited the ACH TBI clinic (n=45). Of the 670 children that were successfully contacted by telephone for follow-up, only 27 (4%) were determined to not be TBI cases upon further questioning. All of the 45 children seen in the TBI clinic were true cases of TBI as originally determined by the REDIS chart review (100%).
Defining TBI in ICD-10 Administrative Data
In this study, TBI coding was validated in two administrative databases: ACCS and the inpatient DAD from April 1, 2002, to March 30, 2011. Both of these databases are coded by trained health technologists who undergo national standardized training. The ACCS captures all emergency visits and hospital-based clinic visits in the province of Alberta, and contains up to 15 coding fields for diagnoses and five for procedures using ICD-10 codes. The DAD captures information including diagnoses and in-hospital outcomes on all patients discharged from every hospital in Alberta. Since April 1, 2002, the DAD has contained up to 50 coding fields for diagnoses and 20 for procedures. Both databases contain a primary diagnosis coding field that contains the diagnosis (ICD-10 code) responsible for most resource use during the visit or inpatient admission.Reference Jette, Quan and Hemmelgarn 15 The rest of the diagnosis codes are classified as secondary diagnoses. Administrative databases were linked with the REDIS cohort data described previously using the first name, last name, date of birth, and medical record number because ethics restricted us from using the more universal personal health number.
ICD-10 codes to be tested for TBI were selected based on the following: (1) an international systematic review of validated ICD codes to study neurological conditionsReference St Germaine-Smith, Metcalfe and Pringsheim 11 ; (2) a review of the grey literature and relevant websites (including TBI codes used by the US Centers for Disease Control and Prevention for their TBI surveillance programReference Faul, Xu LWald and Coronado 1 and codes used by the Canadian National Trauma Registry 20 ; and (3) expert consensus (neurologists, epidemiologists, health services researchers, population health experts) from the Canadian Chronic Disease Surveillance System Neurological Conditions Working Group (unpublished) based on the literature reviews provided previously and expert opinion. Because we were only interested in acute presentations of TBI, some codes used to identify TBI were excluded, such as those used to identify certain TBI-related sequelae (e.g. postconcussive syndrome).
A case of TBI was considered to be present if one or more of the specified TBI-related ICD-10 codes was listed in any of the diagnostic coding fields in either of the linked administrative databases. Table 2 shows the ICD-10 codes that were used to identify TBI in the administrative databases. Nine unique case definitions were subsequently tested to see which produced the best case definition for children with TBI using the REDIS cohort data as the reference standard against the administrative data (DAD and ACCS) (Figure 1). Because TBI is an acute presentation, we only tested different case definitions over a period of 1 or 2 years.
Timeline
ICD-10 codes used to identify TBI in administrative data
NOS=not otherwise specified.
STATISTICAL ANALYSIS
Descriptive statistics were used to describe the breakdown of the study population.
Accepting the REDIS chart data as the reference standard, diagnostic accuracy estimates (sensitivity, specificity, positive predictive value [PPV], and NPV) were calculated for each of the nine case definitions. Sensitivity refers to the proportion of TBI cases identified from REDIS that were coded as TBI cases in the administrative data. Specificity refers to the proportion of non-TBI cases identified from REDIS that were coded as non-TBI cases in the administrative data. PPV refers to the proportion of TBI cases identified in the administrative databases that were deemed “true” TBI cases on the basis of REDIS chart review data. NPV refers to the proportion of non-TBI cases identified in the administrative databases that were deemed “true” non-TBI cases on the basis of REDIS chart review data.
Ethics approval was obtained from the Conjoint Health Research Ethics Board at the University of Calgary.
RESULTS
Study Population
Of the 7101 children identified through REDIS, 1422 (20%) were determined to have TBI and 5679 served as non-TBI cases (Figure 2). After administrative data linkage, 94% (6639/7101) of the original study population was available for analysis, leaving 1343 with TBI and 5296 without TBI (Figure 2). Because of ethical restrictions, we were not able to link our data using the more universal personal health number, which may have contributed to the 6% of patients that we were unable to link. Based on the available linked data, the study population was 56% male (3372/6639), with a mean age of 9.2 years (range, 2.02 days-18.3 years) for males and 8.5 years (range, 2.61 days-18.6 years) for females.
Study population
Validation of ICD-10 Coding
The most frequently used code for TBI cases in our population was S09.9—unspecified injury of head, including injury of ear not otherwise specified (NOS), face NOS, and nose NOS (50.5%)—followed by S06.000—concussion without loss of consciousness without open intracranial wound (22.7%) (Table 3). Similarly, 88.9% of the time, the codes used for TBI were found in the primary diagnosis field of the administrative data.
Frequency of ICD-10 codes used in administrative data to identify TBI
NOS=not otherwise specified.
* Children with TBI in our cohort (true-positive cases) were identified to have the codes listed here.
ICD-10 Codes Used in False-Negative TBI Cases
We investigated which ICD-10 codes were used to identify the false-negative cases (n=406) in our best case definition, to see if we were missing important codes used to capture children with TBI and to determine if our case definition required further enhancement. The majority of false-negative cases were classified with nonspecific codes identifying superficial injuries, or general trauma, which with other information in the REDIS chart may have appeared to be a possible TBI but was not coded as such in the administrative data.
Identification of Best Case Definition to Identify TBI
Of the nine case definitions assessed (Table 4), the one with the best diagnostic accuracy to capture cases of pediatric TBI for future surveillance work was “1 ER or 1 hospital in 1 year” with a sensitivity of 69.8% (95% confidence interval [CI], 67.3-72.2), specificity of 96.7% (95% CI, 96.2-97.2), PPV of 84.2% (95% CI 82.0-86.3), and NPV of 92.7% (95% CI, 92.0-93.3).
TBI validation results for case definitions including hospital or ER visits
CI=confidence interval; ER=emergency room; NPV=negative predictive value; PPV=positive predictive value; TBI=traumatic brain injury.
* This case definition means that at least one TBI-related International Classification of Diseases, 10th revision, code was captured in the hospitalization or ER database during a 1-year period.
DISCUSSION
This study investigated whether two Alberta-specific administrative databases (hospital and ER databases) could be used to develop a validated case definition for pediatric TBI. The best case definition to accurately identify the largest proportion of children with TBI in these ICD-10–coded data was “1 hospital or 1 ER visit in 1 year.” This means that at least one TBI-related ICD-10 code was captured in the hospitalization or ER database during a 1-year period. Currently, there is not a single agreed-upon case definition for TBI. Some experts recommend excluding ICD-10 code S09.9 from TBI case definitions because it has been shown to inflate incidence and prevalence figures. We chose to include this code in our study because it was the most frequently used (50.5%) to identify TBI in our pediatric study population. Had we excluded this code, we would have missed 1145 cases (80.5%). There is concern that minor brain injury cases may be missed if codes such as S09.9 are excluded, given their ambiguous nature.Reference Chen and Colantonio 21 It has also been reported that up to 20% of S09.9 diagnoses may be cases of mild TBI.Reference Chen and Colantonio 21 Therefore, because a large proportion of patients in our study were identified with code S09.9, it is highly plausible that our sample is characterized largely by patients with milder forms of TBI. It is also important to recognize that we did not evaluate false-positive cases occurring as a result of the use of this code in our sample. As such, it is important to recognize that the use of this nonspecific code may have resulted in the identification of more false-positive cases, which is a limitation.
Our recommended case definition, “1 hospital or 1 ER visit over 1 year” was selected because it is important to prioritize sensitivity over other measures when the goal is to identify all persons with a certain characteristic (such as TBI) in a population.Reference Chubak, Pocobelli and Weiss 22 This is relevant to our study because the goal is to use the best case definition for disease surveillance. Although the sensitivity is not perfect, the specificity is very good and, as such, the definition is still valuable for allowing the identification of cohorts for future studies examining a variety of health care issues such as the trajectory of cases through the health care system. This ICD-10 coding validation is important because most countries are now using ICD-10 data to code for morbidity and mortality.Reference Jette, Reid, Quan, Hill and Wiebe 9
The other tested case definitions revealed varying results. The case definition looking at the DAD alone missed the majority of TBI cases (sensitivity, 3.7-4.2%), but was still associated with excellent specificity (up to 99.8%) and PPV (up to 84.7%). Although this group of hospitalized patients may be good for accurately defining cases for hospital-based outcomes or follow-up studies, this is a poor choice for surveillance. Using the hospitalization data alone may also introduce selection bias, because only the more severe cases tend to be admitted. The case definition identifying those from the ER alone may also be good for surveillance, but is likely missing the more severe cases; thus, it is important to use a case definition that captures children from both databases. Incorporating both of these databases for national surveillance of pediatric TBI is feasible in Canada because both hospitalization and ER visits are collected prospectively by provincial governments. It would be time-consuming and prohibitively expensive for a clinical network to develop its own pediatric TBI surveillance program. Administrative data are more feasible and are readily available. In the future, electronic medical records or electronic health records may become another source for TBI surveillance, but alternative case definitions incorporating both free text and ICD codes will then need to be developed. Numerous electronic medical record and electronic health record vendors exist, rendering this option more challenging.
Although there are no cutoffs or standard criteria that specify an optimal sensitivity, specificity, PPV, or NPV for case definitions, the validity of our case definition for pediatric TBI is comparable to other conditions. A systematic review looking at validated case definitions for neurological conditions found low sensitivity for Alzheimer’s disease and dementia, Parkinson’s disease, and spinal cord injury.Reference St Germaine-Smith, Metcalfe and Pringsheim 11 Similarly, two separate studies validating ICD-9 codes for depression found low sensitivity, ranging from 18.1% to 34.0%.Reference Noyes, Liu, Lyness and Friedman 23 , Reference Singh 24 Considering the results of validation studies for different neurological and mental health conditions, ours has high validity and should be considered for use in pediatric TBI surveillance.
There are several limitations to this study. First, our reference standard was based on a pediatric ER electronic chart review only. It is a concern that misdiagnosis or incomplete documentation of clinical data in a more limited electronic record may affect results. However, when children are admitted to the emergency room, they often receive a more comprehensive workup acutely than if solely seen in community-based locations. In addition, these emergency department electronic charts were reviewed by both a trained research assistant and a pediatric neurologist with TBI expertise; therefore, we postulate that this reference standard is most likely reflecting the “true” diagnosis. We also further confirmed the diagnosis by examining the percentage of children who were not deemed to have TBI upon telephone follow-up (4%) and who were not deemed to have TBI upon a clinic visit (0%). Misclassification bias is a common concern with administrative data; however, one of the primary purposes of a validation study is to examine the validity of these codes and to determine how to best identify children with TBI, so that we are capturing the most accurate picture.
To increase the likelihood of capturing children who may have experienced a TBI (in view of the low incidence of TBI based on epidemiological studies), our enriched population included children with primarily trauma, musculoskeletal, or central nervous system issues. As such, the incidence of TBI in our cohort was higher than expected, and as a result, could have influenced our predictive values (e.g. higher prevalence=higher PPV).
Second, we were not able to assess the severity of TBI cases in our population using a gold standard measure. Further research evaluating TBI severity using the Glasgow Coma Scale, the Abbreviated Injury Scale, or other measures would increase our understanding of which types of TBI are seen in different settings and how case definition accuracy may vary by TBI severity subtype. Despite our inability to determine severity, we initially hypothesized that the majority of cases seen in the ER or hospitalized would be the moderate to severe cases. However, considering the high proportion of children identified in our study with the nonspecific code S09.9 (>80%), it is highly likely that many of the children in our sample had milder injuries. The frequent use of this nonspecific code could also relate to the challenge of accurately identifying symptoms of TBI in younger children.
In addition, we were not able to obtain physician claims data that capture all visits, whether hospital or community-based; therefore, it is possible that we are missing a proportion of children with mild TBI who were seen in the community by their family physician or a walk-in-clinic. However, ACCS captures both emergency visits and hospital-based clinic visits; therefore, we are likely to have captured at least the more symptomatic mild TBI cases.
Third, this study population was selected from one hospital in a large city; therefore, the results may not be generalizable to other regions, including rural areas. In fact, it has been stressed in the literature that case definitions validated in one specific area or age group cannot necessarily be applied to other cohorts if you want to ensure accuracy.Reference Benchimol, Manuel and To 25 However, in Canada, professional ICD coders receive standardized national training (17). Therefore, results across different provincial administrative databases should be similar. Generalizing results internationally would likely be difficult because it has been shown there is coding variation between countries.Reference Drosler, Romano, Tancredi and Klazinga 26
In conclusion, we identified that, for surveillance purposes, the case definition “1 hospital or 1 ER visit over 1 year” is best at capturing children with TBI in the two administrative databases using ICD-10 coding. However, there is a need to validate TBI coding in physician claims data to enhance our proposed case definition and to get a more comprehensive picture of all TBI cases whether mild, moderate, or severe. We would also recommend that more efforts be made to educate coders on appropriate use of TBI-related ICD codes to better identify children with the various TBI subtypes, and that physicians receive more training in medical school and residency on the importance of careful chart documentation.
ACKNOWLEDGMENTS AND FUNDING
None.
DISCLOSURES
HQ has received a grant from AIHS and salary support from Alberta Innovates-Health Solutions. NJ is a Principal Investigator for Hotchkiss Brain Institute, the Canada Research Chair Program, and Alberta Innovates Health Solutions; has received a research grant from Pfizer (a shared grant from the University and Pfizer); and has received salary support from a government agency and from principal health research funding sources paid to her university. The remaining authors have nothing to disclose.
