Background
Parental report of otitis media (OM) is an outcome measure used when investigating the relationship of early childhood ear infections with subsequent hearing impairment and poorer psychosocial health (Yiengprugsawan et al., Reference Yiengprugsawan, Hogan and Strazdins2013; Hogan et al., Reference Hogan, Phillips, Howard and Yiengprugsawan2014). However, parental report of OM is not a robust measure of OM primary care visits. In an audit of primary care illness visits for children <18 months old living in Auckland, New Zealand (NZ), parental report underestimated the number of primary care doctor visits for acute respiratory illnesses (Grant et al., Reference Grant, Kaur, Waymouth, Mitchell, Scragg, Ekeroma, Stewart, Crane, Trenholme and Camargo2015). Here, parents reported approximately one-in-three visits identified by clinician review of the primary care records (Grant et al., Reference Grant, Kaur, Waymouth, Mitchell, Scragg, Ekeroma, Stewart, Crane, Trenholme and Camargo2015).
When primary care records are uncoded, clinician review of the free-text description of each primary care visit is possible but impractical for larger studies. As an alternative, software algorithms have been developed to scan and detect diagnoses in free-text records (Aronson and Lang, Reference Aronson and Lang2010). This computer software approach involves scanning free-text records for abbreviations, words and phrases related to the illness of interest.
However, relying solely upon a software algorithm to identify OM healthcare visits introduces the potential for errors. A significant problem is created by ambiguous abbreviations, commonly used in medical records (Sheppard et al., Reference Sheppard, Weidner, Zakai, Fountain-Polley and Williams2008; Collard and Royal, Reference Collard and Royal2015; Awan et al., Reference Awan, Abid, Tariq, Zubairi, Kamal, Arshad, Masood, Kashif and Hamid2016). Abbreviations can have different meanings in different clinical specialties and contexts. For example, ‘ROM’ is an abbreviation used for both ‘right otitis media’ and ‘range of movement’.
Another error that can occur with software algorithms is that they ignore the context of a clinical record entry. For example, the phrase ‘brother has otitis media’ could mistakenly be identified by a software algorithm as a visit for OM in the index child.
This study aimed to develop a screening process to identify OM diagnoses in free-text primary care consultation visits. We intended to create a software algorithm with high specificity to rule-out primary care visits that were not for OM. Keeping false positives to a minimum diminishes the subsequent clinician record review workload required when conducting primary care record-based research about OM.
Methods
This study was completed using the Growing Up in New Zealand child cohort study data (Morton et al., Reference Morton, Atatoa Carr, Grant, Robinson, Bandara, Bird, Ivory, Kingi, Liang, Marks, Perese, Peterson, Pryor, Reese, Schmidt, Waldie and Wall2013). This cohort included 6853 children born to women residing within a geographically defined region of NZ who had an estimated delivery date between April 25th, 2009 and March 25th, 2010 (Morton et al., Reference Morton, Atatoa Carr, Grant, Robinson, Bandara, Bird, Ivory, Kingi, Liang, Marks, Perese, Peterson, Pryor, Reese, Schmidt, Waldie and Wall2013). The NZ Ministry of Health Northern Y Regional Ethics Committee granted ethical approval (NTY/08/06/055) for the study. When the children were 54 months old, the study obtained written informed consent from the child’s caregiver to access the child’s primary care records. In NZ, primary care records are those created by General Practitioners (family doctors) working from primary care practices situated outside of the hospital setting.
The primary care records for a convenience sample of 200 children up to age 4 years from the Growing Up in New Zealand cohort were reviewed (Morton et al., Reference Morton, Atatoa Carr, Grant, Robinson, Bandara, Bird, Ivory, Kingi, Liang, Marks, Perese, Peterson, Pryor, Reese, Schmidt, Waldie and Wall2013). These records contained the free-text description of the child’s visit recorded by the various General Practitioners working in 20 different primary care practices. Inclusion of free text records created by different General Practitioners, working in several practices, allowed us to consider individual variability in free text record documentation.
A team of three clinicians, comprising a paediatrician (CG) and two research nurses (MvA, EW), reviewed these records. First, clinicians independently determined whether OM was a diagnosis considered during the consultation and documented the abbreviations, words, and phrases encountered during this process.
At the second stage, clinicians met and reviewed each consultation record to confirm agreement about the OM diagnoses identified. Where opinions differed, the consultation record was reviewed alongside the list of terms documented by clinicians to classify OM as present or absent, with consensus reached through this process.
During the third stage, the study data manager used the list of terms documented by the clinicians to design the software algorithm and screen the same set of clinician-reviewed primary care records (supplementary material). The development of the screening process for identification of OM diagnoses within primary care records involved comparing the results of the clinician and software algorithm screening methods. The clinicians and data manager met to discuss records where the results of the clinician and computer algorithm review differed, in particular, the false positive records identified as a visit where OM was present by the software algorithm but not the clinician review. The list of terms was also reviewed and corrected during this process.
During the fourth stage, the software algorithm was re-run based upon the updated list of terms. The performance of the software algorithm was assessed, with the clinician review as the gold standard. The results were described using sensitivity, specificity, positive and negative predictive values (PVs), and likelihood ratios (LRs) with 95% confidence intervals (CIs) (Chu, Reference Chu2002). We sought to develop a software algorithm with high specificity and high negative predictive value as we wanted to identify all consultations where OM may have been considered (Akobeng, Reference Akobeng2007), and thus required clinician review.
Results
The 200 children had a total of 10,034 primary care visits (Table 1). Clinician review determined that OM was considered a diagnosis in 917 (9%) primary care visit records. In 9117 (91%) primary care consultation visits, OM was an excluded diagnosis.
Table 1. Summary of primary care visits and primary care otitis media visits for the 200 children from the 20 primary care practices included in the pilot study
Table 2 contains the list of terms documented by clinicians classifying OM as a diagnosis recorded in the primary care records. When any of the terms from Table 2 were combined with words from Table 3, OM was excluded. For example, if a term referring to the tympanic membrane, i.e., ‘T/M’, ‘t/m’, ‘tm’ or ‘TM’ was combined with the word ‘normal’ in Table 3, OM was excluded as a reason for the primary care visit.
Table 2. Search terms which identified a primary care visit where otitis media was a potential reason for the primary care visit
Table 3. Words which in combination with a search term* from Table 3 identified a primary care visit where otitis media was excluded as a potential reason for the primary care visit
* When one of the search terms listed in Table 3 is found in the primary care visit text, it is then compared to all the exclusion terms. If the search term + exclusion term combination matches the visit text for any of the exclusion terms the search term is excluded from the results.
The software algorithm correctly identified 801 of the 917 primary care visits where clinician review determined that OM was considered a diagnosis and 8705 of the 9117 primary care visits where clinician review determined that OM was an excluded diagnosis.
In comparison with the clinician review, the software algorithm had the following diagnostic test properties (95% CI): sensitivity 87% (85–89%); specificity 96% (95–96%); positive predictive value 66% (63–69%); negative predictive value 99% (98–99%); positive LR 19.33 (17.54–21.31); and negative LR 0.13 (0.11–0.16) (Table 4).
Table 4. Diagnostic performance of the electronic record review compared with the clinician review as the reference for identifying primary care visits where otitis media was a potential reason for the primary care visit
Diagnostic test parameter (95% confidence intervals).
-
Sensitivity = 87.35% (85.02% to 89.43%)
-
Specificity = 95.48% (95.03% to 95.9%)
-
Negative predictive value = 98.68% (98.42% to 98.91%)
-
Positive predictive value = 66.03% (63.29% to 68.7%).
Discussion
Statement of principal findings
This study describes the process used to create a software algorithm to search free-text primary care visit records to identify primary care visits where OM was one of the diagnoses made. The intention was to create an algorithm with high specificity to rule out primary care visits that were not for OM. The negative predictive value of the software algorithm developed was 99% (95% CI 98–99%). The positive LR was 19.33 and negative LR was 0.13. The positive and negative LRs derived from this algorithm are of a magnitude that result in large increases in the post-test probability for the positive LR (>10) and moderate increases in the post-test probability for the negative LR (0.10–0.20)(Grimes and Schulz, Reference Grimes and Schulz2005).
The software algorithm identified a subset of primary care records that required clinician review to determine the presence or absence of OM. The clinician review determined that 66% of this subset of primary care visits were due to OM. Thus, the software algorithm reduced the number of primary care records visits requiring clinician review by 88%, from 10034 to 1213 visits. By reducing the number of healthcare visit records requiring clinician review, this greatly reduced the cost in terms of time and budget to extract data from free text primary care records and create variables that precisely and accurately define OM primary care visits.
Study strengths and weaknesses
The use of primary care records that are uncoded, whilst requiring the development of data extraction processes such as described here, has some advantages. Coding of primary care visits varies in its completeness. Respiratory illness visits in preschool aged children is a visit type previously shown to have a relatively high rate of incomplete coding (Hansell et al., Reference Hansell, Hollowell, Nichols, Mcniece and Strachan1999). More recent studies reporting primary care practice audit data document that variability in coding completeness persists (Molony et al., Reference Molony, Beame, Behan, Crowley, Dennehy, Quinlan and Cullen2016). Barriers to complete coding include the limitations of the coding systems, variability in coding skills and motivation, time pressures preventing the structured recording of data during consultations, and the priority given to coding within organizations (de Lusignan, Reference De Lusignan2005).
The study was completed using General Practitioner primary care records only. It may not be applicable to other settings where primary care is delivered, such as hospital emergency departments. This study records of primary care visits made during the preschool years for children born in 2009–10 (Morton et al., Reference Morton, Atatoa Carr, Grant, Robinson, Bandara, Bird, Ivory, Kingi, Liang, Marks, Perese, Peterson, Pryor, Reese, Schmidt, Waldie and Wall2013). Free-text descriptors can change over time and are likely to vary between countries. It is important that the development of any algorithm includes a piloting process, such as that described here, that is completed within the primary care record collection to which the algorithm is to be applied.
The algorithm was developed so it could identify when OM was specifically excluded. The algorithm was intentionally developed to be used in combination with a clinician review and hence, we cannot recommend its use without being combined with clinician review.
The list of search terms we developed included abbreviations and acronyms. Abbreviations and acronyms are commonly used in medical records (Awan et al., Reference Awan, Abid, Tariq, Zubairi, Kamal, Arshad, Masood, Kashif and Hamid2016). Our study showed the numerous abbreviations used for the diagnosis of ‘otitis media’ (Table 2). Patterns of abbreviation and acronym use vary between countries and in different clinical settings (Sheppard et al., Reference Sheppard, Weidner, Zakai, Fountain-Polley and Williams2008, Collard and Royal, Reference Collard and Royal2015). Hence, the process described here would need refinement before being used in different countries and different clinical settings.
Medical record search methods which only include a software algorithm review, without clinician review, miss events where short abbreviations are used in the text (Shah et al., Reference Shah, Martinez and Hemingway2012; MacRae et al., Reference Macrae, Darlow, Mcbain, Jones, Stubbe, Turner and Dowell2015a). Examples cited in the literature of abbreviations not identified correctly include MI for myocardial infarction and ‘regurg’ for regurgitation (Shah et al., Reference Shah, Martinez and Hemingway2012).
Without clinician review, the issue of how acronyms, abbreviations and misspellings are measured or addressed is not possible to define (Wu et al., Reference Wu, Sohn, Ravikumar, Wagholikar, Jonnalagadda, Liu and Juhn2013; MacRae et al., Reference Macrae, Darlow, Mcbain, Jones, Stubbe, Turner and Dowell2015a). With meanings of and patterns of use of abbreviations varying between clinical settings, it would be necessary to repeat the process described in this study for identifying OM diagnoses in other settings. Whilst algorithms can be developed which expand abbreviations into more complete text statements, this is not possible when the abbreviation has multiple and different meanings (Chapman et al., Reference Chapman, Christensen, Wagner, Haug, Ivanov, Dowling and Olszewski2005). Misspellings occur frequently in medical notes created using a keyboard, due to both spelling proficiency and keyboard skills, and also differences in personal practice behaviours related to checking for and correcting misspellings in medical records (MacRae et al., Reference Macrae, Love, Baker, Dowell, Carnachan, Stubbe and Mcbain2015b).
Strengths and weaknesses in relation to other studies
Clinician review and classification of clinical records is acknowledged as time consuming and prone to error. However, it remains a necessary component for extracting data from free-text healthcare visit records. Contemporary systematic reviews of the secondary use of clinical data contained in electronic medical records for research purposes highlight the high frequency of data quality issues and data pre-processing challenges (Edmondson and Reimer, Reference Edmondson and Reimer2020; Lewis et al., Reference Lewis, Weiskopf, Abrams, Foraker, Lai, Payne and Gupta2023).
In this study we had three clinicians classifying visits independently and reaching consensus when initial independent classifications of individual visits differed. Having multiple clinicians involved reduces the bias that can occur if only one clinician is classifying visits, as has been used in another recent study which sought to develop an algorithm for identifying influenza-like illness visits from uncoded General Practitioner records (MacRae et al., Reference Macrae, Love, Baker, Dowell, Carnachan, Stubbe and Mcbain2015b).
In countries, such as NZ, where acute respiratory infection primary care presentations are not consistently coded, reliance solely on a computer programme has insufficient sensitivity to identify visits for childhood respiratory illness presentations. For example, in a study that used 1200 primary care records of children aged 0 to 18 years attending 10 different primary care practices in Wellington, NZ, a software algorithm developed specifically to read primary care records had a sensitivity of 58% and specificity of 99% for identifying primary care OM visits (MacRae et al., Reference Macrae, Darlow, Mcbain, Jones, Stubbe, Turner and Dowell2015a). The computer programme enabled non-OM visits to be excluded, thus significantly reducing the number of records requiring clinician review. However, a clinician review of those records identified by the computer programme as OM visits is still required.
In addition to the presence of acronyms, abbreviations and misspellings, the lack of well-formed grammar or structure in these free-text primary care records limit the capacity to use computer software to identify phrases which reliably correspond to clinical diagnoses (MacRae et al., Reference Macrae, Love, Baker, Dowell, Carnachan, Stubbe and Mcbain2015b).
Meaning of the study: implications for clinicians and researchers
Considerable variability occurs in the quality and completeness of morbidity coding in primary care records (Jordan et al., Reference Jordan, Porcheret and Croft2004). Therefore, a review of the text records appears necessary in order to have confidence in the completeness of description of primary healthcare visits for any particular morbid disease.
The approach described here, of clinicians independently reviewing records that are mostly non-coded, and identifying the abbreviations, words, and phrases that can be used to create a software algorithm, can be applied to other illnesses which result in primary care presentations.
This approach is applicable only to relatively simple diagnoses. Its use for conditions which have very specific and multi-component case definitions, for example acute rheumatic fever or Kawasaki disease, is likely to be limited because of the inconsistencies in documentation of multiple positive or negative findings between different clinicians and by the same clinicians over time who experience competing and varied time pressures.
In summary, we describe the development of a software algorithm to screen primary care records for OM visits, and the utilization of this algorithm to increase the efficiency of identification of OM visits via clinician review of the text record. The process comprised of four steps: clinician review of the visit records, development of the algorithm, application of the algorithm to identify the healthcare visits requiring clinician review, and clinician review of records identified using the algorithm. These four steps are all required to generate precise data when using larger volumes of healthcare visits to describe presentations for specific health conditions. By reducing the number of primary care records visits requiring clinician review by 88%, the algorithm enables large volumes of free text primary care record to be reviewed by relatively small teams of clinician supported by modest budgets.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1463423625100674.
Funding statement
This project was funded by the Health Research Council of New Zealand (project grant number 16_165).
Competing interests
The authors have no competing interest to declare.
Credit statement
Cameron Grant: Conceptualization, methodology, formal analysis, investigation, resources, data curation, writing – original draft, visualization, supervision, project administration, funding acquisition.
Marisa van Arragon: Methodology, investigation, resources, data curation, writing – review & editing, visualization, supervision.
Ellen Waymouth: Validation, investigation, resources, data curation, writing – review & editing.
Alicia Stanley: Methodology, validation, resources, data curation, writing – review & editing.
Mapui Tangi: Validation, investigation, resources, data curation, writing – review & editing.
Eamon Ellwood: Software, investigation, resources, data curation, writing – review & editing.
Carol Chelimo: Conceptualization, methodology, formal analysis, writing – review & editing, visualization, funding acquisition.
Open access
