Hypoplastic left heart syndrome (HLHS) accounts for roughly 2–3% of congenital heart defects (CHDs),Reference Mai, Isenburg and Canfield1 yet it is responsible for 23% of cardiac deaths that occur within the first week of life.Reference Yabrodi and Mastropietro2 Long-term survival for patients with HLHS has continued to improve – contemporary surgical management is associated with over 60–70% 10-year survival.Reference Karamlou and Najm3 Management of these patients is complex with significant parental involvement required for optimal outcomes, particularly during the interstage periods between Norwood (Stage 1), bidirectional Glenn (Stage 2), and Fontan (Stage 3) operations.Reference Ugonabo, Hirsch-Romano and Uzark4,Reference Meakins, Ray, Hegadoren, Rogers and Rempel5 As such, parental education and understanding are essential for the long-term management and follow-up of patients with HLHS. In the modern era, most patients and/or their families access electronic resources to develop an understanding of their health problems.Reference Baker, Devitt, Lynch, Green, Byrne and Kiely6,7 Importantly however, the average US citizen reads at an 8th-grade reading level.Reference Stossel, Segar, Gliatto, Fallar and Karani8 (In the United States of America, students in 8th grade are usually 13 or 14 years old, and 8th grade in the United States of America is equivalent to Year 9 in the United Kingdom, where there is a similar average reading level of 11–14 years old.Reference Stossel, Segar, Gliatto, Fallar and Karani8,9 ) Poor health literacy has been linked to increased mortality in other contexts, including acute heart failure, mental illness, and diabetes.Reference Berkman, Sheridan, Donahue, Halpern and Crotty10–Reference McNaughton, Cawthon, Kripalani, Liu, Storrow and Roumie12 Thus, providing quality online resources about HLHS at a reading level appropriate for the average US/UK citizen is important to improve the usefulness of these materials and ultimately improve the outcomes achieved for these patients. Furthermore, providing quality online resources about HLHS at a reading level appropriate for the average US/UK citizen will enhance overall health equity.
Others have evaluated the quality and readability of publicly available electronic health resources in a variety of different settings.Reference Treffalls, Treffalls and Zachary13–Reference Clothier, Treffalls, Tolbert, Harbin, Yan and Davies16 These studies consistently find that available online health resources are written above the average US citizen reading level and that website quality is often poor, a combination that challenges improved health equity. In one of these analyses evaluating online resources for patients with peripheral artery disease,Reference Treffalls, Treffalls and Zachary13 poor readability was reflected with a median Flesch–Kincaid Grade Level Score of 10.7 (interquartile range = 10–12) and poor quality was reflected with a median total quality score of 19 out of 47 (interquartile range = 15–30), which accounted for accountability, content, interactivity, and structure. While these analyses highlight areas for improvement in other pathologies, the literature lacks analyses of the quality of online resources available to patients with HLHS and their families.
The purpose of this study is to assess the quality and readability of publicly available electronic patient resources about HLHS.
Methods
We queried four search engines for the search terms “hypoplastic left heart syndrome”, “HLHS”, and “hypoplastic left ventricle”. These search terms were chosen in order to avoid irrelevant results that more broad terms would return. The search engines queried were
-
Google,
-
Yahoo!,
-
Bing, and
-
Dogpile.
Google, Yahoo!, and Bing are native search engines, while Dogpile is a “metasearch” engine that compiles results from multiple other native search engines. These queries were completed in December 2022. Queries were completed on a Chrome browser in incognito mode to avoid the impact of browser cookies and search history on results returned by the queries performed.
The first 30 website results for each search term from searches on Google, Yahoo!, Bing, and Dogpile were combined into a single list. Exclusion criteria included:
-
Duplicates,
-
Commercial websites,
-
Physician-oriented resources such as academic research literature,
-
Disability websites that describe how to access governmental benefits, and
-
Broken links.
Of note, disability websites are not designed to help patients and their families understand their conditions.
Websites were grouped into four types:
-
Hospital/healthcare organisation,
-
Governmental agency,
-
Professional medical society, and
-
Open access.
We defined websites as hospital/healthcare organisation if associated with an institution providing direct patient care; governmental agency if associated with a government-operated organisation; professional medical society if associated with a professional society of any speciality of medicine; or open access if unable to be categorised as any of the preceding types.
The assessment tool developed by Ingledew to assess quality of websites in regard to patient–physician collaboration was modified to assess websites about HLHS (Appendix 1). Reference Ingledew17 Using a two-reviewer system (Fig 1), websites were graded for:
Analysis of online HLHS resources by website type. Total (p = 0.03) and accountability (p = 0.001) scores varied significantly by website type, with open access websites scoring the highest and hospital/healthcare organisation websites scoring the lowest. Abbreviations: HLHS, hypoplastic left heart syndrome.
-
Accountability,
-
Content,
-
Interactivity, and
-
Structure.
Reviewers underwent training to become familiar with the evaluation systems. Scoring discrepancies among the two reviewers were resolved by consensus. To assess readability, standardised readability scores were calculated by two reviewers via an open access website that offers tools to assess readability on four scales (Fig 2):
Readability distribution of online patient resources. Distribution of websites by readability as assessed by the Gunning Fog, Flesch–Kincaid, Coleman-Liau, and Linsear Write indices. Using all four indices, the majority of online patient resources for hypoplastic left heart syndrome were written above the 8th-grade reading level recommended by the Food and Drug Administration.
-
Gunning-Fog,
-
Flesch–Kincaid,
-
Coleman-Lau, and
-
Linsear Write.
Nonparametric analysis of variance was performed to assess differences in websites, types of websites, and readability across websites. Inter-rater agreement was assessed by calculating a Cohen’s Kappa coefficient between reviewers.
Results
Websites
After exclusion criteria were applied, 52 websites remained and were assessed by two reviewers. Inter-rater agreement between reviewers was high (Kappa = 0.87). Website types included:
-
35 hospital/healthcare organisation (67.3%),
-
12 open access (23.1%),
-
4 governmental agency (7.7%), and
-
1 professional medical society (1.9%).
Six websites (11.5%) were based out of the United Kingdom, one (1.9%) out of Australia, one (1.9%) out of India, and the remaining 44 (84.6%) websites were based out of the United States of America. Lists of the top five websites in content, readability, and overall score are presented in Table 1.
Top websites in content, readability, and overall
Quality
The quality of online resources for patients with HLHS was variable, with a total score range of 23.5 (range = 8–31.5). The median total score for the 52 websites was 48.7% (19 out of 39 [interquartile range = 15.8–25.3]): accountability was 32.4% (5.5 out of 17 [interquartile range = 2.0–9.3]), content was 66.7% (8 out of 12 [interquartile range = 6.4–10.0]), interactivity was 33.3% (2 out of 6 [interquartile range = 2.0–3.0]), and structure was 75% (3 out of 4 [interquartile range = 2.8–4.0]). Accountability was low with 32.6% of sites disclosing authorship and 26.9% citing sources. Less than half (42.3%) of websites were available in Spanish. Total score varied by website type (p = 0.03), with open access sites scoring the highest (67.9%, median = 26.5; interquartile range = 20.5–28.6) and hospital/healthcare organisation websites scoring the lowest (44.9%, median = 17.5; interquartile range = 13.5–21.5). Score differences were primarily driven by differences in accountability (p = 0.001), with open access websites scoring 67.6%, 11.5 out of 17 (interquartile range = 6.5–13.0) and hospital/healthcare organisation websites scoring 11.8%, 2 out of 17 (interquartile range = 2–7). The timing of all three procedures in staged palliation (Norwood, bidirectional Glenn, and Fontan) was discussed by 76.9% (40 of 52) of websites. One website (1.9%) discussed timing of one or two of the procedures, while 11 (21.2%) did not mention timing at all. All other categories, including content (p = 0.25), interactivity (p = 0.15), and structure (p = 0.33) were similar between groups.
Content
Content was assessed using a custom, HLHS-specific 12-point scale. HLHS was defined in 100% (52 of 52) of websites. However, less than half (40.4%, 21 of 52) discussed the incidence and/or prevalence of HLHS. Presenting symptoms of HLHS (e.g., blue baby, low oxygen saturation, dyspnoea, and cyanosis during feeding) were discussed by 75.0% (39 of 52) of websites. Discussion of diagnostic tests, which required inclusion of echocardiography, was present in 78.9% (41 of 52) of websites. The Norwood, Glenn, and Fontan procedures were discussed by 73.1%, 75.0%, and 75.0% of websites, respectively. Other surgical therapies, including ventricular assist device implantation and heart transplantation, were discussed by 75.0% (39 of 52) of websites. Only 23.1% (12 of 52) discussed surgical complications in more than two sentences, while 13.5% (7 of 52) discussed this topic in one to two sentences. Finally, complications of HLHS (e.g., growth delay, death) were mentioned by 71.2% (37 of 52) of websites.
Readability
Overall readability of the 52 websites was low, with median Flesch–Kincaid Grade Level of 11th grade (interquartile range = 10th–12th grade), which did not vary significantly by website type (p = 0.29). Similar readability levels were found using other calculators, including:
-
The Coleman-Liau index (median = 11th grade; interquartile range = 10th–12th grade; p = 0.83),
-
The Gunning Fog index (median = 13.6; interquartile range = 12.7–14.7; p = 0.99), and
-
The Linsear Write formula (median = 9th grade; interquartile range = 7th–11th grade; p = 0.56).
Using Flesch–Kincaid Grade Level, only 15.4% (8 of 52) of websites were written at or below an 8th-grade reading level, with 17.3% (9 of 52) of websites written at a college or higher reading level.
Discussion
Patients and their families have increasingly taken charge of how they process information about their health; therefore, there is increased need for physicians to help guide them to quality resources. Reference McMullan18,Reference Chen, Li, Liang and Tsai19 The majority of patients now research their conditions before meeting with their physician but remain unlikely to discuss the findings of this research during visits with physicians. Reference Hay, Cadigan and Khanna20 Poor patient literacy is linked to poorer health outcomes and health inequity in many populations. Reference Taylor, Fraser and Dudley21 Many websites involving health data on the internet provide misleading or false information. Reference Jejurikar, Rovak and Kuzon22,Reference Beredjiklian and Bozentka23
The average person in the United States of America reads at a 7th- or 8th-grade level. Reference Stossel, Segar, Gliatto, Fallar and Karani8,24 This fact is one of many reasons why organisations like the Centers for Disease Control and National Institutes of Health of the United States of America advise that websites be written at an 8th-grade level in plain language. 25,26 Readability is an important consideration when healthcare organisations are creating online materials that may be consumed by patients. Reference Stossel, Segar, Gliatto, Fallar and Karani8 Our data, however, show that the majority of websites about HLHS are written at an 11th-grade reading level or higher, and only 15.4% of websites are written at an 8th-grade level or lower. This finding represents a gap between information available and the average reader, and this gap should be addressed by organisations that create health information websites.
Overall, our analysis revealed that hospital/healthcare organisations had the lowest scoring websites, while open access websites had the highest scoring websites. However, overall scores were low, with the average score being 19 out of 39 (48.7%). Quality of websites was especially impacted by accountability, showing a need to increase source citation and authorship disclosure.
While websites frequently discussed the definition of HLHS, diagnostic steps, and surgical procedures, few discussed complications of surgery. Only 23.1% of websites discussed surgical complications in more than two sentences and another 13.5% of websites mentioned surgical complications in one to two sentences, demonstrating a need to increase the number of websites offering insight into surgical complications for patients and their families, to help them in making informed and balanced decisions. As an essential component of informed consent, the discussion of the risks of surgery likely occurs during the patient–surgeon interaction. Reference Slim and Bazin27 However, since patients frequently search for information online, incorporating discussion of surgical complications into publicly available online resources represents an opportunity for improving patient education and enhancing health equity by allowing fully informed consent to occur. Reference Hay, Cadigan and Khanna20 When incorporating discussion of surgical complications, others have found that citing the potential likelihood of complications using point estimates (e.g., 3% risk) or risk ranges (e.g., 1–5%) is associated with improved and consistent patient understanding when compared to using verbal descriptors (e.g., “uncommon”). Reference Rosen, Agrawal, Flum and Liao28 In that survey-based study of 296 American adults, the probability of complications associated with surgical or antibiotic treatment of acute appendicitis was presented using these three techniques (verbal descriptors, point estimates, risk ranges). Reference Rosen, Agrawal, Flum and Liao28 There was significantly higher variability in patient perception of risk when presented with verbal descriptors. Thus, when incorporating increased discussion of surgical complications associated with staged palliation for HLHS into publicly available online resources, using point estimates or risk ranges may be associated with improved patient understanding of these risks, allowing for patients/parents to make informed decisions with realistic expectations.
Spanish-speaking patients, a fast-growing demographic in the United States of America, often have lower health literacy than native English speakers. Reference Brice, Travers, Cowden, Young, Sanhueza and Dunston29,Reference Christy, Cousin and Sutton30 Despite this, less than half (42.3%) of websites about HLHS are available in Spanish. Addressing this gap in accessibility is especially important considering that additional difficulties faced by these patients are often exacerbated by language barriers. Reference Rodriguez, Allison and Daviglus31 Encouragingly, Villa Camacho and colleagues found that online patient education materials written in Spanish for breast cancer were written at a lower grade reading level than equivalent English-language versions (5.49 ± 0.50 versus 7.77 ± 1.95, p < 0.01) and were more likely to meet American Medical Association recommendations for patient educational resources (82.9% versus 40.4%, p < 0.01). Reference Villa Camacho, Pena and Flores32 Thus, while availability of Spanish-language online resources for HLHS is poor, readability of available resources may be appropriate, although this remains unknown for HLHS. Increasing the availability of online HLHS resources translated to Spanish is necessary to improve patient understanding in this subgroup of patients and their families.
Ultimately, the low overall score of most websites and high levels of variability highlight the need to improve websites broadly, especially those designed for patients and their families. This need for improvement is especially true for the domain of accountability, which was overall low and was the main driver of differences in scores among website types. Key methods to improve accountability would be highlighting sources of information and displaying authorship on all patient-facing and family-facing websites.
This study has important limitations. Websites are designed with goals that may not be patient-centred and/or family-centred, and readability and content may vary based on the intended audience. Due to their nature, websites are constantly changing, and this study was only a snapshot of one day in the history of each website. Search engine results are impacted by user search history and other factors like location, which could change some of the top results found by patients. While we excluded websites that were not patient-centred, they may still be encountered by patients and their families in their search results.
Conclusions
Our evaluation of popular websites about HLHS identifies several opportunities for improvement, including increasing accountability by disclosing authorship and citing sources, enhancing readability by providing material that is understandable to readers with the full spectrum of educational background, and providing information in languages besides English.
Acknowledgements
None.
Financial support
This research received no specific grant from any funding agency, commercial entity, or not-for-profit organisation.
Competing interests
None.
Ethical standards
No human or animal experimentation was conducted during the course of this research. All procedures contributing to this work comply with the ethical standards of the relevant national guidelines on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Appendix 1 Website quality assessment questions and scoring
HLHS, hypoplastic left heart syndrome; NA, not applicable; VAD, ventricular assist device.
Open access
