Paediatric acute care cardiology collaborative data registry validation

Abstract Background: The Pediatric Acute Care Cardiology Collaborative (PAC3) was established to improve acute care cardiology outcomes through the development of an accurate and well-validated clinical registry. We report the validation results of the initial PAC3 registry audits and describe a novel regional audit format developed to accommodate a rapidly expanding membership facilitate collaborative learning and allow for necessary modification due to the COVID-19 pandemic. Materials and methods: Six hospitals were audited using a regional audit format and three hospitals were subsequently audited virtually. Critical and challenging-to-collect data elements were audited among at least 40 randomly selected cases. Discrepancies were categorised as either major or minor depending on their relative importance to patient outcomes and clinical care. Results were tabulated and reported. Results: We audited 386 encounters and 27,086 individual data fields across 9 hospitals. The aggregate overall accuracy rate was 99.27% and the aggregate major discrepancy rate was 0.51%. The overall accuracy rate ranged from 98.77% to 99.59%, and the major discrepancy rate ranged from 0.26% to 0.88% across the cohort. No appreciable difference was seen between audit formats. Both the regional and virtual audit methods were viewed favourably by participants. Conclusions: A low data discrepancy rate was found demonstrating that the PAC3 registry is a highly accurate data source for use in quality improvement, benchmarking, and research. Regional audits and virtual audits were both successfully implemented.

The Pediatric Acute Care Cardiology Collaborative (PAC 3 ) was established in 2014 motivated by a mission to improve acute care cardiology outcomes and experience from the perspective of the patient, family, clinical team, and hospital system. The PAC 3 registry began collecting data of February, 2019 with 6 hospitals contributing data and has since expanded to 27 hospitals. Central to the success of PAC 3 's mission is the conviction that improvement will be best measured by metrics derived from a validated, audited, and prospective data registry. 1 Inspired by the high data fidelity standards previously established and verified by the Pediatric Cardiac Critical Care Consortium (PC 4 ), and the opportunity to create shared database elements for patient populations across hospitalisations, as well as the entire lifespan of individual patients, the PAC 3 Database Committee created an audit process for data validation. 2,3 Importantly, while PC 4 and PAC 3 share several data elements and collection methods, they are independent organisations and registries. PAC 3 has partnered with PC 4 throughout its development and has benefitted from their successes. This partnership has facilitated such rapid incorporation of new member hospitals that new methods of data verification are required, as is the development of novel audit processes. Furthermore, the onset of the COVID-19 pandemic in March, 2020 required substantial adjustments to the previously designed in-person audit procedures. Despite these challenges, and maybe because of them, the PAC 3 audit process has proven itself both feasible and effective. PAC 3 completed nine successful audits between February, 2020 and October, 2020: six were regional audits and three occurred in a virtual format.
The PAC 3 audit process is a four-tiered process. Tiers 1-3 involve partnership between the local hospital's data team and the PAC 3 Data Coordinating Center (DCC). Tier 4 involves data element verification and requires the local hospital's data team, the DCC, and PAC 3 auditors. We present a novel registry audit process incorporating both regional and virtual models that could potentially be used by other paediatric registries. We report the results from the first nine audits providing evidence to support the broad use of the registry for benchmarking, quality improvement, and research.

Summary of registry data elements
The PAC 3 registry is built as a hospital encounter-based system recording unique data elements during the acute care cardiology unit encounter. These data are then combined with select shared PC 4 data elements at the hospital level from admission through discharge. PAC 3 encounters, as designated by the encounter start date and time, begin when a patient's care is transferred to a non-ICU cardiology service as the primary service. The PAC 3 encounter ends, as designated by the encounter-level end date and time, when the patient is transferred or discharged from the cardiology service. Additional data fields include demographic and clinical variables, supportive therapies (including respiratory, pharmacologic, devices), feeding and nutrition variables, as well as complications that were actively treated during the acute care cardiology unit encounter. Out of 320 PAC 3 data fields, 295 are mandatory and are required for case submission.
In addition to data fields specific to the acute care cardiology unit-level encounter, shared hospital-based data fields are entered on a single-data platform with the ability to share hospital-level data elements across other data registries such as PC 4 , STS-CHSD (Society of Thoracic Surgery Congenital Heart Surgery Database), and the American College of Cardiology IMPACT (IMproving Pediatric and Adult Congenital Treatments) registry. 4,5 This allows for ease of data entry and data consistency in critical data elements across these registries such as a patient's fundamental cardiac diagnosis.
Hospitalizations are classified as either medical or surgical based on whether the patient underwent a cardiothoracic surgical procedure during their hospital stay. Patients solely admitted to the acute care cardiology unit for routine observation following catheterisation or an electrophysiology study and discharged the following day may be collected for local hospital use but are not included in PAC 3 for submission. Patients physically located on the acute care cardiology unit but with a non-cardiology primary service are not included in the registry.

Data collection team
Data acquisition from the hospital electronic medical record system requires a dedicated data collection team comprised of a clinical champion, typically a cardiologist or an acute care cardiology unit advanced practice provider, as well as data champion(s) who most commonly have a clinical nursing background, but have also come from alternative professional backgrounds such as data analytics and research management. This team is responsible for timely and accurate case submission and for ensuring reliable and consistent data collection processes. A hospital's first audit typically occurs 1 year to 15 months after initiation of data submission. Longer-term follow-up audits are intended to occur in 2-3 year cycles after successfully completing the first audit. We anticipate second-round audits will begin in 2022.

Audit process
The PAC 3 audit process includes four stepwise tiers (Fig 1). The first three tiers are screening points, while the final tier is the audit itself. Tier 4 was developed as in-person regional audits with capacity for multiple centres but was shifted more recently to a virtual format utilising HIPAA-compliant videoconferencing technology due to the COVID-19 pandemic.
The first tier determines the hospital's PC 4 audit history. Since several areas of the PAC 3 registry are shared with PC 4 (demographics and many hospital-level variables), and because of the intentional overlap of the auditing approach, a PC 4 member centre must successfully pass its PC 4 audit to be eligible for PAC 3 auditing. The additional aim of this tier is to eliminate duplicated auditing efforts. For the few hospitals that do not also participate in PC 4 , as this is not a requirement for PAC 3 membership, the initial PAC 3 audit will consist of a separate design requiring an in-person individual centre audit in place of the regional or virtual formats described below.
The second tier of the auditing process is a census check to ensure member hospitals have a robust process for capturing all eligible surgical and medical patients. Hospitals send a record of all surgical cases to the DCC sourced from their local STS data. These records are cross-checked with submitted PAC 3 cases by the DCC. Hospitals then send an admission, discharge, and transfer feed to the DCC which is typically generated by a hospital's billing department or data warehousing department and reports all patient movement in or out of a particular hospital unit or service line. A sample of the admission, discharge, and transfer feed (~10%) is cross-checked with submitted PAC 3 cases to ensure eligible medical cases have been entered. Typically, 300-1000 cases are submitted for verification during the census check depending on a hospital's volume.
The DCC reviews any discrepancies with the hospital's data team during a 2-hour phone call and reviews the hospital's process for ensuring all PAC 3 -eligible patients are captured. If a hospital is found to be under capturing patients by more than three cases, they cannot pass their audit until all eligible patients have been entered and a hospital's process for capturing patients has been improved. This benchmark ensures a high standard of patient capture for all hospitals regardless of volume.
The third tier of the audit is the self-audit and is therefore performed by individual data collection teams on a quarterly basis. Member hospitals can determine their own self-auditing process but are encouraged to participate in quality improvement efforts supported by PAC 3 leadership to both standardise this process and to improve data quality and timeliness. This step may include regularly scheduled meetings to review data collection questions and to simulate audits of submitted cases. The member hospital cannot complete their tier 4 audit unless they have developed an internal auditing process consistent with the aims of tier 3. The results of this internal audit have no bearing on the scoring of the tier 4 audit. The purpose of the internal audit is to ensure that hospitals are revisiting their data on an ongoing basis to reinforce their understanding of data definitions and allow for real-time cleaning of their own data to ensure accuracy and reliability of data entry.
The fourth tier of the audit process verifies individual data elements in the electronic medical record. This was originally designed as an in-person regional audit of 2 or more centres and has more recently been adapted to suit a virtual videoconference format. All programming for audit preparation is done using SAS 9.4 software. The DCC selects 60 hospital encounters representing a broad scope of patient complexity. Standardized forms are developed for both the auditing process itself and for tracking discrepancies. The audit team consists of a data champion from a hospital being audited (the auditee), a clinical champion from another hospital who serves as the auditor, and a neutral thirdparty observer from a separate hospital (the third-party observer can be a data or clinical champion or a member of the PAC 3 leadership team). The audit trios are required to complete at least 40 of the 60 cases over the course of 2 days. This allows the auditee to benefit from the collective data collection expertise of those with in-depth knowledge of the registry and whose data collection processes have been previously vetted. Despite the virtual format, there remains a focus on collaborative learning as provided by the in-person regional audits. For both regional and virtual audit formats, the data champion from the hospital undergoing the audit displays their electronic medical record screen to the auditor and observer. The auditor uses the audit report and verifies each data element with the auditee by examining the electronic medical record as it is navigated by the auditee in real time. The third-party observer is responsible for resolving any disagreements between auditor and auditee and for tracking discrepancies which are then sent back to the DCC for scoring. Audit trios are arranged, when possible, by shared electronic medical record experience.
To accommodate virtual audits, a hospital's data team obtains permission from their hospital to use a HIPAA-compliant video-sharing service to share their screen with the auditor and observer. In part due to the pandemic, hospital policies have become increasingly flexible. Since the auditor and observer are not entering the electronic medical record directly, but rather simply observing, there has been unique opportunity to reimagine the audit process.
Importantly, at the conclusion of each day, time is spent sharing lessons learned from the audit experience. Examples include clarifying specific data definitions, discussing more efficient ways to extract data from the electronic medical record, and sharing data capture of unusual patient types.

Audited fields
The PAC 3 audit subcommittee reviewed the entire registry composed of 320 individual data elements and selected the audit variables that met at least one of two thematic criteria: select variables that are particularly challenging to collect accurately given the absence of a discrete data field in the electronic medical record and select variables that are predicted to be of critical importance to clinical improvement efforts and research. Cumulatively, the audit includes 80 fields for patients less than 1 year of age and 72 fields for all others representing roughly 25% of available data elements in the registry. These audited data elements were categorised into major and minor categories based on relative significance (please see Supplement table). Whenever possible, these categorizations aligned with the approach taken by PC 4 . The selected data elements are mandatory to be filled out in the hospitals' software. PAC 3 has a few optional fields in the case report form, but none of them are audited because they are seldom used for analytic purposes and are primarily used by hospitals for local projects.

Scoring process
After a regional or virtual audit, the DCC enters discrepancies and comments into REDCap and then uploads them into SAS, which is where all scoring calculations take place. If a data champion incorrectly answers a question with a "parent-child" relationship and the parent field was incorrect, the child fields are not additionally scored as discrepant to avoid double-counting. If a systematic error in data entry was identified during the audit as might happen with misinterpretation of a data definition, for example, this error is also excluded in scoring calculations in both the numerator and denominator (please see formula below) and the data champion is required to resubmit these cases in coordination with the DCC prior to the completion of the audit. The reasoning behind excluding systematic errors due to a misunderstanding of the data definition from scoring is that they are not representative of a data champion's ability to find the appropriate source of truth in the electronic medical record for a data element. For the purpose of this paper, we performed an additional analysis which rescored audits accounting for each systematic error, as well.
Two calculation metrics are included in the scoring report: the major discrepancy rate and the overall accuracy rate. A hospital must have a major discrepancy rate of 1.5% or less and an overall accuracy rate of 97% or more to pass their audit. The major discrepancy rate is counted using this formula: Major Discrepancy Rate ¼ Number of major discrepancies Number of fields with potential for major discrepancy Â 100 The overall accuracy rate is calculated using this formula: Overall accuracy ¼ Number of fields without any discrepancies Total audited fields Â 100 For fields that can have multiple responses, such as feeding route at hospital admission, if the data champion misses these questions entirely, it is considered a major discrepancy. If the data champion missed these questions partially, for example, the patient was fed orally and via a nasogastric tube and the data champion only captured the nasogastric tube, this is considered a minor discrepancy and is not included in the calculation for the major discrepancy rate.

L. Khadr et al.
After scoring each hospital, a scoring report is made in Microsoft Access that includes the major discrepancy rate, the overall accuracy rate, discrepancies for each encounter, and any additional audit notes for the auditee. Data champions are required to resubmit any cases where discrepancies were found.

Results and data accuracy
After the registry leadership worked to develop this extensive process, PAC 3 completed nine successful audits in 2020: six were regional audits and three occurred in a virtual format. Hospitals 1 and 2 were included in the first regional audit, which occurred February, 2020, while hospitals 3-6 were audited during the second regional audit March, 2020. Immediately following the second regional audit, strict workplace travel restrictions were implemented due to the COVID-19 pandemic. Hospitals 7, 8, and 9 underwent virtual audits in May, July, and October, 2020, respectively.
Across all 9 hospitals, 386 encounters and 27,086 individual data fields were audited. The aggregate overall accuracy rate was 99.27% and the aggregate major discrepancy rate was 0.51%. As seen in Table 1, among the hospitals that were audited in-person at the regional audits, major discrepancy rates ranged from 0.3% to 0.9% and overall discrepancy rates ranged from 98.8% to 99.6%.
The virtual audits had similar scores and discrepancies: hospitals 7, 8, and 9 had major discrepancy rates ranging from 0.3% to 0.6% and their overall accuracy rates ranging from 99.4% to 99.5%. The results of the registry domains by hospital are summarised in Tables 2 and 3. When all errors were considered including systematic errors marking such errors as minor or major discrepancies accordingly, they accounted for 0.05% of the data fields audited and all hospitals met our passing threshold with overall discrepancy rates ranging from 98.7% to 99.5% and their major discrepancy rates ranged from 0.3% to 0.9%.

PAC 3 data validation
The results of this initial series of PAC 3 audits demonstrate that the data fields of the registry are accurate, and the data can be adjudicated using a model that is thorough and sustainable. All nine centres demonstrated low rates of major and minor discrepancies. Additionally, and of great value to the processes and sustainability of PAC 3 , the auditing process can be effectively executed in both a regional manner and a virtual format without sacrificing the standards of previous approaches. 2 Not only has this initial series of audits demonstrated that the PAC 3 registry is a valid data source for the purposes of quality improvement, research, and hospital benchmarking, but we have come to learn that the auditing process itself has the potential to further the culture of collaborative learning.
The overall accuracy of the PAC 3 registry is similar to the accuracy previously reported by PC 4 and STS. 2,6 While distinct entities, there are many similarities across these registries. Shared data elements, many of which exist at the hospital level such as demographics and diagnoses, help ensure consistency across platforms while simultaneously reducing redundant data collection efforts. Similarities among registries are also evident at the local level. Data champions often cross-train on multiple registries which helps foster a local culture of registry partnership. With evidence of its validated accuracy and precision, the PAC 3 registry data can now be incorporated into the Cardiac Networks United environment and provide new and important insights from the acute care cardiology unit perspective such as detailed feeding and nutrition information, particularly in the infant population, and important metrics surrounding ICU transfers and hospital readmissions that do not currently exist in other data registries.
From the perspective of PAC 3 leadership, the aim of the audit is not only to ensure accurate and complete data capture but also to provide specific data extraction feedback to each member hospital, as well as to share generalisable information in an expeditious manner with other member institutions. For this reason, the PAC 3 leadership chose not to use an external vendor for audits because vendors are not familiar with the nuances of the data collection process and cannot provide educational feedback or advice on how to extract data more efficiently from the electronic medical record as auditors who are also members of the registry. We also found that systematic data collection errors where rare but when they occurred offered yet another opportunity for education and feedback. After this initial audit of nine hospitals, PAC 3 leadership examined the most missed data fields (Table 3) to bolster registry education efforts. In turn, local teams at hospitals have used this feedback to work with their clinical staff and data warehouses to improve charting of feeds and therapy documentation.

Benefits of regional and virtual audits
At the outset of the PAC 3 audit process, a regional audit framework was developed to allow for a timely audit process across a rapidly expanding membership. Regional audits offered two distinct benefits: economy of scale and more immediate access to collaborative learning. A regional audit of four centres, such as the one in March, 2020, provided an opportunity for networking with fellow registry members and dissemination of best practices as they were being discovered in real time. Pairing auditors and auditees that shared the same electronic medical record brand allowed data teams to share and reflect on data collection methods and strategies.
Typically, each hospital develops a local "source of truth" within the electronic medical record for each data element based on local documentation practices. Sharing these "sources of truth" among data teams was a commonly cited strength of the regional audit framework. Prior to the first regional audit, there was concern that this format would result in the data champions receiving less individualised attention than that given during a one-on-one audit at the hospital being audited. However, because the structure of the regional format in which a data champion works with two other people in the trio model, there was no perceived lack of individualised feedback reported. Additionally, when centres were asked to reflect on the debrief sessions, during which all regional centres shared learnings in a communal fashion, participants appreciated the transparent approaches and the immediacy of the learnings.
As a result of the COVID-19 pandemic and the workplace travel restrictions across hospitals within the United States and Canada, PAC 3 was forced to consider a shift to a virtual audit model. While virtual audits do not offer the same immediacy in terms of networking opportunities and collaborative learning, there was obviously even greater potential economic benefit as travel expenditures were eliminated. To foster a personable experience during a virtual audit, audit trios turn on their video cameras and have debriefing sessions at the end of each day like the in-person regional audits to discuss any learnings between the audited centre and auditors. Although the group participating in the virtual audit is smaller than the group participating in a regional audit, virtual auditees continue to receive constructive feedback from the auditors. Given the positive feedback the PAC 3 leadership has received from audited hospitals about the virtual audits in terms of personalised attention and learning opportunities while being more costeffective than the in-person regional audits, the registry is considering continuing to do virtual audits even after COVID-19 travel restrictions are lifted.

Components of data collection for audit success
At the core of any registry's data validity is the data team, who sit closest to the source data, particularly in the case of PAC 3 , the data champion. To ensure a member hospital's audit success, the data team must develop an efficient workflow that ensures accurate data collection without overburdening the data champion. Much has been learned from the tier 3 portion of the auditing process regarding the importance of the collaboration between the data champion and the clinical champion. The need for open communication, either in-person or digital, between these parties has proven paramount to developing a reliable and valid data collection process.
In summary, the PAC 3 registry is highly accurate and the fourtiered auditing system has performed consistently across an inperson regional format and the more recently necessary virtual audit format. The results of the auditing process have demonstrated the feasibility of accurate PAC 3 data collection, and as a result, the registry meets its aims to be a highly accurate and validated data source for use in quality improvement, hospital benchmarking, and research.