The Pediatric Acute Care Cardiology Collaborative (PAC3) was established in 2014 motivated by a mission to improve acute care cardiology outcomes and experience from the perspective of the patient, family, clinical team, and hospital system. The PAC3 registry began collecting data of February, 2019 with 6 hospitals contributing data and has since expanded to 27 hospitals. Central to the success of PAC3’s mission is the conviction that improvement will be best measured by metrics derived from a validated, audited, and prospective data registry. Reference Kipps, Cassidy and Strohacker1 Inspired by the high data fidelity standards previously established and verified by the Pediatric Cardiac Critical Care Consortium (PC4), and the opportunity to create shared database elements for patient populations across hospitalisations, as well as the entire lifespan of individual patients, the PAC3 Database Committee created an audit process for data validation. Reference Gaies, Donohue and Willis2,Reference Gaies, Anderson and Kipps3
Importantly, while PC4 and PAC3 share several data elements and collection methods, they are independent organisations and registries. PAC3 has partnered with PC4 throughout its development and has benefitted from their successes. This partnership has facilitated such rapid incorporation of new member hospitals that new methods of data verification are required, as is the development of novel audit processes. Furthermore, the onset of the COVID-19 pandemic in March, 2020 required substantial adjustments to the previously designed in-person audit procedures. Despite these challenges, and maybe because of them, the PAC3 audit process has proven itself both feasible and effective. PAC3 completed nine successful audits between February, 2020 and October, 2020: six were regional audits and three occurred in a virtual format.
The PAC3 audit process is a four-tiered process. Tiers 1–3 involve partnership between the local hospital’s data team and the PAC3 Data Coordinating Center (DCC). Tier 4 involves data element verification and requires the local hospital’s data team, the DCC, and PAC3 auditors. We present a novel registry audit process incorporating both regional and virtual models that could potentially be used by other paediatric registries. We report the results from the first nine audits providing evidence to support the broad use of the registry for benchmarking, quality improvement, and research.
Materials and methods
Summary of registry data elements
The PAC3 registry is built as a hospital encounter-based system recording unique data elements during the acute care cardiology unit encounter. These data are then combined with select shared PC4 data elements at the hospital level from admission through discharge. PAC3 encounters, as designated by the encounter start date and time, begin when a patient’s care is transferred to a non-ICU cardiology service as the primary service. The PAC3 encounter ends, as designated by the encounter-level end date and time, when the patient is transferred or discharged from the cardiology service. Additional data fields include demographic and clinical variables, supportive therapies (including respiratory, pharmacologic, devices), feeding and nutrition variables, as well as complications that were actively treated during the acute care cardiology unit encounter. Out of 320 PAC3 data fields, 295 are mandatory and are required for case submission.
In addition to data fields specific to the acute care cardiology unit-level encounter, shared hospital-based data fields are entered on a single-data platform with the ability to share hospital-level data elements across other data registries such as PC4, STS-CHSD (Society of Thoracic Surgery Congenital Heart Surgery Database), and the American College of Cardiology IMPACT (IMproving Pediatric and Adult Congenital Treatments) registry. Reference Martin, Beekman and Ing4,Reference Gaies, Cooper and Tabbutt5 This allows for ease of data entry and data consistency in critical data elements across these registries such as a patient’s fundamental cardiac diagnosis.
Hospitalizations are classified as either medical or surgical based on whether the patient underwent a cardiothoracic surgical procedure during their hospital stay. Patients solely admitted to the acute care cardiology unit for routine observation following catheterisation or an electrophysiology study and discharged the following day may be collected for local hospital use but are not included in PAC3 for submission. Patients physically located on the acute care cardiology unit but with a non-cardiology primary service are not included in the registry.
Data collection team
Data acquisition from the hospital electronic medical record system requires a dedicated data collection team comprised of a clinical champion, typically a cardiologist or an acute care cardiology unit advanced practice provider, as well as data champion(s) who most commonly have a clinical nursing background, but have also come from alternative professional backgrounds such as data analytics and research management. This team is responsible for timely and accurate case submission and for ensuring reliable and consistent data collection processes. A hospital’s first audit typically occurs 1 year to 15 months after initiation of data submission. Longer-term follow-up audits are intended to occur in 2–3 year cycles after successfully completing the first audit. We anticipate second-round audits will begin in 2022.
Audit process
The PAC3 audit process includes four stepwise tiers (Fig 1). The first three tiers are screening points, while the final tier is the audit itself. Tier 4 was developed as in-person regional audits with capacity for multiple centres but was shifted more recently to a virtual format utilising HIPAA-compliant videoconferencing technology due to the COVID-19 pandemic.
PAC3 audit process.
The first tier determines the hospital’s PC4 audit history. Since several areas of the PAC3 registry are shared with PC4 (demographics and many hospital-level variables), and because of the intentional overlap of the auditing approach, a PC4 member centre must successfully pass its PC4 audit to be eligible for PAC3 auditing. The additional aim of this tier is to eliminate duplicated auditing efforts. For the few hospitals that do not also participate in PC4, as this is not a requirement for PAC3 membership, the initial PAC3 audit will consist of a separate design requiring an in-person individual centre audit in place of the regional or virtual formats described below.
The second tier of the auditing process is a census check to ensure member hospitals have a robust process for capturing all eligible surgical and medical patients. Hospitals send a record of all surgical cases to the DCC sourced from their local STS data. These records are cross-checked with submitted PAC3 cases by the DCC. Hospitals then send an admission, discharge, and transfer feed to the DCC which is typically generated by a hospital’s billing department or data warehousing department and reports all patient movement in or out of a particular hospital unit or service line. A sample of the admission, discharge, and transfer feed (∼10%) is cross-checked with submitted PAC3 cases to ensure eligible medical cases have been entered. Typically, 300–1000 cases are submitted for verification during the census check depending on a hospital’s volume.
The DCC reviews any discrepancies with the hospital’s data team during a 2-hour phone call and reviews the hospital’s process for ensuring all PAC3-eligible patients are captured. If a hospital is found to be under capturing patients by more than three cases, they cannot pass their audit until all eligible patients have been entered and a hospital’s process for capturing patients has been improved. This benchmark ensures a high standard of patient capture for all hospitals regardless of volume.
The third tier of the audit is the self-audit and is therefore performed by individual data collection teams on a quarterly basis. Member hospitals can determine their own self-auditing process but are encouraged to participate in quality improvement efforts supported by PAC3 leadership to both standardise this process and to improve data quality and timeliness. This step may include regularly scheduled meetings to review data collection questions and to simulate audits of submitted cases. The member hospital cannot complete their tier 4 audit unless they have developed an internal auditing process consistent with the aims of tier 3. The results of this internal audit have no bearing on the scoring of the tier 4 audit. The purpose of the internal audit is to ensure that hospitals are revisiting their data on an ongoing basis to reinforce their understanding of data definitions and allow for real-time cleaning of their own data to ensure accuracy and reliability of data entry.
The fourth tier of the audit process verifies individual data elements in the electronic medical record. This was originally designed as an in-person regional audit of 2 or more centres and has more recently been adapted to suit a virtual videoconference format. All programming for audit preparation is done using SAS 9.4 software. The DCC selects 60 hospital encounters representing a broad scope of patient complexity. Standardized forms are developed for both the auditing process itself and for tracking discrepancies. The audit team consists of a data champion from a hospital being audited (the auditee), a clinical champion from another hospital who serves as the auditor, and a neutral third-party observer from a separate hospital (the third-party observer can be a data or clinical champion or a member of the PAC3 leadership team). The audit trios are required to complete at least 40 of the 60 cases over the course of 2 days. This allows the auditee to benefit from the collective data collection expertise of those with in-depth knowledge of the registry and whose data collection processes have been previously vetted. Despite the virtual format, there remains a focus on collaborative learning as provided by the in-person regional audits.
For both regional and virtual audit formats, the data champion from the hospital undergoing the audit displays their electronic medical record screen to the auditor and observer. The auditor uses the audit report and verifies each data element with the auditee by examining the electronic medical record as it is navigated by the auditee in real time. The third-party observer is responsible for resolving any disagreements between auditor and auditee and for tracking discrepancies which are then sent back to the DCC for scoring. Audit trios are arranged, when possible, by shared electronic medical record experience.
To accommodate virtual audits, a hospital’s data team obtains permission from their hospital to use a HIPAA-compliant video-sharing service to share their screen with the auditor and observer. In part due to the pandemic, hospital policies have become increasingly flexible. Since the auditor and observer are not entering the electronic medical record directly, but rather simply observing, there has been unique opportunity to reimagine the audit process.
Importantly, at the conclusion of each day, time is spent sharing lessons learned from the audit experience. Examples include clarifying specific data definitions, discussing more efficient ways to extract data from the electronic medical record, and sharing data capture of unusual patient types.
Audited fields
The PAC3 audit subcommittee reviewed the entire registry composed of 320 individual data elements and selected the audit variables that met at least one of two thematic criteria: select variables that are particularly challenging to collect accurately given the absence of a discrete data field in the electronic medical record and select variables that are predicted to be of critical importance to clinical improvement efforts and research. Cumulatively, the audit includes 80 fields for patients less than 1 year of age and 72 fields for all others representing roughly 25% of available data elements in the registry. These audited data elements were categorised into major and minor categories based on relative significance (please see Supplement table). Whenever possible, these categorizations aligned with the approach taken by PC4. The selected data elements are mandatory to be filled out in the hospitals’ software. PAC3 has a few optional fields in the case report form, but none of them are audited because they are seldom used for analytic purposes and are primarily used by hospitals for local projects.
Scoring process
After a regional or virtual audit, the DCC enters discrepancies and comments into REDCap and then uploads them into SAS, which is where all scoring calculations take place. If a data champion incorrectly answers a question with a “parent–child” relationship and the parent field was incorrect, the child fields are not additionally scored as discrepant to avoid double-counting. If a systematic error in data entry was identified during the audit as might happen with misinterpretation of a data definition, for example, this error is also excluded in scoring calculations in both the numerator and denominator (please see formula below) and the data champion is required to resubmit these cases in coordination with the DCC prior to the completion of the audit. The reasoning behind excluding systematic errors due to a misunderstanding of the data definition from scoring is that they are not representative of a data champion’s ability to find the appropriate source of truth in the electronic medical record for a data element. For the purpose of this paper, we performed an additional analysis which rescored audits accounting for each systematic error, as well.
Two calculation metrics are included in the scoring report: the major discrepancy rate and the overall accuracy rate. A hospital must have a major discrepancy rate of 1.5% or less and an overall accuracy rate of 97% or more to pass their audit. The major discrepancy rate is counted using this formula:
$$Major{\mkern 1mu} Discrepancy{\mkern 1mu} Rate = {{Number{\mkern 1mu} of{\mkern 1mu} major{\mkern 1mu} discrepancies} \over {Number{\mkern 1mu} of{\mkern 1mu} fields{\mkern 1mu} with{\mkern 1mu} potential{\mkern 1mu} for{\mkern 1mu} major{\mkern 1mu} discrepancy}} \times 100$$
The overall accuracy rate is calculated using this formula:
$$Overall\,{\mkern 1mu} accuracy = {{Number\,{\mkern 1mu} of{\mkern 1mu} fields{\mkern 1mu} \,without\,{\mkern 1mu} any{\mkern 1mu} \,discrepancies} \over {Total\,{\mkern 1mu} audited{\mkern 1mu} fields}} \times 100$$
For fields that can have multiple responses, such as feeding route at hospital admission, if the data champion misses these questions entirely, it is considered a major discrepancy. If the data champion missed these questions partially, for example, the patient was fed orally and via a nasogastric tube and the data champion only captured the nasogastric tube, this is considered a minor discrepancy and is not included in the calculation for the major discrepancy rate.
After scoring each hospital, a scoring report is made in Microsoft Access that includes the major discrepancy rate, the overall accuracy rate, discrepancies for each encounter, and any additional audit notes for the auditee. Data champions are required to resubmit any cases where discrepancies were found.
Results and data accuracy
After the registry leadership worked to develop this extensive process, PAC3 completed nine successful audits in 2020: six were regional audits and three occurred in a virtual format. Hospitals 1 and 2 were included in the first regional audit, which occurred February, 2020, while hospitals 3–6 were audited during the second regional audit March, 2020. Immediately following the second regional audit, strict workplace travel restrictions were implemented due to the COVID-19 pandemic. Hospitals 7, 8, and 9 underwent virtual audits in May, July, and October, 2020, respectively.
Across all 9 hospitals, 386 encounters and 27,086 individual data fields were audited. The aggregate overall accuracy rate was 99.27% and the aggregate major discrepancy rate was 0.51%. As seen in Table 1, among the hospitals that were audited in-person at the regional audits, major discrepancy rates ranged from 0.3% to 0.9% and overall discrepancy rates ranged from 98.8% to 99.6%. The virtual audits had similar scores and discrepancies: hospitals 7, 8, and 9 had major discrepancy rates ranging from 0.3% to 0.6% and their overall accuracy rates ranging from 99.4% to 99.5%. The results of the registry domains by hospital are summarised in Tables 2 and 3. When all errors were considered including systematic errors marking such errors as minor or major discrepancies accordingly, they accounted for 0.05% of the data fields audited and all hospitals met our passing threshold with overall discrepancy rates ranging from 98.7% to 99.5% and their major discrepancy rates ranged from 0.3% to 0.9%.
Audit results
* Virtual audit.
Audit results by domain
* Virtual audit.
Most missed fields from audit
Discussion
PAC3 data validation
The results of this initial series of PAC3 audits demonstrate that the data fields of the registry are accurate, and the data can be adjudicated using a model that is thorough and sustainable. All nine centres demonstrated low rates of major and minor discrepancies. Additionally, and of great value to the processes and sustainability of PAC3, the auditing process can be effectively executed in both a regional manner and a virtual format without sacrificing the standards of previous approaches. Reference Gaies, Donohue and Willis2 Not only has this initial series of audits demonstrated that the PAC3 registry is a valid data source for the purposes of quality improvement, research, and hospital benchmarking, but we have come to learn that the auditing process itself has the potential to further the culture of collaborative learning.
The overall accuracy of the PAC3 registry is similar to the accuracy previously reported by PC4 and STS. Reference Gaies, Donohue and Willis2,Reference Overman, Jacobs and O’Brien6 While distinct entities, there are many similarities across these registries. Shared data elements, many of which exist at the hospital level such as demographics and diagnoses, help ensure consistency across platforms while simultaneously reducing redundant data collection efforts. Similarities among registries are also evident at the local level. Data champions often cross-train on multiple registries which helps foster a local culture of registry partnership. With evidence of its validated accuracy and precision, the PAC3 registry data can now be incorporated into the Cardiac Networks United environment and provide new and important insights from the acute care cardiology unit perspective such as detailed feeding and nutrition information, particularly in the infant population, and important metrics surrounding ICU transfers and hospital readmissions that do not currently exist in other data registries.
From the perspective of PAC3 leadership, the aim of the audit is not only to ensure accurate and complete data capture but also to provide specific data extraction feedback to each member hospital, as well as to share generalisable information in an expeditious manner with other member institutions. For this reason, the PAC3 leadership chose not to use an external vendor for audits because vendors are not familiar with the nuances of the data collection process and cannot provide educational feedback or advice on how to extract data more efficiently from the electronic medical record as auditors who are also members of the registry. We also found that systematic data collection errors where rare but when they occurred offered yet another opportunity for education and feedback. After this initial audit of nine hospitals, PAC3 leadership examined the most missed data fields (Table 3) to bolster registry education efforts. In turn, local teams at hospitals have used this feedback to work with their clinical staff and data warehouses to improve charting of feeds and therapy documentation.
Benefits of regional and virtual audits
At the outset of the PAC3 audit process, a regional audit framework was developed to allow for a timely audit process across a rapidly expanding membership. Regional audits offered two distinct benefits: economy of scale and more immediate access to collaborative learning. A regional audit of four centres, such as the one in March, 2020, provided an opportunity for networking with fellow registry members and dissemination of best practices as they were being discovered in real time. Pairing auditors and auditees that shared the same electronic medical record brand allowed data teams to share and reflect on data collection methods and strategies. Typically, each hospital develops a local “source of truth” within the electronic medical record for each data element based on local documentation practices. Sharing these “sources of truth” among data teams was a commonly cited strength of the regional audit framework. Prior to the first regional audit, there was concern that this format would result in the data champions receiving less individualised attention than that given during a one-on-one audit at the hospital being audited. However, because the structure of the regional format in which a data champion works with two other people in the trio model, there was no perceived lack of individualised feedback reported. Additionally, when centres were asked to reflect on the debrief sessions, during which all regional centres shared learnings in a communal fashion, participants appreciated the transparent approaches and the immediacy of the learnings.
As a result of the COVID-19 pandemic and the workplace travel restrictions across hospitals within the United States and Canada, PAC3 was forced to consider a shift to a virtual audit model. While virtual audits do not offer the same immediacy in terms of networking opportunities and collaborative learning, there was obviously even greater potential economic benefit as travel expenditures were eliminated. To foster a personable experience during a virtual audit, audit trios turn on their video cameras and have debriefing sessions at the end of each day like the in-person regional audits to discuss any learnings between the audited centre and auditors. Although the group participating in the virtual audit is smaller than the group participating in a regional audit, virtual auditees continue to receive constructive feedback from the auditors. Given the positive feedback the PAC3 leadership has received from audited hospitals about the virtual audits in terms of personalised attention and learning opportunities while being more cost-effective than the in-person regional audits, the registry is considering continuing to do virtual audits even after COVID-19 travel restrictions are lifted.
Components of data collection for audit success
At the core of any registry’s data validity is the data team, who sit closest to the source data, particularly in the case of PAC3, the data champion. To ensure a member hospital’s audit success, the data team must develop an efficient workflow that ensures accurate data collection without overburdening the data champion. Much has been learned from the tier 3 portion of the auditing process regarding the importance of the collaboration between the data champion and the clinical champion. The need for open communication, either in-person or digital, between these parties has proven paramount to developing a reliable and valid data collection process.
In summary, the PAC3 registry is highly accurate and the four-tiered auditing system has performed consistently across an in-person regional format and the more recently necessary virtual audit format. The results of the auditing process have demonstrated the feasibility of accurate PAC3 data collection, and as a result, the registry meets its aims to be a highly accurate and validated data source for use in quality improvement, hospital benchmarking, and research.
Supplementary material
For supplementary material accompanying this paper visit https://doi.org/10.1017/S1047951121005047
Acknowledgements
The authors recognise the immense effort and dedication of the hospital data collection teams without which the PAC3 registry would not exist. We would also like to acknowledge the Pediatric Cardiac Critical Care Consortium (PC4) for their pioneering work, leadership, and partnership.
Financial support
The Heart Institute at Cincinnati Children’s Hospital and Cardiac Networks United.
Conflicts of interest
None.
Open access
