INTRODUCTION
There are 1·4 million people living in about 16 000 nursing homes in the United States [1]. These residents are typically at an increased risk of acquiring influenza and other illnesses. They are older, typically have multiple chronic conditions, and live in an institutional environment where they are in close contact with other residents and staff [Reference Nichol2, Reference Menec, MacWilliam and Aoki3]. Influenza vaccine is safe and effective, but many nursing-home residents do not receive the vaccine. Only 83% of nursing-home residents were vaccinated in 2009 [Reference Cai, Feng, Fennell and Mor4].
The Center for Medicare and Medicaid Services (CMS) is a federal agency that administers the Medicare and Medicaid programmes. It regulates healthcare facilities that provide services to patients enrolled in these programmes. In 2005, CMS issued a regulation requiring long-term-care facilities to offer influenza vaccinations to all residents and document their vaccination status [5, 6]. According to the regulation, every resident should be vaccinated unless it is contraindicated, the resident or legal party declines the vaccine, or there is a vaccine shortage [5, 6]. CMS projected that an increase in the resident vaccination rate from 74% (CMS's estimate of the vaccination rate at the time of the analysis) to 90% would avert over 5000 hospital admissions and over 2000 deaths [5]. CMS requires nursing homes to report residents' influenza vaccination status on the Resident Assessment Instrument (RAI). The RAI is a standardized, primary screening and assessment tool for all residents in Medicare- or Medicaid-certified long-term-care facilities.
The US Centers for Disease Control and Prevention uses RAI data to measure vaccination rates among nursing-home residents. CMS uses RAI data to measure resident vaccination rates for each nursing home. CMS reports resident vaccination rates for each nursing home publicly on the Nursing Home Compare website [7]. CMS incorporates nursing-home-level resident vaccination rates into a five-star quality ranking for each nursing home, which is also publicly reported on the Nursing Home Compare website.
The accuracy of influenza vaccination rate measures calculated using RAI data is unknown. If rates are measured with error, policy-makers may draw incorrect inferences about the impact of programmes to increase the vaccination rate, and nursing-home quality ratings will provide an inaccurate signal of true quality. The purpose of this study was to evaluate the validity of influenza vaccination status as recorded on the RAI through reviews of nursing-home RAIs and medical charts (the gold standard). Using medical chart abstraction, we assessed how nursing homes document the vaccination status of their residents in three study states (Florida, Georgia, Wisconsin) and determined whether or not there was agreement between the two documentation sources.
METHODS
Sample selection
This study, which was part of a larger project to assess the policies and practices related to influenza vaccination in nursing homes, examined the documentation of resident influenza vaccination status in 39 nursing homes in Florida, Georgia, and Wisconsin.
We performed power calculations to determine a target sample size. Our goal was to recruit 15 nursing homes per state based on an estimate of the number of charts and RAI records we would need to review so that (1) there was an 80% chance that the confidence intervals for our sensitivity and specificity estimates would include the true values and (2) the point estimates for sensitivity and specificity would be within five percentage points of the true value.
The nursing-home associations in each state sent emails to their members describing the study and asked them to contact us if they were interested in participating. All of the facilities in our study contacted us. There are 1431 total facilities in these states [1], so our sample represents slightly less than 3% of total. These three states were selected due to their variability in immunization rates of nursing-home residents by race, state laws regarding standing order programmes, and geographical diversity. In each state, we conducted site visits with nursing homes to assess their resident and staff vaccination policies and procedures. We recruited nursing homes that had a resident population that fell into one of three categories: (1) majority white (>90% white); (2) majority black (>50% black); and (3) racially mixed (50% white/5–50% black). Final recruitment ranged from 12 to 15 nursing homes per state, with a mix of for-profit and non-profit nursing homes. Site visits were conducted from June 2011 to August 2012. The Institutional Review Board at Emory University approved this study.
Data collection
During each site visit, we sampled 20% of nursing-home residents' medical charts and matched them to residents' RAI records. Most facilities gave us a list of residents. We selected every fifth name from the list. In the handful of facilities where we did not have a list, we selected charts sequentially from the shelves in nursing stations. In some cases nursing-home staff brought carts with charts on them into a conference room for us to review. Residents whose charts were being updated or reviewed by nursing facility staff may have been excluded from our sample. In some cases our sample fell below or exceeded the 20% threshold based on the time we had available to perform the reviews. Residents who had entered the facility within 4 weeks of our visit were not included in our sample because, according to the facilities, it takes about a month before all of a resident's information is fully entered into the RAI.
We obtained paper medical charts from nursing stations in the facility. From the medical chart, we recorded resident's year of birth, race, gender, original admission date, and the most recent admission date. Both the original admission date and most recent admission date were recorded in case the resident was sent to hospital or returned home at any point during the influenza season. We recorded whether the resident received influenza vaccine in the facility during the current or most recent influenza season, either 1 October 2010 to 31 March 2011 or 1 October 2011 to 31 March 2012 depending on when the site visit took place. If a resident did not receive the vaccine, we recorded the reason. Reasons for declination included: (1) not in facility during influenza season, (2) received vaccine outside facility, (3) not eligible – medical contraindications, (4) offered and declined, (5) not offered, (6) inability to obtain vaccine, and (7) none of the above.
We collected information on residents' vaccination status from RAI records. In most facilities RAI records were stored electronically. A staff member, usually a nurse, is assigned to enter information from patients' charts into the RAI quarterly. Some facilities maintain paper-based copies of the RAI. These are included in patients' charts.
Facilities that stored RAIs electronically gave us access to these records through a desktop computer at the facility. We collected information on residents' vaccination status, whether the resident needed an interpreter, and whether the resident had any cognitive impairments, which could influence his or her ability to accept or decline the vaccine. We considered a resident to have received influenza vaccine during the most recent influenza season if he/she received the vaccine in or out of the facility. Nursing-home staff are supposed to determine if residents who were hospitalized received influenza vaccine during their hospital stay and indicate that the resident received influenza vaccine in the medical chart and RAI. We were unable to review residents' hospital charts or outpatient records, and so we did not directly measure the proportion of residents who were vaccinated outside the facility.
In some cases were unable to determine residents' vaccination status because the vaccination information page in the medical chart was missing, the writing was illegible, or vaccination status was not recorded in the chart or the RAI even though the resident was in the facility during the current or most recent influenza season. In these instances we initially classified vaccination status as ‘unknown’. For the purposes of this analysis, we grouped ‘not vaccinated’ and ‘unknown’ together.
Analysis
The primary independent variable of interest is the agreement, between the medical chart and the RAI. Medical charts and RAIs were considered be in agreement if both indicated that the resident received the influenza vaccine, regardless of whether the resident was vaccinated in or out of the facility, or if both indicated the resident was not vaccinated or vaccination status was unknown. We used a z test for proportions to assess the significance of differences in agreement rates between groups. We measured the sensitivity and specificity of RAIs relative to medical records. We considered medical records to be the ‘gold standard’ for vaccination status documentation. In most nursing homes, a nurse is assigned to the task of updating residents' RAIs quarterly. The nurse obtains information on vaccines received from medical charts or a vaccination log.
The distribution of observations across states in our sample is different from the distribution of nursing facility residents across states. For example, observations from Florida account for 29% of our sample, but nursing-home residents in Florida are 55% of the total across Florida, Georgia, and Wisconsin (129 000 residents) [8]. We calculated weighted sensitivity and specificity estimates where the weights reflect the share of nursing-home residents in each state relative to the total across the three states.
We used z tests for proportions to compare sensitivity and specificity to 0·50, a standard benchmark for assessing the accuracy of diagnostic tests. A random guess as to whether a resident was vaccinated or not would be right 50% of the time. We estimated the association between resident and facility characteristics and agreement between the medical record and RAI, a dichotomous variable, using a logistic regression with random effects (also known as a mixed-effects model) for facility. We estimated the model in Stata v. 11·0 (Stata Corporation, USA). The regression included the following resident characteristics: age, gender, race (white vs. other), dementia (as indicated by the RAI), and whether the resident requires an interpreter (also as indicated by the RAI). Facility characteristics included bed size, the percent of residents who are white, the overall quality rating (measured on a scale of 1–5), and location (Florida, Georgia, Wisconsin). All facility characteristics were obtained from the CMS Nursing Home Compare website [7]. We also report the association between these factors and residents' receipt of influenza vaccine as indicated by the medical records.
RESULTS
We sampled medical records for 961 unique residents. Records for 121 persons were excluded from the sample because they were not residents of the facility during the most recent influenza season. Our final sample included 840 medical records matched with RAIs. There were 271 records from Florida (12 nursing homes), 345 from Georgia (15 nursing homes), and 224 from Wisconsin (12 nursing homes). According to medical charts, 643/840 (77%) residents were vaccinated. According to RAI data, 628 (75%) residents were vaccinated.
Table 1 shows results from the medical chart and RAI audit. When in-facility and out-of-facility vaccination and ‘no vaccine’ and ‘unknown vaccination status’ were considered as distinct categories, the agreement rate was 78%. When we collapsed these categories, as shown in Table 2, the agreement rate increased to 86%. When we weighted the sample to account for differences in the distribution of observations across states and the distribution of nursing facility residents nationally, the agreement rate was 83%.
Agreement between the medical record and Resident Assessment Instrument
Agreement between the medical record and Resident Assessment Instrument by subgroup
Vaccinated, Vaccinated in or outside the facility; not vaccinated, not vaccinated or status unknown.
Agreement rates were 77% in Florida, 88% in Georgia, and 92% in Wisconsin. Agreement rates were significantly different between Florida and Georgia (P < 0·001) and Florida and Wisconsin (P < 0·001) but not between Georgia and Wisconsin (P = 0·17).
The sensitivity of RAIs for measuring receipt of influenza vaccination was 90% [95% confidence interval (CI) 0·87–0·92], which was significantly different from 0·50 (P < 0·001). Eighty-nine percent of residents who, according to medical charts, were vaccinated (either in or outside of the facility) were correctly identified as having been vaccinated in the RAI. When we weighted the sample to account for differences in the distribution of observations across states and the distribution of nursing facility residents nationally, the sensitivity was 85% (95% CI 0·83–0·88).
The specificity of RAIs was 74% (95% CI 0·67–0·80), which was also significantly different from 0·50 (P < 0·001). Seventy-three percent of residents who, according to the medical record, were not vaccinated or whose vaccination status was unknown were correctly identified as such in the RAI. When we weighted the sample to account for differences in the distribution of observations across states and the distribution of nursing facility residents nationally, the specificity was 77% (95% CI 0·71–0·83).
Table 2 displays results separately for whites and non-whites. There were 523 whites in the sample, of whom 410 (78%) received the influenza vaccine according to the medical record. There were 317 non-whites of whom 233 (73%) received influenza vaccine. The difference in agreement rates was not statistically significant (P = 0·21).
Figure 1 displays facility-level agreement rates in vaccination documentation between the medical record and the RAI. Bars indicate 95% confidence intervals. We truncated the upper bound of the interval at 1·00 because agreement rates cannot exceed 1·00. Four facilities had perfect agreement rates. Two facilities had agreement rates lower than 60%.
Facility-level medical record–Resident Assessment Instrument agreement rates. Error bars represent 95% confidence intervals. (Source: Authors' assessment of records in 39 nursing homes.)
Table 3 displays variable means and results from the logistic regressions, expressed in odds ratios. The odds ratios for male sex and resident age were positive and statistically significant. Contrary to expectations, we did not find that agreement rates were lower for residents with dementia or residents who need an interpreter. The odds ratios associated with both of these resident characteristics were not significantly significant. The odds ratio for for-profit facilities was significantly lower than 1, but the odds ratio associated with facilities' quality rating was not significantly different from 1. The odds ratios for Georgia and Wisconsin were greater than 1 and statistically significant.
Sample characteristics and estimates of the impact of patient and facility characteristics on Resident Assessment Instrument–medical record agreement and receipt of influenza vaccination
OR, Odds ratio; CI, confidence interval; N, Number in the sample; %, percent in the sample; S.D., standard deviation.
The final columns in Table 3 display the association between resident and facility characteristics and receipt of influenza vaccine as measured from medical charts. The odds ratios for male sex and dementia were greater than 1 and statistically significant. The odds ratio for for-profit facilities was significant and smaller than 1, while the odds ratios for Georgia and Wisconsin were statistically greater than 1.
DISCUSSION
We found that the sensitivity and specificity of the RAIs for measuring receipt of influenza vaccine were 85% and 77%. There was a high degree of variation between facilities and states. Resident vaccination rates measured using RAIs were only 1·8 percentage points lower than rates measured from medical charts (74·8 vs. 76·6).
Monitoring vaccination rates in the population is important for gauging the impact of programmes and policies to promote adherence to vaccination recommendations. Our results indicate that vaccination rate measures calculated using RAIs can serve as reliable indicators of the vaccination status of nursing-home residents nationally. Policy-makers can draw inferences about trends in vaccination rates and the impact of programmes and policies to increase vaccination rates and reduce disparities using RAI-based measures. However, facility-based measures constructed from RAIs should be interpreted with caution.
Variation between facilities in the accuracy of vaccination coding may result from differences in coding practices. The 2005 CMS regulation requiring nursing homes to offer and document resident influenza vaccination did not specify how nursing-home administrators should accomplish these tasks. Nursing homes have great flexibility to develop their own documentation procedures, which also allows for discrepancies to emerge between documents. For instance, instead of having a strict protocol to follow, some of the nursing homes we visited indicated that clinical staff maintain internal vaccination ‘logs’. Data from these logs are transferred to the medical chart. In some facilities vaccinations and the documentation of receipt of vaccinations are conducted by one or two staff members. In others, many different staff members engage in these activities. For instance, nurses enter the information into the log and then another nurse may transfer the information from the log into the individual medical chart. From there, a coordinator usually enters the vaccination information into the RAI. As the number of individuals involved in the process increases, the likelihood of transcription errors in the records may also increase.
Errors may also arise from differences in the way in which out-of-facility vaccinations are recorded. In some cases medical charts simply ask staff to record whether or not the resident received influenza vaccine. Staff may indicate whether vaccine was received in or outside of the facility. However, the RAI asks ‘did the resident receive the influenza vaccine for this influenza season in this facility [emphasis added]?’ A separate field enables nursing-home staff to record if the resident received the vaccine outside the facility. Differences in wording may contribute to disagreement between charts and RAIs.
We did not find differences in documentation between white and black residents, nor between facilities with high or low proportions of black residents. This finding suggests racial differences previously documented are, in fact, real and therefore may not be due to errors in documentation [Reference Bardenheier9]. Cai et al. report that there is a six percentage point gap in influenza vaccination rates between white and black nursing-home residents [Reference Cai, Feng, Fennell and Mor4]. Vaccination rates were similar between white and non-white residents in our sample.
We found that the accuracy of RAIs was lower in Florida compared to Georgia or Wisconsin. This disagreement may be due to the cultural differences between Florida, Georgia, and Wisconsin. Nursing-home administrators in Florida told us that cultural differences and mistrust of vaccines prevents their immigrant residents and staff from accepting vaccinations. In 2010, immigrants made up 19·4% of Florida's population compared to only 9·7% and 4·5% for Georgia and Wisconsin, respectively [Reference Camarota10]. While examining North Carolina, Martinez found that most immigrants are unfamiliar with the United States healthcare system and the healthcare practitioners are often unfamiliar with the traditions of their patients [Reference Martinez11]. This unfamiliarity and cultural tension also may make it difficult to ascertain the resident's vaccination history and accurately document their status.
When examining how resident and facility characteristics influence the agreement between medical charts and RAIs, we found that older residents and male residents are more likely to have accurate documentation than either younger residents or female residents, respectively. We also found that non-profit nursing homes were more likely to have accurate documentation than for-profit long-term-care facilities and that residents in for-profit facilities were less likely to receive the influenza vaccine. These results are consistent with the findings from a study that reported that for-profit nursing homes have lower staffing levels than non-profit facilities and are more likely to be cited for deficiencies that could result in harm to residents [Reference Harrington12].
The facilities that participated in our study may have better processes for administering and documenting influenza vaccine compared to non-participating facilities, and results should be interpreted with this limitation in mind. Moreover, due to practical difficulties, we did not draw a random sample of nursing-home residents. The likelihood that a nursing-home resident was sampled differs between states and, within states, facilities based on the racial composition of residents. Sensitivity and specificity estimates based on a truly random sample may differ from ours.
In this study we treated medical records as a gold standard. Based on our discussions with nursing-home staff, we believe that vaccinations are accurately recorded in medical records. However, vaccination status could not be accurately determined for 10% of medical charts reviewed because the vaccination documentation page was missing, the writing was illegible, or vaccination status was not recorded. Electronic medical records may improve the ability of facilities to accurately document patient care and transfer information between charts and RAIs, but all the facilities we visited used paper charts. None had plans to switch to electronic records.
Data from RAIs provide a reasonable approximation to vaccination rates in nursing-home residents. However, there is some disagreement between RAIs and medical records, and RAI-based vaccination records may slightly understate true vaccination rates. Reliance on paper charts and vaccination logs, the involvement of multiple staff in the vaccination administration and documentation process, and the frequency with which residents are hospitalized and vaccinated in the hospital, may contribute to coding errors.
ACKNOWLEDGEMENTS
This study was supported by Centers for Disease Control and Prevention cooperative agreement 1U01IP000411-01. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
DECLARATION OF INTEREST
None.
