Introduction
Many older adults are at an increased risk for preventable events such as falls, delirium, and functional decline.Reference Fried, Ferrucci and Darer 1 One in five seniors lack the support they need to function daily.Reference Desai, Lentzner and Weeks 2 Gurley et al. chillingly captured the potential severity of the problem: those found helpless or dead accounted for 3.7% of calls to emergency medical services (EMS).Reference Gurley, Lum and Sande 3 This population-based study showed that the rate of being “found down” increased dramatically with age, from 3/1,000 person years in the 60–64 age group, to 27/1,000 person years among patients 85 and older.Reference Shah, Glushak and Karrison 4 More than 35% of those found down had been incapacitated for over 12 hours; those found down after 72 hours had a mortality of 67% during their hospitalization.Reference Gurley, Lum and Sande 3 Clearly, there are isolated vulnerable older people living in the community with unmet needs who could benefit from a safety net.Reference Desai, Lentzner and Weeks 2
Due to social isolation and a lack of support and timely access to primary care, such frail older adults frequently depend on EMS to function as their safety net.Reference Shah, Glushak and Karrison 4 This may explain why, despite their increased disease burden, the proportion of non-urgent calls increases with age and is twice as common among patients 65 years and older compared to younger adults.Reference Clark and FitzGerald 5 Up to one-third of ambulance calls among older adults are for assistance with mobility (e.g., being unable to rise from the toilet or after a non-injurious fall).Reference Shah, Glushak and Karrison 4 Older adults are more likely to call 911 on multiple occasions and to decline transportation to hospital after calling 911, compared to younger adults.Reference Marks, Daniel and Afolabi 6 Repeat 911 calls account for 18% to 40% of ambulance calls among people 65 years of age or older.Reference Weiss, Ernst and Miller 7 Thus, an older adult calling 911 for a seemingly minor problem may be a “sentinel” event, signaling a diminished ability to continue functioning independently.Reference Bernstein 8 However, interventions to prevent adverse outcomes such as falls,Reference Tinetti, Baker and McAvay 9 hospitalization,Reference McCusker, Jacobs and Dendukuri 10 and functional declineReference McCusker, Jacobs and Dendukuri 10 have not been shown to be either effective or economical when applied broadly to all older adults.Reference McCusker, Jacobs and Dendukuri 10 - Reference Wilber and Gerson 12 Interventions must be targeted at high-risk individuals to be effective.Reference Miller, Lewis and Nork 13 , 14 The Identifying Seniors at Risk (ISAR) tool was developed to identify high-risk older patients in the ED, but has never been validated for use by paramedics.Reference McCusker, Bellavance and Cardin 15
Given their frequent encounters in the homes of older adults, and the substantial proportion of older EMS patients who are not transported to ED, paramedics may be uniquely positioned to identify those at risk for adverse outcomes.Reference Gerson, Schelble and Wilson 16 This potential has increasingly been recognized by the creation of “community paramedicine” programs,Reference Ball 17 - Reference Weiss, Chong and Ong 20 but paramedics have historically received little specialized geriatric training to help them identify older adults at risk for adverse outcomes.Reference Wilber and Gerson 12 Thus we sought to develop the “Paramedics assessing Elders at Risk for Independence Loss” (PERIL) clinical prediction rule to permit paramedics to identify high-risk older adults.
Our specific objectives were: 1) to test the reliability of 43 candidate questions for a clinical prediction rule; 2) to derive the PERIL clinical prediction rule, a checklist for use by paramedics to assess whether an older adult is at high risk for repeat ED visits, hospitalization or death in the month following an EMS encounter; 3) to compare our prediction rule to a proxy measure of paramedic global assessment of risk; and 4) to compare our prediction rule to the existing ISAR prediction tool when employed by paramedics in the prehospital setting.
Methods
Study design
This program of research included the development of potential variables, reliability testing, and the derivation of the PERIL clinical prediction rule following methodological standards for decision rules.Reference Stiell 21 A pool of 43 candidate predictor variables was selected based on three published systematic reviews.Reference Aminzadeh and Dalziel 11 , Reference Grief 22 , Reference McCusker, Karp and Cardin 23 Our aim was to ensure that the rule was “clinically sensible,” i.e., all required data elements would be available to paramedics and feasible to collect during the EMS encounter.Reference Beveridge, Ducharme and Janes 26 Development of the data collection form was guided by an expert panel composed of a clinical epidemiology and decision rule expert, two geriatricians, and a paramedic with research experience, as well as paramedic focus groups (see Figure 1). All items were framed as simple yes or no questions. One question was included to act as a proxy measure of paramedics’ global assessment of risk for adverse outcomes: “Given the current home situation, are there any problems that would prevent this client from being safely discharged home from the ED, or contribute to recurrent EMS/Emergency use?” We recognized that when completing our proxy variable of clinical judgment, paramedics in the study would have the benefit of the training they received and the completion of the 42 other variables at the same time. This meant that our proxy measure of clinical judgment would be a “conservative” comparison standard for the performance of the PERIL rule relative to truly unaided clinical judgment by untrained paramedics. We also included all questions from the Identifying Seniors at Risk (ISAR), and trained paramedics in how to use the ISAR questions.Reference McCusker, Bellavance and Cardin 24
Figure 1 PERIL (Paramedics assessing Elders at Risk for Independence Loss) data collection form.
We then developed a curriculum to educate paramedics about the study rationale, objectives, and use of the PERIL data form.Reference Lee, Verbeek and Morrison 25 We used existing one-hour EMS continuing educational sessions to train 1,185 paramedics, including advanced and basic care providers. Paramedics received a didactic lecture about the presentation of common geriatric syndromes and signs of reduced function evident in the home setting. Paramedics then viewed a video-taped simulated encounter between a paramedic crew and an older patient, and recorded any problems they observed. Finally, we conducted a prospective observational cohort study to derive the PERIL clinical prediction rule. We assessed inter-rater reliability for all candidate predictor variables in a convenience sub-sample when two paramedics were able to complete the PERIL data for the same patient.Reference Lee, Verbeek and Morrison 25
Study participants
Eligible patients of three Canadian EMS systems, including Toronto, Ottawa, and Renfrew County, Ontario, were 65 years of age and older and assessed in their home by a paramedic responding to a 911 call. Recruitment occurred from January 1, 2005 to March 15, 2008. We excluded patients who were critically ill, as determined by the Canadian Triage Assessment ScaleReference Beveridge, Ducharme and Janes 26 (CTAS category 1); who lived in a nursing home, as determined by a list of such institutions for each community; or who were attended to in a public place rather than in the their home, because our hypothesis was that paramedics could observe clues of declining function in the patient’s home.
Methods of measurement
Baseline measures
Baseline demographic measures included patients’ age and sex. Information abstracted from the paramedic Ambulance Call Record (ACR) included the reason for the 911 call, chronic medical conditions, regular prescription medications, vital signs, and whether the patient was transported to an ED or not.
PERIL data-collection form
The standardized PERIL checklist comprised of 38 dichotomous variables (see Figure 1) that paramedics observed when attending to an older adult in the patient’s home. The five ISAR items were based on asking the patient questions instead of direct observations, for a total of 43 items (See Figure 1).
Outcome measures
Our primary composite measure of adverse outcomes was based on previous studies looking at adverse outcomes in older ED patients.Reference McCusker, Bellavance and Cardin 24 , Reference McCusker, Bellavance and Cardin 27 , Reference Meldon, Mion and Palmer 28 It included any repeat visit to the ED, hospitalization, or death within 30 days of the initial EMS encounter. We excluded adverse outcomes that were related to the initial EMS call, i.e., occurred on the same day as the initial EMS call with no ED discharge occurring prior to the adverse event. Patients’ data were linked to provincial administrative databases, including the National Ambulatory Care Reporting System, the Discharge Abstract Database, and the Registered Persons Database. We used deterministic matching for 79% of the cohort where Ontario Health Insurance Plan (OHIP) numbers were available and probabilistic matching based on name and date of birth for the rest of the cohort. These datasets were linked using unique, encoded identifiers and analyzed at the Institute for Clinical Evaluative Sciences (ICES).
Statistical analysis
Sample size calculation
We estimated that a sample size of 1,000 older adults would provide sufficient power to examine a minimum of 35 predictive variablesReference Peduzzi, Concato and Kemper 29 and detect odds ratios of 1.6 or greater, assuming a power of 80%, alpha level of 0.05, and adjusting for an r-squared of 0.4 between the independent variables in the model, and a primary outcome rate of 35% observed in our pilot data.Reference Peduzzi, Concato and Kemper 29 , Reference Hsieh, Bolock and Larsen 30
Model derivation
We chose logistic regression modeling over recursive partitioning to derive and maximize the specificity of the PERIL prediction rule,Reference Cook and Goldman 31 given the limited resources available for frail older adults. A highly sensitive rule with low specificity would identify too many “false positive” individuals who would consume the limited available resources without actually requiring them. We used Harrell’s technique to build logistic regression models.Reference Harrell 32 In summary, only clinically relevant predictor variables were chosen and variables were not screened based on univariate association with the primary outcome to protect against type-I error. We refined the list by excluding predictors that lack variables with greater than 95% of observations in a single category, or variables with poor inter-rater agreement (κ<.60). We used correlation matrix analysis to assess for high multicollinearity (VIF > 2.5), and if variables showed high multicollinearity, we chose the most clinical relevant variable, or ran bootstrap simulations for sets of equally clinical relevant variables with high multicollinearity.Reference Efron and Tibshirani 33 We then built candidate predictive models using backward stepwise regression and selected the final model based on predictive ability, ease of use, and parsimony (i.e., fewest number of variables). Receiver Operating Curve (ROC) of the final PERIL prediction rule, paramedic clinical judgment, and the ISAR tool were also reported and areas under the curve (AUC) were compared.Reference Lee, Schwindt and Langevin 34
Ethics
The Sunnybrook and Ottawa Hospital Research Ethics Boards approved the study. As a minimal risk observational study, the requirement for written informed consent was waived.
Results
Characteristics of study subjects
Paramedics screened 1,418 older adults, and 1,065 (75.1%) were eligible. All 1,065 eligible subjects were linked with provincial databases to acquire patient outcomes and 764 (71.7%) had complete data allowing logistic-regression (see Figure 2). There was no difference in adverse outcome rates between patients with and without complete data (269/764 or 35.2%, 95% CI 31.8% to 38.7%, for subjects with complete data, vs. 372/1,065 or 34.9% , 95% CI 32.1% to 37.9%, among subjects missing at least one variable). Their mean age was 80.9 years (interquartile range 75.3 to 86.3 years), 297 (38.9%) were male and 54 (7%) were not transported to hospital (see Table 1).
Figure 2 PERIL derivation study flow chart.
Table 1 Demographic characteristics of eligible subjects in the cohort used to derive the PERIL decision rule
Note: CI=confidence interval; CTAS=Canadian Triage and Acuity Scale.
Primary outcome
A total of 269 individuals experienced 291 adverse outcomes, for a primary outcome rate of 35.2% (269/764). The three components of the primary outcome broke down as 169/764 (22.1%) older adults who were hospitalized, 89/764 (11.6%) who had a subsequent ED visit, and 33/764 who died after the initial EMS encounter (4.8%). All adverse events occurred within 30 days of the initial call to 911. Of the total 291 composite adverse outcomes, 18 older adults experienced two events and two experienced three events.
Selection of variables
Five variables were answered “yes” in less than 5% of patients, and were excluded from analysis: home too cold, chronic renal failure, agitation, delusions, and suspected elder abuse. Given the extremely low frequency and their weak univariate association with our primary outcome, they did not influence the final model in a sensitivity analysis. Two sets of variables showed multicollinearity: a) “previous 911 use” and “previous ED use”; and b) the three variables “lacks help needed to bathe/shower,” “lacks help needed to get to toilet,” and “lacks help needed to clean living quarters.” For the first sets of collinear variables, we selected “previous 911 use,” given our focus on prehospital screening. For the second set of collinear variables, “lacks help needed to bathe/shower” was selected because it showed the strongest association with the outcome in bootstrap simulation analysis in the absence of any clear superiority clinically. Thus a total of 38 of the original 43 variables were entered into the logistic regression model (see Table 2).
Table 2 Inter-observer reliability, adherence and univariate association with adverse outcomes of 38 potentially predictive variables included in the logistic regression analysis
Reliability and adherence
There was “good” (kappa ≥0.6) to “excellent” (kappa ≥0.8) reliability for 40/43 of the questions.Reference Landis and Koch 35 Among the 38 questions retained for logistic regression, reliability was “good” to “excellent” for 37/38 questions and “substantial” for one question (history of chronic medical conditions—see Table 2). Thus no questions were excluded on the basis of poor reliability or kappa <0.6. Paramedic adherence in completing individual questions is also listed in Table 2. In addition, rates of missing data among the 38 questions ranged from 0 to 328/1065 (30.8%). The ISAR variables that required paramedics to question the patient directly were missing in 27% of cases or more, whereas variables based on paramedic observations had high adherence rates (see Table 2).
Model derivation
Four significant independent predictors remained after model selection: 1) 911 use in the past 30 days; 2) “Given the current home situation, are there any problems that would prevent this client from being safely discharged home from the ED, or contribute to recurrent EMS/Emergency use?”; 3) overall lack of social support, defined as “Patient has unmet needs for social support that, in your opinion, will contribute to recurrent EMS/Emergency use”; and 4) male sex (see Table 3).
Table 3 The Paramedics assessing Elders at Risk for Independence (PERIL) Rule
Note: ED=Emergency Department; EMS=emergency medical services.
The four-item PERIL prediction rule was significantly associated with the composite adverse primary outcome measure, and the Hosmer-Lemeshow statistic failed to show any lack of fit (χ2=2.51, df=7, p=0.93). The PERIL prediction rule had an AUC of 0.62, compared to the AUC of 0.57 for clinical judgment as captured by the single proxy variable of paramedic overall risk assessment (p=0.02). Since the odds ratios for all variables in the final predictive model were approximately equal, we constructed the PERIL prediction rule, with one point assigned for each positive question resulting in a maximum of four points, with a range of one to four points. (See Table 4). The sensitivity and specificity for all cut-off values are listed in Table 5.
Table 4 Predictive performance of the Paramedics assessing Elders at Risk for Independence (PERIL) clinical prediction rule
Note: CI=confidence interval; ED=emergency department; EMS=emergency medical services.
Table 5 Comparing adverse outcomes, sensitivity and specificity of the Paramedics assessing Elders at Risk for Independence Loss (PERIL) compared to single global risk assessment variable
Note: ED=Emergency Department; EMS=emergency medical services.
Finally, we compared the predictive performance of the ISAR to the four-item PERIL rule in a subsample of 633 subjects with complete ISAR variables. The AUC for the ISAR was similar to the PERIL tool in this subsample (0.57 vs. 0.61 respectively, p=0.28).
Discussion
This large multi-centred observational study demonstrated the feasibility of Canadian paramedics identifying older adults at high risk of future adverse events using a clinically sensible prehospital rule, i.e., one that uses information that can be reasonably collected by paramedics at the time of the 911 call. We found that paramedics could reliably observe data needed for the PERIL rule when attending to older patients; inter-observer reliability was substantial to excellent for all variables considered for the model.
This study adds to the evidence supporting the feasibility of community paramedicine, a model of care whereby paramedics apply their training and skills in “non-traditional,” preventative, community-based roles.Reference Mason, Knowles and Colwell 19 - Reference McCusker, Bellavance and Cardin 24 Gerson et al. first studied screening by paramedics for clues of unmet needs in the home environment of older adults.Reference Dixon, Mason and Knowles 18 In 121/124 cases where paramedics identified older adults as having unmet needs, this was confirmed during a home-visit by a “gold-standard” geriatric assessor from a community agency. Gerson et al. also reported that 91% of the paramedics supported the screening of vulnerable elders and wished to continue doing so.
Paramedic ability to identify high-risk older adults using unaided clinical judgment had not been studied, so we assessed whether a single proxy variable assessing paramedic global assessment of risk would be sufficient to identify high-risk older adults. Addition of the three other PERIL questions had superior predictive ability compared to our single proxy variable of “unaided” clinical judgement (see Table 5). An additional advantage of the four-question prediction rule, over the single proxy measure of paramedic judgment alone, is that it allows the use of different cut-off levels tailored to the needs of the paramedic service (see Table 5). The specificity achieved with the single proxy variable of clinical judgement was 41.5%, whereas specificity was improved to 83.8% when a cut-off of three was chosen for the PERIL rule (see Table 5). High-volume systems that need to limit the number of referrals made may choose a more specific threshold. In low-volume rural settings, a more sensitive threshold can be chosen. It should also be noted that participating paramedics had undergone 60 minutes of training on assessing the 42 other risk factors for adverse outcomes, and this may have improved the predictive ability of our proxy variable compared to truly “unaided” clinical judgement among untrained paramedics.
McCusker et al. were the first group that followed methodological standards for developing an ED-based clinical prediction rule targeting older adults. The five-item ISAR tool was originally designed to identify ED patients over 70 at high risk for adverse events within six months of the original ED visit. The adverse events observed included functional decline in 54%, death in 35%, and nursing-home admission in 10.8%. The ISAR showed an AUC of 0.71, sensitivity of 72%, and specificity of 58% for this composite outcome.Reference Harrell 32 The data from the original study were subsequently used to assess the ability of the ISAR to predict return to the ED within 30 days; the predictive performance for early return to ED was substantially lower (AUC 0.63).Reference McCusker, Bellavance and Cardin 27 Our literature search revealed that our study is the first to examine use of the ISAR by paramedics.
While there was no statistically significant difference in the predictive performance of ISAR compared to the PERIL rule, the completion of the ISAR questions was lower—all five of the ISAR questions were missing in more than 27% of eligible patients. In contrast, paramedics completed the three observation-based PERIL items nearly 100% of the time. Thus, paramedics provided sufficient data to compute a meaningful score based on three items of the PERIL score nearly 100% of the time.
Our composite outcome was chosen based on existing ED screening rules, and was intended to represent adverse outcomes that might be prevented by appropriate preventative management. There is some debate as to whether return ED visits should be considered as an adverse outcome, as McCusker has shown that most ED visits by older people are appropriate.Reference McCusker, Karp and Cardin 23 One approach is to look at whether repeat ED visits are for the same condition. However, it is problematic to use administrative databases such as NACRS to capture whether a return visit is for the same problem. For example, a patient with non-cardiac chest pain could have several ICD-10 discharge codes, even if they returned with a similar complaint. However, 169/269 (62.8%) of adverse outcomes in this study were subsequent hospitalization, and only 89/269 (33.1%) were for return to the ED.
This study has several limitations. First, we were unable to collect data on patients paramedics failed to screen due to problems with paramedic system information systems at the time of the study, and thus we have no data on the potential number of patients that were not screened. There was a further loss of 28.3% of eligible subjects due to missing variables. Clearly, our data show that paramedics are more likely to complete observation-based items, as opposed to items that require paramedics to ask patients additional questions that were outside their usual workflow. The only variable in the final PERIL prediction rule that was collected directly from the patient was “previous 911 use,” which was asked of the patients in 71.8% of cases. The recent adoption of electronic patient care records by many paramedic and EMS services will allow for the automated collection of previous 911 use in the future, and improve the accuracy and capture rate of this variable.
The predictive performance of the PERIL prediction rule was only moderate, with an area under the curve of 0.62, and low sensitivity. However the PERIL rule was designed to maximize specificity over sensitivity, to avoid patients falsely identified as being at high risk. We chose this strategy because in most Canadian health care systems, there is more demand for community health care services than supply, and the current standard for referral to such services is unaided clinical judgement. Multiple studies have shown that ED staff have difficulty identifying older adults at high risk for adverse outcomes using clinical judgement alone.Reference Aminzadeh and Dalziel 11 , Reference McNamara, Rousseau and Sanders 36 , Reference Singal, Hedges and Rousseau 37 Our data confirm that paramedics are also challenged to identify high-risk older individuals with their clinical judgement alone, even when aided by training and completion of 42 other variables to assess risk at the same time.
In summary, this study demonstrated the feasibility, high reliability, and adherence of trained paramedics with completing a 43-item assessment of older patients to identify those at high risk for adverse outcomes within one month of an EMS encounter. The PERIL prediction rule has advantages compared to a proxy measure of paramedic clinical judgement, including better predictive performance and the ability to set different thresholds tailored to the needs of the paramedic service. While the ISAR is an acceptable alternative in terms of predictive ability, PERIL is a simpler rule that had better adherence. Further research should validate the PERIL predication rule in an independent population of older patients, validate the four-item version of PERIL, and explore methods to optimize screening of older adults at high risk for adverse outcomes. If validated, future research should explore whether emergency physicians’ and paramedic services’ use of the PERIL prediction rule can be used to target high-risk individuals for referrals to preventative community-based interventions (e.g., primary care, home care, community paramedicine teams, or geriatric outreach teams), and whether such referrals reduce adverse outcomes.
Acknowledgments
JSL, RPV, MJS, LM, IS, DR, AK and GN conceived the study idea, and designed the trial. JSL, RPV, MJS, LC, IS, JT, LM, MN, BHR, SS, DR, AK and GN helped obtain research funding. JSL, LC, MN, BHR, and SS undertook recruitment of participating centers supervised the conduct of trial and data collection. JSL conducted training, executed recruitment of patients, data collection and managed the data, including quality control. MS, GN, IS, RPV and AK provided statistical advice on study design and analyzed the data. MS, LM, DR, AK, and GN provided advice on analysis. GN is the senior author and acted as academic mentor to JSL. JSL drafted the manuscript, and all authors contributed substantially to its revision and provided final approval. JSL takes responsibilities for the paper as a whole.
Competing Interests: The authors acknowledge the Canadian Institutes of Health Research, Institute of Aging, who supported this work. Dr. Rowe is supported as a Tier I Canada Research Chair in Evidence-based Emergency Medicine by the Canadian Institutes of Health Research (CIHR) through the Government of Canada (Ottawa, ON). Dr. Naglie is supported by the George, Margaret and Gary Hunt Family Chair in Geriatric Medicine, University of Toronto. Dr. Morrison holds the Robert and Dorothy Pitts Chair in Acute Care and Emergency Medicine, Li Ka Shing Knowledge Institute, St Michael’s Hospital and the University of Toronto. She also receives salary support from the National Institute of Health, USA for studies related to cardiac arrest and trauma.
