Introduction
Demographic and economic changes, such as the ageing population; the increased number of chronically ill patients with more intensive and technically complex nursing needs; the changing preferences of patients; and the financial constraints in health-care budgets nurture a shift of care from the hospital to the home-care setting (Jansen et al., Reference Jansen, Kerstra, Abu-Saad and Van der Zee1996; Mistiaen et al., Reference Mistiaen, Duijnhouwer, Wijkel, de Bont and Veeger1997; Blegen et al., Reference Blegen, Goode and Reed1998; McCorkle et al., Reference McCorkle, Strumpf, Nuamah, Adler, Cooley, Jepson, Lusk and Torosian2000). This evolution increases the focus on health care at home. The role of home nurses, however, is not well defined, resulting in a large dispersion of role implementations in different countries (Keller et al., Reference Keller, Strohschein, Lia-Hoagberg and Schaffer1998; Hughes et al., Reference Hughes, Robinson, Cooley, Nuamah, Grobe and McCorkle2002; Frich, Reference Frich2003; Robinson et al., Reference Robinson, Dugger, Fong, Heintzman, Hnizdo, Libby, McDaniels, Miles, Rempel, Taylor and Wrashaw2006). Indeed, it is not precisely known what home-care nurses do; what kind of care is received by patients; what the performance of home nurses is; and what the impact of the increasing need for home nursing is on the current and future role of home nurses. In this respect, it is necessary to gain a clear insight into the activity profile of home nurses. Internationally, research on nursing interventions is mainly conducted in intramural settings. Research in the field of home nursing is much more limited. The few existing studies confirmed a wide diversity of job functions in home nursing (Keller et al., Reference Keller, Strohschein, Lia-Hoagberg and Schaffer1998; Hughes et al., Reference Hughes, Robinson, Cooley, Nuamah, Grobe and McCorkle2002; Frich, Reference Frich2003; Robinson et al., Reference Robinson, Dugger, Fong, Heintzman, Hnizdo, Libby, McDaniels, Miles, Rempel, Taylor and Wrashaw2006). However, these studies are hampered by the lack of a sound measurement. There is no gold standard in the assessment of home nurses’ activities. The measurements that were used had different levels of abstraction in the description of the work of home nurses (Martin et al., Reference Martin, Scheet and Stegman1993; Schumacher and Marren, Reference Schumacher and Marren2004; De Vliegher et al., Reference De Vliegher, Paquay, Grypdonck, Wouters, Debaillie and Geys2005), and they used different classification systems, of which not all appropriately captured the dimensions of physiological, technical, social, and psychological activities of home nurses’ practice. Furthermore, validity and reliability of these measurements are uncertain. Hence, to gain insight into the work of home nurses, valid and reliable instruments need to be developed to adequately measure the activities of these nurses. Therefore, we developed a tool to capture the activity profile of home nurses: ‘the 24-h recall instrument for home nursing’. The present article describes (i) the development and pilot testing of this 24-h recall instrument and (ii) the evaluation of its content and face validity, and its interrater reliability.
Methods
Instrument development
The instrument was developed in three phases. First, to capture the actual everyday work of home nurses, the concept of a 24-h recall was chosen to develop the instrument under study. The 24-h recall is a method that is mainly used in the dietary and food setting to record a person’s food and eating habit over a period of 24 h (Conway et al., Reference Conway, Ingwersen, Vinyard and Moshfegh2003; Park et al., Reference Park, Paik, Skinner, Spindler and Park2004; Subar et al., Reference Subar, Thompson, Potischman, Forsyth, Buday, Richards, NcNutt, Hull, Guenther, Schatzkin and Baranowski2007). Non-participatory observations of home nursing rounds and individual interviews at the end of these rounds allowed us to develop a first checklist of activities, which comprised 74 items. Second, this first checklist was evaluated by experts in home nursing and was compared with the activities listed in the nomenclature (reimbursement system in home nursing in Belgium) and the Nursing Interventions Classification (NIC). The comparison of the checklist with the nomenclature resulted in adding more technical activities, such as bladder irrigation, positioning, transfer, and replacement of a suprapubic probe. On the basis of the comparison with the NIC, we subdivided the list of activities into domains (basic physiological, complex physiological, psychosocial support, recording/administration, and communication) and categories (self-care facilitation, elimination management, nutrition support, immobility management, drug and perfusion management, skin/wound management, diabetes care, and other care). Altogether, these adaptations allowed us to make a preliminary version of the 24-h recall instrument for home nursing. In this version, we included a time recording per activity, to assess the time spent per intervention.
This version of the 24-h recall instrument contained three parts: (i) items related to the demographic data of the patient; (ii) items measuring the home nursing activities; and (iii) items related to the demographic data of the participant (home nurse). This preliminary version of the instrument was further evaluated in the present study.
Psychometric properties
A selection of psychometric properties of the 24-h recall was tested, using the Standards for Educational and Psychological testing as framework (American Educational Research Association, American Psychological Association and National Council on Measurement in Education, 1999): validity on test content and interrater reliability. The terminology as used in the Standards was used in the present manuscript.
Validity: evidence based on test content
In testing psychometric properties, the validity on test content (content validity and face validity) is fundamental. It refers to the themes, wording, and format of the items, tasks, or questions in the test, as well as to the guidelines for procedures regarding the administration and scoring of the test (American Educational Research Association, American Psychological Association and National Council on Measurement in Education, 1999). We evaluated the validity on test content by means of non-participatory observations of home nurses, in which we observed and recorded their activity profile. Each home nursing round was concluded with an individual interview to allow the home nurses to add activities that were not included in the instrument. Furthermore, comments on the layout, relevance, clarity, and feasibility of the instrument could be given. In this phase, both items that investigated the activities and the time spent at each activity during working rounds were evaluated. On the basis of the information obtained in this validity testing, adaptations to the instrument were made, and subsequently, reliability was tested.
Interrater reliability
The interrater reliability was tested to assess the degree to which two individuals would yield the same results. Home nurses and a researcher completed the 24-h recall, simultaneously and independently, for each patient on a same nursing round.
Participants
Home nurses were eligible to participate in this study if they (i) were Dutch-speaking; (ii) employed in an organization for home nursing in Flanders (Belgium); (iii) employed in the organization for at least five years; (iv) working on a nursing round that reflects the diversity of home nursing activities; (v) agreed to be observed during the nursing round and to be interviewed by the researcher; and (vi) had the permission of the patients to be accompanied by an observer.
Five home nurses who met the inclusion criteria were selected, and all were willing to be accompanied by a researcher on their round and to record the performed activities on one round (morning or afternoon). They were all women, had a mean age of 45 years (range between 31 and 51 years), worked part time (mean of 24 h per week), and had an average working experience of 24 years as a nurse and 13 years in the home nursing organization. During this validation study, a total of 69 individual patients received care from these five home nurses, 71% of which were female, and having a mean age of 79 years (range between 43 and 97 years).
Procedure
The selected home nurses received an envelope with an invitation letter, explaining the purpose of the study and the instrument, an informed consent form, and copies of the 24-h recall for each patient in care. During the nursing round, the researcher recorded all activities immediately after each patient’s visit and this recording was set out to be the standard measurement. The five participating nurses completed the 24-h recall form independently from the researcher. Three of the participating nurses completed each form immediately after a patient’s visit. The two other nurses completed the forms for all patients at the end of their nursing round. This strategy allowed us to investigate whether there is an impact of the moment of recording the performed activities on the reliability of the data. The home nurses were assured that the results would remain confidential. At the end of each round, the researcher collected the recorded forms for analysis.
Data analysis
To assess the interrater reliability of the 24-h recall instrument for home nursing, we calculated the proportion of observed agreement (P 0), the Kappa coefficients (K), the prevalence index (PI), and the bias index (BI). The proportion of observed agreement is the ratio of exact agreement between the raters per total number of assessments. To correct for agreement by chance, we also calculated Cohen’s Kappa coefficient (Fleiss and Cohen, Reference Fleiss and Cohen1973). The 24-h recall instrument consists of binary items (performed or not performed). Therefore, the unweighted Cohen’s Kappa was used. Kappa values are scaled to range from −1 to +1: a negative value indicates poorer than chance agreement, zero indicates exactly chance agreement, a positive value indicates better than chance agreement, and a value of unity indicates perfect agreement (Fleiss and Cohen, Reference Fleiss and Cohen1973). Furthermore, the strength of agreement for the Kappa coefficient is considered to be poor for Kappa values below 0.40, moderate from 0.41 to 0.60, substantial from 0.61 to 0.80, and almost perfect above 0.80 (Landis and Koch, Reference Landis and Koch1977). For binary items, paradoxes in agreement parameters can be owing to bias and prevalence effects (Byrt et al., Reference Byrt, Bishop and Carlin1993; Hoechler, Reference Hoechler2000; Sim and Wright, Reference Sim and Wright2005; Vach, Reference Vach2005). Therefore, the PI and BI are reported in addition to the Kappa to address these paradoxes of the Kappa statistic (Sim and Wright, Reference Sim and Wright2005; Cunningham, Reference Cunningham2009). The PI for binary items was calculated as follows: the absolute value of the difference between the number of cases rated as positive by the researcher and the nurse, and the number of cases rated as negative by the researcher and the nurse, divided by the total number of assessments (Sim and Wright, Reference Sim and Wright2005; Cunningham, Reference Cunningham2009; Wellens et al., Reference Wellens, Van Lancker, Flamaing, Gray, Moons, Verbeke, Boonen and Milisen2012). The BI calculates the extent to which the researcher and the nurse disagree on the proportion of cases in a specific category: the absolute value of the difference between the number of cases rated as positive by the researcher and as negative by the nurse, and the number of cases rated as negative by the researcher and as positive by the nurse; divided by the total number of assessments (Sim and Wright, Reference Sim and Wright2005; Cunningham, Reference Cunningham2009; Wellens et al., Reference Wellens, Van Lancker, Flamaing, Gray, Moons, Verbeke, Boonen and Milisen2012). The Kappa coefficient was calculated using SAS 9.2, and the PI and BI were calculated using SAS 9.2 and Excell.
Results
Validity: evidence based on test content
The home nurses, who revised the preliminary instrument during a nursing round, provided the following suggestions to improve the instrument: the addition of the interventions ‘communication with colleagues’, ‘communication with other health care workers’, ‘giving education and information to patients’, ‘providing emotional support to patients and their informal caregivers’, ‘aid with daily care’, and ‘home dialysis’; a columnar listing of the nursing activities; a removal of the time recording per activity (they forgot to record the time or they did not know when to start or end the recording or they performed a lot of activities simultaneously, which made it difficult to rate each activity separately, and there was a huge discrepancy between the nurses’ and the researcher’s time rating). These adaptations resulted in an instrument containing 137 items (cf. annex): six items with regard to patient characteristics; two items with regard to the time recording per visit (time entering and time leaving the home of the patient); and 129 binary items with regard to the performed activities (‘psychosocial support: subject’, ‘aid with daily care: specify’, and ‘Other Care – Specify clearly’ are not part of these 129 items).
Interrater reliability
The interrater reliability of the 24-h recall could be evaluated for 55 of the 129 binary items (Table 1). The remaining 74 activities were not performed during the test period. The mean observed proportion of agreement for the 55 items was 0.94. For all, but two items (observation overall health status; giving information/advice) the observed agreement was ⩾0.80. This suggests that the agreement between raters is high.
Observed proportion of agreement, kappa (CI of kappa), PI and BI for each item of the 24-h recall (n=69 patients)
PI=prevalence index, BI=bias index.
The 55 items had a mean kappa value of 0.71, indicating that the level of agreement was substantial. Applying the conventional cut-offs on these 55 items for interpreting kappa values, reliability was perfect to almost perfect (K=1 or K⩾0.81) for 24 items (44%), substantial (0.61≤K<0.81) for 15 items (27%), moderate (0.41≤K<0.61) for eight items (15%), and poor (K<0.41) for eight items (15%).
The PI ranged between 0.16 and 0.97, showing great variety. In general, prevalence indices reflect the homogeneity of a sample: if the PI is known to be high (eg, 0.97), the magnitude of kappa might be reduced (Sim and Wright, Reference Sim and Wright2005; Burn and Weir, Reference Burn and Weir2011). The BI was very low, ranging between 0 and 0.14. This finding suggests that there is no tendency between the raters to score the performed activities to a different pattern.
To investigate the potential impact of the moment of completing the 24-h recall form on the reliability of the data, we calculated the mean kappa value for each nurse/researcher pair. The kappa values for the three nurses who completed the forms after each patient visit were K=0.86; K=0.90; and K=0.83. For the two nurses who completed the 24-h recall forms for all patients at the end of their nursing round, the mean kappa values were K=0.61 and K=0.84. This indicates that the moment of recording of the performed activities did not influence the interrater reliability.
Discussion
In the present study, we offered evidence to support the face validity, content validity, and the interrater reliability of the 24-h recall instrument for home nursing in measuring the activity profile of home nurses.
The iterative process of the developmental and testing phase resulted in an instrument that is valid on test content. Twelve categories of activities are withheld that are relevant to measure the activity profile of home nurses. In each category, respondents have the possibility to record additional activities that are not included in the list. This has both positive and negative consequences. It is positive that the full spectrum of activities can be reported, even if they are not listed in the instrument. A negative consequence is that it can result in an ‘easy-way recording’: participants write down activities that are in fact listed in the category or that are listed in another category. Therefore, it is important to thoroughly explain the purpose of the free-writing space. The time recording per activity was demonstrated to fail in terms of validity, and therefore was replaced by a time recording per patient visit. Although the instrument captures the whole spectrum of home nursing activities, it fails to reflect the process of care. Further research is needed to refine the ‘psychosocial support’ section in a way that the instrument shows the shift to more complex homecare in which technical activities are performed in combination with more intellectual activities. Despite the fact that psychosocial support items are less directly observable nursing care activities, 7 of the 55 items for which the interrater reliability could be measured were psychosocial support items. All but one item (information/advice) had an observed proportion of agreement higher than 0.90, indicating that the level of agreement was almost perfect and that both the home nurse and the researcher observed and recorded these less directly observable activities. However, it is important to indicate that working with a predetermined list of activities tends to influence nurses’ description of the care they provide. This influences them to record activities that they did not actually provide and can also help them to focus on the less observable intellectual activities. The instrument was developed to record the actual work of home nurses, but it does not give an indication of the quality of the performed activities.
In general, the proportions observed agreement were very high, and the strength of kappa agreement was substantial to almost perfect. Only two items had low Kappa values in combination with observed agreement values below the threshold of 0.80 (observation of overall health status and giving information/advice). This suggests potential inequality between raters. A total of 14 items had kappa values lower than 0.60, despite high-observed agreement values (⩾0.80). This reflects homogeneity of the sample on the particular items, which reduces the kappa value (Wellens et al., Reference Wellens, Van Lancker, Flamaing, Gray, Moons, Verbeke, Boonen and Milisen2012). This finding is supported by the fact that the PI was >0.80 for 11 of these 14 items. For binary items, a high percentage of agreement along with a poor kappa agreement for the same item can be because of bias and prevalence effects (Byrt et al., Reference Byrt, Bishop and Carlin1993; Vach, Reference Vach2005). The prevalence of item scores affects the stability of kappa. If the ratings of the sample of patients lack variability (ie, are homogeneous), it is unlikely that kappa will be close to 1. This phenomenon is independent of the sample size (Wellens et al., Reference Wellens, Van Lancker, Flamaing, Gray, Moons, Verbeke, Boonen and Milisen2012). Therefore, for binary items, the PI should be calculated: the absolute value of the difference between the number of cases rated as positive by both raters, and the number of cases rated as negative by both raters, divided by the total number of assessments (Sim and Wright, Reference Sim and Wright2005). Furthermore, the BI was very low, indicating that there is no tendency between the raters to record the performed activities according to a different pattern. Consequently, it can be concluded that bias did not play a role in the magnitude of kappa. Taken together, we can conclude that the interrater reliability of our 24-h recall instrument is good.
Hence, the 24-h recall instrument for home nursing can be considered to be a new and unique instrument in the landscape of home health care: it is an important first step in capturing the everyday activities of home nurses; it does not take too much of home nurses’ time (no more than 5 min per patient); and it has shown evidence on face and content, and interrater reliability. Because of these characteristics, this instrument is used in a nationwide study on the activity profile of home nurses in Belgium, in which 81 493 patients were included. The results of that study will be reported in a related article.
The 24-h recall instrument for home nursing can be applied for different purposes. It can be used to investigate the current activity profile and job characteristics of home nurses. However, the instrument can also be used to study the activities performed by auxiliary nurses or health-care assistants in home nursing. This would provide important information for managers and policymakers. Profiling the activities of home nurses and health-care assistants would allow managers to proactively anticipate to the fast evolving and challenging change in needs towards home nursing. Furthermore, the instrument is an important tool for policymakers in terms of adequate decision making and future policy planning, for example, determining the future need for and competences of home nurses. From a research perspective, the newly developed 24-h recall instrument for home nursing could be a step towards the creation of a gold standard to measure the activity of nurses and auxiliary services.
This study had some methodological limitations that need to be considered in the interpretation of the findings. First, with regard to the development of the instrument the non-participating observations might have influenced the nurses in a way that they did not perform their activities as they usually do. Second, a small number of home nurses participated in this study. Hence, the distribution of activities performed by these nurses is unlikely to be representative, and therefore was not reported in the present article. The results of a nationwide study including more than 80 000 patients will provide more accurate data on home nursing activities. They will be reported in a related article. Third, this study focused only on the validity evidence based on test content and on the interrater reliability to evaluate the psychometric properties of the 24-h recall instrument for home nursing. Other aspects of validity and reliability of this instrument should be evaluated when using the instrument in large-scale projects. Fourth, this study was conducted with home nurses in Belgium. It might be possible that the replication of this study with other health-care professionals, in other settings and in another health-care system would result in different findings. Hence, we need further research on other settings and countries to see to what extent our findings are transferable.
Conclusion
We developed a 24 h recall instrument to measure the activities of home nurses. The findings in this study support the validity evidence based on test content and the interrater reliability of this instrument. This instrument can help to shape practice and policy by making the home nursing profession more transparent: a clear insight into the kind of care that is provided by home nurses and is received by the patients in primary care contributes to the development of a clear definition of the role of home nurses in health care. However, further research is necessary to examine other aspects of validity and reliability of the instrument.
