Introduction and policy context
Successive UK governments have enacted welfare policies that aim to instil active levels of consumer agency into the ways in which individuals access and engage with welfare provision (Clarke et al., Reference Clarke, Newman, Smith, Vidler and Westmarland2007; Simmons et al., Reference Simmons, Powell and Greener2009).
In the English NHS, patients now need to make choices about their care providers, and are increasingly expected to make choices about their own care pathways (Greener, Reference Greener2003; Greener, Reference Greener2007; Greener and Mannion, Reference Greener and Mannion2009; Department of Health, 2010; Armstrong, Reference Armstrong2014; Harding et al., Reference Harding, Medina Lara, van Teijlingen, Wood, Galpin, Baron, Crowe and Sharma2014).
Increasing numbers of adults of all ages, eligible for state funded health and social care, now operate ‘cash for care’ and individual budget schemes, whereby individuals use state funds to interact with markets in order to procure goods and services to meet their own needs (Glendinning, Reference Glendinning2008; Glendinning et al., Reference Glendinning, Challis, Fernandez, Jacobs, Jones, Knapp, Manthorpe, Moran, Netten, Stevens and Wilberforce2008; Jones et al., Reference Jones, Welch, Caiels, Windle, Forder, Davidson, Dolan, Glendinning, Irvine and King2011; Stevens et al., Reference Stevens, Manthorpe, Glendinning, Moran, Jacobs, Challis, Wilberforce, Fernandez, Knapp, Jones and Netten2011; Netten et al., Reference Netten, Jones, Knapp, Fernandez, Challis, Glendinning, Jacobs, Manthorpe, Moran, Stevens and Wilberforce2012; Moran et al., Reference Moran, Glendinning, Wilberforce, Stevens, Netten, Jones, Manthorpe, Knapp, Fernandez, Challis and Jacobs2013). Furthermore, the gradual tightening and restriction of eligibility criteria for entitlement to statutory social care has increased the number of ‘self-funders’ purchasing their own care and support, and, in the process, navigating a complex care market (Hudson and Henwood, Reference Hudson and Henwood2009; Henwood, Reference Henwood2011).
In housing, despite having a longer and more established relationship with markets and consumer agency (Mills, Reference Mills, Simmons, Powell and Greener2009), maintaining independence often means having to reassess the home environment and make complex choices in relation to types of retirement and or supportive living (Burgess, Reference Burgess2010; Burgess and Morrison, Reference Burgess and Morrison2015).
Implemented throughout welfare provision, these policies position consumer theory as a key driver in allocating resources (Institute for Public Care and Care Quality Commission, 2014). In health and social care, therefore, the primary criterion underpinning resource provision has shifted from professional expertise and assessed need to consumer perceived need, wants and, for some services, the ability to buy.
Published work so far outlines some of the difficulties some people face when making consumer-like choices about their own welfare (Greener, Reference Greener2002; Reference Greener2003; Clarke et al., Reference Clarke, Newman, Smith, Vidler and Westmarland2007; Glendinning et al., Reference Glendinning, Challis, Fernandez, Jacobs, Jones, Knapp, Manthorpe, Moran, Netten, Stevens and Wilberforce2008; Simmons et al., Reference Simmons, Powell and Greener2009; Glennerster, Reference Glennerster2010; Jones et al., Reference Jones, Welch, Caiels, Windle, Forder, Davidson, Dolan, Glendinning, Irvine and King2011; Netten et al., Reference Netten, Jones, Knapp, Fernandez, Challis, Glendinning, Jacobs, Manthorpe, Moran, Stevens and Wilberforce2012; Armstrong, Reference Armstrong2014). A common theme highlights the difficulties consumers face. Welfare provision does not seem to have much in common with mainstream markets, is complex and is characterised by difficulties in accessing and understanding information. Indeed, it can be a problem to gather information on all the possible options available. In addition, the need to access welfare can often be associated with ‘vulnerabilities’, such as changes in health or relationship and financial circumstances.
There is some evidence, however, that the provision of information and advice (I&A) is an important tool that could enable welfare consumers to make informed choices (Baxter et al., Reference Baxter, Glendinning and Clarke2008; Baxter and Glendinning, Reference Baxter and Glendinning2011; Baxter, Reference Baxter2013) and is associated with higher self-reported levels of social care-related quality of life (van Leeuwen et al., Reference Van Leeuwen, Malley, Bosmans, Jansen, Ostelo, van der Horst and Netten2014). However, it is also evident that accessing information does not always have its intended effect in informing and enabling consumers in a welfare context. Furthermore, because formal information is often inaccessible, informal and subjective, the sources of support often prove to be more important in shaping welfare decision-making (Turnpenny and Beadle-Brown, Reference Turnpenny and Beadle-Brown2014). However, these sources may not always be of high quality or be based on robust evidence.
The information and advice economy
Considering the importance of information and advice in the contemporary policy context, definitions that outline the distinction between the terms are surprisingly absent. The differences may appear obvious, but a lack of definition and description in existing empirical work, some of which is reviewed in this article, means readers are left unsure as to what reported outcomes and impact are based on. It is important, therefore, to define ‘information’ and ‘advice’. Margiotta and colleagues (Reference Margiotta, Raynes, Pagidas, Lawson and Temple2003: 9) describe information as ‘material deemed to be of interest to a particular population. This can be either passively available or actively distributed’ and advice as ‘guidance and direction on a particular course of action which needs to be undertaken in order to realise a need, access a service or realise individual entitlements’.
Many providers of formal I&A on welfare are located in the third sector (Windle et al., Reference Windle, Netten, Caiels, Masrani, Welch and Forder2010). Services range from those designed to enhance the uptake of state benefits, often including advocacy and caseworker support (Moffatt et al., Reference Moffatt, White, Stacy, Downey and Hudson2004; Adams et al., Reference Adams, White, Moffatt, Howel and Mackintosh2006), to health and social care services (Baxter et al., Reference Baxter, Glendinning and Clarke2008; Baxter and Glendinning, Reference Baxter and Glendinning2011; Baxter, Reference Baxter2013). Some services integrate many areas, such as housing and care for younger disabled people (Simons, Reference Simons2000) and older people (Ritters and Davis, Reference Ritters and Davis2008; Burgess, Reference Burgess2010; Spiers, Reference Spiers2012; Adams and Green, Reference Adams and Green2015). While access to I&A services may depend on a referral being made, often from a professional source (e.g. benefits advice and advocacy given by Citizen's Advice Bureau advisors in primary care settings – Adams et al., Reference Adams, White, Moffatt, Howel and Mackintosh2006), a significant number of I&A services concerning welfare are reactive and rely on individuals themselves initiating a principal–agent contact (Stiglitz, Reference Stiglitz, Durlauf and Blume2008). In simple terms, this is when a principal (someone in need of I&A) contacts and seeks the expertise of an agent, who will provide I&A to the principal.
With it being notoriously difficult to illustrate the economic value and the cost–benefit of telephone I&A (Windle et al., Reference Windle, Netten, Caiels, Masrani, Welch and Forder2010), compared to face-to-face counsel for instance, there is evidence from the wider literature that telephone I&A is a relatively financially efficient service model (CCHPR, 2012). An engagement with a telephone I&A provider does not preclude clients from receiving written information relating to their enquiry, either by post or email (CCHPR, 2012) and many existing telephone services are also embracing other means of disseminating their material – most notably via the internet.
The internet presents considerable opportunity for cost-effective and widespread dissemination of I&A across most areas of welfare, particularly when key groups are becoming increasingly internet savvy (e.g. disabled and older groups). Yet, the internet also has limitations. Firstly, while the use of search engines and key words might direct users to large amounts of formal information (most providers of formal I&A have a website), search engines may also direct users to informal and poor quality information. Secondly, in most cases the internet can only be used to disseminate information and not the bespoke guidance that characterises advice (Margiotta et al., Reference Margiotta, Raynes, Pagidas, Lawson and Temple2003).
With welfare provision often tainted by significant information asymmetries, information often needs to be imparted in conjunction with bespoke advice, particularly when circumstances around the welfare of an individual are complex. While it is possible to combine information and advice in a virtual format, for example by using email and ‘live chat’, such services tend to run alongside established telephone based services. However, despite some limitations, the internet is having an increasingly important role – particularly in relation to recent welfare reform agendas.
I&A has a central focus in recent UK welfare reform. For example, through the Care Act (HM Government, 2014), local authorities are now obliged to provide I&A on welfare in multiple and accessible formats (Hunt and Lamb, Reference Hunt and Lamb2013). However, local authorities are also under significant and continued pressure to reduce overall expenditure. While it is emerging that many local authorities are favouring low cost ‘off the shelf’ online platforms that have yet to undergo rigorous evaluation, anecdotal evidence also suggests that local authorities signpost people to existing third sector telephone services. For example, many local authorities in the UK signpost older people to the FirstStop service, a national housing options service run by Elderly Accommodation Counsel (e.g. Bassetlaw District Council, 2014). On this basis, despite the important and emerging role internet services are having in the I&A economy, the dual function of information and advice that characterise telephone services offered by third sector agencies assumes a critical importance.
This article
A rich empirical literature is emerging on the marketised and consumerist approaches to organising welfare provision and the implications this might have for individuals. However, one relatively underdeveloped area of research concerns the efficacy of telephone I&A services. An important question for the emerging area of a telephone I&A service on housing options for older people is: what criteria should be employed to assess the efficacy of these services? Existing discussion indicates that service efficacy represents a complex and under-developed area. While there is research on information use, outcomes and impact, these areas tend to be distinctive and separate bodies of research. Although each area is an important component in relation to efficacy, the extent to which the current foci of enquiry is singular and not ‘joined up’ is problematic when attempting to illustrate efficacy.
Informed by what are largely separate bodies of research and literature on (1) information use, and (2) outcomes and impact, the next section identifies a framework of five key areas to consider when undertaking ‘joined up’ research on the efficacy of telephone I&A.
The new ‘efficacy framework’ identified in this article is employed to evaluate the strengths and weaknesses of current studies that focus on outcomes and or impact of telephone I&A services concerning welfare. Critical weaknesses are found in current (predominately quantitative) research in this area, and we outline methodological considerations for developing future robust research designs. It is proposed that a realist evaluation approach (Pawson and Tilley, Reference Pawson and Tilley1997) is capable of addressing the critical weaknesses in existing research on the efficacy of telephone I&A.
Theoretical ‘efficacy framework’
The following section presents a new framework with which to evaluate telephone I&A. It builds on the wider work of Saxton et al. (Reference Saxton, Naumer and Fisher2007) and Netten and Forder (Reference Netten and Forder2008: 18) who developed an individual outcome framework that described how the initial impacts of being imparted with I&A (e.g. peace of mind, empowerment and knowledge) will have intermediate actions (e.g. benefits accessed, services accessed, debts repaid) and long-term outcomes and impacts (e.g. quality of life, health, financial stability).
However a key component of research into the efficacy of I&A should be how it is used, and what role it plays for individuals in attaining an outcome. The importance of this is highlighted in a recent systematic review on the use of formal information on health and social care that found that formal information is often not utilised as it is hard to interpret. Indeed, subjective impressions and informal sources often supersede formal I&A as the key factor in decision-making (Turnpenny and Beadle-Brown, Reference Turnpenny and Beadle-Brown2014). It is clearly important, therefore, to include the use and role of imparted formal I&A in research frameworks when considering efficacy. This will provide the necessary focus on whether imparted formal I&A has had its intended effect.
In this article, we combine the outcome approach with the concepts of individual agency practices (Greener, Reference Greener2002) around information provision, use and its role in decision-making to produce a ‘joined up’ or holistic ‘efficacy framework’ with which to assess the efficacy of telephone I&A. The rationale behind the phases of the framework is outlined below (see Figure 1).
Theoretical framework for ascertaining efficacy of information and advice services
The five dimensions of the new ‘efficacy framework’
There are five key dimensions proposed in this new efficacy framework for I&A services:
Context
The socio-economic profile, reasons, motives and wider structural context for seeking I&A (e.g. choice of social care provider or form of retirement/independent housing) will have a bearing on the content, form of I&A that is imparted and how or if it is acted upon.
Imparted information and advice
I&A services will likely impart a vast array of different types and forms of information and or advice to different people in different circumstances. In order to establish how effective imparted I&A has been in relation to an outcome or impact, it is important to examine what was imparted. Particularly important is the content of the imparted support, whether imparted support took the form of generic information or bespoke advice (Margiotta et al., Reference Margiotta, Raynes, Pagidas, Lawson and Temple2003), and whether this was verbal, written or both. Furthermore, the characteristics of imparted support and, for example, what level of detail was included in the information, will likely determine how it is used and, ultimately, its efficaciousness. A critical issue is whether imparted information is sufficiently detailed and definitive enough to lead to a clear outcome or whether it encourages more informed interpretation by users.
Agency: impact, role and use of information and advice
It is important to focus on how imparted I&A is used and its role in decision-making, particularly in relation to other formal or informal sources of support from individuals or organisations. As discussed earlier, less formal sources of I&A in related fields are sometimes used even when formal sources of I&A have been imparted (Turnpenny and Beadle-Brown, Reference Turnpenny and Beadle-Brown2014). Within the context of engaging with a provider of I&A and receiving I&A on welfare, this element calls for a focus on an individual's agency practices – how they act, why they act in a given manner and their capabilities (Greener, Reference Greener2002) when imparted with I&A.
Outcome
Netten and Forder's (Reference Netten and Forder2008) adapted framework, with which we concur, suggests it is important to focus on the intermediate action-based outcome associated with the use of the imparted I&A. In other words, what caused the outcome to occur, and what role did the use of imparted formal I&A play in shaping an action-based outcome?
Impact
Again, using Netten and Forder (Reference Netten and Forder2008), the intermediate-action based outcome, if associated with an individual's use of imparted formal I&A, will have a wider impact.
This framework, described above, is now used as a basis to review existing original research in relation to telephone I&A.
The literature search
Ten social science academic databases were identified on the basis of their relevance to social policy and welfare. After considering variations and synonyms around key words, a literature search was designed so that records containing either ‘information’ or ‘advice’ and one or more of ‘impact’, ‘outcome’ or ‘result’ in the title were found. ‘Telephone’ was not included as a search term in order to avoid not locating studies that did not use the term in the title (it transpired that some did not use ‘telephone’ in the title). Only original research in English was sought. With consumerist policies in the UK gaining prominence since the early 1990s, the search was restricted to articles published after 1989 until the present. The search is documented in Table 1.
Literature search of academic databases
A strategy was devised in order to locate original research studies with a focus on telephone I&A. As per Table 1, 7,031 records were manually scanned for sources on the individual outcomes and or impact of I&A relating to any aspect of welfare deemed relevant. Of the 7,031 records, 316 matched this criterion. However, 137 were duplicates. Phase 1 of the search yielded 179 sources.
The abstracts, and in some cases the publications themselves, were reviewed in order to apply the key inclusion criteria – concerning a focus on telephone I&A; eight studies were found. These studies are summarised in Table 2.
Summary of articles
Of these eight academic peer-reviewed studies, one compared both a telephone and email service (Mevissen et al., Reference Mevissen, Eiling, Bos, Tempert, Mienties and Schaalma2012) and another focused on professional and consumer enquiries (Melnyk, Reference Melnyk, Schevchuck and Remillard2000). The discussion in this article purely focuses on the characteristics of consumer telephone I&A enquiries.
Methodological and theoretical discussion
Quantitative research, underpinned by epistemological and ontological approaches grounded in positivism and objectivism, suggests that there is relative uniformity to social action – that it is objective in nature and it is possible to test and measure such objective theories (Bryman, Reference Bryman2016). These perspectives are prevalent throughout the reviewed studies.
The following section uses the proposed framework introduced above (see Figure 1) to discuss the characteristics of studies.
Context
Ward et al.’s (Reference Ward, Baum, Ter Maat, Thomsen and Maibach1998) research, concerning telephone I&A on cancer, indicates that motives for engaging with services were often multiple. However, their questionnaire only allowed participants to express one motive for enquiry. Despite this, the researchers cross-tabulate and report and discuss findings by motive – even though this questionnaire design is likely to have produced skewed data.
Other studies that elicit data on motives for seeking I&A also illustrate that the reasons for doing so are multiple and complex. Considering the areas of focus, this is not surprising. Although Dale et al.’s (Reference Dale, Crouch, Patel and Williams1997) study presents selected qualitative data on ‘reasons for calling’ that illuminate the complexity and personal nature of the caller's motives, these are not easily reflected in the associated categorised quantitative findings. In addition, Ward et al.’s (Reference Ward, Baum, Ter Maat, Thomsen and Maibach1998) study demonstrates the dangers of cross-tabulating and reporting findings by motive when it is acknowledged that ‘seeking motives’ are multiple. In these studies, quantitative tools do not adequately capture what is illustrated or what is likely to be complex, personal and subjective motives for seeking I&A. More sophisticated approaches, such as multivariate analysis, could be considered as a means to capture such data.
Data relating to socio-economic profiles, mostly denoted by level of education, are included in some studies with varying detail. For example, Ward et al. (Reference Ward, Baum, Ter Maat, Thomsen and Maibach1998) cross tabulate satisfaction ratings with the level of education of I&A seekers, and Lechner and De Vries (Reference Lechner and De Vries1996) find that those with higher levels of education are over-represented in the study.
Wider structural factors relating to the area in which I&A was imparted were not discussed in relation to research findings. Instead, such issues form the basis of some limited discussion in the introduction to reported findings. However, considering the health, and treatment focus on what I&A was imparted, this is perhaps less surprising than if the focus required a more complex array of participants such as wider social care packages or housing options.
Imparted information and advice
Using the definition of I&A offered by Margiotta and colleagues (Reference Margiotta, Raynes, Pagidas, Lawson and Temple2003) earlier, there is a clear distinction between the generic material that characterises information, and the bespoke guidance that defines advice.
With motives for seeking I&A often tending to be multiple and or personal and complex, with imparted support reflecting these motives, it is reasonable to suggest that the types of information and or advice imparted will be similarly diverse. However, existing studies do not always outline what forms of I&A are imparted. For example, the details of the specific areas of I&A imparted, or what form they took (e.g. verbal or written) or whether what was imparted had the generic qualities of information or the bespoke characteristics of advice are not discussed in depth. Indeed, a lack of theoretical or any discussion around the nature of the I&A, as a causal mechanism and resource, means that it is not possible to contextualise reported outcomes or impact.
Mevissen et al. (Reference Mevissen, Eiling, Bos, Tempert, Mienties and Schaalma2012) present quantitative data that outlines the content of participants’ consultation with advisors, and Lechner and De Vries (Reference Lechner and De Vries1996) capture data on the clarity, usefulness, amount and overall quality of information. However, neither study describes the form or type of information and or advice that was imparted – either in quantitative categories or even as more specific case study examples.
The quantitative methods used in these studies have a limited focus on individual contexts, motives for seeking I&A and the form of I&A that is imparted. These studies also have significant limitations in outlining the impact of I&A on individual use. Therefore, it is not clear from these studies how I&A was used, what role it had and whether the imparted I&A acted as the dominant causal mechanism in relation to the reported outcomes and impact – thus negating some of the logic behind the methods used and possibly requiring additional layers of complexity in research design to effectively explore these areas.
Agency: impact, role and use of information and advice
This body of research does not satisfactorily examine how I&A is used and its role in determining associated outcomes. Ward et al. (Reference Ward, Baum, Ter Maat, Thomsen and Maibach1998) and Darrow et al. (Reference Darrow, Speyer, Marcus, Ter Maat and Krome1998) present findings on the helpfulness of imparted I&A. Olofinjana et al. (Reference Olofinjana, Connolly and Taylor2009) make some attempt at capturing quantitative data on what other sources were used, but only in the event of the helpline being unavailable. None adequately ascertain how imparted I&A was used, in relation to agency practices, and the role agency practices in relation to the imparted information and or advice had in shaping reported outcomes.
The use and role of I&A giving tend not to be subject to data collection, and by implication any critique is based around a lack of enquiry and evidence. However, it is worth discussing the assumptions that not focusing on the use of I&A and its role in outcomes makes about human agency and the reported outcomes.
Need for deeper enquiry and qualitative methods around agency
By not exploring how participants used the imparted information, and or advice, and what specific role it had in their decision-making and outcome, existing studies do not fully address the possibility that other sources of information, and or advice, or other resources and mechanisms (such as the support of friends or family) may have made a significant contribution to the reported outcome or impact. This uncertainty is an area that should be addressed in further research.
There are two possible and different implications for how such studies are designed. Firstly, study designs are based on the proposition that the individuals who seek I&A are detached from any other influences. However, this proposition seems unlikely and in need of further exploration. More likely is the conjecture that agency practices are objective and that agency based on different circumstances and different forms of information and or advice is objective (Bryman, Reference Bryman2016). This assumption is not underpinned by evidence, but clearly influenced by rational choice theorists such as Becker (Reference Becker1976), who outlines that, in his view, emotional and non-emotional decisions are conceptually the same, but that emotive decisions require a greater investment of information. Implicit in this notion is that individuals behave with a significant degree of uniformity and do not behave differently depending on their circumstances, motive for enquiring or the form of I&A that is imparted.
However, this proposition does not fully consider how individual agency practices, I&A use and, ultimately, the efficacy of I&A will vary, or differ, depending on different economic, social, political or cultural contexts (Bourdieu, Reference Bourdieu1999; Collet, Reference Collet2009). Nor does it fully consider different motives for seeking I&A, different individual practices and interactions with different forms of information and or advice or other yet to be determined nuances – or even whether the formal imparted I&A was used at all (Turnpenny and Beadle-Brown, Reference Turnpenny and Beadle-Brown2014).
While the assumptions about agency in these studies are based on rational choice theory as espoused by Becker (Reference Becker1976), the critique outlined here is drawn from a different view of agency. Without abandoning the idea that rationality underlies behaviour and agency, bounded rationality accommodates the role of uncertainty, context, individual taste, competencies and agency practices (Jones, Reference Jones1999). Bourdieu's ‘habitus’ represents a type of bounded rationality that offers a behavioural model, which Collet (Reference Collet2009) suggests is suited to framing complex and disruptive events in economic sociology and related fields.
Although ‘habitus’ is itself complex, it can be summarised as the culmination of subjective individual dispositions (with particular primacy given to early experiences) and contextual conditions (Collet, Reference Collet2009). The disposition element of habitus refers to the lasting or enduring dispositions and capacities posited by socialised norms and integrated into individual behaviours, the physical embodiment of social, symbolic and cultural capital (Bourdieu, Reference Bourdieu1999). This can change or be affected by unexpected situations or by wider socio-cultural developments or specific economic, social or cultural conditions in the field of practice (Navarro, Reference Navarro2006).
This view of agency suggests that it is unlikely that people share the same predispositions, motivations and behave uniformly; ‘habitus . . . makes it possible to produce an infinite number of practices that are relatively unpredictable’ (Bourdieu, Reference Bourdieu1999: 55). This is a proposition that requires further empirical enquiry.
On this basis, there is a need for greater focus on individual practices and processes, or what Bourdieu (Reference Bourdieu1999) terms ‘habitus’, in I&A giving. For example, in what is a literal illustration of the significance of early experiences in Bourdieu's concept of ‘habitus’, Baxter and Glendinning (Reference Baxter and Glendinning2011) find that earlier life experiences of using health and social care often act to give older people an ‘insider knowledge’ advantage and increased confidence in finding and acting on health and social care information.
Even within the studies subject to discussion in this article, there is an acknowledgement of diversity inherent in agency practices, and, importantly, that quantitative methods alone are inadequate when focusing on I&A giving services. When discussing study limitations, Olofinjana et al. (Reference Olofinjana, Connolly and Taylor2009) suggest that reporting quantitative outcome measures alone is a weakness of their study. In their call for the inclusion of qualitative methods, the authors emphasise that eliciting data on an individual's narrative or individual processes is important in contextualising and understanding outcomes:
open questions would have allowed helpline staff to further explore with callers the outcomes of helpline use. This could have provided a more thorough analysis of outcome given the narrative nature of each caller's history. (Olofinjana et al., Reference Olofinjana, Connolly and Taylor2009: 367)
Despite its acknowledged importance in the study and in the wider literature (see above), what Olofinjana et al. (Reference Olofinjana, Connolly and Taylor2009: 367) describe as ‘the narrative nature of caller's history’ is not captured in quantitative studies. On this basis, it is not possible to adequately highlight or report on the efficacy, effectiveness or impact of I&A that is subject to study.
Outcome and impact
It is worth noting that, despite being the central focus of study, definitions around outcome and impact are absent in existing research. There is a great deal of variation in the use of terminology, with little explanation or justification. Some definitional clarity of ‘outcome’ and ‘impact’ is required.
The organisation ‘Charities Evaluation Services’ describes an outcome as a change that happens as a consequence of a service and ‘impact’ as the broader effect, long-term or otherwise, of an outcome (Charities Evaluation Services, 2014). A similar definition is offered by the Centre for Nonprofit Management, which states that:
While the outcome evaluation tells us what kind of change has occurred, an impact evaluation paints a picture as to how a program might have affected participants’ lives on a broader scale. (Center for Nonprofit Management, 2014)
An ‘outcome’ in this sense is a finite and objective indicator of a change brought about by engaging with I&A. Although, as outlined above, by not exploring use or role of I&A in the outcome, there is little attention to exploring the extent to which purported outcomes can solely be attributed to the imparted I&A. For example, existing frameworks outline that an example of the impact of an outcome around engaging with I&A on welfare might be an increase in wellbeing or quality of life (Windle et al., Reference Windle, Netten, Caiels, Masrani, Welch and Forder2010).
Depending on the characteristics and nature of imparted support (e.g. whether objective outcomes are attributable to the I&A), a further distinction to outline is that, with an outcome associated with I&A being an objective indicator or measure of change, and with impact referring to the broader and subjective effect of an outcome, these concepts may lend themselves to opposing ontological and epistemological positions. For example, objective indicators may be more appropriately captured using quantitative methods. On the other hand, qualitative methods may better lend themselves to the broader and subjective characteristics of impact (Bryman, Reference Bryman2016). With these definitions and methodological distinctions in mind, it is illuminating that Lechner and De Vries (Reference Lechner and De Vries1996) and Venn et al. (Reference Venn, Darling, Dickens, Quine, Rutter and Slevin1996) choose to use the inherently subjective and qualitative terms ‘impact’ and ‘experiences’ when describing their quantitative studies.
A lack of discussion around definitions, and thus possibly understanding around key concepts, does little to illustrate the scope of reported quantitative outcome data. Yet, like the critique around existing studies inadequately focusing on the use and role of I&A in reported outcomes, similarly, most studies tend to be restrictive when reporting outcomes.
For example, while outcomes are reported, the impact of those outcomes is often not reported. Darrow et al. (Reference Darrow, Speyer, Marcus, Ter Maat and Krome1998) report that over a third of participants found additional support in their local community after engaging with a cancer I&A service. However, it is not reported what impact this had on individuals. Whilst this is assumed a positive finding, from the data presented it is not possible to tell whether the impact of finding local support was indeed positive. Wider methodological considerations are discussed in the following section.
Discussion: realigning from positivism to post-positivism and realism
The critical weakness in the design of existing studies is an inadequate focus on individual (e.g. socio-economic) and structural (e.g. external environment) contextual drivers of the agency practices and the causal processes that lead to outcomes. The studies reviewed in this paper may benefit from adopting a realist proposition that, while there is an objective reality independent of us knowing it, what it is to know (reality) is multi-layered, complex, and individual's engage interact, engage and experience this multi-layered and complex reality differently (Danermark et al., Reference Danermark, Ekstrom, Jakobsen and Karlsson2002). Existing studies do not adopt this stance because outcomes are reported almost with no consideration as to how they were arrived at. Outcomes, whilst being objectivist properties of knowledge, do not exist independently of causal processes that impact upon individual actions and wider external and structural factors.
The tenets of critical realism (Bhaskar, Reference Bhaskar1978, Reference Bhaskar1979) represent a position particularly suited to exploring the areas of research highlighted in this article because it acknowledges and focuses on uncovering the processes that lie behind objective forms of reality. It is critically important to adopt this mode of enquiry because, as Easton (Reference Easton2010: 121) states, ‘the most fundamental aim of critical realism is explanation; answers to the question “what caused those events to happen?”’. However, more specifically the need to focus on how I&A works, why it works (for some and not others) and in what circumstances requires an approach to research grounded in realist evaluation (Pawson and Tilley, Reference Pawson and Tilley1997).
Realist evaluation
Realist evaluation (Pawson and Tilley, Reference Pawson and Tilley1997) is an increasingly popular approach (Greenhalgh et al., Reference Greenhalgh, Wong, Jagosh, Greenhalgh, Manzano, Westhorp and Pawson2015) that proposes social programmes or interventions rest on the conceptual theory of ‘outcome = mechanism + context’. In other words, the outcomes of social programmes or interventions are a product of ‘mechanisms’ ‘firing’ in context (Pawson and Tilley, Reference Pawson and Tilley1997).
According to Pawson and Tilley (Reference Pawson and Tilley1997), central to mechanisms are the resources that are imparted to the participants of social programmes (e.g. I&A), individual reasoning and agency. Context refers to the conditions in which mechanisms function (e.g. possibly, and not limited to, individual circumstances and motives for seeking I&A and wider external and structural factors) and also relevant economic, social, political and cultural conditions of the field (Bourdieu, Reference Bourdieu1999). Based on (and not necessarily limited to) wider literature, wider theory, stakeholder consultations and or observations, realist research involves theorising around the contexts in which mechanisms fire. On this basis, thinking retroductively and inferring beyond data (Meyer and Lunnay, Reference Meyer and Lunnay2013), the purpose of data collection is to reject, refine or accept programme theory (Pawson and Tilley, Reference Pawson and Tilley1997). It is on this basis that realist evaluation provides an important frame for research in relation to efficacy of I&A, and more specifically why I&A works (and why not), how, for whom and in what contexts (Pawson and Tilley, Reference Pawson and Tilley1997). In doing so, the critical weaknesses of the studies discussed above may be addressed.
Methods
Pawson and colleagues have become increasingly prescriptive in their call for a combination of quantitative tools (as a means of delineating context patterns and observing outcomes) to compliment qualitative methods, with the latter recommended for ‘mining mechanisms’ (Pawson and Manzano-Santaella, Reference Pawson and Manzano-Santaella2012: 182). This prescriptive blueprint largely reflects a philosophical realignment from the general logic of realism and Bhaskar's early pioneering work on generative causation (1978) to empirical realism in Pawson's more recent work (Pawson, Reference Pawson2013, Reference Pawson2016a, Reference Pawson2016b, personal communication). However, realist evaluation as per Pawson and Tilley (Reference Pawson and Tilley1997), imposes no set blueprint in relation to methods. Indeed, others emphasise the importance of qualitative methods in relation to realist enquiry (Maxwell, Reference Maxwell2012) and realist evaluation (Jagosh et al., Reference Jagosh, Bush, Macaulay, Greenhalgh, Wong, Cargo, Green, Herbert and Pluye2015; Adams et al., Reference Adams, Sedalia, McNab and Sarker2016), particularly in relation to eliciting outcomes (Jagosh, Reference Jagosh2014).
Considering the diverse, complex and distinctively different areas of welfare, and corresponding I&A services, it is likely that different fields will necessitate the implementation of different methods. What follows are some key issues for consideration when choosing methods.
Firstly, it is unlikely that a researcher will be able to observe outcomes of a telephone I&A service as an empirical stance calls for (Pawson and Manzano-Santaella, Reference Pawson and Manzano-Santaella2012). However, generative mechanisms and possibly outcomes of telephone I&A, if not directly observable, may likely form the basis of agents’ theoretical accounts – something that is congruent with critical realism (Bryman, Reference Bryman2016).
Secondly, it is important to consider the area of welfare that the I&A service focuses on. While reviewed studies in this article tend to have a health or medical focus, where the efficacy of I&A might be able to be more objectively defined (e.g. around measurable health outcomes), this is less likely to be the case in other areas of welfare (e.g. such as wider social support or housing). In the latter, individual experiences are much more likely to include much wider I&A seeking (beyond the service under study), possibly from multiple sources both formal and informal (Turnpenny and Beadle-Brown, Reference Turnpenny and Beadle-Brown2014; Burgess and Morrison, Reference Burgess and Morrison2015), and the scope and breadth of imparted formal I&A (from the service under study) is likely to include multiple options. On this basis, outcomes may not be clearly measurable (like a health outcome), but could be based around more focused and informed agency (e.g. around different for social support or housing) thus being less clearly defined and appropriate for the use of qualitative methods.
Thirdly, it will be important to consider the design and characteristics of the I&A mechanism, or the nature of the intervention and what exactly is imparted. For example, it might be important to consider whether imparted I&A is able to give seekers a definitive answer or attain a definitive or objective outcome, or whether I&A is non-definitive and only able to inform agency, or a continued and perhaps more focused search. It is useful to use an example to illustrate such differences. For example, in relation to retirement housing, definitive information might include advice that clarifies an individual's position, or information on vacancies, cost, eligibility criteria or other types of information that may lead to a definitive outcome. On the other hand, non-definitive information would not include the aforementioned areas and may, for example, take the form of a directory of possible housing providers. The former, where definitive outcomes may be attributable to the I&A might suit quantitative outcome measures, but the latter non-definitive example might require deeper qualitative inquiry around ascertaining how, why, for whom and in what circumstances in relation to efficacy.
Conclusion
This article has proposed a theoretical framework which highlights key areas of focus to delineate the efficacy of I&A services. Using the framework to discuss the strengths and weaknesses of existing research on the outcomes and impact of telephone I&A, this article has found that a current critical weakness is a lack of expansive data collection and a lack of focus on how mechanisms and agency practices operate in context. In terms of further research, a realist evaluation theoretical perspective, and the role of mixed and multiple (and particularly qualitative) methods is suited to addressing these areas.
The use of markets to allocate resources, in relation to the responsiveness of providers, is only as successful as the extent with which consumers are able to make informed and good quality choices. In turn, considering complex information asymmetries, informed choices for many will likely rest on engagements with formal I&A services. An enhanced understanding around what forms of I&A are more effective – how consumers act when seeking and acting on I&A, why they act like they do and in what circumstances I&A enables high quality decision-making and outcomes – is a robust approach to ensuring services are responsive, which then enables consumers to access welfare provision that reflects their needs.
