Skip to main content

Methods for the economic evaluation of changes to the organisation and delivery of health services: principal challenges and recommendations

  • Rachel Meacock (a1)

There is a requirement for economic evaluation of health technologies seeking public funding across Europe. Changes to the organisation and delivery of health services, including changes to health policy, are not covered by such appraisals. These changes also have consequences for National Health Service (NHS) funds, yet undergo no mandatory cost-effectiveness assessment. The focus on health technologies may have occurred because larger-scale service changes pose more complex challenges to evaluators. This paper discusses the principal challenges faced when performing economic evaluations of changes to the organisation and delivery of health services and provides recommendations for overcoming them. The five principal challenges identified are as follows: undertaking ex-ante evaluation; evaluating impacts in terms of quality-adjusted life years; assessing costs and opportunity costs; accounting for spillover effects; and generalisability. Of these challenges, methods for estimating the impact on costs and quality-adjusted life years are those most in need of development. Methods are available for ex-ante evaluation, assessing opportunity costs and examining generalisability. However, these are rarely applied in practice. The general principles of assessing the cost-effectiveness of interventions should be applied to all NHS spending, not just that involving health technologies. Advancements in this area have the potential to improve the allocation of scarce NHS resources.

Corresponding author
*Correspondence to: Rachel Meacock, Research Fellow in Health Economics, Manchester Centre for Health Economics, The University of Manchester, 4.311 Jean McFarlane Building, Oxford Road, Manchester M13 9PL, UK. Email:
Hide All
Anderson, R. and Hardwick, R. (2016), ‘Realism and resources: towards more explanatory economic evaluation’, Evaluation, 22: 323341.
Appleby, J. (2016), ‘A 7/7 NHS: what price equity?’, BMJ, 352: i404.
Asaria, M., Griffin, S. and Cookson, R. (2016), ‘Distributional cost-effectiveness analysis’, Medical Decision Making, 36: 819.
Barratt, H., Campbell, M., Moore, L., Zwarenstein, M. and Bower, P. (2016), ‘Randomised controlled trials of complex interventions and large-scale transformation of services’, in R. Raine, R. Fitzpatrick, H. Barratt, G. Bevan, N. Black, R. Boaden, et al. (eds), Challenges, solutions and future directions in the evaluation of service innovations in health care and public health, Health Services Delivery Research, 4(16): 19–36.
Claxton, K., Martin, S., Soares, M., Rice, N., Spackman, E., Hinde, S., Devlin, N., Smith, P. and Sculpher, M. (2015), ‘Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold’, Health Technology Assessment, 19: 1–503.
Congressional Budget Office (2007), ‘CBO’s Health Insurance Simulation Model: A Technical Description’, Congressional Budget Office, Washington, DC.
Department of Health (2008), ‘Guidance on the routine collection of patient reported outcome measures (PROMs)’, Department of Health, London.
Drummond, M., Griffin, A. and Tarricone, R. (2009), ‘Economic evaluation for devices and drugs – same or different?’, Value of Health, 12: 402404.
Drummond, M., Sculpher, M., Claxton, K., Stoddart, G. and Torrance, G. (2015), Methods for the Economic Evaluation of Health Care Programmes, 4th edn, Oxford: Oxford University Press Inc.
Finkelstein, A. and Taubman, S. (2015), ‘Randomize evaluations to improve health care delivery’, Science, 347: 720722.
Gillies, C., Freemantle, N., Grieve, R., Sekhon, J. and Forder, J. (2016), ‘Advancing quantitative methods for the evaluation of complex interventions’, in R. Raine, R. Fitzpatrick, H. Barratt, G. Bevan, N. Black, R. Boaden, et al. (eds), Challenges, solutions and future directions in the evaluation of service innovations in health care and public health, Health Services Delivery Research, 4(16): 37–54.
Gomes, M., Gutacker, N., Bojke, C. and Street, A. (2016), ‘Addressing missing data in patient‐reported outcome measures (PROMS): implications for the use of PROMS for comparing provider performance’, Health Economics, 25: 515528.
Graves, N., Walker, D., Raine, R., Hutchings, A. and Roberts, J.A. (2002), ‘Cost data for individual patients included in clinical studies: no amount of statistical analysis can compensate for inadequate costing methods’, Health Economics, 11: 735739.
Hunter, R. M., Davie, C., Rudd, A., Thompson, A., Walker, H., Thomson, N., Mountford, J., Schwamm, L., Deanfield, J., Thompson, K., Dewan, B., Mistry, M., Quoraishi, S. and Morris, S. (2013), ‘Impact on clinical and cost outcomes of a centralized approach to acute stroke care in London: a comparative effectiveness before and after model’, PLoS One, 8: e70420.
Kristensen, S., Meacock, R. and Sutton, M. (2015), ‘Methodology for Modelling Changes in Health Services Delivery and Policy’, Vignette prepared for the MRC-NIHR Methodology Research Programme Advisory Group, London.
Lamont, T., Barber, N., Pury, J., de, Fulop, N., Garfield-Birkbeck, S., Lilford, R., Mear, L., Raine, R. and Fitzpatrick, R. (2016), ‘New approaches to evaluating complex health and care systems’, BMJ, 352: i154.
Lilford, R. J., Chilton, P. J., Hemming, K., Girling, A. J., Taylor, C. A. and Barach, P. (2010), ‘Evaluating policy and service interventions: framework to guide selection and interpretation of study end points’, BMJ, 341: c4413.
Maynard, A. (2012), ‘The powers and pitfalls of payment for performance’, Health Economics, 21: 312.
Meacock, R., Kristensen, S. R. and Sutton, M. (2014), ‘The cost-effectiveness of using financial incentives to improve provider quality: a framework and application’, Health Economics, 23: 113.
Meacock, R., Doran, T. and Sutton, M. (2015), ‘What are the costs and benefits of providing comprehensive seven-day services for emergency hospital admissions?’, Health Economics, 24: 907912.
Meacock, R., Sutton, M., Kristensen, S. R. and Harrison, M. (2016), ‘Using survival analysis to improve estimates of life year gains in policy evaluations’, Medical Decision Making 37(4): 415–426.
Medical Research Council (2008), ‘Developing and evaluating complex interventions: new guidance’, Medical Research Council, London.
Medical Research Council (2009), ‘Using natural experiments to evaluate population health interventions: guidance for producers and users of evidence’, Medical Research Council, London.
National Institute for Health and Care Excellence (NICE) (2013), Guide to the methods of technology appraisal 2013. The-reference-case, [31 March 2016].
National Institute for Health and Care Excellence (NICE) (2014), Guide to the processes of technology appraisal, [30 November 2016].
Raine, R., Fitzpatrick, R., Barratt, H., Bevan, G., Black, N., Boaden, R., Bower, P., Campbell, M., Denis, J.-L., Devers, K., Dixon-Woods, M., Fallowfield, L., Forder, J., Foy, R., Freemantle, N., Fulop, N. J., Gibbons, E., Gillies, C., Goulding, L., Grieve, R., Grimshaw, J., Howarth, E., Lilford, R. J., McDonald, R., Moore, G., Moore, L., Newhouse, R., O’Cathain, A., Or, Z., Papoutsi, C., Prady, S., Rycroft-Malone, J., Sekhon, J., Turner, S., Watson, S.I. and Zwarenstein, M. (2016), ‘Challenges, solutions and future directions in the evaluation of service innovations in health care and public health’, Health Services and Delivery Research, NIHR Journals Library, Southampton.
Reed Johnson, F. (2009), ‘Editorial: moving the QALY forward or just stuck in traffic?’, Value in Health, 12: S38S39.
Sanders, G. D., Neumann, P. J., Basu, A., Brock, D. W., Feeny, D., Krahn, M., Kuntz, K. M., Meltzer, D. O., Owens, D. K., Prosser, L. A., Salomon, J. A., Sculpher, M. J., Trikalinos, T. A., Russell, L. B., Siegel, J. E. and Ganiats, T. G. (2016), ‘Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine’, JAMA, 316: 10931103.
Shah, K., Praet, C., Devlin, N., Sussex, J., Appleby, J. and Parkin, D. (2012), ‘Is the aim of the English health care system to maximize QALYs?’, Journal of Health Services Research & Policy, 17: 157163.
Shiell, A., Hawe, P. and Gold, L. (2008), ‘Complex interventions or complex systems? Implications for health economic evaluation’, BMJ, 336: 12811283.
Smith, M. D., Drummond, M. and Brixner, D. (2009), ‘Moving the QALY forward: rationale for change’, Value in Health, 12: S1S4.
Sutton, M., Elder, R., Guthrie, B. and Watt, G. (2010), ‘Record rewards: the effects of targeted quality incentives on the recording of risk factors by primary care providers’, Health Economics, 19: 113.
Tunis, S. R., Stryer, D. B. and Clancy, C. M. (2003), ‘Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy’, JAMA, 290: 16241632.
Turner, S., Goulding, L., Denis, J.-L., McDonald, R. and Fulop, N. J. (2016), ‘Major system change: a management and organisational research perspective’, in R. Raine, R. Fitzpatrick, H. Barratt, G. Bevan, N. Black, R. Boaden, et al. (eds), Challenges, solutions and future directions in the evaluation of service innovations in health care and public health, Health Services Delivery Research, 4(16): 85–104.
Yao, G. L., Novielli, N., Manaseki-Holland, S., Chen, Y.-F., Klink, M., van der, Barach, P., Chilton, P. J. and Lilford, R. J. (2012), ‘Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers’, BMJ Quality & Safety, 21: i29i38.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Health Economics, Policy and Law
  • ISSN: 1744-1331
  • EISSN: 1744-134X
  • URL: /core/journals/health-economics-policy-and-law
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 13
Total number of PDF views: 132 *
Loading metrics...

Abstract views

Total abstract views: 736 *
Loading metrics...

* Views captured on Cambridge Core between 20th April 2018 - 21st August 2018. This data will be updated every 24 hours.