Origins of this book

This book is the third in a series of critical reflections on the state of statistics supported by the Radical Statistics Group, following on from Demystifying Social Statistics (Irvine et al, 1979) and Statistics in Society (Dorling and Simpson, 1999). Both earlier books were mainly concerned with UK official statistics, as tools for understanding and sometimes changing the economy and society. In the 20 years since Statistics in Society appeared, a far wider range of data sources have gained influence, so this book must be concerned not just with ‘official statistics’ but with administrative data from many different parts of the state, data produced and owned by companies, and data ‘harvested’ from social media.

Our language has also changed. We now often talk about ‘data’ produced by data scientists, not ‘statistics’ produced by statisticians. Traditional statistical methods are grounded in probability theory, which assumes data are random samples, or ‘behave’ similarly, so requiring formal hypothesis testing. The new data science ignores much of this and requires instead a familiarity with computer programming, often accessing live ‘data feeds’ via ‘APIs’ (Application Programming Interfaces) such as those provided by Twitter – or Transport for London. Rather than test hypotheses, these new approaches create machine learning models, using methods which minimise the importance of representative sampling, and the distinction between causal relations and mere correlations.

This Introduction, and the chapters in this book, relate these and other changes in statistics and data to key trends in society. In these we include the 2007–08 financial crisis; the subsequent ‘austerity’ regimes in the UK and elsewhere; uneven economic and financial performance; crises of governance at local, national and international levels; and longer term trends of increasing inequality and deepening globalisation.

We introduce individual chapters at the start of each Part. In Part I, we consider recent challenges to traditional ways of doing statistics, sources of data and methods of analysis. Part II considers some consequences of an increasingly globalised world. Parts III and IV review changes in how the state functions, and in the economy. Part V covers changes in health and health policy.