1 Introduction
Over 40 years ago, the aerospace engineering sector made a strategic decision to develop the capability to predict the performance of different design options for their new jet engines using computer simulation techniques (see Blazek Reference Blazek2015, for the full history summarized here). Their logic was simple – if they could simulate alternative design options in silico, they would reduce the number of alternative designs built and tested in expensive test-beds, making substantial savings and reducing development times. The expectation was that such simulations would not be immediately beneficial; rather this was a long-term investment with pay-offs promised only at some future date, well beyond the job tenure of the people who initiated the investment. In practice, this initial investment in simulation capability paid off. Forty years on, the sector can now simulate with high accuracy the performance of alternative future engine designs.
Our work described here uses the same medium- to long-term logic. Thus, we ask whether we can develop predictive capability regarding the performance of complex systems involving both human and technical factors (i.e., socio-technical systems; see e.g., Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014). The expectation being that this may pay off in some (as yet unspecified) medium- to long-term future. In the aerospace sector, above, developing simulation capability for jet engines requires contributions from several engineering disciplines – which understand the behaviours of technical systems and sub-systems – and computer scientists – with the necessary computational expertise. Thus, developing predictive capability for complex socio-technical systems requires the application of organizational psychology as a design science, to harness its understanding of human behaviour and cognition in work environments.
Accordingly, our long-term goal is to predict performance in complex socio-technical systems so that organizational effectiveness, efficiency, and safety can be optimized, and the risks of severe malfunctions can be designed, led, or managed out. Our work to date has focused on system failures, primarily because the potential for immediate impact is high and the necessary data are readily available. Thus, this paper introduces a new method to predict such failures and illustrates its application to two case studies: one attempting to predict breakdowns in patient care within the UK’s National Health Service (NHS), representing chronic malfunctions, and one attempting to predict where accidents may occur on the UK’s rail network, representing acute malfunctions.
There are three wider strands to our argument. First, design science has devoted less effort to studying human behaviour in socio-technical systems than it has to product development and associated technical processes; so, applying organizational psychology will greatly benefit design science. Second, like other social sciences, organizational psychology has not invested much energy in trying to develop predictive capabilities in this area or more generally; so, striving to design systems and processes upfront rather than responding reactively to problems will greatly benefit organizational psychology. Third, what little predictive work there is about human behaviour in socio-technical systems has been dominated by human factors and ergonomics specialists, focusing on interactions between technical and human parts of such systems (see e.g., Stanton et al. Reference Stanton, Salmon, Harris, Marshall, Demagalski, Young, Waldmann and Dekker2009). Unfortunately, though, there has been little input from design scientists or organizational psychologists to date. This has limited progress as organizational psychology could support whole system perspectives including human, organizational, and technical elements. We are therefore advocating that organizational psychology has much to offer as a design science, by predicting where malfunctions may occur in socio-technical systems and how to design these faults out.
A key benefit of applying organizational psychology as a design science is that it would lead to increased opportunities for organizations to consider human behaviour at an earlier stage in the development of new systems and/or the re-design of existing systems (Simon Reference Simon1996; Hevner Reference Hevner2007), working proactively rather than responding reactively to problems that have already occurred (e.g., Clegg & Shepherd Reference Clegg and Shepherd2007). This would also add value by incorporating well-founded theories of human behaviour into design processes. In the longer term, this could create opportunities for bringing advanced engineering design methods and tools, such as visualization and optimization, into organizational psychology and the design of socio-technical systems. Such attempts will make new empirical and theoretical demands, encouraging us to ask new challenging questions and work with new clients. Consequently, we believe that attempts to develop predictive capabilities will be good for the development of both design science and organizational psychology.
As such, the objectives of this paper are to: (1) introduce a method for predicting malfunctions in socio-technical systems, (2) describe examples of the variables that form the core of the predictive models and the kinds of data required, (3) present a multi-stage model of how the approach works in practice, (4) discuss the major advantages and disadvantages of the method, (5) give a preliminary assessment of future prospects for the method, and (6) argue that organizational psychology should be applied as a design science to engage in new predictive paradigms for socio-technical systems, developing and testing new methods and data analytic techniques.
We start, however, by providing an overview of organizational psychology and socio-technical systems, to illustrate their applicability to design science.
1.1 An overview of organizational psychology and socio-technical systems
Organizational psychology examines human behaviour and cognition in work contexts, both individually and collectively, and includes a dual focus on performance and wellbeing (Spector Reference Spector2008). While the discipline has made great progress in examining and understanding these areas (Patterson Reference Patterson2001), most organizational psychology research and practice focuses primarily on the people themselves and often neglects the broader contextual aspects of work such as processes and technology. Consequently, we view the predictive method introduced in this paper as a step towards expanding this narrow focus, to enable organizational psychology to be applied as a design science. To do so, we draw on socio-technical systems theory as a unifying framework to enable organizational psychology to be applied to a broader range of topics and domains.
Although the term ‘socio-technical’ was used sporadically during the early 20th century (e.g., Kayden Reference Kayden1920), the foundations of socio-technical systems theory lie in the work of the UK’s Tavistock Institute of Human Relations during the 1950s (Trist Reference Trist1953; Emery Reference Emery1959). Here, psychologists examined the teamwork of miners and the complex technical processes they used to extract coal deep underground (Trist & Bamforth Reference Trist and Bamforth1951). This dual focus on human and technical aspects of work continued through the 1960s in various guises, such as Leavitt’s (Reference Leavitt1964) conceptualization of organizations as a system comprising people interacting with tasks, technology, and structures. However, it was Cherns’ (Reference Cherns1976; Reference Cherns1987) landmark papers which arguably established modern socio-technical theory, proposing a number of principles for effective system design such as aligning a system with its objectives, ensuring it facilitates the desired behaviours, and providing timely and targeted information.
These socio-technical principles were subsequently updated for the digital age by Clegg (Reference Clegg2000) – to address meta, content, and process design – who also developed the hexagonal socio-technical framework we use here in our predictive method (Clegg Reference Clegg1979; Challenger, Clegg & Robinson Reference Challenger, Clegg and Robinson2010; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014), as shown in Figure 1. This hexagonal socio-technical framework conceptualizes work as a complex system, comprising both socio elements – people, culture, and goals (the left side of Figure 1) – and technical elements – technology, infrastructure, and processes (the right side of Figure 1). Changes in any one element, or node, will cause and necessitate changes elsewhere in the system due to its complex interactive nature, as illustrated by the nodes’ interconnecting lines representing causal relationships. Such a system is complex as it comprises multiple elements interacting in multiple ways, often concurrently, to yield non-linear outputs (Choi, Dooley & Rungtusanatham Reference Choi, Dooley and Rungtusanatham2001).
Hexagonal socio-technical systems framework (adapted from Clegg Reference Clegg1979; Challenger et al. Reference Challenger, Clegg and Robinson2010; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014).
The framework can be used to represent any socio-technical system, at a micro (individual), meso (team), and/or macro (organization and/or entire industry) level of analysis (Klein & Kozlowski Reference Klein and Kozlowski2000), as the following examples show, respectively. For instance, socio-technical theory has been applied to understand the interactions between individuals and computer interfaces (González-Torres, García-Peñalvo & Therón Reference González-Torres, García-Peñalvo and Therón2013), engineering team work (Crowder et al. Reference Crowder, Robinson, Hughes and Sim2012), pro-environmental initiatives in a manufacturing organization (Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014), and regulatory change in energy industries (Verbong & Geels Reference Verbong and Geels2007). The capability of socio-technical systems theory to integrate these various socio and technical elements, both within and across these different levels of analysis (e.g., individuals nested within teams, which in turn are nested within organizations) (Klein & Kozlowski Reference Klein and Kozlowski2000; Verbong & Geels Reference Verbong and Geels2007), makes it ideally suited to examining complex systems where multiple components interact across different hierarchical levels in often seemingly unpredictable ways (Johnson Reference Johnson2009).
Within socio-technical research, there are two broad perspectives. The first perspective (‘Big-S Socio-technical’), as described above, largely extends its social science foundations by focusing mainly on the human or socio elements of complex systems and is largely undertaken by psychologists, human factors specialists, and other social scientists. The second perspective (‘Big-T socio-Technical’) focuses mainly on the technical elements of complex systems and is largely undertaken by engineers, computer scientists, and other technical specialists. For instance, de Weck, Roos & Magee (Reference de Weck, Roos and Magee2011) view socio-technical systems as engineering systems – exhibiting both technical and social complexity – focused on meeting societal needs such as energy, transport, communication, health, and education. They argue that the performance and impact of such systems can be evaluated with reference to various ‘-ilities’, such as quality, safety, reliability, flexibility, and maintainability. Similarly, others have focused on designing engineering systems to be both resilient and robust to withstand changes in their operating environment (Pavard et al. Reference Pavard, Dugdale, Saoud, Darcy and Salembier2006).
There is some debate about whether these two different perspectives on socio-technical systems theory can be reconciled. For instance, Kroes et al. (Reference Kroes, Franssen, van de Poel and Ottens2006) debate whether humans can be considered integral parts of an engineering system itself or merely actors in the environment in which the system operates. They further note the conceptual difficulties of integrating humans into such systems due in part to the complexities of technical engineering systems. However, many of these debates are likely to reflect disciplinary biases. For instance, research shows that specialists often perceive more complexity in their own disciplinary topics than those of other disciplines (Sagie & Magnezy Reference Sagie and Magnezy1997). Furthermore, for every psychologist arguing for a greater focus on human elements in socio-technical systems (Challenger et al. Reference Challenger, Clegg and Robinson2010), there is an engineer arguing for more focus on the technical elements (Coiera Reference Coiera2007). Ultimately, a pragmatic approach is best so researchers should select levels of theoretical abstraction that are both equivalent and understandable to specialists from either perspective (e.g., component function rather than fluid dynamics for technical aspects of the system, and human behaviour rather than cerebral synapses for the socio aspects). Indeed, many see no such conceptual problems with socio-technical systems and praise the advantages of such an integrated approach (Mumford Reference Mumford2000).
Finally, not only does socio-technical systems theory provide a bridge between organizational psychology and the analysis of malfunctions in complex systems, but it also aligns closely with the discipline of design science. Indeed, the pioneering book The Sciences of the Artificial by Simon (Reference Simon1969), the eminent psychologist and computer scientist, is widely acknowledged as the foundation of design science (Huppatz Reference Huppatz2015; see also the debate between editors in the inaugural issue of the journal Design Science – Papalambros et al. Reference Papalambros, Ahmed-Kristensen, Badke-Schaub, Bauerly, Boujut, Cagan and Gero2015). Furthermore, in his acceptance speech for the 1978 Nobel Prize in Economics, Simon argued for inter-disciplinary research involving behavioural and technical sciences, of the type we advocate here, stating ‘…all the great problems that face our world today have both technical and human content – the one intermingled inseparably with the other’ (Simon Reference Simon1978).
Indeed, the optimal design of jobs and tasks, for performance and wellbeing, has been a fundamental topic of interest to organizational psychologists for the last half century (Hackman & Oldham Reference Hackman and Oldham1976; Parker Reference Parker2014). Therefore, broadening this focus to examine the role of human behaviour in the design of whole socio-technical systems would be a natural progression. Furthermore, considering the structure, function, and behaviour of such systems – and its human actors – upfront in this way, proactively, would enable performance and safety to be predicted and therefore optimized, thereby designing out or mitigating the risk of malfunctions. In so doing, this approach would enable organizational psychology to be applied as a true design science (see e.g., Papalambros et al. Reference Papalambros, Ahmed-Kristensen, Badke-Schaub, Bauerly, Boujut, Cagan and Gero2015).
1.2 Previous research examining malfunctions in socio-technical systems
To understand how we might develop the capability to predict malfunctions within complex systems, and therefore identify actions to prevent their occurrence, we first consider where such approaches have been employed most effectively to date. There is a strong history within human factors and ergonomics of articulating post-event analyses and explanations of disasters (see Hall & O’Day Reference Hall and O’Day1971; Turner & Pidgeon Reference Turner and Pidgeon1978; Johnson Reference Johnson1980; Gherardi et al. Reference Gherardi, Turner, Pidgeon and Ratzan1999). This tradition has evolved to emphasize more systemic and combinatorial explanations whereby accidents are seen to arise as the result of unusual combinations of circumstances (see e.g., Perrow Reference Perrow1984; Taylor Reference Taylor1989, Reference Taylor1990; Reason Reference Reason1990; Stanton et al. Reference Stanton, Salmon, Harris, Marshall, Demagalski, Young, Waldmann and Dekker2009; Salmon, Cornelissen & Trotter Reference Salmon, Cornelissen and Trotter2012; Underwood & Waterson Reference Underwood and Waterson2014).
Supporting this perspective, several useful methods have been developed for post-event analysis, including for example: AcciMap (Rasmussen Reference Rasmussen1997; Grant, Goode & Salmon Reference Grant, Goode and Salmon2015), Functional Resonance Analysis Method (FRAM) (Hollnagel & Goteman Reference Hollnagel and Goteman2004), Systems Theoretic Accident Modelling and Processes Model (STAMP) (Leveson Reference Leveson2004), systems dynamics simulation (e.g. Cooke Reference Cooke2003), and causal loop diagrams (e.g., Goh, Brown & Spickett Reference Goh, Brown and Spickett2010). A comparison and more detailed consideration of such methods can be found in Underwood & Waterson (Reference Underwood and Waterson2014).
Similarly, organizational psychologists adopting a socio-technical framework have analysed a number of accidents and disasters involving human behaviour, including the Hillsborough football stadium disaster, King’s Cross underground station fire, and Bradford City Football Club fire (see Challenger & Clegg Reference Challenger and Clegg2011; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014). Despite these contributions by organizational psychologists to post-event analyses, however, it must be acknowledged that this field has been dominated to date by human factors and ergonomics specialists.
Yet, as we shall see later, issues and variables that are core aspects of socio-technical systems design and organizational psychology are central to an understanding of such malfunctions. For example, the job designs and working practices of frontline staff in the Mid-Staffordshire NHS Foundation Trust hospital and of the maintenance workers in Network Rail are key elements in understanding the failures examined in the two case studies presented in this paper. Similarly, there is evidence that inappropriate or poor leadership are factors in many disasters and malfunctions (see e.g., Challenger & Clegg Reference Challenger and Clegg2011). We therefore argue that, as a design science, organizational psychology has a direct and important contribution to make to our understanding of these systemic malfunctions, not only retrospectively but proactively, in a predictive fashion, to design them out. Indeed, it would be surprising if the discipline did not have something to offer these problem domains.
We turn now to work focused on predicting where malfunctions may occur in complex systems. There has been relatively little energy invested in this domain and we can speculate on the underlying reasons (see Gherardi et al. Reference Gherardi, Turner, Pidgeon and Ratzan1999, for a fuller discussion of potential contributing factors). First, the low frequency of such events can mean that they appear to occur almost randomly in large distributed and complex systems. Second, the dominant theoretical ideas and underlying mindsets stress their combinatorial origins, whereby a set of unique factors come together in statistically unlikely ways (see e.g., Perrow Reference Perrow1984; Reason Reference Reason1990). It could be argued, therefore, that this has dissuaded most experts from pursuing predictive capability, instead encouraging the development of wider organizational strategies focusing on general-purpose prevention, such as developing high reliability organizations (see Rochlin Reference Rochlin1986; Weick Reference Weick1987). However, with a predictive capability and the underlying science, practitioners could identify when a complex socio-technical system’s operation was approaching a malfunction, akin to condition-based real-time jet engine monitoring systems which inform maintenance decisions (see e.g., Kobayashi & Simon Reference Kobayashi and Simon2007).
Nevertheless, there have been recent attempts to develop some predictive capabilities in this area, largely building on the post-event methods discussed earlier, such as STAMP (Rong & Tian Reference Rong and Tian2015), and AcciMap (Salmon et al. Reference Salmon, Read, Stanton and Lenné2013). While these approaches show promise, they focus insufficiently on the ways in which people’s behaviour is influenced and shaped by the organizational contexts in which they work. For instance, of the six system levels typically considered by AcciMap analyses, only one explicitly focuses on micro human behaviour and even then only partially and at a relatively high level of abstraction (e.g., ‘actor activities’ such as communication failures) (Debrincat, Bil & Clark Reference Debrincat, Bil and Clark2013). Our aspiration, therefore, is for the widespread adoption of socio-technical system design methods and tools that enable the design of systems with more predictable performance characteristics, arising from both inherent characteristics of the systems themselves and the ways in which they are operated.
Unsurprisingly perhaps, given the above, work by social scientists in this predictive area has again been dominated by the same specialists in human factors and ergonomics, with design scientists and organizational psychologists not participating. For rare exceptions see Challenger & Clegg (Reference Challenger and Clegg2011), who used a socio-technical model to predict what might go wrong ahead of the London 2012 Olympic Games, and Ridgway et al. (Reference Ridgway, Clegg, Williams, Hourd, Robinson, Bolton, Cichomska and Baldwin2013) who applied a similar socio-technical approach to forecasting requirements for future factories. It should also be acknowledged briefly that psychologists have undertaken some predictive research in other specialist domains, such as predicting future job performance through various personnel selection tests (Schmidt & Hunter Reference Schmidt and Hunter1998) and identifying future competency requirements in response to business changes (Robinson et al. Reference Robinson, Sparrow, Clegg and Birdi2005). However, this other predictive research examines much simpler and more specific scenarios than the behaviour of the complex socio-technical systems examined here.
Essentially, we are adopting a prototyping-based research method, typically used in design research, to explore what might be possible. Thus in part, this becomes an experiential and empirical issue and we will learn best the limits to predictive work by attempting it. We see this as equivalent to debates on the extent to which we are able to computer-model and simulate human behaviour in complex systems (Pan et al. Reference Pan, Han, Dauber and Law2007; Crowder et al. Reference Crowder, Robinson, Hughes and Sim2012; Hughes et al. Reference Hughes, Clegg, Robinson and Crowder2012). Experience indicates it is possible to develop adequate computer simulation models of the behaviour of travellers in rail terminals, of customers in supermarket queues, of emergency evacuation, and of team working (see e.g., Johnson & Feinberg Reference Johnson and Feinberg1997; Zarboutis & Marmaras Reference Zarboutis and Marmaras2004; Challenger et al. Reference Challenger, Clegg and Robinson2010; Crowder et al. Reference Crowder, Robinson, Hughes and Sim2012; Hughes et al. Reference Hughes, Clegg, Robinson and Crowder2012), for instance. These may well be circumstances where human behaviour is relatively simple and not heavily influenced by organizational structures and cultures, but it does raise a logical issue of how far we can go in developing such models and what the circumstantial limits are.
1.3 Case study problem domains
To explore these predictive possibilities we examine two problem domains, each with a different focus, timescale, and system level (Klein & Kozlowski Reference Klein and Kozlowski2000; see Section 1.1). We introduce these examples both to highlight the range of potential applications when considering malfunctions in complex systems and to illustrate our method with worked examples. The first concerns the severe breakdowns in patient care at the UK’s Mid-Staffordshire NHS Foundation Trust hospital between 2005 and 2009, part of the UK’s National Health Service (NHS), representing a chronic malfunction in a macro-level socio-technical system (i.e., whole organization). The second concerns the fatal Grayrigg rail accident in Cumbria, UK, in 2007, representing an acute malfunction in a meso-level socio-technical system (i.e., large team). Our analyses of the case studies are described in detail in Section 2 and summarized in Figures 2 and 3, respectively, where the directional arrows indicate the nature of the causal relationships we describe.
Socio-technical systems analysis of the organizational problems in Mid-Staffordshire NHS Foundation Trust hospital 2005–2009.
Socio-technical systems analysis of the organizational problems underlying the Grayrigg rail accident 2007.
2 Method for Predicting Malfunctions in Socio-Technical Systems (PreMiSTS)
Malfunctions in socio-technical systems can be regarded as a failure mode and are therefore of particular interest in the design of such systems. In this section, we present our 9-stage method, PreMiSTS, to advance our work on developing systematic capabilities to predict system malfunctions and performance. The method’s stages are summarized in Table 1 and detailed below, illustrated by the two example problem domains. The method is cyclic and iterative, rather than linear, to be repeated over time to facilitate learning and continuous improvement. Finally, although the stages and structure of the PreMiSTS method are novel, many of the data collection and data analysis methods used within the stages – such as interviews and the analysis of archival company data – are standard social science research methods, except where indicated otherwise by the references to the supporting literature.
Predicting Malfunctions in Socio-Technical Systems (PreMiSTS)
2.1 Stage 0: Check that the selected problem domain has precedents and appears consistent with a socio-technical approach
As a pre-method activity, there needs to be an initial check that the problem domain selected (e.g., healthcare quality in NHS hospitals) has well and independently analysed precedents. Without a high quality evidence base, the levels of speculation will almost certainly be too high to make the exercise worth the risk. With system malfunctions, such evidence may arise from independent enquiries and/or independent research.
A second pre-condition is that the method has a strong socio-technical orientation and the existing evidence base must indicate that such an approach is appropriate. Socio-technical systems theory underpins the method, whereby the systems in question are seen as comprising inter-related socio and technical elements (Cherns Reference Cherns1976; Clegg Reference Clegg2000). Such socio-technical frameworks have been used to analyse a number of acute disasters including the Hillsborough football stadium disaster, the King’s Cross underground station fire, and the Bradford City Football Club fire (see Challenger et al. Reference Challenger, Clegg and Robinson2010; Challenger & Clegg Reference Challenger and Clegg2011; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014).
Our accident analyses are combinatorial in nature, whereby accidents are believed to result from unique combinations of circumstances interacting in statistically unlikely ways (see Perrow Reference Perrow1984; Reason Reference Reason1990; Rasmussen Reference Rasmussen1997; Salmon et al. Reference Salmon, Cornelissen and Trotter2012). However, as discussed below, we also acknowledge there are common factors across different disasters and that some of their uniqueness originates from how these manifest and interact, alongside some factors entirely unique to particular disasters (Challenger & Clegg Reference Challenger and Clegg2011; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014).
2.2 Stage 1: Identify generic common factors (from previous literatures and enquiries)
Previous analysis of several major accidents has identified some common underlying factors (Challenger & Clegg Reference Challenger and Clegg2011), as listed in Table 2. One common factor concerns the dominant goals and metrics driving local behaviour in the system where the malfunction occurs. For example, in Mid-Staffordshire NHS Foundation Trust hospital, management were focused on financial requirements to achieve ‘Trust’ status to the relative exclusion of patient care and these priorities were passed onto the staff on the ground (see Figure 2). Similarly at Grayrigg, the local system prioritized the rail track modernization programme to the relative exclusion of ongoing maintenance and indeed reduced the amount of maintenance time available (see Figure 3). In each case, these goals strongly influenced the behaviour of local staff, promoting and supporting inappropriate working practices and job designs. These priorities also reflected poor and ineffective leadership. Thus, as discussed earlier, such issues are central to applying organizational psychology as a design science, representing areas where both disciplines can and should make a joint contribution.
Common factors across a range of disasters
2.3 Stage 2: Identify domain-specific factors
We now consider each problem domain, in turn, through the two selected case studies. We analyse each using an established hexagonal socio-technical framework comprising socio factors (people, culture, goals) and technical factors (technology, infrastructure, processes) linked as a complex inter-related system (see e.g., Challenger & Clegg Reference Challenger and Clegg2011; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014).
2.3.1 Case study 1: Mid-Staffordshire NHS Foundation Trust hospital
Several reports (e.g., Berwick Reference Berwick2013; Francis Reference Francis2013; The King’s Fund 2013) detail the appalling suffering of many patients between January 2005 and March 2009 at the Mid-Staffordshire NHS Foundation Trust hospital (abbreviated here to Mid-Staffs Hospital). There were serious and repeated failures of care and a much higher than expected mortality rate. Another analysis has examined how to design NHS healthcare using a socio-technical approach for a greater focus on patient care and safety (Clegg et al. Reference Clegg, Bolton, Offutt and Davis2014).
Drawing on these sources, we conducted a post-event analysis of the reported failings using a socio-technical framework as shown in Figure 2 and detailed below. The socio-technical local factors at Mid-Staffs Hospital (see Francis Reference Francis2013) are also summarized in Table 3.
Local factors at Mid-Staffordshire NHS Foundation Trust hospital 2005–2009
Essentially, the failings at Mid-Staffs Hospital resulted from a sub-optimal or inappropriate organizational culture which exacerbated inadequacies with the people’s skills and abilities to design, acquire and use technology and physical infrastructure sufficiently well to conduct their work processes, in order to achieve their organizational goals of delivering quality care to patients. We conclude that interacting and interdependent system-wide factors led to the chronic breakdown of patient care over a number of years.
Within Mid-Staffs Hospital, conflicting system goals were a fundamental reason for the breakdown of patient care. The nurses’ job designs and working practices were focused on financial and operational performance outcomes rather than on the quality of daily patient care. Although these measures gave a healthier financial picture, this led to a shortage of skilled nurses as patient care was not prioritized. This shortage was explicitly recognized as a problem by the hospital at the time. Such people factors evidently contributed to the care failure as the hospital staff’s performance metrics were focused on financial targets and bureaucratic self-assessment and self-declaration rather than on quality of patient care. This led to disengagement by nursing staff from managerial and leadership responsibilities, resulting in inadequate nursing standards with caring neglected and undervalued.
Furthermore, staff were not effectively guided or aided by organizational processes. There were no feedback tools or systems regarding quality of care, while strong processes existed regarding financial performance and waiting times, for instance. These factors are inter-related with the goals of the hospital, again being focused primarily on targets. Regulatory gaps were evident, such as safety monitoring with little, if any, measurement of care quality, and checks also missed these problems. Risk-based regulation was still being developed and individuals were not engaged with the national government standards. No tools were in place to enforce these standards into the work design, and developments of further feedback or technological monitoring systems were slow and lacked sophistication. The hospital had no insight and awareness of the reality of patient care, leading to isolation from better practice elsewhere. This created a lack of openness and transparency to the public and external agencies (i.e., looking inwards not outwards). The hospital therefore failed to separate what was essential from what was merely desirable. There was a failure in communication between many agencies to share their knowledge or concerns.
Additionally, infrastructure was a significant factor in the breakdown of care. Due to the complexity and size of the NHS, organizations operated in silos leading to a lack of connection between the strategic overview and performance management. The constant reorganization of NHS structures contributed to this disconnection, and such restructuring also resulted in loss of corporate memory and misunderstandings about responsibilities and functions, such as the lack of regulatory or supervisory systems. This affected people’s roles resulting in the diffusion of responsibility, particularly among less senior staff. Neither the impact nor risk to quality of patient care was considered during restructuring. There were many stakeholders linked to Mid-Staffs Hospital (e.g., Trust Board, CCGs, The Healthcare Commission), yet the top-down system failed to engage with patients and the public (e.g., Patient and Public Involvement Forums).
All the above were exacerbated by a negative culture, which tolerated poor standards, ignored staff concerns, and prioritized financial goals and managerial top-down priorities over patient care. The punitive treatment of whistleblowers discouraged employees from showing concern and created passivity (a ‘heads down’ ethos), and the working practices discouraged interventions against wrongdoing. Patients were not at the centre of the system as the hospital was defensive to criticisms and frequently overlooked complaints.
2.3.2 Case study 2: Grayrigg rail accident
At 20:12 hours on 23 February 2007, an intercity passenger train carrying 109 passengers and crew, travelling at 95 mph, derailed near the village of Grayrigg, Cumbria, in the UK (RAIB 2011). All nine of the train’s carriages left the track as the train passed over Lambrigg 2B points, resulting in one fatality, 30 serious injuries, and 58 minor injuries (RAIB 2011). The UK’s Rail Accident Investigation Board’s (RAIB) official report identified the unsafe condition of the Lambrigg 2B points as the cause of the accident, specifically the failure of stretcher bars and their joint to the points’ switch rails (see RAIB 2011, pp. 32–57, for technical details). However, several system-wide factors contributed to creating the conditions for the acute technical failure (RAIB 2011; Kim & Yoon Reference Kim and Yoon2013; Underwood & Waterson Reference Underwood and Waterson2014). The main findings from our socio-technical analysis of events leading up to the Grayrigg rail accident are discussed below and summarized in Figure 3. The socio-technical local factors are also summarized in Table 4.
Local factors at location of Grayrigg rail accident 2007
A series of latent and acute factors contributed to the technical failure of the Lambrigg 2B points. Our analysis of the RAIB (2011) report highlights a focus on the goals of track modernization, line performance, and the clearance of a backlog of routine maintenance, which may have influenced many other behaviours and decisions. These performance-driven goals fed into a culture which prioritized operational targets at the expense of safety, did not value strong reporting standards, and relied upon routine and historical inspection and maintenance practices. This culture was reflected in the processes, many inherited from maintenance predecessors, and the lack of independent audits or checks of infrastructure assets.
Track access arrangements were difficult following the focus on the track modernization programme, limiting time for routine track inspections. Deficiencies in job designs and specified working practices were evident. Notably, working practices were not fit for purpose, with key checks and procedures relating to the failed points underspecified at an organizational level. There were inadequate checks to ensure that patrol staff had the relevant competencies, and evidence of missed training for the joint points team. Furthermore, the design of work and team structures divided reporting responsibilities for the track and there was an absence of local ownership of this track section. The poor work design was reflected in the lack of continuity of maintenance staff in this track section, due to shift patterns and work allocation, and there were insufficient checks between reporting levels to identify missed inspections. The people involved were stretched by staff shortages and inadequate supervision.
These people factors interacted with the poor track access arrangements and goals which contributed to high workload, fatigue, excessive delegation, staff shortages, and a lack of continuity to inspections. All of these factors directly contributed to the track section manager forgetting to perform a basic track inspection, including the Lambrigg 2B points, which he had agreed to undertake in addition to his usual workload in order to free up staff for other tasks. The management information technology systems did not support the organization in identifying common faults across the rail network and may have contributed to discrepancies in the communication and reporting processes. All of these system-level factors contributed to conditions which allowed the infrastructural failures in the Lambrigg 2B points’ components. This technical failure may have been prevented if sufficient earlier routine checks had been performed.
Our interpretation of the evidence supports Reason’s (Reference Reason1990) assertion that accidents result from combinations of human, technical, and organizational issues. Our analysis of the system-wide factors contributing to the Grayrigg rail accident is also consistent with other analyses that have emphasized the broader context, human factors, procedures, and relational aspects (e.g., Kim & Yoon Reference Kim and Yoon2013; Underwood & Waterson Reference Underwood and Waterson2014).
Having analysed both case studies, we return now to our key aim of presenting a method for predicting malfunctions. Both analyses highlighted warning signs – or variables within behaviour and the system set-up which could have been used to anticipate the potential for such major failings before they occurred. We now consider how organizational psychologists might use such material to predict malfunctions so that they can be designed out of the system.
2.4 Stage 3: Select potential critical predictor variables and identify potential indicators
This stage involves the identification of important predictor variables along with data that can be used as indicators, which we now illustrate with examples. The predictor variables in each case are likely to include a mix of both socio and technical factors and may include both common and unique elements.
From Mid-Staffs Hospital, key predictors include the dominant goals and metrics held and promulgated by senior managers, coupled with the impact these have on local priorities and working practices. Indicators here would include interview and questionnaire data on the priorities and metrics perceived by staff on the ground, as opposed to managerial statements of intent or policy, along with any data on how staff spend their work time (see e.g., Robinson Reference Robinson2012).
From Grayrigg, key technical predictors include the nature of this track section, the accompanying high train speeds, and the type of points used. Key predictors in the social system include the organization of work and working practices among the local maintenance staff. Technical indicators here would include data on the nature of the track, maximum train speeds on particular track sections, and the presence of certain types of points. Socio indicators would include data on the inspection frequency at each track section, whether particular sections have continuity of inspectors (i.e., ‘track ownership’), and the skills and capabilities held by maintenance staff.
When identifying predictor variables it is important to distinguish whether they are acute and immediate precursors to malfunctions or chronic and longstanding causes. This distinction resonates with previous work by Wagenaar & Reason (Reference Wagenaar and Reason1990) distinguishing between ‘token’ causes and ‘type’ causes of accidents, respectively. They argue that tokens are those causes immediately preceding an accident which often occur suddenly, while types are background causes which have been present for a long time during routine operations. While there are only a few types, there are many different tokens, so they advocate addressing types as a more efficient and effective preventative approach. Some examples of types they identified include deficiencies in hardware, procedures, maintenance, and training, together with competing goals, non-compliance, and minimal organization; all factors which resonate strongly with the socio-technical approach we are advocating here (see Figure 1). Further research of this nature by Reason (Reference Reason2000) suggests that accidents are caused by dangers in latent conditions aligning with active failures, in the same way the holes in adjacent slices of Swiss cheese might align to create a deep gap through all slices. Finally, a similar strategy was advocated by Leplat & Rasmussen (Reference Leplat and Rasmussen1984) who recommended identifying points earlier in the accident chain that could be readily improved, thereby ‘breaking the sequence’, rather than focusing mainly on immediate causes of past accidents.
2.5 Stage 4: Discuss with domain experts
Ideally, domain experts will be involved at each stage of this process, but this is critical at this stage. Genuine multi-disciplinary collaboration is needed, with the particular combination of disciplines being determined by the problem domain.
For example, predicting breakdowns in patient care so that the system can be redesigned requires inputs from doctors, nurses, carers, health service managers, organizational psychologists, and simulation experts. Similarly, predicting accidents on the rail system requires inputs from human factors experts, infrastructure engineers, safety engineers, maintenance engineers, organizational psychologists, operational rail staff, and simulation experts. The organizational psychologists would guide the PreMiSTS method in each case, at least until the method is more developed, operating as researchers and consulting with these experts inside the organization when required. In the future, once the method matures and its procedures become routine, it would be possible for organizations to apply the method themselves in-house.
Given the nature of the data and information these experts provide, about past and potential malfunctions or even disasters, it is possible that this method could become a highly political process in organizations. Stakeholders may be keen to avoid blame or suspicion and therefore withhold information or divert attention elsewhere (Hood Reference Hood2002), for instance. However, a key strength of adopting a socio-technical systems approach to accident analysis is that the blame is shifted away from the acts of individuals towards examining multiple causes within the wider system (Junior et al. Reference Junior, e Santos, Vidal and de Carvalho2012), thereby making the admission of error easier for individual stakeholders. Nevertheless, it would still be important to develop trust in the process and steps could be taken to facilitate this. For instance, introducing a voluntary amnesty process for highly sensitive information – where errors could be reported free from blame – would encourage honestly (Cohen Reference Cohen2000). Furthermore, fostering a collective identity for all stakeholders would enhance trust, thereby reducing potential rivalry and the resultant ‘blame game’ (Kramer & Lewicki Reference Kramer and Lewicki2010).
The purposes of such discussions include confirming and elaborating results from Stages 1–3, collecting inputs for Stages 5–6, checking for important omissions, and identifying potential data sources.
2.6 Stage 5: Build a combinatorial model (or models) that are as specific as possible
At this stage, these models take the form of propositions (about which data are later collected and analysed during Stages 6–8). For example, problems of patient care are more likely when there is a dominant emphasis by local senior managers on performance metrics, combined with staff spending substantial time collecting and reporting managerial and financial data, combined with low staff morale, and where senior managers are complacent regarding patient care.
Similarly, rail accidents are more likely when the track section has a combination of high operating speed, a certain type of ‘high risk’ points, and is located far from the maintenance depot, combined with maintenance staff with a lack of ownership of the track, short-staffing, low morale, and where senior managers are complacent about safety.
There is an inherent risk that such propositions seem intuitive or even obvious. However, the fact that such accidents were not foreseen and continue to occur without warning is a compelling indication that they are not. Indeed, such hindsight bias is well documented following accidents and errors (Roese & Vohs Reference Roese and Vohs2012). To address this, our PreMiSTS method offers a means to build on this understanding of previous malfunctions systematically – to shift from accurate hindsight to accurate predictions – and therefore prevent malfunctions before they occur, or at least minimize their frequency and impact.
We note here that there is a natural tension between developing simple, parsimonious models that incorporate relatively few variables and developing more complete models (Bonabeau Reference Bonabeau2002; Vandekerckhove, Matzke & Wagenmakers Reference Vandekerckhove, Matzke, Wagenmakers, Busemeyer, Wang, Townsend and Eidels2014). Importantly, though, models with few variables and simple rules can also produce complex emergent effects if run dynamically over time (see Nowak Reference Nowak2004). This is reminiscent of Thorngate’s (Reference Thorngate and Strickland1976) notion of trade-offs between simplicity, generalizability, and accuracy in behavioural models (see also Waterson, Clegg & Robinson Reference Waterson, Clegg and Robinson2014), and Szabo’s (Reference Szabo and Leonides1993) discussion of trade-offs related to reliability and generalizability in computational models.
In practice, evaluation of simulation results against empirical results and experience will be used to determine a model’s optimal size. It is entirely possible that this varies across problem domains and across different areas in a given problem domain – akin to meshes used for physics-based analyses where a finer mesh is used in regions of highest interest.
This approach also raises the prospect, in time, of developing more complex typologies and profiles. Thus, for example, it is possible that there is a profile, X, where poor care is the result of a particular combination of factors (e.g., Mid-Staffs Hospital). However, there may well be another profile, Y, for other locations where poor care is the result of a different combination of factors. As this approach becomes elaborated over time, increasingly sophisticated profiles and typologies of poor care could be developed. This is probably more realistic than seeking a single combination for events such as poor healthcare or rail accidents.
Furthermore, as the underlying predictive science is developed, it could become possible to compute visualizations of system performance characteristics for a given system design, potentially highlighting new areas of malfunction before they occur in the real world.
Finally, we note that there is a risk here and in previous stages that the focus will be on predictors and malfunctions with which people are familiar, thereby failing to address other potential malfunctions. This common risk is often driven by ‘confirmation bias’ – the tendency to seek information to confirm rather than refute expectations (e.g., Jonas et al. Reference Jonas, Schulz-Hardt, Frey and Thelen2001) – and ‘groupthink’ – misperceptions of unanimity in groups leading to risky decisions (e.g., Bénabou Reference Bénabou2013). A useful strategy to counter these biases is to mandate the consideration of alternative options (Kassin, Dror & Kukucka Reference Kassin, Dror and Kukucka2013) and this procedure could be incorporated into our method.
2.7 Stage 6: Collect data to test the models
Once the models have been elaborated and discussed, data need to be collected on each of the component variables. We note that some of these data will already exist in the host organizations and may be readily available. However, such data are likely to be dispersed and distributed around different parts of the organization, for example in separate departments with responsibilities for human resources, operations, engineering, maintenance, and safety. Thus, collecting such data is no trivial task and will need to be supported by senior managers. Furthermore, our experience of organizational information systems suggests there will likely be substantial gaps and inconsistencies in the data to reconcile.
There are also likely to be data that are needed but not currently collected. For example, if low staff morale is a potential indicator of problems in the system and these data are not currently collected, then it may be worth including this variable in staff surveys. We also anticipate there will be trade-offs between the use of existing data that is an approximation of the requirement – what ‘big data’ researchers call proxy variables (e.g., Giannotti et al. Reference Giannotti, Pedreshi, Pentland, Lukowiczs, Kossmann, Crowley and Helbing2012) – versus the cost of collecting new data that may constitute more precise indicators. For example, absence and/or staff turnover data may be a ‘good enough’ proxy for staff morale, at least initially. We anticipate this may be a difficult judgment call, particularly in the early stages of using this approach, and one challenge will be assessing if existing data are adequate.
This challenge is not new, however. Collecting appropriate proxies for human behaviour and organizational performance is a perennial challenge for organizational psychologists, for instance. What is a new challenge, however, is attempting to amass such a diverse range of indicators and to use them to predict events. While one could argue that stress surveys and related measures are already used by psychologists to target potential flashpoints, the difference with our approach would be the range of indicators used and the nature of the occurrences we intend to predict – moving beyond individual behaviours and outcomes to large systems or organizational-level events.
2.8 Stage 7: Analyse data and run models to identify highest risk hotspots and include sensitivity analyses
The requirement here is to run the predictive models using data ideally from the full population of sites. Thus, when predicting breakdowns in healthcare, this would require data from all relevant hospitals; and for rail accidents, data from all parts of the rail network. This will require clarity over the unit of analysis – for instance, deciding how many parts of the healthcare system and rail network should be treated as separate units for comparative purposes. As discussed above, much of these data will already exist distributed throughout organizations, but the collection and collation are not trivial tasks. Even if available, the data may be organized in units of analysis that do not cleanly match the sites selected in this stage so compromises may be needed.
Sensitivity analyses should also be conducted where the key factors in the models are varied, at least initially, to identify particularly critical factors. In the early days of using this process, we are relaxed that this will generate different answers. The overall goal of this stage is to identify areas which are most at risk of malfunctions, seeking the top 5% or 10% of risk hotspots as defined by the clients.
There are two choices for the underlying nature of the model, each with different benefits and costs. One version could be a straightforward, multivariate statistical regression type model. Alternatively, computer simulation models could be developed using techniques such as agent-based or discrete event modelling (see e.g., Bonabeau Reference Bonabeau2002; Hughes et al. Reference Hughes, Clegg, Robinson and Crowder2012). In either case, it would be possible to calculate the numerical probability of malfunctions occurring given a particular set of input levels for the predictor variables.
2.9 Stage 8: Visit, audit, improve, and validate
The identified hotspots or high risk sites from Stage 7 will be used to trigger normal inspection regimes (e.g., top 5% of risk hotspots) but also an equivalent number of other sites for comparison. Findings will be fed back into the organizations concerned, systematically initiating inspection visits and audits, the results of which will then be used to improve local practice and thereby reduce ongoing risk. The inspection data will also be fed back into the models to improve their accuracy. Thus, the process is an iterative circular one with feedback loops to enable refinement and continuous improvement.
The identification of hotspots and their inspection, along with inspections of an equivalent number of other non-identified sites allows some validation (Robinson Reference Robinson, Cash, Stanković and Štorga2016). Specifically, this process allows us to compute hits (how many of the identified hotspots from this process are actually problematic during inspection?), false alarms (how many of the identified hotspots are operating adequately during inspection?), and misses (how many of the other sites that were not identified as hotspots are actually problematic during inspection?) (see Dixon, Wickens & Mccarley Reference Dixon, Wickens and Mccarley2007). The goal is to have high hits, low false alarms, and low misses. This yields quantifiable metrics to evaluate the PreMiSTS method and improve it over time.
This systematic quantification promotes further learning. Thus, for example, a high score for misses may subsequently reveal that there are other combinatorial explanations (or profiles) for poor care and could help identify what these may be (see our earlier discussion under Stage 6). In this view, it is unlikely that there is a single predictive model for poor care and there may be several, of which the Mid-Staffs Hospital example is just one.
The ‘other sites’ – that were not identified as hotspots – can either be selected randomly from the population of sites or selected to match key characteristics of the identified hotspots, depending on which selection strategy is more likely to help identify misses. The former strategy is likely to be best for initial exploratory research, while the latter strategy would be useful for more focused research examining particular variables.
3 Evaluation of the PreMiSTS method
We now evaluate the PreMiSTS method and identify its advantages and disadvantages. To do so, we use the fourteen criteria identified by Read et al. (Reference Read, Salmon, Lenné and Stanton2015) for evaluating socio-technical human factors methods, to which we add the criterion practical impact. For simplicity, we have merged these criteria into the following six broader categories based on conceptual similarity: (1) theoretical foundation and creativity; (2) involvement of stakeholders, workers, and users; (3) holism, integration, and tailorability; (4) structure, iteration, and traceability; (5) reliability and validity; and (6) efficiency, usability, and practical impact. The evaluation of the PreMiSTS method against these categories of criteria is presented further below.
As discussed throughout this paper, the PreMiSTS method we present here is intended as a framework for ultimately developing the capability to predict malfunctions in complex socio-technical systems. However, currently, we view this as a capability building exercise rather than the finished product, as we have explained throughout. Consequently, we have not yet tested our method with real people in real organizations and we acknowledge that this is a major limitation of our work to date. In the short-term, it requires testing in real word organizations and scenarios to establish initial predictive capability, which can then be refined and improved in the long-term with future research to deliver the accurate predictions required. Thus, to some extent, the method is untested. However, though the stages and structure of the PreMiSTS method are novel, the methods within these stages are generally based on established social science approaches and previous research literature. Consequently, we are still able to evaluate the method from these existing perspectives and we do this below now.
3.1 Theoretical foundation and creativity
The method builds on prior knowledge and expertise in this area, particularly socio-technical systems theory (Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014), especially in Stages 0–4. Thus, for example, it can cope both with the underlying combinatorial logic that is prevalent in the literature (e.g., Rasmussen Reference Rasmussen1997; Svedung & Rasmussen Reference Svedung and Rasmussen2002; Salmon et al. Reference Salmon, Cornelissen and Trotter2012), and with the idea that there may also be some common underlying problems from which we can learn (e.g., Challenger et al. Reference Challenger, Clegg and Robinson2010).
The method also helps open up the topic to discussion in new and creative ways. It provides one way of getting started and is specific enough to allow empirical testing. It allows us to ‘learn by doing’ which is an effective way of making progress in these areas (Cassell & Johnson Reference Cassell and Johnson2006; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014). Thus, we believe we will learn more about the difficulties and prospects of work in this area by trying it out.
Finally, we believe the approach opens up new opportunities and challenges for both design science and organizational psychology, leading to methodological developments and theoretical advances. Thinking predictively will make new demands of both disciplines and, if successful, will open up massive opportunities to develop our capabilities.
3.2 Involvement of stakeholders, workers, and users
In principle, the method can be highly participative and inclusive. It encourages the inclusion of different people, from different disciplines and backgrounds (see Mumford Reference Mumford1983; Clegg Reference Clegg2000; Eason Reference Eason and Graves2008, for discussion of the benefits of such an approach), thereby fostering shared mental models (Mathieu et al. Reference Mathieu, Heffner, Goodwin, Salas and Cannon-Bowers2000) and facilitating innovation (Alves et al. Reference Alves, Marques, Saur and Marques2007).
3.3 Holism, integration, and tailorability
The method incorporates multivariate explanations of malfunctions and, over time, we anticipate that such explanations will become more nuanced and sophisticated. Drawing on its foundations in socio-technical systems theory (e.g., Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014), it can examine a full range of variables covering both socio elements (people, culture, goals) and technical elements (technology, infrastructure, processes) as shown in Figure 1. It is also able to examine socio-technical systems at a micro (individual), meso (team), and/or macro (organization and/or entire industry) level of analysis (Klein & Kozlowski Reference Klein and Kozlowski2000). Thus, PreMiSTS can examine socio-technical systems in a holistic and integrated way, and is also sufficiently flexible to be tailored to a full range of work scenarios.
3.4 Structure, iteration, and traceability
The method provides a highly structured 9-stage process for predicting malfunctions in socio-technical systems, providing a clear audit trail and traceability for decisions that have been made at each stage. Furthermore, PreMiSTS can be used on an ongoing basis, repeating the nine stages periodically to refine the performance and safety of socio-technical systems in an iterative way.
3.5 Reliability and validity
Reliability refers to the consistency of measurement while validity refers to whether the method measures what it claims to (Robinson Reference Robinson, Cash, Stanković and Štorga2016), with the former a prerequisite of the latter (Cook Reference Cook2009). Both constructs would have to be assessed statistically using data collected while running the PreMiSTS method. In this context, reliability would concern the agreement between experts concerning the likely predictors of malfunctions and the consistency with which the organizational data were measured, while validity would concern whether the likely predictors identified do indeed predict malfunctions in reality. We now discuss some issues of relevance to these constructs.
First, there is a concern that the method is too reliant on expert judgement on what variables to include in the predictive models. Consequently, it could be biased by the mental models of those heavily involved in the process, making it idiosyncratic and potentially unreliable. To counter this, the inclusion of further domain experts could balance out such biases and enable reliability to be assessed. Although the same criticism can be aimed at existing post-event analyses, at least in those circumstances there are more data available. Our view is that the application of organizational psychology as a design science, alongside expertise from other disciplines, will be highly beneficial in this regard. Indeed, this risk is arguably considerably greater without this involvement.
Second, the method allows for computation of hits, false alarms, and misses (Dixon et al. Reference Dixon, Wickens and Mccarley2007), thereby enabling potential validation and providing opportunities for learning and development. If it helps direct inspection efforts and demonstrates good hit rates, it consequently becomes useful in reducing the likelihood of future malfunctions. We believe this is a major strength. However, even if initial prediction accuracy is poor, we will still learn a great deal. For instance, in the case of health care breakdowns, the approach has already helped us understand that there may well be more than one profile of predictive circumstances. The failures at Mid-Staffs Hospital may represent just one such profile and it may therefore make sense to develop a typology of breakdowns in care using this method.
Finally, the method is currently unproven in various ways. For example, it is unclear to what extent the method can adequately address: (1) low frequency and high impact mishaps, or ‘black swan’ events (Taleb Reference Taleb2007); (2) distributed, messy, and/or incomplete data; and (3) contested domains with little consensus about underlying causes. Furthermore, the behaviour of some systems may simply be too complex to predict accurately, demonstrating emergent malfunctions that are challenging or impossible to anticipate (Pavard et al. Reference Pavard, Dugdale, Saoud, Darcy and Salembier2006). However, such extreme scenarios are thankfully rare and, in today’s digital age, the increasing availability and accuracy of ‘big data’, and the technology to mine it (Wu et al. Reference Wu, Zhu, Wu and Ding2014), makes even these scenarios potentially predictable in future. Furthermore, even with seemingly ‘weak signals’, the establishment of systematic procedures to raise awareness, identify risks, and share lessons may help identify problems sufficiently early to prevent them from incubating (Macrae Reference Macrae2014). Even if highly accurate predictions remain elusive, though, such efforts will still identify areas of higher risk to enable resources to be targeted and precautions to be taken.
3.6 Efficiency, usability, and practical impact
The method is not hugely expensive, not requiring large investments in technological support for instance. The main costs arise in Stage 6 – collecting dispersed data in large organizations – and in Stage 8 – where the outputs are used to trigger site inspections. Yet our assumptions here are that much of these data already exist, so the costs are therefore primarily in collation, and that site inspections already occur in organizations such as the NHS and Network Rail (RAIB 2011; Berwick Reference Berwick2013). Thus, in Stage 8, this method need not add new costs, but rather helps direct and prioritize existing effort. Indeed, the method would help trigger the inspection regime by identifying potential hotspots – areas of high risk – and gives a strong rationale for both breadth of coverage – using this method – and subsequent depth of coverage – by follow-up inspection. This combined approach to managing risk and predicting failure is useful in its own right.
On the negative side, modern organizations are under extreme pressure to deliver results to short-term deadlines and may not be able to accept open-ended commitments of this kind. Furthermore, the method may be seen as disruptive by client organizations. For example, it is not clear how this method will fit with existing practices in organizations such as the NHS and Network Rail, each of which already has heavy, historical investments in their own ways of collecting and collating data (see Stage 6 above), and in selecting sites for inspections (see Stage 8 above). A lack of alignment here would reduce the chances of successful adoption and use.
Finally, there is a danger that the method may generate misplaced confidence that such work can predict failings so accurately that disasters can be prevented entirely. Thus, efforts in this area may do more harm than good by leading to reduced vigilance. Indeed, there may be an ironic danger that this approach may even propagate the types of organizational complacency and inertia that have been highlighted as contributing to high-profile disasters (Challenger & Clegg Reference Challenger and Clegg2011; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014).
4 Future prospects and implications
The application of organizational psychology as a design science can greatly enhance the prediction of system malfunctions and the discipline has much to gain by becoming actively involved in such research. We describe and use our PreMiSTS method focused on malfunctions as illustrative examples, not as the sole best way forward. Development of the method has followed the rationale adopted by the aerospace engineering sector in their long-term investment in modelling system performance in silico. As such, we are attempting to develop cumulative predictive capability using organizational psychology as a design science to predict human behaviours, as opposed to the physics that are used to predict performance of physical systems.
The PreMiSTS method is best suited to situations with the following characteristics: (1) systemic problems with a strong socio-technical basis, including behavioural and organizational issues; and (2) problems that appear to have combinatorial roots and that display a mix of underlying common and unique factors. To enable the further development of the method, we also need to work with problems that have independently analysed precedents, with analyses strong enough to enable validation of predictive models. Our sense is that this profile fits with many malfunctions and problems in the real world, such as IT failures, problems with infrastructure and the provision of services, social care, child protection, and environmental disasters (cf., Clegg & Shepherd Reference Clegg and Shepherd2007; Broadhurst et al. Reference Broadhurst, Wastell, White, Hall, Peckover, Thompson, Pithouse and Davey2009; Davis et al. Reference Davis, Challenger, Jayewardene and Clegg2014).
Although the shift towards a predictive design science will make new demands of organizational psychology, we believe such challenges will be healthy, requiring new methods of the kind we have elaborated in this paper, and new techniques such as the use of ‘big data’ and computer simulation. Recent research using big data to design supply chains (Waller & Fawcett Reference Waller and Fawcett2013), and the computer simulation of team behaviour (Crowder et al. Reference Crowder, Robinson, Hughes and Sim2012) are two examples of such paradigms. Indeed, we would argue that it is not only desirable but essential that organizational psychology engages in predictive work of this nature to establish itself firmly as a design science. Now, at the beginning of the digital age, there have never been more opportunities to study human behaviour across the full spectrum of complex socio-technical systems due to the huge volume of big data that are now available about such activities (IBM 2016). As the volume, range, accuracy, and availability of such data continues to grow at an exponential rate thanks to digital advances (Wu et al. Reference Wu, Zhu, Wu and Ding2014), so too do the capabilities to analyse and understand these data with the development of new methods such as advanced computer simulation and networked artificial intelligence (Giannotti et al. Reference Giannotti, Pedreshi, Pentland, Lukowiczs, Kossmann, Crowley and Helbing2012). Indeed, as experts at understanding human behaviour and cognition (both individually and collectively) in such complex systems, no other discipline is better placed to exploit these opportunities; however, it is essential that organizational psychology does so, as other disciplines surely will.
We do acknowledge that work in this area will be high risk, but we are confident it will be worthwhile even if it fails to make satisfactory predictions in the short-term. Examples of potential pay-offs include the development of new partnerships and new relationships with potential clients in the problem domains. Working with such clients will make heavy demands on our capabilities and encourage us to develop in creative ways. In so doing, it will help us attract the most talented people to work with us on these problems.
In addition to such indirect potential benefits, there are likely to be pay-offs in content. For example, in recent years, there has been an increasing recognition of the importance of complex systems as a research area for organizational psychology. Emerging research into highly complex multi-team systems, or ‘teams of teams’, is just one example of this expanding focus (see e.g., Davison et al. Reference Davison, Hollenbeck, Barnes, Sleesman and Ilgen2012; Luciano, Dechurch & Mathieu Reference Luciano, Dechurch and Mathieuin press). Such work will be promoted by an emphasis on developing a predictive design science capability.
Such an emphasis will also make new demands on us for collating and analysing big and dispersed data of the kind we will need to populate our combinatorial models. For instance, the technology and methods required to store, process, and analyse such data will necessitate the acquisition of more advanced statistical and computational skills (see e.g., Cohen et al. Reference Cohen, Dolan, Dunlap, Hellerstein and Welton2009). This will also require multi-disciplinary collaborations.
A predictive emphasis will also encourage us to try to contribute to some of the most challenging problems of the day. To use a topical example, the Mid-Staffs Hospital failing has been and continues to be a national topic of interest and it is, we believe, important that organizational psychologists develop and use a predictive capability to make an impact. This has been demonstrated in the analyses of what went wrong at Mid-Staffs Hospital but holds even more potential value as we move into predictive mode.
Such an emphasis on prediction will also encourage the development of organizational psychology as a design science, one in which it offers more than wisdom after the event and helps design new complex systems that are more effective and less prone to failure (see Wastell Reference Wastell2011).
At the same time, we readily acknowledge this is not the only approach to predicting such events (see e.g., Stanton et al. Reference Stanton, Salmon, Harris, Marshall, Demagalski, Young, Waldmann and Dekker2009). Only time will tell how each of these investments fares and it is clearly important that people working in this area find ways of both collaborating and competing to address common problems and opportunities.
Debate is needed on how such efforts are taken forward. Our current view is that this is high risk work and highly speculative. It is at levels 1–3 of the Technology Readiness Level (TRL) schema, where we are identifying key principles and establishing feasibility (see Mankins Reference Mankins2009, for an overview of TRLs). Currently, the work is therefore best suited to being undertaken by research and development organizations, such as universities, working closely with clients but funded initially by public schemes sought competitively. The next step would then be to develop and demonstrate the ‘technology’, during TRL levels 4 and 5, to show the value and applicability of this approach. If and when this bears fruit, the nature of the work then becomes more focused on refinement and application during TRL levels 6, 7, and 8, and may then be more appropriately taken forward by private sector funding.
Finally, in predictive engineering simulation, consolidating results from multiple simulations based on different branches of physics has required several issues to be resolved (Trippner, Rude & Schreiber Reference Trippner, Rude, Schreiber, Stjepandić, Wognum and Verhagen2015). For example, a key enabler was the establishment of computer-based shape models that allowed designs to be specified in a form suitable for simulation (Requicha & Voelcker Reference Requicha and Voelcker1983). Thus, as we develop PreMiSTS simulation models, equivalent questions surrounding the computer representation of socio-technical systems will emerge. These issues will influence the types of simulation models developed and the underlying predictive design science they represent, challenging organizational psychology to evolve and improve.
5 Conclusion
We should be clear on the claims we are making of this work. In principle, the proposed method can involve users and domain experts and can cope with multi-disciplinary concerns, as it uses a socio-technical and combinatorial framework. We also believe it can build on previous work in this domain, drawing on the evidence of previous malfunctions and disasters and on what appear to be common underlying problems. The method builds in opportunities for the computation of hits, false alarms, and misses, and thereby offers opportunities for validation, learning, and development. Such metrics have the potential to be a great asset. Reactions on the part of clients and domain experts will also be important in helping to take the work forward.
We are, of course, not claiming this method and its underlying rationale is the answer to the problem of prediction. Rather, we are claiming it is one promising way forward, worthy of further development, and offering opportunities to ‘learn by doing’. We are also claiming we will learn a great deal even if it does not deliver what we hope. Our aspiration is that the presentation of this PreMiSTS method here will help stimulate further research and development in this area. One obvious way in which such predictive work could be developed is to attempt it in the more general and multi-dimensional domain of system performance, which we believe includes but is not limited to system malfunctions.
If successful, however, such predictive capability would be a major step forward for people working with and in complex organizational systems, helping to make them more effective, reliable, and safe. In short, these are exciting and challenging times to develop organizational psychology as a design science. A predictive paradigm, and the methods and techniques associated with it, offer opportunities to make a real impact.
Open access
