Chapter 16 establishes the concept of an externality, a situation where a decision carries additional costs or benefits to a party not involved in the decision-making process. The chapter classifies several types of externalities and explores how and why they lead to undesirable outcomes. Then potential solutions to externalities as well as their strengths and weaknesses are discussed: subsidies, taxes, regulation, and the assignment of property rights. The chapter ends with a discussion of cost–benefit analysis and its role in evaluating interventions aimed at combatting externality problems.

This chapter begins by applying the theory of the firm to the context of a medical practice. The concepts of profit maximization, inputs to production, and outputs from production are all applied to this example. Then the chapter moves on to discuss sources of efficiency in production and specific applications to medical care production. The chapter then develops the basics of supply curves, and how they are interpreted and used, and then brings supply together with demand to describe the basics of equilibrium in an efficient market. The last part of the chapter discusses threats to efficiency in markets, giving a brief overview of the largest sources of difficulty in applying efficient markets to the healthcare context. This sets up the rest of the book: the healthcare sector is filled with problems that make efficient markets unlikely, meaning that understanding the underlying economics is vital.

Chapter 9 explores the hospital as an economic entity. The chapter discusses the differences between for-profit and not-for-profit hospitals, the organization of hospital workforces, and how the for-profit/not-for-profit distinction interacts with the organization. Then the chapter explores how different parties involved with the hospital seek to exert influence over how hospital resources are used: what they want and how they get it. Finally, the chapter covers topics in how hospitals operate within the sector: hospital growth, hospital competition, and operation of hospital systems.

Chapter 11 focuses on the adverse selection problem in the health insurance market, which is sometimes referred to as the “death spiral.” First, the concept of “uninsurable” individuals is developed and the rationale for eliminating differential pricing based on pre-existing conditions (experience rating) is explored. Then the chapter discusses the consequences of this choice: the adverse selection problem. Finally, alternative solutions to adverse selection are explored – both single-payer systems and market-based solutions such as the Affordable Care Act. The end of chapter supplement explores the historical reasons for differences between the health insurance system in the United States and the system of a close ally, the United Kingdom.

This chapter introduces the machine learning side of things of this book. Although we assume some prior experience in machine learning, we start off with a full recap of the basic concepts and key terminology. This includes a discussion of learning paradigms, such as supervised and unsupervised learning, and the machine learning life cycle, articulating the steps to go from data collection to model deployment. We cover topics like data preparation and preprocessing, model evaluation and selection, and machine learning pipelines, showing how all the stages of this cycle are susceptible to being compromised when we talk about large-scale data analytics. After that, the rest of the chapter is devoted to the machine learning library of Spark, MLLib. Basic concepts such as Transformers, Estimators, and Pipelines are presented with an example using linear regression. The example provided forces us to use a pipeline of methods to get the data ready for training. This allows us to introduce some of the data preparation packages of Spark (e.g., VectorAssembler or StandardScaler). Finally, we explore evaluation packages (e.g., RegressionEvaluator) and how to perform hyperparameter tuning.

Chapter 12 deals with how prices are set within the healthcare sector. Prices do not come from direct interaction between buyers and sellers but are instead arrived at via a complex negotiation process between multiple parties. This chapter covers the lifecycle of a medical bill, the role of chargemasters (master price lists) in billing, how providers negotiate with insurers, and how the uninsured are treated in billing. The chapter then discusses price competition and quality competition between providers, as well as the role of price transparency.

Chapter 19 explores differences in spending on healthcare and health outcomes across countries. The main result of interest from these comparisons is that the United States spends vastly more than other OECD countries but gets only middle of the road health outcomes. The second part of the chapter explores leading theories as to why this is the case. The final part of the chapter compares different strategies for setting up a healthcare and health insurance sector: the Bismarck model, the Beveridge model, national health insurance, and hybrid models.

Chapter 8 explores how incentives influence the behavior of providers. The first part of the chapter is an application of personnel economics, discussing how different pay schemes create incentives for different types of effort. The chapter goes on to discuss the latent variable problem when looking at pay schemes and behavior and covers literature that helps to unpack the causal story. The second half of the chapter discusses demand inducement and its variants (fee splitting and self-referral) from both a theoretical and empirical perspective. The end of chapter supplement gives a brief introduction to behavioral economics and how cognitive biases can also be used to influence provider behavior.

MapReduce is a parallel programming model that follows a simple divide-and-conquer strategy to tackle big datasets in distributed computing. This chapter begins with a discussion of the key distinguishing features and differences of MapReduce with respect to similar distributing computing tools like Message Passing Interface (MPI). Then, we introduce its two main functions, map and reduce, based on functional programming. After that, the notation of how MapReduce works is presented using the classical WordCount example as the Hello World of big data, discussing different ways to parallelize it and their main advantages and disadvantages. Next, we delve into MapReduce a bit more formally, and its functions in terms of key–value pairs, as well as the key properties of the map, shuffle, and reduce operations. At the end of the chapter we cover some important details as to how to achieve fault tolerance, how to exploit MapReduce to preserve data locality, how it can reduce data transfer across computers using combiners, and additional information about its internal working.

The goal of this chapter is to present complete examples of the design and implementation of machine learning methods in large-scale data analytics. In particular, we choose three distinct topics: semi-supervised learning, ensemble learning, and how to deploy deep learning models at scale. Each of them is introduced, motivating why parallelization to deal with big data is needed, determining the main bottlenecks, designing and coding Spark-based solutions, and discussing further work required to improve the code. In semi-supervised learning, we focus on the simplest self-labeling approach called self-training, and a global solution for it. Likewise, in ensemble learning, we design a global approach for bagging and boosting. Lastly, we show an example with deep learning. Rather than parallelizing the training of a model, which is typically easier on GPUs, we deploy the inference step for a case study in semantic image segmentation.

Hadoop is an open-source framework, written in Java, for big data processing and storage that is based on the MapReduce programming model. This chapter starts off with a brief introduction to Hadoop and how it has evolved to become a solid base platform for most big data frameworks. We show how to implement the classical Word Count using Hadoop MapReduce, highlighting the difficulties in doing so. After that, we provide essential information about how the resource negotiator, YARN, and its distributed file system, HDFS, work. We describe step by step how a MapReduce process is executed on YARN, introducing the concepts of resource and node managers, application master, and containers, as well as the different execution models (standalone, pseudo-distributed, and fully-distributed). Likewise, we talk about the HDFS, covering the basic design of this filesystem, and what it means in terms of functionality and efficiency. We also discuss recent advances such as erasure coding, HDFS federation, and high availability. Finally, we expose the main limitations of Hadoop and how it has sparked the rise of many new big data frameworks, which now coexist within the Hadoop ecosystem.

This chapter puts forward new guidelines for designing and implementing distributed machine learning algorithms for big data. First, we present two different alternatives, which we call local and global approaches. To show how these two strategies work, we focus on the classical decision tree algorithm, revising its functioning and some details that need modification to deal with large datasets. We implement a local-based solution for decision trees, comparing its behavior and efficiency against a sequential model and the MLlib version. We also discuss the nitty-gritty of the implementation of decision trees in MLlib as a great example of a global solution. That allows us to formally define these two concepts, discussing the key (expected) advantages and disadvantages. The second part is all about measuring the scalability of a big data solution. We talk about three classical metrics, speed-up, size-up, and scale-up, to help understand if a distributed solution is scalable. Using these, we test our local-based approach and compare it against its global counterpart. This experiment allows us to give some tips for calculating these metrics correctly using a Spark cluster.

This chapter deals with the topic of inequality in the healthcare sector. The chapter begins by looking at some summary data and the inequality in various measures of health that are obvious from simple differences. Then the chapter discusses some basic social philosophy on the topic of how to judge fairness in an abstract society. More detailed data on health outcomes is then explored, highlighted inequality along different demographic lines. Then there is a discussion of theories that fit the facts presented: why do these inequalities exist, and what do we learn by unpacking them: both social implications as well as clinical. Finally, inequalities in the labor market for healthcare workers are discussed.