Acting in good faith is an elemental requirement in fulfiling duties where the rightholder has reposed trust or faith in the dutyower. Sometimes, these dutyowers are fiduciaries. From an ethical perspective an essential feature of the trust and agency relationships is that a principal is entitled to repose ‘trust’ in a trustee or agent – by ‘trust’ I mean the commonsense notion of trust, not the technical legal relationship of trustee and beneficiary. Even so, the practical relevance of the duty is typically misunderstood, in part because of common misunderstandings about the good faith duty’s relationship with fiduciary powers, trustworthiness, and good will. Acting in bad faith is a breach of faith or trust that has relevance to isolated pockets of trust and agency law. At an abstract and general level, where a trustee or agent acts in bad faith this allows the principal or the court to remove the trustee or agent, extinguishing his right to that role. The duty of good faith has nothing in particular to do with the exercise of fiduciary powers.

Why you care: We begin with an end-to-end example of the design (with explicit assumptions), execution, and interpretation of an experiment to assess the importance of speed. Many examples of experiments focus on the User Interface (UI) because it is easy to show examples, but there are many breakthroughs on the back-end side, and as multiple companies discovered: speed matters a lot! Of course, faster is better, but how important is it to improve performance by a tenth of a second? Should you have a person focused on performance? Maybe a team of five? The return-on-investment (ROI) of such efforts can be quantified by running a simple slowdown experiment. In 2017, every tenth of a second improvement for Bing was worth $18 million in incremental annual revenue, enough to fund a sizable team. Based on these results and multiple replications at several companies through the years, we recommend using latency as a guardrail metric.

Why you care: To design and run a good online controlled experiment, you need metrics that meet certain characteristics. They must be measurable in the short term (experiment duration) and computable, as well as sufficiently sensitive and timely to be useful for experimentation. If you use multiple metrics to measure success for an experiment, ideally you may want to combine them into an Overall Evaluation Criterion (OEC), which is believed to causally impact long-term objectives. It often requires multiple iterations to adjust and refine the OEC, but as the quotation above, by Eliyahu Goldratt, highlights, it provides a clear alignment mechanism to the organization.

This chapter instills an appreciation for the powerful effects (both positive and negative) of performance pay on employee behavior. It opens with a performance-pay success story, namely a field experiment by Shearer (2004) in which the piece-rate compensation of Canadian tree planters was changed. It then develops some examples of the darker side of performance pay, including the Wells Fargo employees who opened false accounts to meet a quota. Section 9.2 provides visual representations of performance pay in which the pay graph has a positive slope (i.e., it increases when the worker’s performance measure increases), sometimes linearly as with piece-rate pay and sometimes nonlinearly as with bonuses. The chapter emphasizes the incentive and sorting effects associated with performance pay as well as its prevalence. Workers’ attitudes towards risk (of earnings fluctuations) and how risk affects performance pay is covered, along with performance measurement, various drawbacks of performance pay, and how to design performance-pay contracts. Readers will finish the chapter with an understanding of the advantages and disadvantages of performance pay and when it can be effectively used.

This chapter on executive compensation and stock options is effectively a continuation of Chapter 9 on performance pay. It provides an overview of executive compensation and an intuitive, non-technical treatment of stock options that focuses on the worker incentives that options create. There is a lot of discussion of risk (of income loss) that builds on Chapter 9, and the “pay for luck” discussion that ends the chapter concerns the possibility of firms’ reneging on CEOs’ bonus payments, which echoes the wage-theft themes from Chapter 2. Section 10.2 covers the executive bonuses known as “80/120” plans, representing them pictorially as nonlinear functions of a performance measure (that are upward-sloping in some parts, as in the performance-pay graphs of Chapter 9). The section on stock options is detailed and explains all of the key terminology and the most important concepts in this area. The distinction between the intrinsic value and the market value of an option is made carefully, with an intuitive, non-technical discussion of the Black–Scholes–Merton options valuation formula, and the role of risk is explained in detail.

In Chapter 1, we reviewed what controlled experiments are and the importance of getting real data for decision making rather than relying on intuition. The example in this chapter explores the basic principles of designing, running, and analyzing an experiment. These principles apply to wherever software is deployed, including web servers and browsers, desktop applications, mobile applications, game consoles, assistants, and more. To keep it simple and concrete, we focus on a website optimization example. In Chapter 12, we highlight the differences when running experiments for thick clients, such as native desktop and mobile apps.

Why you care: Randomized controlled experiments are the gold standard for establishing causality, but sometimes running such an experiment is not possible. Given that organizations are collecting massive amounts of data, there are observational causal studies that can be used to assess causality, although with lower levels of trust. Understanding the space of possible designs and common pitfalls can be useful if an online controlled experiment is not possible.