Why you care:  Sometimes the effect that you care to measure can take months or even years to accumulate – a long-term effect. In an online world where products and services are developed quickly and iteratively in an agile fashion, trying to measure a long-term effect is challenging. While an active area of research, understanding the key challenges and current methodology is useful if you are tackling a problem of this nature.

Why you care: While experimentation is widely adopted to accelerate product innovation, how fast we innovate can be limited by how we experiment. To control the unknown risks associated with new feature launches, we recommend that experiments go through a ramp process, where we gradually increase traffic to new Treatments. If we don’t do this in a principled way, this process can introduce inefficiency and risk, decreasing product stability as experimentation scales. Ramping effectively requires balancing three key considerations: speed, quality, and risk.

Why you care: When running experiments, you also need to generate ideas to test, create, and validate metrics, and establish evidence to support broader conclusions. For these needs, there are techniques such as user experience research, focus groups, surveys, human evaluation, and observational studies that are useful to complement and augment a healthy A/B testing culture.

Why you care: As your organization moves into the “Fly” maturity phase, institutional memory, which contains a history of all experiments and changes made, becomes increasingly important. It can be used to identify patterns that generalize across experiments, to foster a culture of experimentation, to improve future innovations, and more.

Acting in good faith is an elemental requirement in fulfiling duties where the rightholder has reposed trust or faith in the dutyower. Sometimes, these dutyowers are fiduciaries. From an ethical perspective an essential feature of the trust and agency relationships is that a principal is entitled to repose ‘trust’ in a trustee or agent – by ‘trust’ I mean the commonsense notion of trust, not the technical legal relationship of trustee and beneficiary. Even so, the practical relevance of the duty is typically misunderstood, in part because of common misunderstandings about the good faith duty’s relationship with fiduciary powers, trustworthiness, and good will. Acting in bad faith is a breach of faith or trust that has relevance to isolated pockets of trust and agency law. At an abstract and general level, where a trustee or agent acts in bad faith this allows the principal or the court to remove the trustee or agent, extinguishing his right to that role. The duty of good faith has nothing in particular to do with the exercise of fiduciary powers.

This chapter instills an appreciation for the powerful effects (both positive and negative) of performance pay on employee behavior. It opens with a performance-pay success story, namely a field experiment by Shearer (2004) in which the piece-rate compensation of Canadian tree planters was changed. It then develops some examples of the darker side of performance pay, including the Wells Fargo employees who opened false accounts to meet a quota. Section 9.2 provides visual representations of performance pay in which the pay graph has a positive slope (i.e., it increases when the worker’s performance measure increases), sometimes linearly as with piece-rate pay and sometimes nonlinearly as with bonuses. The chapter emphasizes the incentive and sorting effects associated with performance pay as well as its prevalence. Workers’ attitudes towards risk (of earnings fluctuations) and how risk affects performance pay is covered, along with performance measurement, various drawbacks of performance pay, and how to design performance-pay contracts. Readers will finish the chapter with an understanding of the advantages and disadvantages of performance pay and when it can be effectively used.