Introduction

Since the dawn of time, humans have been shaped by technology. To survive, humans built tools and re-engineered the environment and themselves. The roots of contemporary techno-social engineering are ancient. Accordingly, in this chapter, we provide a historical primer on the transformative power of tools.¹

Tool Use: The Basics

While the myth persists that technology is applied science, humans have always developed and used tools, even during pre-scientific times.² Some define the human essence by this capability, characterizing us as homo faber, beings who make and use tools. From this perspective, tool use enables us to accomplish ends that otherwise would remain out of our reach. Many of these ends continue to elude other species on the planet. There is considerable evidence of non-human tool use, even quite sophisticated techniques and inventions, among animals such as crows, chimpanzees, and cephalopods.³ Nonetheless, we don’t have to worry about any of them starting a nuclear war any time soon.

Technology doesn’t always do what we want it to do. Inventors can’t always determine how a technology will be used once it’s integrated into society. To believe otherwise is to commit the mistake known as the “designer’s fallacy.”⁴ Technology use routinely involves unintended consequences and trade-offs. Take the discovery of fire and early inventions that applied it to keep people warm and cook meat (a quick source of protein). The process had obvious benefits for our ancestors, including profound cognitive ones.⁵ But the Promethean gift also posed risks. Fire can burn people, possessions, and shelters. Most importantly, the development and use of tools to make and manage fire shaped who we are and who we are capable of being. As anthropologists and historians have explained, these tools opened new possible paths for humans to live and develop. Other tools have done the same.

Yet over time we’ve lost many capabilities by learning to solve problems with newer tools. Our predecessors could do many things that most of us simply can’t do anymore. Know anyone who uses “dead reckoning” to navigate a boat? That’s a pre-GPS orientation to determining positioning, revolving around factors like time, direction, and speed. Or, imagine if, in the future, the only cars that exist are autonomous, self-driving models. If they collectively went on strike, how many would be able to ride a horse to work? Or pilot a “dumb” automobile, after stealing it from a museum?

Tool use also has expanded our capabilities. Delegating tasks to technology has freed up time and resources and enabled us to move on to new and often more advanced problems. Keeping the innovation cycle going requires imagining, creating, and using new tools.

Consider the trajectory of fishing tools as a progression from spear fishing, to using a fishing pole, to using nets, and so on.⁶ Along the way, capabilities have been gained and lost. At one time, many humans had considerable skill in handling a fishing spear; doing so was necessary to obtain food. Over time, that skill gradually became less useful. At some point, a fishing pole proved more efficient. And that efficiency ended up paling in comparison to subsequent large-scale fishing from boats with nets, a historical turn that turned pole fishing into a recreational hobby (rather than a professional endeavor) for many cultures.⁷ In the end, progress in fishing appears to be a net gain in terms of both efficiency and capabilities. Less fishing effort by fewer people yields more fish for consumption, and, as a result, people in a historically based fishing community can diversify their skills and learn to use other tools.⁸ Upon closer inspection, however, complex trade-offs become apparent. Displaced fishermen who cannot find work doing other jobs are a problem. Not everyone can transition from skill to skill – not least because training can be too costly and time-intensive. In some contexts, eroded familial and community traditions lead to a loss of social identity and capital. We should be skeptical of overly rosy narratives about how the future of automation will liberate us to spend more time doing artistic and related endeavors.⁹

While it’s tempting to assume innovation yields ever increasing net gains, we must avoid being lulled into complacency by simple and comforting explanations. Patterns of this sort aren’t uniform or inevitable. They can change, be localized, and even be manipulated to affect the distribution of gains and losses. There are always winners and losers as wealth, power, and capabilities concentrate and dissipate.

Shaping and Being Shaped by Tools

As media scholar John Culkin put it in his rephrasing of a famous Winston Churchill quote, “We shape our tools and, thereafter, our tools shape us.”¹⁰ This truism seems banal. It’s easy to state but much more difficult to employ in a manner that elucidates and enables evaluation of tools or humanity. We need to understand how we shape and get shaped by our tools.

This mutual shaping occurs on an existential level, through self-understanding and modeling. Philosopher John Searle thus observes:

On other levels, the dynamic constitutive relationship between humans and tools requires that we look beyond the particular function of a tool. According to Weizenbaum, “Whatever their primary practical function, [tools] are necessarily also pedagogical instruments.”¹² In other words, our tools allow us to teach each other about the tools and their functions, but also to teach us about who we are, what we can do, what is possible, and who we may become. Tools become part of the environment that shapes our beliefs, preferences, and capabilities.

In the history of techno-social engineering, language may be the most important tool ever invented. Historian Yuval Noah Harari claims that three major revolutions shaped the course of human history: the Cognitive Revolution, the Agricultural Revolution, and the Scientific Revolution. The Cognitive Revolution, which occurred about 70,000 years ago, involved the emergence of “new ways of thinking and communicating” that relied on “fictive language.”¹³ While many animals communicate, examples being the buzzing of bees and the howls of monkeys, only humans have developed language capable of describing things that don’t exist – imagined things.

Fictive language is important because of the collective affordances it provides, especially our ability to coordinate activities and work collectively in flexible ways.¹⁴ With fictive language, humans could create common myths and construct complex social institutions, such as churches and governments. People believe in “the existence of laws, justice, human rights,” for example, but “none of these things exist outside the stories that people invent and tell one another.”¹⁵ Language begets myths and large-scale human cooperation or subjugation. Myths sustain empires. Remarkably, on two consecutive pages of his book, Harari displays the Code of Hammurabi and the Declaration of Independence.¹⁶ Both claim to be rooted in “universal and eternal principles of justice.” Nevertheless, the Code proclaimed a social hierarchy among superiors, commoners, and slaves, while the Declaration proclaimed all are created equal. Neither is natural or objectively true. Both are powerful myths, imagined orders made possible by the tool of fictive language.¹⁷

The Agricultural Revolution also profoundly transformed human societies across the world. Unlike foraging societies, agricultural societies needed to manage huge amounts of mathematical data.¹⁸ Human brains were not up for the task, forcing our ancestors to develop tools to store and process data. The Sumerians developed a written language, initially “limited to facts and figures.”¹⁹ At first, this partial script did not cover the whole spoken language, but, over time, this changed. For example, cuneiform emerged as a full script allowing humans to speak not only to those around them but also across longer distances and even generations.

The Scientific Revolution introduced an array of powerful tools, starting with a change in mindset: embrace science as a means for transcending our natural human ignorance.²⁰ As humans began to accept their own ignorance, they invested more and more resources in scientific research to explore the unknown and use the acquired knowledge to develop new tools. Science incorporates systematic observation with experimentation, new forms of imagination, theorizing, and the logic and language of mathematics. Isaac Newton’s 1687 masterpiece, The Mathematical Principles of Natural Philosophy, might be “the most important book in modern history.”²¹ Newton’s theory was a powerful, general-purpose tool. It could be used to explain all sorts of physical phenomena, from apples falling from trees to the trajectory of artillery. It reflected and contributed to our changing mindset. “Newton showed that the book of nature is written in the language of mathematics.”²²

Tools are also products of human imagination. We “create little without first imagining that [we] can create it.”²³ Yet tools also shape our “imaginative reconstruction of the world.”²⁴ Tools are “pregnant symbols in themselves,” meaning that they “symbolize the activities they enable.”²⁵ The fishing spear is a tool for fishing, and it represents the capability associated with its use. As it comes or goes, so does our imagined construction of the world and ourselves within it.²⁶

Joseph Weizenbaum discusses a series of examples of this phenomenon, ranging from spears, the six-shooter, and other weapons, to “ships of all kinds,” to the printing press, the cotton-picking machine, and industrial machines. He discusses tools for measurement, such as telescopes and microscopes, and various other prostheses that extend human power, reach, and control over the environment. With each of these examples, humans gain and lose functional capabilities while their imagined world and their place within it are transformed.

Perhaps the “paramount change that took place in the mental life” of humans was our perception of time and space.²⁷ For most of our time on Earth, humans perceived time “as a sequence of constantly recurring events” rather than “a collection of abstract units (i.e., hours, minutes, and seconds).”²⁸ The clock changed everything. It was the first autonomous machine, not a prosthetic extension of human power. As the historian and philosopher Lewis Mumford observed, the clock “disassociated time from human events and helped create the belief in an independent world of mathematically measurable sequences: the special world of science.”²⁹ The clock transformed human perception of nature and consequently humanity’s role as “creature of and living in nature to nature’s master.”³⁰ Weizenbaum not only connects the clock to the rise of scientific rationalism, but he also links the clock to the fall of direct experience as a guide for human judgment and knowledge.

Historians, economists, science and technology studies scholars, and experts from various disciplines have examined other transformative tools that shaped the developmental path for human society over the past few centuries. The steam engine transformed human society, as did the telephone, the automobile, radio, television, and many other forms of technology. And today, we have the computer.

Weizenbaum fixates on the computer as a symbol and implementation of the dramatic transformations at the end of the twentieth century. Like the clock, the computer is an autonomous machine that can run on its own and perform various functions without needing human intervention. Like the clock, the computer has transformed us.

Initially, computers simply performed existing computing tasks more rapidly. Calculations done by humans in their heads, with paper and pencil, slide rules, or tab cards could be done more efficiently with computers. But, as is often the case with transformative tools, the range of uses for the computer expanded substantially with experience. Problems that were comprehensible in the language of computation could be tackled with computers. Computers gradually became integrated into an incredibly wide array of business and government processes and systems, becoming part of the structures upon which these systems depended, part of their background environment. Business decisions and, more importantly, the methods for making business decisions, such as systems analysis and operations research, grew increasingly reliant on the power of computers. As the power of computers grew, so did the perceived power of the methods and consequently their prestige and scope of application or their domain. The expanded scope of systems analysis, operations research, and a host of related computer-aided decision-making tools extended their influence on society. They became their own fields and entered the mainstream.

Fetishized Computers and Idealized Computation

Weizenbaum makes a remarkable observation that resonates with much of what we say in this book. He states:

If a computer greatly improves the carrying out of calculations used to cast a horoscope – performing a series of complex symbol manipulations, etc., and doing so much more rapidly and efficiently than an unaided human astrologer – the “improvement in the technique of horoscope casting is irrelevant to the validity of astrological forecasting.” And thus, “If astrology is nonsense, then computerized astrology is just as surely nonsense.”³³

Weizenbaum identifies a fundamental problem: We have fetishized computers (and other tools), and, as a result, we have “reified complex systems that have no authors, about which we know only they were somehow given us by science and that they speak with its authority, permit no questions of truth or justice to be asked.”³⁴ The “science” he refers to is a type of rationalism and instrumental reason that can be boiled down to “computability and logicality.” For example, he criticizes B. F. Skinner³⁵ for elevating “behavioral science” over “common sense,” and this means failing to appreciate “a common sense informed by a shared cultural experience [and that] balks at the idea that freedom and dignity are absurd and outmoded concepts.”³⁶

Weizenbaum sensed a shift in the pattern, in the co-evolution of humans and our social and technological tools.³⁷ There seemed to be an all too convenient marriage between means and ends. The tools – computers, systems analysis, science, instrumental reason – work together synergistically to define reality, just as the light under a lamp-post defines the territory within which a drunk might look for his lost keys. “[I]nstrumental reason, triumphant technique, and unbridled science are addictive. They create a concrete reality, a self-fulfilling nightmare.”³⁸

It’s difficult to appreciate how powerfully the tools we develop shape us. One of the most important ways is by shaping our imagined reality, our very beliefs about ourselves, and our preferences and values. If the ends worth pursuing are determined by our tools, by their constructed reality (the contours and contents of the lit space under the lamp-post), then nothing less than our very humanity may be at risk of being whittled away. Our imagination could become bounded by the constraints embedded in the tools and the logics they perpetuate. We are not there, at least not yet. In many ways, experience suggest that our tools generally have expanded our horizons. This is especially true when it comes to knowledge.^39,40 Nonetheless, we must remain vigilant and continue to examine our tools, our reliance on them, and what ends might linger just beyond the light.

Let us put the key point simply, as Weizenbaum did.

Problems comprehensible in the language of computation, in theory, can be solved with computers, systems analysis, science, instrumental reason, and so on. Conversely, problems incomprehensible in the language of computation, by definition, cannot.⁴¹ Weizenbaum explains how many incomprehensible problems are improperly assumed to be comprehensible. Justice, for example, is not in and of itself comprehensible in the language of computation. Many different conceptions of justice have been articulated. None of them are fully reducible to computation problems. Recall Harari’s comparison of the two codes of justice – Hammurabi’s Code and the Declaration of Independence. The subject matter of both uses numerical relationships to express conceptions of justice, and one might argue that the codes aim to reduce justice to a computational problem, but neither really does. Both rely on the prior judgment of the collective human society – or those in power – to construct the imagined reality reflected in the codes. Humans set the baselines for justice. Similarly, policy-makers and value theorists may frame social choice problems in terms of social welfare functions and thereby structure choices and trade-offs in a manner that seems quantitative, formulaic, and possibly reducible to problems comprehensible in the language of computation. However, beneath the framing itself are a host of human judgments about values and relative weights, that set the baselines necessary for computational processes to run their course.⁴²

A major social problem is rooted in the imperialism of instrumental reason and the improper assumption that all problems are comprehensible in the language of computation and thus can be solved with the same set of social and technological tools. This assumption sometimes results from erroneous understanding of problems but also from myopic infatuation with the power of our tools. We might modify Culkin’s phrase as follows: “We shape our tools, fall in love with them, and, thereafter, our tools shape us.” Weizenbaum’s argument remains fundamental, particularly as we identify ever more powerful means for solving problems comprehensible in the language of computation.

Extending the concept of the “designer’s fallacy,” we label this issue the problem of engineered determinism. What we mean to evoke by the term is the idea that society can engineer in a deterministic fashion a world that operates deterministically. This is not to say the world is naturally deterministic, or predetermined by fate or natural physical processes. Rather, it is the grand hubris that we can socially construct a perfectly optimized world if we only have the data, confidence in our tools, and willingness to commit.

“Against the Imperialism of Instrumental Reason,” a chapter in Weizenbaum’s book, opens with a parable of how the “enormous power” humans have attained through the tools of (computerized) science and technology have left humans impotent. To hammer home the point, he quotes the historian Studs Terkel:

Powerful as Weizenbaum’s account is, it remains incomplete. It has influenced our work substantially, as we’ve hopefully made clear. But Weizenbaum focused on two related sets of tools, one technological and the other social. The technological tool set centered on the computer; he emphasized its symbolic role. Throughout this work, we’ll discuss other tools within that set, such as communication networks, algorithms, and big data.

Taylor’s Scientific Management of Human Beings

There is something fundamental missing, or perhaps implicit, in Weizenbaum’s account that we need to draw out. It’s the paradigm shift that occurred at the turn of the twentieth century with the emergence of Frederick Taylor’s theory of scientific management, commonly referred to as Taylorism.⁴⁴ Taylor revolutionized the relationships between management and labor, and it’s no surprise that all the people in the Studs Terkel passage that Weizenbaum quoted were workers.

In his biography of Taylor, Robert Kanigel offers the following description:

We connect Taylorism to Weizenbaum’s account for a few reasons. First, the link strongly supports Weizenbaum’s observations about the rise of systems analysis, operations research, and computer-assisted decision-making within business management circles. It even buttresses Weizenbaum’s emphasis on the importance of the clock as a techno-social engineering tool. After all, Taylor’s system depended heavily on “efficiency experts” using stopwatches to conduct time studies, a critical source of data used in scientifically managing workers. Second, the connection helps explain the rise of instrumental reason and scientific approaches to managing human beings and their social relationships in the workplace and elsewhere. Third, the association focuses on a set of techniques that preceded and did not depend on the computer. These techniques have been strengthened greatly by computers and adjacent technologies such as sensors, data analytics, communications networks, and so on. Yet the normative validity or legitimacy of the techniques must be evaluated independently, and, to do so, we must resist the pull of fetishized innovation and unwarranted claims of technological inevitability.

Taylor developed his techniques, his theory of scientific management of humans in the workplace, in the late nineteenth century and early twentieth century. Taylor saw substantial inefficiencies in factories and other workplaces, and he attributed many of the inefficiencies to mismanagement of labor. As a young man, Taylor had worked as a shop foreman, attempted to get the most out of his workers, and begun to diagnose the inefficiencies he observed as a product of poorly structured incentives, unmotivated and sometimes shirking laborers, and, perhaps most importantly, a tremendous knowledge gap that rendered management ineffective. Managers knew too little about the workers, their tasks, their capabilities, and what motivated them to work.

Over decades and across different workplaces and even industries, Taylor carefully studied workplaces, workers, and their work. He examined minute details of tasks performed, and, based on the data collected, sought to optimize performance in terms of increased efficiency and productivity. Taylor’s system was generalizable. In other words, his system was not limited to a particular workplace, nor was it limited to any set of time and motion studies.

At one level, Taylor’s scientific management system is a type of data-dependent technology.⁴⁶ Taylorism is one of the best early examples of data-driven innovation, a concept currently in vogue.⁴⁷ Taylor’s system included the techniques for both gathering data and putting such data to use in managing people. Taylor’s system thus encompassed the surveillance techniques employed by the “efficiency experts,” their use of stopwatches and careful visual observation of task performance under varied incentive schemes. For example, he would offer a worker being studied a much higher wage than the prevailing market wage to test worker capability and task performance under different conditions, and, if possible, push prevailing views about what workers could accomplish and increase productivity. Taylor and his disciples relied on personal observations written in notebooks and careful analysis of various inputs, outputs, processes, and procedures across the many workplaces they studied.

Taylor’s critics emphasized that Taylor’s scientific management was anything but scientific. They alleged (accurately in many cases) that Taylor’s prescriptions for management often had an ad hoc flavor to them. When the data was incomplete, Taylor relied on his own judgment, which amounted to little more than a fudge factor or unwarranted exercise of managerial discretion and could not be considered scientific.

Yet the managerial data gaps would close. Twentieth-century technological innovations, ranging from the computer to the camera, have dramatically upgraded the capability of managers to gather, process, evaluate, and act upon data.⁴⁸ Not surprisingly, Taylorism spread like wildfire across industries and beyond the factory floor, to hospitals, schools, and various other contexts.⁴⁹ As Kanigel put it, “Taylor’s credo of rational efficiency has burned its way into the modern mind.”⁵⁰

Taylorism is an applied version of the instrumental reason and rationalism discussed by Weizenbaum. Consider how Taylorism defines both means and ends. As a technology or management technique or system, Taylorism is obviously branded as a means. The problem to be solved also was unambiguous: inefficiencies plagued the workplace leading to waste and lost productivity. Taylorism and Fordism are famous both for their underlying objective, namely, to increase efficiency, quality, and productivity for the ultimate benefit of managers, owners, and capitalists, and means, specifically by managing factory workers in various ways that get them to behave like machines.

Deeply embedded throughout Taylorism, the ends of productivity and efficiency are not only assumed to be paramount but also to be comprehensible in the language of computation. That is the heart of Taylor’s claim that his system constituted scientific management; it is reflected throughout the system itself. Workers were, in fact, conceived as inputs, cogs, resources, etc.; their work was broken down, analyzed, and programmed.⁵¹ Taylor and his disciples assumed it was all comprehensible in the language of computation. At a fundamental level, Taylorism was a revolutionary system for engineering humans. As Taylor famously declared, “In the past the man was first; in the future the system must be first.”⁵²

The assembly line is a particularly salient and culturally recognized example.⁵³ An assembly line is a manufacturing process involving the progressive assembly of parts into a whole product, where the semi-finished assembly moves from one work station to the next in a linear fashion. While assembly lines pre-dated Taylor and Ford, Ford famously optimized the process for mass production. Fordism combined product standardization, systematized use of assembly lines wherein unskilled laborers used special purpose tools at different stages, and the principle that workers should be paid higher “living wages” to both provide better incentives and enable them to purchase the products they made.⁵⁴

A critically important aspect of this type of techno-social engineering is the environmental nature of the means, the way in which the managers employing the management practices advocated by Taylor (and adapted by Ford) reconstructed the physical and social environments within which their workers worked. Managers could leverage control over the environment to control those within the environment in various subtle but powerful ways. Similar to how the clock reconstructed our environment and us,⁵⁵ time and motion studies fueled task and schedule management in the workplace. As Harari describes:

The factory thus not only produced whatever widget the company eventually sold (e.g. Harari’s shoes or Ford’s automobiles), but it also produced machine-like humans, sometimes referred to as automatons.⁵⁷ Kanigel states:

Taylorism Criticized Yet Expanding

Critics of Taylorism recognized and railed against these effects on workers, but architecting the environment (optimizing it, really) to achieve these particular effects is the technological innovation to note.⁵⁹ “The Industrial Revolution turned the timetable and the assembly line into a template for almost all human activities … [Soon] schools too adopted precise timetables, followed by hospitals, government offices and grocery stores.”⁶⁰ These are interesting examples because they define and are defined by the physical spaces, social institutions, and increasingly technologies that together constitute particular environments designed to engineer humans.

Today, even though the assembly line “defines surprisingly little of modern manufacturing,”⁶² Taylorism is pervasive. Taylorism had its ups and downs across business schools, management consultancies, and factory floors throughout the twentieth century. Some companies and even industries moved away from it to alternative systems for managing labor.⁶³ Nonetheless, the basic principles of Taylorism have become deeply embedded in how society conceptualizes all sorts of management, ranging from businesses to government to schools to amateur athletics to child rearing. Again, Robert Kanigel put it well:

With ever growing data about human labor, task performance, and so on, the trend in workplace surveillance and management has only grown and expanded in scope,⁶⁵ and it is likely to continue. Until when? How far can it go? What happens if taken to the extreme? What would it mean to Taylorize human labor fully? One thing it would mean is that we would have accepted, even if only tacitly, the contention that management of human labor is a problem comprehensible in the language of computation. Another thing it would mean is that any boundary around the workplace, employment, or even the idea of work itself would dissipate because human labor is not constrained to any such boundary.

Modern data-driven micro-management of human resources (time, attention, effort, etc.) across various industries is simply a form of Taylorism extended beyond formal employer–employee contexts. Like vehicles, physical space, and computing resources, human physical labor can be optimized for on-demand allocation determined by data and algorithms. The Taylorist vision of efficient management is focused on minimizing costs associated with misallocated or wasted human capital, effort, and attention. Ironically, soon, eliminating productive inefficiencies that arise from mismanagement of labor might entail getting rid of human managers altogether and turning instead to smart technologies.⁶⁶

There is no reason to limit technologically-optimized-and-implemented Taylorism to traditional work, however. The logic easily extends to a much wider range of actions that depend upon human labor (time, attention, effort, etc.), whether driving a car, caring for one’s children, exercising our bodies and minds, or any other human activity.⁶⁷ In the not-so-distant future, intelligent technological systems – not necessarily sentient ones – may be deployed to maximize human productivity throughout our lives.

Humans are naturally inefficient. We are often unproductive and costly to sustain. One way to understand the power of the Taylorist logic, particularly as extended beyond the workplace, is that it entails minimization of various costs associated with humans being human. For humanists, this is deeply troubling. Some will emphasize the potential upsides, rooted in increased convenience, entertainment, happiness, and welfare. They’ll argue that, overall, we’ll all be much better off in a world optimized to deliver cheap bliss. In subsequent chapters, we’ll revisit this debate about humanity and the world we’re building.

Introduction

Previous chapters explained techno-social engineering and considered illustrative examples. Here, we discuss in detail the current legal and technical architecture of electronic contracting, a surprising case of techno-social engineering. The design of this environment might incline people to behave like simple stimulus-response machines and become increasingly predictable and programmable.

Contract law shapes the transactional environments where people formulate legally respected and binding commitments and relationships. In general, contract law is understood to be a form of liberating social infrastructure that enhances individual and group autonomy. Unfortunately, conventional understanding seems to be wrong. Contracting practices have changed dramatically over the past half-century to accommodate changes in economic, social, and technological systems. Today’s contracts might be more liberating for some (e.g. firms) than others (e.g. consumers).¹ As implemented in electronic architecture, contracts may be oppressive.

Designers arrange the digital contracting environment to create a practically seamless, transaction cost minimized user experience. Rather than requiring people who intend to use online services to read lengthy pages filled with boilerplate legal jargon – jargon they can’t reasonably be expected to understand and won’t be able to negotiate with – a simple click of the mouse, with mere conspicuous notice of the existence of terms, suffices to manifest consent for entering legally binding contractual relationships.

There’s plenty of legal debate about the legitimacy of contracts that this mechanism creates, including heated disputes over whether opportunities for opting-out do enough to preserve autonomy. Some celebrate efficiency: the seamlessness of the interaction and the minimization of transaction costs. From this perspective, electronic contracts are a perfectly rational means for consumers to quickly access desired services and goods. Others, however, lament unfairness, the one-sidedness of take-it-or-leave-it contracts of adhesion that don’t foster the “meeting of the minds” that once seemed to be the core principle of contract law.

While a significant debate about the content of contracts is taking place amongst scholars and in the courts, a crucial omission limits the critical conversation and prevents us from appreciating the full power of online contracting. Few discuss the negative impact the electronic contracting environment has on our habits and dispositions, and, more generally, on who we are as human beings.² It can be uncomfortable to focus on these issues and the possibility that electronic contracts are objectionable as a matter of public policy because they condition us to devalue our own autonomy.

We examine two related ways to characterize the contracting problem.

First, we claim that the electronic contracting environment should be understood as a techno-social tool for engineering human beings to behave automatically, like simple machines. To validate this claim, we develop testable hypotheses and explicate the underlying theory.

Next, we describe the contracting problem in Taylorist terms, as a system of scientific management that’s directed toward consumers.³ This view highlights how consumers, like laborers in Taylorist workplaces, are conditioned (and possibly deskilled) to behave in ways that largely are determined by efficiency-oriented system designers. Viewing electronic contracting through the lens of Taylorism connects our discussion to a broader set of techno-social engineering problems.

Although we don’t believe the click-to-contract mechanism necessarily was intended to be a tool for techno-social engineering humans, it nonetheless may have become one because of its gradual optimization and the expanding scale and scope of its deployment. Emergent Taylorism might seem oxymoronic because Taylor developed tools for management who would use them in a direct and deliberate manner. However, this line of thinking places too much emphasis on intentionality and managerial responsibility. We aren’t making claims about the intentions of designers. Instead, we’re focusing on environmental tools and their impacts on human behavior, and we’re questioning the underlying logic of optimization.

If the electronic contracting environment conditions human beings to behave like simple stimulus-response machines, and if repeated interaction with this environment has lasting effects, then systemic reform of contract law might be warranted for reasons that go beyond the arguments proffered in the standard literature.⁴ Our argument is not about the goodness or badness of contract terms per se. Nor is it about the outcomes in specific contracts, transactions, or cases. Rather, our concern is with the social costs associated with rampant techno-social engineering that devalues and diminishes human autonomy and sociality.

One caveat before proceeding:

Throughout this chapter, we use endnotes marked with the header “Empirical Query” to identify hypotheses and claims that need to be verified through empirical investigation. These statements are not defensive. Nor are they signs of weakness in our theoretical argument. Good theory leads to good empirics. It reveals questions that are worth investigating and provides structure and boundaries for further theoretical and empirical inquiries. In short, we note these empirical queries to encourage others to work with us or independently on pressing research questions.

The Experience of Electronic Contracting

In her book, Boilerplate, law professor Peggy Radin presents two conceptually familiar worlds to help orient our thinking about contracts.⁵ In World A (for Agreement) “contracts” are actual bargained-for exchanges between parties who each consent to the exchange. Traditionally, this is how many imagine contracts work in an ideal world. In World B (for Boilerplate)⁶ “contracts” are standardized form contracts, also known as contracts of adhesion and take-it-or-leave-it contracts. The logic of this world captures what we often experience as consumers. Radin explores how the use of boilerplate has expanded significantly and what the implications of such expansion might be for consumers, contract law, and society. Most importantly, she explains how “boilerplate creep”⁷ gradually erodes public ordering (e.g. law of the people, political and social institutions, government) and replaces it with private ordering (e.g. law of the firm, market).⁸ Her analysis is comprehensive and includes many examples of boilerplate offline and online.

The beauty of the Internet is its scope and diversity, the incredibly wide range of websites, interactions, conversations, communities, activities, and transactions available to each of us. Not surprisingly, navigating webpages and online content carries an incredible amount of legal baggage. Much of it is governed by electronic contracts. If the Internet followed Radin’s model of World A, transaction and information costs could be stifling. Lawyers have worked to reduce friction by drafting boilerplate agreements. And website designers who architect the digital environment have played a role in this process, too. Both are “choice architects” who frame the choices or options that consumers are presented with.⁹

Consider a system where a choice architect designs the online environment to make using digital services as seamless as possible. Essentially, the choice architect structures the environment to minimize the burden placed on the user when she is consenting to terms. It’s an environment where the rational response to terms of use pages (links, really) requires little thought and imposes no burden on the user. After all, “acceptance” is reduced to a mere click of the mouse. If this is starting to sound familiar, it’s because it is our current online contracting environment.¹⁰ Technically, the contracting environment is choice-preserving in the sense that users retain their autonomy. They can opt out of the web applications services – so long as they’re willing to accept that the social or professional costs of doing so can be high. The key point, though, is that, in this context, it is completely rational for a user to blindly accept the terms of use. To read the proposed and, frankly, imposed terms would be a complete waste of time, an irrational and ultimately futile exercise.¹¹

It seems natural to distinguish this scenario from those that involve the government. Contracting is, after all, a private affair: the holy grail of private ordering. While many of us may feel ashamed, cheated, disappointed, or otherwise less than satisfied with our contracting behavior, we cannot complain about coercion, much less government paternalism. Or can we? The answer depends on just how dramatically contracting has changed over the past half-century to accommodate changes in economic, social, and technological systems, both off- and online. According to law professors Robert Hillman and Jeffrey Rachlinski, “[t]he Internet is turning the process of contracting on its head.”¹²

One aspect of the dramatic change in contracting practices is its pervasiveness and relevance in our everyday digitally networked lives. The current scale and scope of private ordering through written contract is unprecedented. It may truly be a massive “orgy of contract formation.”¹³ We haven’t attempted to quantify the number of written contracts the average person enters during her lifetime. However, we suspect the following: (i) the number has steadily, if not exponentially, increased over the past half-century;¹⁴ (ii) the rate of meaningful participation in negotiating terms has steadily decreased;¹⁵ and (iii) the number of written contracts concerning mundane affairs has increased, if not skyrocketed. By mundane, we mean ordinary, everyday affairs for which a written contract would be cost-prohibitive and inefficient in the absence of boilerplate.¹⁶

We could add a fourth hypothesis about the increasing number of written contractual agreements concerning trivial affairs, which we imagine to be something like the offline purchase of a lollipop. Such “lollipop contracts” are unnecessary and wasteful. Why bother to cement relationships with a written contract when it concerns trivial affairs? One answer becomes apparent when we flip the question around and ask: Why not? It’s no bother because the transaction costs associated with forming a written contract are trivially low in the electronic contracting context. Such contracts might not really be about the legal relationship. They serve other purposes, such as setting standards, incrementally contributing to boilerplate creep, and further replacing public ordering with private ordering.

Let’s put this in personal terms. Do you consider yourself experienced in contracting? Have you negotiated many contracts? If so, what did the negotiation entail? For many people, contracts are significant legal affairs that involve insurance, loans, employment, and other major life transactions. We also enter many contractual agreements that concern less significant affairs. For example, a contract for a single service – say, to have a porch painted – or an ordinary sales contract – say, for a household item.¹⁷ Consider how much time you spend online each day, how many different service providers you interact with during such time, and the percentage of those interactions governed by contracts. Now perhaps you understand the intuitions behind our hypotheses.¹⁸

How many written contracts have you entered into during your lifetime? If we asked this question in the distant past, decades ago or a century ago, the answer probably would be orders of magnitude less than the answer provided by current readers.¹⁹ Future readers of this chapter may find the question odd because the idea of distinct, identifiable contracts may be at odds with their experience of completely seamless contractual governance. This raises an interesting theoretical issue. Freedom of contract requires the correlative freedom from contract.²⁰ When contract becomes automatic and ubiquitous, both disappear. There is no freedom.²¹

Our modern, digital, networked environment is architected with technological systems that operate mostly in the background, behind user interfaces that magically hide the complexity and incredible number of actual and virtual machines, processes, data flows, and actors. These interfaces also happen to be means by which we enter a substantial number of legally binding relationships with service providers and other parties.

We routinely enter contracts by clicking a virtual button, whether using a mouse, touchpad, touchscreen, or remote control. The experience is hardly a momentous occasion and often is barely notable. This is a designed feature and not an accidental bug – a point we’ll subsequently revisit. Experientially, it feels²² nothing like signing on the dotted line of your mortgage, employment contract, or insurance agreement.²³ But the legal effect is the same. When you click “I agree,” you manifest consent to enter a legally binding contractual agreement. With whom? Most often, the other party is the service provider. Providers include website operators, software or app providers, smart television companies, and so on. Sometimes, there are other parties – affiliates or third parties – who also are part of the deal. For example, you may enter an agreement with the website owner and agree to let her share your data with affiliated entities. But these entities are not typically contracting parties. They’re just beneficiaries of the contract between you and the direct service provider. Typically, they have side-agreements with the service provider, but not with you.²⁴

The electronic contracting environment we’re all familiar with is thus a product of, and completely contingent upon, evolved contract law and practice. Critically, this includes technological systems through which we interact, communicate, transact, and form relationships. Both could be different. Contract law could have accommodated changes in economic, social, and technological systems differently.²⁵ What we have now is neither necessary nor inevitable. Fortunately, contract law still can change.

The technological systems through which we interact, communicate, transact, and form relationships also could be different. They are designed and optimized to obtain predetermined results given the legal and technological constraints and the predictable behavior of visitors. The technological systems reflect a series of design choices in their deployment, in their architecture.²⁶ Contract law has permitted and encouraged the development of an electronic contracting environment in which it would be irrational for users to read the terms of the contract. The technological design of the user interface – a click-to-agree button coupled with a link to a separate file with potentially endless pages of terms – is merely an implementation of what contract law has allowed.

It’s efficient. Each online contract we enter is presumably in our interest and cost-benefit justified. Otherwise, we’d choose to abstain. Price and service are presumably the driving factors. All else is a mere transaction cost to be minimized, buried in terms of service that no rational person would read. You retain your autonomy and may choose to leave, but that’s it. Quite simply, you may take it or leave it.

At least, that is how it seems. But what exactly is the price? Often, we have no idea because the true price is not money exchanged. Frequently, users do not pay money for services. The apparent sticker price is zero. Even when users pay money (e.g. for a subscription or a $0.99 app), the sticker price often is discounted; other side-payments exist. The actual price users pay for the services websites provide includes all the information the sites collect about them.²⁷ As Radin shows, the actual price also includes the various legal rights we may have given up.²⁸ Further, it includes the commodification of users through pseudo-relationships. Websites act as brokers for user data and relationships to generate a weird sort of B-world social capital that greases whole series of transactions as well as the slippery slope.

How, then, do we evaluate the relationships being formed as users visit websites when the relationships extend well beyond the website and users to include third parties, such as advertisers, website affiliates, and others who may be direct or incidental third-party beneficiaries of the user-site transaction? One might reject our characterization of the relationships as part of the price that users pay. This puts the skeptic in an awkward position. How else can we reconcile the flow of third-party benefits? Website users are the objects of the various side-agreements, after all. As people like to say about Facebook and Google, users are not really the consumers. Rather, users are the product being consumed by all the advertisers and other third parties with whom Facebook and Google have side-agreements.²⁹ The point generalizes well beyond Facebook and Google. Apple’s App Store, Amazon, Microsoft, and many other online services work this way. User as product describes much of the digital networked environment, and consequently much of our everyday lives.³⁰

Still, given the numbers (of users, sites, transactions, third parties, data, contracts), the design of the electronic contracting interface seems perfectly rational. In many offline contexts, consumers don’t read most terms and only deliberate over a small number of salient variables, like price, quality, and timing. Comparable deliberation in the online context might take too long, be too complicated, and lead to fewer transactions. It’s much easier to hide the complex details³¹ and nudge users to click “I agree” without deliberation over any terms.³²

Our current online contracting regime is a compelling example of how our legal rules coupled with a specific technological environment can lead us to behave like simple stimulus-response machines – perfectly rational, but also perfectly predictable and ultimately programmable. The environment disciplines us to go on auto-pilot and, arguably, helps create or reinforce dispositions that will affect us in other walks of life that involve similar technological environments. These similar environments might turn out to be everywhere given the direction of innovation.³³ Although Radin does not frame the issue in terms of conditioning or programming, she notes how status quo bias can lead us to continue down a familiar path:

It turns out, however, that the effect may be more powerful than Radin suggests. A laundry list of heuristics (i.e. “rules of thumb”) and cognitive biases might reinforce our behavior.³⁵ Decision fatigue can be overwhelming.³⁶ The opportunity costs of slowing down and deliberating can be high.³⁷ And, habits with their automaticity and corresponding behavioral path dependencies are incredibly powerful.³⁸

Not surprisingly, boilerplate creeps, which only exacerbates the effects as we become more comfortable, complacent,³⁹ and easier to nudge.⁴⁰ Particularly worrisome is how boilerplate creep enables both surveillance and nudging, which are both creep phenomena as well.⁴¹

Consider how the electronic contracting environment optimized for websites has migrated to mobile devices and apps, and further to smart televisions and beyond. The parties, legal relationships, technologies, services provided, data generated and collected, and implications vary dramatically across these contexts. Nonetheless, in general, our behavior remains the same: perfectly predictable, seemingly rational, and hyper-efficient, check the box, click “I agree.”⁴²

Just think for a moment about how the relationships and privacy implications differ when you shift from website to smart television. There are different service providers, different third-party affiliates in the background, different technologies and services, and different types of data. A smart TV might be in your living room, and it might even have a microphone (for those karaoke sessions). Others have diagnosed the problem. We only want to emphasize how the stimulus-response mechanism works similarly despite how different the implications might be.

We might be disposed to deliberate for at least some of these transactions, to stop and think about what we’re getting ourselves and other people (e.g. family members who share the smart television) into. But we’re being conditioned not to do so.⁴³ For example, if you decide to investigate the privacy policy for your smart television, you’ll likely see the “1/50” page count at the bottom of the screen, shrug, and click the back arrow, behaving just as you’re supposed to.⁴⁴

Some will resist our characterization and believe that while the objective theory of contracts is flawed,⁴⁵ nobody is in danger of mindlessly following scripted programming. You might believe that you really decide for yourself when you click “I agree” and are decidedly not pre-programmed to do so. This is a common reaction. It was ours at first. Many others have reacted similarly. People assume that, at some point in their past, they consciously adopted a strategy to deliberate once in a while and otherwise to trust in markets,⁴⁶ the wisdom of crowds,⁴⁷ and watchdog journalists and advocates.⁴⁸ If a contract has deep flaws, surely others will identify egregious terms and presumably a court would refuse to enforce them.

But what justifies the assumption or trust? Put aside the merits of such a strategy and the beliefs upon which it is based. The question is whether we in fact ever really adopt such a strategy based upon actual deliberation about the merits. Given the empirical difficulty of determining what’s actually causing us to act, self-assessing the contents of our minds is an introspective blunder.⁴⁹ For starters, optimized environments might be architected to make you feel as though you’re making deliberate choices when you’re not actually doing so.⁵⁰ In other words, one effect of techno-social engineering might be that you’re disciplined to overestimate how much freedom actually lies at your disposal – that you mistake the illusion of choice for the real thing.⁵¹ Moreover, just as an immediate click may be rational in the immediate context, trusting in markets, the wisdom of crowds, watchdogs, and courts may be the only rational choice across contexts given the incredible number of interactions mediated in the same manner. Crucially, the seemingly efficient rationality of both the micro and macro choices is completely contingent on the designed architecture of the electronic contracting environment and the scale and scope of its deployment. Crowds, watchdogs, and courts do catch some shockingly egregious terms and conditions, serving a useful function. However, they can only do so much.

Another complication concerns the very definition of choice. When we make decisions in the electronic contracting context (e.g. to click or not), it remains unclear whether we’re exercising judgment and making a genuine choice. The electronic contracting environment may be designed to stimulate instinctive, heuristic thinking (referred to in the scientific literature as System 1 of the brain). Yet the user may feel or be led to believe that she has engaged in rational deliberation (System 2).⁵³ In the moment and even in hindsight, the stimulus-response behavior of simply clicking is perfectly rational – again, reading terms and conditions is a waste of time and insisting upon negotiation is futile – and so it’s easily mistaken as a product of deliberation.⁵⁴

Put otherwise, suppose choice architects – website, application, or other interface designers – arrange the relevant stimuli in an electronic contracting interface to trigger an automatic click. They might describe their optimization problem in terms of minimizing decision time or time-to-click or some comparable metric. The designer’s goal is to consummate transactions with minimal transaction costs, which generally means rapid click-through to agreement. Consumers often have the same goal. For most, the electronic contracting interface is a mere hurdle, an obstacle to getting the content or service they’re looking for. Deliberation is generally wasteful in this context, with significant opportunity costs. Triggering System 1 seems to be a win-win(-win-win- … winⁿ).⁵⁵ This gradually becomes a truism. When we talk about this with people, they often suggest that they are behaving rationally, which is true, given the futility and costs associated with deliberating or reading terms. But they also insist that they deliberated at some prior point in time (which is left unspecified because they can’t recall precisely when it occurred) and decided, rationally, to adopt a strategy of clicking through quickly and waiting for bad news to arrive later.

System 1 thinking does not always entail behaving like a simple machine,⁵⁶ but in this context, we think it does. First, the relevant human capability being engineered and examined by us is cognitive processing of an instrumental decision.⁵⁷ In this setting, System 2 corresponds with deliberation or “thinking slow,” and System 1 corresponds with automatic behavior or “thinking fast.”⁵⁸ Second, in this setting, the techno-social environment nudges at the micro level of a single interaction and contributes incrementally to decision fatigue. Other factors also affect participants and reinforce the techno-social environment at the macro level. People who repeatedly interact with this system end up performing simple scripts (whether learned or programmed) in response to stimuli, and, in this particular fashion, behave like a simple machine. Thus, we might go a bit further and say that, in this setting, System 1 corresponds with not thinking (rather than thinking fast) because the automatic behavior is scripted or programmed. It may be counterintuitive to equate rational behavior with scripted or programmed behavior because the former seems good and the latter bad. But the two characterizations are not mutually exclusive. Neither is inherently good or bad. Everything depends on the context. Sometimes following a script is perfectly rational. For example, we all do so daily when making pleasantries with others. What makes the electronic contracting environment special is that it is designed to make it irrational to break away from the click-to-contract script.

One strong objection to our analysis is that we have long behaved this way, that automatic assent is neither new nor unique to online contracts. Most people don’t read the terms of insurance contracts, mortgages, or the vast majority of offline contracts. Supposedly, we have long given up on reading and negotiating contracts, especially in business-to-consumer contexts. But this counterargument is predicated upon a subtle mistake. Not reading does not mean abandoning deliberation. Skipping over the fine print does not mean we bypass exercising System 2 thinking altogether and rely exclusively on System 1. We still focus on the most important and salient terms of insurance contracts, mortgages, etc. At some point during contracting, we at least deliberate over the magnitude of price.⁵⁹ The same is likely true for online purchases that involve “big ticket” items. That is, when the price or quality of the good or service being purchased is salient to the consumer, the consumer presumably deliberates, both online and offline. But as noted earlier, such deliberation is absent from many electronic contracts precisely because the apparent price is zero and the hidden price is in the unread terms, the data exchanges, and the attenuated and sometimes uneasy relationships brokered with various third parties. By design, we’re led to trust blindly, as if we had relationships worthy of such trust when we really don’t.⁶⁰

The electronic contracting environment is thus another illustration of techno-social engineering of humans. If our characterization is correct and the environment effectively programs human beings to behave like simple stimulus-response machines – perfectly rational, predictable and programmable – then it’s important to determine if the process is dehumanizing.

Thus far, we haven’t taken a strong normative position. Instead, we’ve primarily aimed to draw attention to a phenomenon that deserves more attention from legal theorists and ethicists. The majority of them seem to deny its existence or fail to recognize its contours. The techno-social engineering we’ve described affects two basic human capabilities, the capability to deliberate and the capability to relate to others. As we discuss below and in later chapters, we believe these capabilities are fundamental to being human and at risk of being lost through rampant techno-social engineering.

Electronic Contracting and Taylorism

In this section, we again turn to Taylorism. It’s a useful lens for examining the design of electronic contracting and the phenomenon of electronic contracting creep.

As discussed in Chapter 4, in the late nineteenth century and the early twentieth century, Frederick Taylor developed his theory of scientific management of humans in the workplace.⁶¹ His work was motivated by concerns about efficiency. Taylor saw substantial inefficiencies in factories and other workplaces. He attributed many of the inefficiencies to mismanagement of labor. Taylor carefully studied workers and their work, examining minute details of tasks performed in the workplace, and, based on the data collected, he developed a system for optimizing their performance with the objective of increasing efficiency and productivity. We consider Taylorism to be one of the building block philosophies that today supports widespread techno-social engineering of humans.

The human-computer interface that we’re calling the electronic contracting environment is but one example of an unheralded modern extension of Taylorism outside the workplace. There are many others, some of which we discuss in subsequent chapters. The crucial point we would like to make here is that the underlying structure, logic, and effects of electronic contracting and Taylorism may be the same. That is, like an idealized Taylorist workplace, the electronic contracting environment is optimized to minimize transaction costs, which, in this context, largely consist of human time and attention. Human deliberation, especially in the electronic contracting context, tends to be unproductive. Moreover, as described previously, the impact on consumers also has a Taylorist flavor, in the sense that consumers perform scripted routines, habitually, automatically, like simple machines. Yet, in contrast with laborers who at least understand they are being managed and optimized like cogs in a machine, consumers are blissfully unaware of the techno-social engineering that they’re experiencing.

The electronic contracting environment, interface and architecture evolved considerably over the past few decades. Our hypothesis is that it was optimized along the lines that Taylorism suggested, although in a more emergent and organic fashion.⁶² Given this hypothesis, we formulated the following sets of questions:

1. i. What was the metric or value being optimized? How was it determined? Did it change? How was it measured and communicated? Did designers test it with humans? How so?

2. ii. Did people do the functional equivalent of time and motion studies to figure out how human beings interacted with the interface and how they “performed” certain predictable tasks (such as browsing and clicking)?

3. iii. Were studies reported in professional journals? What was common knowledge? Industry custom and practice?

Our research hasn’t yielded fully satisfactory answers. In part, this is because we have found only limited published studies that directly address the design of the electronic contracting environment, interface, or architecture.⁶³ This leaves us with no choice but to rely on sources describing website design more generally⁶⁴ and informal personal conversations with experts, web designers, lawyers, and others about their experiences.⁶⁵

Website design varies considerably in terms of the design objectives. Among the most common variables to be optimized are the following: user experience, user attention, time spent on the site, task performance (which can range from clicking on advertisements to interacting with other users or features of the site), and communication of messages or ideas (which can range from advertising to educational). Each of these can be broken down, measured, and optimized in various ways.

In the context of pages of a website associated with electronic contracting, the main design objectives include:⁶⁶

• minimize transaction costs (i.e. all costs attributable to the pages and associated with consummating a transaction, including time and effort spent by the consumer);

• maximize retention (i.e. the rate at which consumers agree and don’t leave the site);

• minimize design and operational costs (meaning the costs associated with creating and operating the contracting architecture);

• maximize enforceability (i.e. the resulting contract is enforceable in court).

Other design variables might be included, such as attractiveness, simplicity of the design, or how well the design fits with the brand or the rest of the website. “The design of an online interface involves many … choices [that can shape impressions], whether choices about font, lines, colors that ‘convey mood and provide a setting for the information’ or choices about how interactivity can evoke different personalities.”⁶⁷ It’s worth noting, however, that most of the major objectives for website pages – especially user experience and communication of particular messages or ideas – aren’t relevant for electronic contracting pages, except in the negative sense that designers would like to minimize any interference with the website’s primary pages.⁶⁸ The electronic contracting pages are functional and task-oriented. They usually have little to no personality. Accordingly, while the prioritization of these objectives might vary across sites, we believe the first objective – minimize transaction costs – matters most. The second could be framed in terms of opportunity costs and folded into the first. The third seems less important, mainly because the marginal costs associated with designing the electronic contracting pages of a website are quite small. Keep in mind this does not include the costs associated with a lawyer drafting the terms and conditions, which is not a design or operational cost.⁶⁹ Anecdotally, our informal conversations confirmed that these were among the least interesting but easiest pages of a website for a designer to create and operate, especially given the ease of copying and standardization. The priority given to legal enforceability varies considerably across websites, depending on, among other things, the degree to which service providers anticipate potential litigation. The gradual evolution of the electronic contracting interface in response to court decisions demonstrates the significance of this factor and how the shock of a court decision can lead to new design practices spreading relatively rapidly and becoming industry standard.⁷⁰

We need to distinguish two types of website pages related to electronic contracting. First, there are website pages displaying contract terms and conditions. These static pages vary in terms of where they are located within a website, how a customer accesses the pages, their format, length, style, content, readability, and so on. Usually, people access them through a hyperlink, which may be a general Terms of Service link found on every page, or, if coupled with a required action on the part of the consumer (e.g. “Click I agree” or filling out a form), the hyperlink may be proximate to the button or form.⁷¹ In general, these pages are not terribly different from hard copies of offline contracts. They’re filled with legal jargon that’s non-negotiable and incomprehensible to the average person.⁷² The substance of some of the terms and conditions are meaningfully different because of the nature of the website services, relationships, data practices, and so on. Regardless, these are not design features, and so we generally leave them aside in the ensuing discussion.

A significant difference is that online contracts often suffer from bloat because the marginal costs of adding legal language are vanishingly small. This is an important design feature. The nature of the digital medium allows for cheap bloat, and one consequence of seemingly endless terms is obfuscation and raising the costs for consumers to read, engage, or deliberate. Non-negotiable, incomprehensible, and seemingly endless contract terms can be so daunting that no one bothers. This design feature affects consumer behavior on a micro contract-by-contract basis and on a macro endless-stream-of-contracts basis. To their credit, some firms have innovated in their design of these terms and conditions pages to improve readability or otherwise help consumers (consumer interest groups and markets more generally) better understand the contracts and identify important terms.⁷³

The second type of website page includes some active mechanism for formally entering a contract, where the mechanism purports to satisfy the legal requirements for contract formation. Today, the most salient example is the click-to-contract mechanism, often referred to as a “click-through” or “click-wrap” agreement. For smartphones and tablets, it is the tap-to-contract mechanism.⁷⁴

This architecture originated with software delivered in a box. Upon attempting to load the software on a computer, the consumer would be unable to proceed unless the consumer clicked a box confirming “I agree” or something similar.⁷⁵ The design innovation has three important characteristics. First, it forces the consumer to act physically and affirmatively in a manner creating a definitive record, which has both evidentiary and doctrinal significance.⁷⁶ The click-to-contract mechanism satisfies the objective theory of contract, which bases contract formation on objectively reasonable manifestations of assent, and, consequently, courts generally have upheld them.⁷⁷

Second, the click-to-contract design innovation created obstacles, or speed bumps, of various sizes, that slow down the consumer. The smallest is an “I agree” button immediately clickable. Slightly more effort is required for a customer to check a box acknowledging that she has read the terms and condition before she can click the “I agree” button. Even more substantial are the software interface designs requiring a user to scroll over some (or all) of the terms and conditions before clicking the “I agree” button. Even more options are possible, even though they aren’t practiced much, to our knowledge. For example, one could require consumers to type particular text from the terms and conditions or answer substantive questions about the contract. In the end, many different designs were possible. What would be the point of adding more substantial speed bumps? Why slow down customers? Why prevent them from getting what they are really after – the content or service just beyond the speed bump? As privacy and security scholars have noted, getting people to slow down and engage with the terms and conditions – even security warnings – takes quite substantial speed bumps.⁷⁸

Third, the click-to-contract design innovation provides an aesthetically pleasing mechanism for executing standard form contracts.⁷⁹ The separation of the two types of pages is critical. In contrast with paper standard form contracts, the terms can be hidden on another page. As a result, the interface is more aesthetically appealing and the user experience is practically seamless. These design features contribute to ease and effectiveness of techno-social engineering.

In practice, as the click-to-contract mechanism evolved both off and on the Internet, the optimizing logic of Taylorism seems to have taken hold. So, returning to the questions we posed:

Website designers and other architects of the electronic contracting environment primarily approached the optimization problem in terms of click-rates, which strongly correlated with time and attention. They may not have understood what they were doing as “an optimization problem.” Their basic goal was to get visitors to observe and then click the “I agree” button as soon as possible.⁸⁰ The primary transaction cost to be minimized was the consumer’s time and attention.

In the field of human-computer interaction, designers recognize the need to minimize user “interaction cost.” This is defined as the sum of mental and physical efforts users must expend while interacting with a site to reach their goals or expected benefits.⁸¹ There may be a variety of user actions and goals, depending upon the website. Raluca Budiu, the Director of Research at the Nielson Norman Group, a leading user experience research and consulting firm, explains that the following user actions contribute differently to the total interaction cost:

• reading

• scrolling

• looking around in order to find relevant information

• comprehending information presented to you

• clicking or touching (without making mistakes)

• typing

• page loads and waiting times

• attention switches

• memory load – information that users must remember to complete their task.⁸²

Not surprisingly, electronic contracting interfaces evolved to minimize the costs associated with these actions.⁸³ Often, though not always, the click-to-contract mechanism is designed to eliminate most of these actions and thus interaction costs. The only action that really matters is clicking.⁸⁴

Rules of thumb, industry custom, and standard design practices percolate in the web design and human-computer interface communities. The placement of text and images, font size and style, and various other factors are well-studied, although not always documented in academic literature.⁸⁵ In a sense, these communities have performed the functional equivalent of Taylor’s time and motion studies to figure out: (i) how human beings interacted with designed interfaces; and (ii) how they “performed” certain predictable tasks (e.g. browsing and clicking). Anecdotally, we heard of various in-house “motion studies” of how people clicked with a mouse, how they used the up-down-left-right arrows and space bar to navigate, and eyeball tracking.⁸⁶ Time studies are legion.⁸⁷

It’s difficult for us to draw firm conclusions about how, if at all, these studies influenced the click-to-contract interface. We don’t know of studies focused exclusively on this design subject. We hypothesize that the click-to-contract interface was gradually optimized based (perhaps only implicitly) on the more general web design “time and motion studies” to minimize time-to-click-to-contract and motion-to-click-to-contract and thus transaction costs. If we’re right, then the analogy with Taylorism is strong.

Some legal scholars have suggested that website user experience designers might employ images, animation, and other seductive features to distract consumers or prime them so that the speed bump (even if only minimal) is one they speed over.⁸⁸ As many others have noted, the internet environment more generally might lead consumers to be more impulsive or impatient.⁸⁹ Let’s suppose this is true. We would add that such behavior and feelings are not inherent, inevitable, or natural. They are constructed and contingent upon the built environment. Web designers in general contribute to such behavior, and they exploit it. The eye-tracking, click-rate, and other “time and motion” web design studies might not be framed in terms of exploitation. Then again, neither were Taylor’s time and motion studies. Both sets of studies are managerial and, more specifically, aimed at developing the data necessary to scientifically manage human subjects – workers and consumers – to maximize productivity and efficiency.

The electronic contracting interface fits within the same pattern, at least in some respects. Yet the legal system has gradually imposed some constraints. For example, in Specht v. Netscape, 306 F 3d 17 (2nd Cir. 2002), the Court of Appeals for the Second Circuit required conspicuous notice to protect consumers against surreptitious contracts. The court directly evaluated Netscape’s design choices and concluded that customers who downloaded a software “plug-in” called “SmartDownload” could not have reasonably known of the software’s associated license terms. To download the software, customers visited a webpage and clicked on a “Start Download” button. The “sole reference to … license terms on the … webpage was located in text that would have become visible to plaintiffs only if they had scrolled down to the next screen.” The website design itself concealed the license terms in a “submerged screen.” This influential precedent has shaped design choices, both directly – by emphasizing conspicuous notice – and indirectly – by identifying potential legal risks from design choices that undermined the integrity of the click-to-contract mechanism as a means to satisfy the objective theory of contracts.⁹⁰ Not surprisingly, many major websites have reduced the clutter and potential distractions on their electronic contracting interface.

Humanity’s Techno-Social Dilemma Revisited

So far, we have suggested that contract law and practice, especially the electronic contracting environment, are more than meets the eye. The standard account correctly stipulates that they enable people to formulate legally binding commitments and relationships. Additionally, we’ve suggested the Taylorist design of the electronic contracting environment conditions humans to behave like simple machines. Such an attack on our autonomy might warrant systemic reform of contract law.⁹¹

Again, our argument is not about the goodness or badness of contract terms per se, or even the outcomes in specific contracts, transactions, or cases. The oppression we’re identifying is not unfair or exploitative terms. Nor is it Radin’s concern about private ordering replacing public ordering. These are significant problems for electronic contracts. For now, let’s assume away such problems and presume the terms of electronic contracts are fair, consumer-friendly, and even better than their paper counterparts.⁹²

Our focus has been and remains on the social costs external to the contracts. These are the social costs associated with rampant techno-social engineering that devalues and diminishes human autonomy and sociality⁹³ as we become accustomed to being nudged, conditioned, and, more broadly, engineered to behave like simple stimulus-response machines. In the Introduction, we characterized this type of threat against core human aptitudes and capabilities as humanity’s techno-social dilemma, an analogue to the tragedy of the commons. Here, we’re presenting the electronic contracting dilemma as an example of humanity’s techno-social dilemma. Like sheep herders who act in a perfectly rational manner when adding sheep to their flock without fully accounting for the social costs manifest through the degradation of the pasture, consumers rationally enter into electronic contracts that, as we have assumed, are fair and consumer-friendly, without fully accounting for the social costs manifest through the degradation of their autonomy, and, we would go so far as to say, the diminution of our collective humanity.⁹⁴ Our concern is thus with the macro effects of many micro-level decisions to contract that on their face are perfectly rational and efficient. This may seem to put too much weight on the shoulders of contract law. However, the same can be and has been said for many tragedies of the commons.

The electronic contracting dilemma raises concerns about human autonomy because people may be subject to Taylorist techno-social engineering nudging them to behave automatically. If this were an isolated or rare occurrence, the impact would not likely be meaningful or lasting. However, the click-to-contract human-computer interface has crept across varied contexts ranging from websites, smartphone apps, smart TVs, to the Internet of Things.

To make matters worse, electronic contracting creep is accompanied by surveillance and nudge creep. The human-computer interface itself nudges, and it enables surveillance by site owners and third parties. Such surveillance feeds data back and thereby further enables and contributes to nudging. It might seem unfair to put the burden of this problem on contract law. Perhaps one might argue that surveillance and nudging are wholly separate affairs and fall outside the purview of contract law. While that’s a convenient argument, we fail to see the merit of putting on such blinders. The three creep phenomena seem inexorably intertwined, a vicious rather than virtuous cycle in our current digital environment.⁹⁵ All three are fundamental parts of modern Taylorism and the techno-social engineering of humans.

Contract Creep

Autonomy and sociality are critical to being human and maintaining a flourishing society. Without autonomy, we can’t live self-directed lives.⁹⁶ And without sociality, we can’t create meaningful personal and professional relationships upon which collective endeavors depend.⁹⁷

Contract law is one of many important institutions aiming to support and even extend these basic capabilities (for more detail, see Appendix E). In general, it has been quite successful. But continued success is not inevitable. It’s contingent upon many factors, ranging from the competencies of lawyers and judges to the technical media through which contracts are executed. Critically, contract law’s impact upon our autonomy and sociality depends upon social practices and the built environments we construct that have distinctive affordances.

We’ve argued that the electronic contracting environment has inverted the primary aims of contract law. We explained how the technical architecture of the electronic contracting environment nudges consumers to behave automatically. The design of the architecture may have emerged gradually over the past few decades, but, nonetheless, it has a distinctly Taylorist imprint. In accordance with optimality conditions (i.e. efficiency and productivity), consumers follow the click-to-contract script. The choice architecture retains minimal decisional autonomy in simple take-it-or-leave-it fashion, but the fiction of actual choice only contributes to gradual creep of the human-computer interface from websites to apps to smart TVs to smart homes and beyond.

If the postulated conditioning not only exists, but also creeps across contexts, extending the range of situations where we behave automatically, then there is real cause for concern. When such creep leads us to be complacent, to follow the herd and passively accept matters that should require deliberation, our humanity itself is at risk.

Introduction

This chapter furthers our analysis of techno-social engineering by discussing a philosophical position called extended mind theory. According to this view, throughout history humans have extended their minds beyond their physical brains and bodies by using technological tools as aids for performing cognitive processes. Proponents insist that calculators, address books, and other so-called “cognitive prosthetics” are reliable and integral parts of our minds, and, consequently, our selves.

Consider the humorous yet glowing terms New York Times columnist David Brooks uses to describe his relationship with GPS technology. The language resonates with our Chapter 2 discussion of outsourced navigation.

Brooks misses the big picture. And this isn’t surprising. In the philosophical literature, folks are fighting about all kinds of topics related to the extended mind without asking the most important questions, which concern techno-social engineering of human minds.

• Who is doing the thinking when humans use mind-extending technologies?

• In mind-extending situations, what types of thinking do humans and technologies each do and how transparent are the different forms of thinking to the humans whose minds are being extended?

• How does technologically extended thinking impact the development of human capabilities?

What we object to the most is that leading extended mind advocates regard many of the technologies that extend the human mind as normatively neutral, in the sense that the tools themselves are neither good nor bad.² They say cognitive prosthetics empower users, leave their autonomy intact, and can be characterized as “cognitive upgrades.”³ We challenge these assumptions and the Silicon Valley lingo that’s used to convey them. As many of the examples discussed in previous chapters demonstrate, seemingly harmless mind-extending technologies can be powerful instruments for techno-social engineering. The more potent ones make us pawns in other people’s games.

We advance the following novel claim: Mind-extending technologies run the risk of turning humans into simple machines under the control or influence of those in control of the technologies, at least in contexts where the technologies are networked and owned and controlled by others.

Many readers may balk at this claim, viewing it to be speculative science fiction at best, or at worst, a fear-mongering attack on the technologists and technologies we hold dear. Such reactions are understandable but ultimately misguided and too easily used as dismissive parries. The fear-mongering argument is itself mere fear-mongering. We are not attacking technologists or technologies in general. Generalization poses a serious problem for extended mind advocates because technologies are not all the same. Given their affordances, technologies are rarely if ever neutral. Instead, they have context-dependent impacts (costs and benefits, if you prefer). Mind-extending technologies not only afford users beneficial opportunities to extend their minds and think differently with their tools, but the technologies also afford other parties opportunities to perform various thinking tasks and even to exercise mind control. The science fiction argument is reasonable, but it only highlights that we’re talking about the extreme end-state when we say humans are made into machines. The incremental steps toward that end-state are what we are hoping to detect and examine. To do so, we must pay close attention to the relationships between different technologies, our extended minds, and our autonomy.

Extending Ourselves: Background and Examples

Is the extended mind thesis on target or wildly off base? It depends on how you see things. Before giving you our answer, we’ll bring you up to speed on three related issues: the extended body, extended cognition, and cognitive technology. These topics are less controversial than the extended mind itself. They are a great baseline for you to determine how solid or porous the connections are between mind, body, and world.

Extended Mind

In 1998, philosophers Andy Clark and David Chalmers wrote a provocative article, “The Extended Mind.” They present scientific accounts of how the mind naturally copes with its inherent limitations (e.g. being good at pattern recognition but not at memorizing long lists of information). And they contend that humans have a biological propensity to extend their minds outside of their brains and bodies and into the environment. Since its initial formulation, a substantial literature has been engaging with these ideas. Clark himself has defended and refined the argument in a series of books and articles.

The core idea is that humans think in various ways through and with technologies that are external to their biophysical selves (bodies and brains), and by doing so literally extend the boundaries of their minds. This transformation happens by tightly coupling cognitive processes in the brain with those performed in or with the technology. If you don’t believe that this expansive merger (or act of incorporation) regularly occurs, then, according to Clark and Chalmers, you’ve fallen victim to “biochauvinistic prejudice” and are mistakenly presuming that thinking only happens in the head. To help us avoid this error, Clark and Chalmers articulate an ideal they call the “parity principle”:

Simply, humans effectively extend their minds through technologies when the mental processes or steps for solving a problem are like what would otherwise be completed internally. The difference between extended cognition theory and extended mind theory is that only the latter views the whole cognitive process as occurring in and being part of the subject’s mind. For example, both extended cognition and extended mind advocates agree that when a person uses paper and pen to perform a calculation, she takes a cognitive task that could be performed in her head with varying degrees of difficulty and simplifies how it’s solved. But only the extended mind theorist views paper and pen as genuine (not metaphorical) parts of the mind itself.

Let’s consider another example. Clark and Chalmers discuss a man named Otto who “suffers from Alzheimer’s disease, and like many patients, … relies on information in the environment to help structure his life.”²¹ In particular, Otto relies completely on a notebook that he carries with him for all sorts of information about the world. For Otto, the notebook is more than an external memory device. He develops an intimate relation to it, relying on it automatically and with as much ease and confidence as one would rely on biological memory. Clark and Chalmers thus emphasize that Otto thinks using the notebook. It’s an integral part of his thinking circuit, basically, and a bonafide part of his mind.²²

Not all environmental, non-biological resources used while performing cognitive processes are “candidates for inclusion into an individual’s cognitive system.” Clark and Chalmers thus stipulate additional restrictive criteria, such as:

1. 1. “the resource be reliably available and typically invoked”;

2. 2. “any information thus retrieved be more or less automatically endorsed [and thus] not usually be subject to critical scrutiny (e.g., unlike the opinions of other people). It should be deemed about as trustworthy as something retrieved clearly from biological memory”;

3. 3. “information contained in the resource should be as easily accessible as and when required”;

4. 4. “information [stored in the resource] has been consciously endorsed at some point in the past and indeed is there as a consequence of this endorsement.”

Let’s consider another familiar example of a mind-extending technology in light of these four criteria. For many people, the GPS navigation system is functionally equivalent to Otto’s notebook, with respect to location and navigational information. Like Otto, many people rely on their GPS device to navigate, often while driving but also while riding a bike or walking. For most, GPS is “reliably available and typically invoked;” the “information thus retrieved [is] more or less automatically endorsed [and thus] not usually … subject to critical scrutiny [and] deemed about as trustworthy as something retrieved clearly from biological memory;” and the “information contained in the [GPS device is] as easily accessible as and when required.” While the fourth criterion arguably is not met, when people rely on GPS to navigate the physical world, they appear to be extending their minds in the sense that Clark and Chalmers suggest.

Strangely, Clark and Chalmers appear to claim that the person extending her mind retains full autonomy and alone does the thinking required to determine: (i) what procedure should be followed when using technology to perform cognitive functions; and (ii) how to judge when a problem that’s pursued with the help of such technology is successfully solved. Indeed, criteria (1) and (2) imply conscious choice and control over the resource, while criterion (4) requires some conscious endorsement. The problem is that these features don’t guarantee an autonomous relation to technology. At the extreme, “reliably available,” “typically invoked,” “automatically endorsed,” “not usually be subject to critical scrutiny,” and “about as trustworthy as something retrieved clearly from biological memory” all would be satisfied in the extreme case of a brainwashed human who is indistinguishable from a machine.²³ In the next section, we consider more common and less extreme forms of techno-social engineering.

Extended Mind and Techno-Social Engineering

A recent statement of the core extended mind hypothesis is the Hypothesis of Organism-Centered Cognition (HOC):

According to this articulation of the theory (which is compatible with previous ones), human minds are not only embodied, but they are situated within and integrated into their environment. The dynamic interplay between human minds and the constructed socio-technical environment is the main link connecting extended mind theory with our analysis of techno-social engineering.

To explore the juncture between extended minds and techno-social engineering, we have to bracket other debates, including some ongoing philosophical discussions.²⁵ For our purposes, we’ll accept the extended mind thesis as a metaphysically valid account of the human mind and its basic relationship with technologies.²⁶ This means that rather than asking if the extended mind thesis departs too greatly from more widely accepted theories, like extended cognition, we’ll focus on something else entirely: the normative implications that are hidden when the extended mind thesis is taken to have only descriptive and explanatory value. Our trajectory takes us into ethical and political territory. These are areas that theorists like Clark address somewhat lightly (as we discuss below) and which most philosophers of mind would contend fall outside their purview.²⁷

For us, the central point of contention is that extended mind proponents tend to depict technologies that extend the human mind as neutral, thereby presuming that the humans who technologically extend their minds remain autonomous decision-makers. We’re deeply skeptical of these presumptions. Mind-extending technologies run the risk of turning humans into simple machines²⁸ under the control or influence of those in control of the technologies, at least in contexts where the technologies are networked and owned and controlled by others.

Consider Chalmers weighing in and saying something that might have made Steve Jobs – the man who viewed computers as akin to bicycles for the mind – blush.

Now, a skeptic would say this testimony sounds less like a metaphysical attestation and more like script written for a paid Apple spokesperson,³⁰ especially since warnings about proprietary technological extensions eroding our autonomy date back at least as far as the 1960s. Media theorist Marshall McLuhan then cautioned: “Who owns your extended eyes? Once we have surrendered our senses and nervous systems to the private manipulation of those who would try to benefit from taking a lease on our eyes and ears and nerves, we don’t really have any rights left.”³¹ McLuhan recognized the risk of mind control.

To a degree, so does Chalmers. After all, in a TED talk that seems to represent how he conceptualizes his relationship with mind-extending technologies, he says that the iPhone “has already taken over some of the central functions of [his] brain.”³² Yet he does not see this phenomenon as risky because he assumes he retains autonomy when his mind and an iPhone merge, a form of “biotechnological symbiosis” as Clark describes it. The mistaken assumption, however, unwittingly works a kind of “moral magic” that transforms mind control from serious transgression into acceptable and desirable practice. (We discuss another example of such moral magic in Appendix E.)

What Chalmers fails to appreciate is that the iPhone is designed to grant access and control privileges to others – a feature that isn’t deliberately built into Otto’s notebook.³³ This blind spot is apparent when Chalmers proudly describes using Facebook to crowd-source the question of what to present to a TEDx audience. He never questions the role that Facebook plays as a mediator of all communicative exchanges that take place on its platform. Facebook’s proprietary algorithms (which are not available for public scrutiny) determine who can easily see a post (because it’s placed near the top of the News Feed), who must exert effort to see a post (because it’s placed a good distance away from the top of the News Feed), and who won’t see the post at all (because the algorithms determine it isn’t relevant enough to appear on that person’s News Feed). Nor does he consider the impact of social expectations and social pressures on his decisions. Only the truly resolute can ask for help on Facebook (where friends, colleagues, and family can personally respond as well as look at what different people have to say and comment upon their suggestions) without feeling the pull of peer pressure. (All of this is potentially the tip of the iceberg. In the next chapter, we’ll discuss Facebook’s emotional contagion experiment. Facebook not only affects what we think, but also how we feel.)

With this view of mediation in mind, let’s revisit the GPS example. The GPS system processes location information in real-time and develops navigational planning. The system determines the steps in each route and, usually, the order of routes presented. If each of these functions were performed inside a person’s head, we would describe the functions as thinking. Since Clark and Chalmers ask us to avoid “biochauvinistic prejudice” and admit that these functions are still thinking when performed externally, outside of the person’s head and through the technology, we’ll do so for the sake of argument. We concede that the person using a GPS system has extended her mind.

But there’s more to the extended mind perspective. While a GPS navigational system provides suggested routes, Clark and Chalmers should be consistent and committed to the view that people themselves choose which turns to make and which routes to take. Viewed this way, the technology only provides recommendations for consideration; it allows users to fully determine for themselves whether they want to follow the selected pathways.

And yet, doesn’t it seem odd to say that only the person using the GPS is in charge of the cognitive functions embedded in the GPS system? Instead, responsibility and the execution of various cognitive tasks seem to be distributed among different agents within the cognitive system. It seems more reasonable to say that the GPS system user is not the only person doing the thinking; the system designers and choice architects also participate. We can describe this form of extended mind as techno-social thinking because the person extending his or her mind with the GPS system is calling on cognitive resources embedded in the technology by other people. In some cases, GPS navigational systems employ real-time contributions from peers on a network that communicate traffic information and use that information to modify suggested routes and their ranking. Unfortunately, the extended mind thesis tells us nothing about a host of important issues, such as:

• the psychology of how people respond to information presented by machines, such as “automation bias” and “automation complacency,” which are meticulously detailed by author Nicholas Carr;³⁴

• the relationships amongst all these minds providing cognitive contributions (e.g. Are there more compelling reasons to see them as all equal or to see some as some deserving priority over others? Is there or should there be hierarchy by technological design?);

• the technological affordances on both the supply and demand sides of the system (e.g. How can navigational, logistical, and route planning services be exploited? How does the presentation of routes and associated data shape what users can do and how they behave?).

For example, one of us used a GPS to help navigate a long trip. The device interface specified how fast the car was traveling and how much longer it would take to get to the destination. Those numbers effectively nudged the driver to speed up by conveying how much quicker he could arrive. We suspect others have had this experience. While this may have been an inadvertent nudge, marketers have identified and clamored to exploit geographically targeted advertising; route planning is a natural extension – an attractive means for bringing customers closer to the point of purchase.³⁵ (Recall our extended discussion of GPS creep in Chapter 2.)

The other author has had the experience of using a GPS-based route-planning app that provides real-time updates based on feedback from other app users. At times, while using the app, he has wondered why the app suggests new route segments when there is no traffic on any segments (because he is driving very early in the morning) and a shorter route is apparent. After talking to several taxi drivers, he concluded that the app probably is directing him down route segments to help the system gather data. The suggested route is not necessarily the optimal one for his trip, but it might be better for the system.³⁶ (Recall Taylor’s credo.)

The matter of who is doing what type of thinking might seem trivial in some cases. In others, however, it raises challenging questions about how mind-extending technologies convey power. Such power may be subtly employed to shape beliefs and preferences, among other things, at least, so long as you believe that beliefs and preferences are the province of individual human beings. As philosopher John Danaher put it, “the locus of control is the all-important variable.”³⁷ In his book Throwing Rocks at the Google Bus, Douglas Rushkoff argues:

Intrusion and Uncontrollability

In the final chapter of Natural-Born Cyborgs, Clark himself considers various lurking “specters” raised by the extended mind theory and the prospect of biotechnological symbiosis. We focus on two: intrusion and uncontrollability.³⁹ His discussion of intrusion presciently anticipates the Internet of Things and opens with:

Clark recognizes how the mind-extending technologies of our emerging smart world allow, encourage, and indeed depend upon others intruding upon our lives. He casts the intrusions as privacy issues caused by specific aspects of the technologies – e.g. cookies, globally unique identifiers, ubiquitous computing, and smart-badge systems. He briefly discusses responses, such as biting the bullet and accepting the trade-offs, implementing technological fixes that allow people to choose to protect their privacy, and shifting democratic norms that might mitigate anxieties and harms from others knowing sensitive things about us. Unfortunately, Clark casts the privacy concerns mainly as others learning something about us. But as we saw in the discussion of the deployment of fitness tracking devices in schools, such a shallow conception of privacy is an insufficient means for understanding and evaluating the underlying normative trade-offs that occur when tools and systems provide us with benefits but also subject us to techno-social engineering.

Clark touches on our concern very lightly when he turns to uncontrollability: “Some suggest that we should actively limit our reliance on technological props and aids, not just to protect our privacy but to control our own destinies and preserve our essential humanity.”⁴¹ His response is quite simple:

Clark doesn’t go much further into the analysis. He notes the prevalence of the utopian and dystopian outlooks on the humanity-technology relationship, but then concludes: “the kind of control that we, both as individuals and as society, look likely to retain is precisely the kind we always had: no more, no less.”⁴³ Perhaps he’s right. His prediction is hard to evaluate. Still, in light of the extended creep of techno-social engineering that’s occurring, the prognostication seems to be wishful thinking at best.

Since Silicon Valley joined Hollywood in becoming one of the largest US exporters of fantasy, it’s not surprising that billionaire and technology entrepreneur Elon Musk is starting to sound a lot like Clark. Musk insists that the only way for humans to avoid becoming “‘house cats’ to artificial intelligence” is for us to evolve into cyborgs.⁴⁴ To further this end, Musk is developing an “injectable mesh-like ‘neural lace’ that fits on your brain to give it digital computing powers.”⁴⁵ Rhetorically, this sounds great. To avoid being controlled by hyper-intelligent artificial intelligence, all we need to do is build technologies that enable “lag-free interactions between our brains and external devices.”⁴⁶ This option, however, begins to lose its appeal once we ask the questions raised in this chapter, which neither Clark nor Musk take too seriously: Who is doing what type of thinking? And what will the technology companies that provide us with this service demand in return? The unstated but presumed answer is that markets will sort it all out. But as we’ve been arguing throughout this chapter and the book, it’s dangerous to let the economy decide the fate of our humanity.

Mind Extension and Human Capabilities

Having accepted the descriptive thesis of the extended mind theory for purposes of analysis and argument, there is a second important question that we need to ask. How might mind-extending technologies affect the development of human capabilities? This question brings us back to ideas explored in previous chapters.

Let’s return to the GPS example. Consider the damning assessment that philosophers Hubert Dreyfus and Sean Kelly offer:

Dreyfus and Kelly emphasize the hidden costs realized while one uses the GPS, and they describe those costs in terms of lost knowledge, sensory perception, sensitivity, and navigational skills. Critically, these costs entail both static and dynamic consequences. They constitute lost personal experiences (static costs), and, as such, lost opportunities to practice and develop capabilities (dynamic costs). The immediate trade-off is easily understood; it’s no surprise that GPS is so popular. Who wants to experience the hassle of navigating? And yet, the dynamic trade-off, like so many we face in daily life and which we highlight in this book, is not easily marked, much less understood and evaluated. Still, that shouldn’t stop us from trying. We must ask how our capabilities are developed and shaped by our experiences, by our struggles and practices.

Pre-eminent legal scholar Cass Sunstein maintains GPS can have “antidevelopmental consequence[s].” As noted in Chapter 2, he describes a 2000 study of London taxi drivers:

Sunstein focuses on choice-making as the relevant type of thinking and considers how defaults that require active choosing can support learning as people develop their own preferences, knowledge, and perhaps other forms of capital and capabilities. This argument should be broadened to include other types of thinking besides choice-making and to include many other types of mind-extending technologies besides GPS.

There’s nothing new in noting that skill-based trade-offs can accompany technological development. As people, including Clark, frequently point out, back in antiquity Socrates expressed concern that the invention of writing would lead to atrophy of biological memory. Over time, this seems to have proven true. But even with the decline of oral cultures, it seems clear that, all things considered, we’re much better off for it, considering all the amazing things written language can do.

Still, net gains are by no means inevitable. The key question, then, is how we should evaluate the trade-offs that occur where technology and skill intertwine and provide potential for both gains and losses. Philosopher Richard Heersmink puts the point this way:

At first glance, Heersmink seems to offer a sensible framework. If we’re keen on examining skills from a pragmatic perspective because we value them for the practical advantages their development and use can bring, it seems like a no-brainer to run a cost-benefit analysis whenever we’re trying to decide if new uses of technologies are better than older alternatives.⁵⁰ However, things are more complicated than this popular outlook suggests.

For one, it can be difficult to identify negative costs once an overly optimistic view of technology has been adopted. Take the United States Navy, which largely is considered to be the most impressive navy in the entire world. Integrating cutting-edge technology into its operations is a top priority. That’s why “the Navy stopped training its service members to navigate by the stars about a decade ago” and focused “instead on electronic navigational systems.”⁵¹ Sensible as this seemed at the time, things have started to reverse course and Navy education is once again teaching how to navigate by stars. This is because “the Navy and other branches of the U.S. military are becoming increasingly concerned … that they may be overly reliant on GPS. In a big war, the GPS satellites could be shot down. Or, more likely, their signal could be jammed or hacked.”⁵² In short, while many military systems are powerfully connected to GPS technology to strengthen all kinds of functionality, cybersecurity concerns have arisen to illuminate how cognition enhancement and vulnerability can go hand-in-hand once systems dependency emerges. If a cyberwar erupts and GPS technology becomes a high-profile target for enemies to strike, saying it seemed impossible to fight the tide of innovation would sound like an excuse for not committing to thoughtful defense.

To further appreciate why it can be myopic to evaluate every new technology by determining whether it’s cost-benefit justified, consider the following thought experiment that highlights the limits of using micro-level considerations to weigh pros and cons. Imagine that in the future GPS chips are implanted in every baby’s head at birth. These technologically advanced chips are painless to install and their intracranial storage can’t cause any biological damage or complications. Furthermore, the chips are constantly updated with accurate, real-time information, and cannot ever err, be hacked, or stop working during a user’s lifetime. The chips don’t impede concentration (or any other cognitive function) because they only provide logistical information when the user’s brain sends explicit signals that request such detail. And, finally, the chips work everywhere in the world. There would be no such thing as a dead zone or place where a satellite signal can’t be obtained.

Such powerful chips would improve everyone’s sense of direction. We’d never have to worry about becoming separated from the flawless technology that’s always ready to tell us how to get around. And yet, if GPS technology belongs to the class of techno-social engineering tools we’ve discussed throughout this book, then we run into the problem posed in Chapter 2. Once we’re fully comfortable using GPS to handle all navigation, it becomes hard to resist techno-social engineering creep. As time passes, there will be ever-increasing opportunities to automate our mobility, our labor, our bodies, our minds, ourselves. Cost-benefit analysis is ultimately the wrong framework to evaluate our steps down this path.

We end where we began, with two fundamental questions: Who is doing what type of thinking when humans use mind-extending technologies? What are the impacts of technologically extended thinking on the development of human capabilities? We’ll consider them further in the subsequent chapters as we ramp up to a robust discussion of free will, autonomy, predictability, and programmability.

Introduction

Smart is in. The latest buzzword in the technology industry and policy circles is smart. (Or maybe it’s intelligent or autonomous. Buzzwords change like the winds.) We’ve built massive networked surveillance systems with the rise of the Internet that seem poised to inject intelligence into every aspect of our lives. The Internet may have transformed virtually every socio-technical system on the planet, but arguably it was just a step (or leap) down the slippery-sloped path we’ve been on for decades (if not centuries). What lies ahead?

Imagine a world that’s aggressively engineered for us to achieve happiness at minimal social cost. In this hypothetical future, ubiquitous techno-social tools will govern – or micro-manage – our world to prioritize three distinctive yet interrelated normative ends: optimized transactional efficiency, resource productivity, and human happiness. In two words, cheap bliss.¹

Now, even though we don’t currently live in such a world, the technologies required for it to exist are being developed and deployed. Proponents of the Internet of Things, big data, sensors, algorithms, artificial intelligence, and various related technologies make seductive promises, including that increased intelligence – “smart” phones, grids, cars, homes, classrooms, clothing, and so on – will minimize transaction costs, maximize productivity, and make us perfectly happy. (Nudging entails a very similar seductive promise.)

It’s important to note that society isn’t really structured to optimize social institutions and systems to maximize efficiency, productivity, or happiness. Though it may sound counterintuitive, we usually take the opposite approach. Simply put: We don’t optimize. (Apologies to economists and engineers.) The social value of leaving a wide range of opportunities open for the future generally exceeds the value that society could realize by trying to optimize its systems in the present.² In other words, at least in the United States, the default operating principle of social governance of people and shared resources is to leave things underdetermined; this allows individuals and groups to engage in self-determination with different outcomes, depending on the context and changing conditions.³ As law professor Julie Cohen succinctly put it, we need ample room for play.⁴

Optimization presumes an identifiable set of values. One of the reasons why society generally does not aim to optimize for specific values, such as efficiency or happiness, is that people often are committed to many different values. Another reason is that these values are often incommensurable and that makes prioritization contentious and trade-offs inevitable. Yet another reason is that the means used to optimize are highly and probably inevitably imperfect. Further, our understanding of the complex causal relationships between means and ends is incredibly limited. Still another reason is that, even putting aside the prior concerns, what looks like an optimal equilibrium in a specific time, place, or context often may only be locally optimal and globally suboptimal. In other words, what looks great might turn out to be relatively crappy.

Nonetheless, our world is changing rapidly, and seductive promises of intelligent optimization are difficult to resist. As presaged by the computer scientist Joseph Weizenbaum, technologies govern many of our day-to-day activities, and do so with such powerful consequences that it can be difficult for social institutions to keep pace. Assuming we continue along the path we’re on, in the near future we’ll rely even more thoroughly on technologies to intelligently govern our behavior. To be clear, we don’t believe this reliance will come about because technologies will have become sentient, autonomous AIs that enslave humanity. Instead, our hypothesis is much simpler and, we think, more plausible than the Frankensteinian alternatives. We imagine that within the next few decades: (1) we will have gradually built and connected smart techno-social environments that deliver on their promises; (2) the scope of deployment will expand to the point where there is seamless interconnection and likely integration across all environments within which humans live; and (3) the normative agenda executed throughout all this construction and deployment will be optimal efficiency, productivity, and happiness.

The path of engineered determinism we are heading on surely allows for many different futures; a change in direction, even 180 degrees, is always possible. Nothing, other than entropy (as Isaac Asimov suggested in The Last Question), is inevitable. But there are many reasons to believe that the future envisioned here is plausible and may even be a reasonable approximation of what lies ahead.

If the world we’re envisioning seems stark, know that its intellectual seeds have already been sown. We’ve discussed some of them in previous chapters, and we’ll have more to say later in the book. Recall from Chapter 4 that Weizenbaum worried that the computer would lead to the computerization of all human and social problems on the faulty assumption that all such problems are comprehensible in the language of computation; all that would be needed to solve the world’s problems were more powerful computers, programs, and data. The imperialism of instrumental reason Weizenbaum warned of is no different than the optimization logic we’ve just discussed. Though incredibly prescient, he may have missed the critical, complementary role of converging communications media and networks in engineered environments. Accordingly, this chapter begins with mass media and the Internet before turning to smart techno-social environments.

Mass Media and the Reconfiguration of Our Lived-In Environments

The relationships between humans and tools often have an environmental component. At a minimum, the pedagogical function of tools shapes how humans perceive and imagine their reality. More concretely and acutely, techno-social tools for engineering humans often reconstruct the physical, social, and other environments within which humans are situated. The assembly line and public school examples are illustrative. Each is understood to be a special space constructed to achieve specific ends by social engineering.

Media, treated as a collective noun, is an important and broad set of techno-social engineering tools not tied to any specific space. Media “encompasses the myriad technologies and means for communicating with, informing, and entertaining individuals and the masses.”⁶ The history of humans and media is as old as the history of humans and tools. An important example is language.⁷ Innovations in media and extending access to the masses empowered our capacity to generate, cultivate, share, and sustain imagined realities that enable large-scale cooperation.

Media scholar Sharon Kleinman defines mass media as “media that aims its messages at large, mainstream audiences, the masses.”⁸ Mass media are communication tools designed to reach a large audience, often an audience that is geographically distributed.⁹ Conventional mass media entail asymmetrical communications – one-to-many or at least a small number of content producers and a much larger audience. Examples include print, radio, and television. Some of the conventional boundaries that separated mass media from other media may be disappearing. “The once apparent and mostly rigid boundaries between media content creators and media audiences, and between interpersonal communication and mass media, have blurred tremendously in the past few decades such that the term mass media has lost its precision in the digital age.”¹⁰

Mass media seem to be a precursor with a strong legacy and still a relevant part of the ongoing wave of technological, social, and cultural change we discuss below. We focus on media as a techno-social engineering tool and do not discuss market structures, content analysis, or socio-political concerns such as the relationships between mass media and democracy.¹¹

Media are both constitutive and reflective of society. For all mass media, the ideas communicated, facts described, stories told, images displayed, agendas set, personalities shown, and so on contribute to a process of cultural development and exchange that affects and shapes the audience and feeds back upon itself and thereby affects and shapes authors and distributors – or more generally, the media systems themselves. Media scholars have extensively studied the various media and highlighted their important differences according to different models and metrics. Here, we emphasize affordances and effects. As we shall see, these features are relevant to our critical analysis of techno-social engineering tools more generally.¹²

Affordances are human capabilities extended, enhanced, or enabled by a tool. Affordances themselves may have effects on people even if those people never exercise the capabilities, where the opportunity itself shapes their beliefs and perception of reality and their place within it. More often, however, analysts focus on the effects attributable to users’ exercise of capabilities afforded by a tool – what happens when someone actively takes advantage of an opportunity? Apart from the affordances of different media, there also can be direct and indirect effects attributable to consumption of media content – for example, entertainment and learning.

The affordances of mass media vary on both the supply side (e.g. content producers, editors, distributors, advertisers, etc.) and the demand side (e.g. audiences). For example, pre-digital print mass media – such as books, magazines, and newspapers – depends on the distribution of tangible copies. This feature of the technology involves production and distribution costs, imposes constraints on the nature of the communication, and shapes how audiences perceive and interact with the content. Once consumers possess a copy, they may decide where and how they choose to read – for example, in a public or private space.¹³ Consumers may retain copies, annotate or otherwise modify them, sell or share them, or even destroy them. These affordances vary with the materials used to fix copies. For example, clay tablets are more durable, heavier, and less modifiable than paper. These differences affected – or biased, according to media and communication scholar Harold Innis – cultural development. Similarly, if we compare books, newspapers, and magazines in terms of their forms, style, production process, formats, conventions, and so on, we find many differences affect their content and consequently shape the corresponding “print culture.”¹⁴

“[E]ach communication channel codifies reality differently and thereby influences, to a surprising degree, the content of the message communicated.”¹⁵ The medium matters because it shapes, structures, and controls the scale, scope, reach, pace, and patterns of human communications; it extends the human capability to communicate. As media theorist Marshall McLuhan emphasized, for “any medium or technology … the change of scale or pace or pattern … introduce[d] into human affairs” is what really matters: this is what he meant by his famous aphorism, “the medium is the message.”¹⁶

Radio and television broadcasts are more ephemeral than paper. Radio and television broadcasts depend upon different production and distribution technologies than print. Book distribution resembles many other tangible goods; think of trucks delivering boxes of books to stores. Radio and television broadcasting requires completely different infrastructure, institutional structures, and equipment. Of the various media, the First Amendment provides the least protection to broadcasting.¹⁷ Broadcasters must obtain a license from the Federal Communications Commission and comply with an incredibly complex regulatory regime.

Each medium affords content producers different means with which to communicate to audiences and presents content producers with different challenges for garnering and sustaining audience attention. Radio depended on, and perhaps revived, storytelling and oral traditions while television’s synchronization of audio and visual was more akin to dramatic performance.¹⁸ Radio and television dramatically expanded the scale with which news and culture could disseminate practically instantaneously. The entire nation tuned in and experienced momentous events, such as the assassination of John F. Kennedy and American astronauts landing on the moon. This change in the scale and immediacy of mass media reverberated throughout society.

On the demand side, consumers must possess a radio or television. If consumers wish to retain a copy of a broadcast, they must make their own with a recording device. Radio and television broadcasts are not the only way to experience music and video. Like print, discrete copies can be purchased, and those media offer some of the affordances that print provides, such as control over the time and place of consumption, but not others, such as relatively easy annotation. Consumers experience the content of radio and television akin to live performances, by listening and watching as an audience member. Yet there are some important differences. Audiences must possess equipment to tune in. They flip a switch or push a button; they choose a channel; and they pick their place. Radio and television lean more toward communal or group-oriented experience than print. Friends and families listen and watch together in their chosen environment, such as living rooms, clubs, or bars.

Both radio and television tend to be spliced with advertising, and regular interruptions are normalized. For many, advertising is a necessary evil to be tolerated as the means for supporting otherwise free broadcasts. As legal and media theorist Katherine Strandburg explains:

Strandburg explains that the “plausible assumption that broadcast advertising is experienced by consumers mostly as a disutility or cost … is strongly supported by the empirical fact that consumers go to great lengths to avoid broadcast advertising, at least in the television context.”²⁰

The affordances of conventional mass media tended to reinforce asymmetrical communications, with content producers generally catering to mainstream audiences consisting mostly of passive consumers who often trusted what they were told and were happy to be entertained. This is a gross generalization; there are plenty of exceptions and counterarguments. Competition sometimes (but not always) brought diversity and competing perspectives; and the differences within and across media as well as between for-profit and non-profit suppliers complicate the story. Still, the (uncontroversial) point is that mass media are powerful techno-social engineering tools that cater to and create passive mainstream audiences.

There are various theories and explanations for why and how the mass media cater to the mainstream and what consequences follow. Advertising played an important role. As Strandburg succinctly explains:

Our characterization of the audience as passive consumers refers to the fact that the consuming audience plays no role on the supply side of mass media systems. Mass media systems do not afford most people with access to the production or distribution facilities necessary to communicate to the public. Such power is generally afforded to – or reserved for – a relatively small number of people.

Let’s turn our attention to mass media consumption and its effects. Mass media involve the presentation of perceptible content to consumers. The manner of perception and corresponding effects vary based on which senses and intellectual capacities the media are most directly attuned to.²² Print involves text and visual stimuli and requires reading, imagining, and visualizing; radio involves auditory stimuli and requires listening; and television and cinema involve synchronized visual and auditory stimuli, requiring watching and listening. Generations of media scholars have analyzed and debated the scale, scope, nature, and causes of the political, cultural, and psychological effects of different media and the different ways that audiences can be empowered by critical media literacy to become active interpreters of content.

We’re sure you are already aware of the decades-long debates over television. Has TV turned everyone into couch potatoes? No, and even for those of us who are prone to vegging out on the sofa and binge watching programs, most of us don’t succumb to the siren call of television all the time. Studies suggest TV watching occupies quite a lot of people’s lives, but it is difficult to assess the effects. Some, such as media theorist Neil Postman, contend “that TV … attained the power to control education because it dominates the attention, time, and cognitive habits of the youth.”²³ Other scholars suggest that televised content is anything but homogeneous. For example, philosopher of technology Don Ihde argues:

On the Internet and across various devices and platforms, mass media continue to change dramatically; just think about how most people consume print, music, and video today. While it is beyond the scope of this chapter to summarize the extensive media debates and findings, the following takeaways are the most relevant for our discussion.

Mass media shape our cultural environment as they reach into and reconfigure our lived-in environments, our workplaces, schools, homes, automobiles, clubs, restaurants, taverns, and so on. The reconfiguration is often infrastructural and architectural because it operates structurally in the background and in a manner that tends to be overlooked and taken for granted by those situated within the environment. As with the clock and other tools, our perception and understanding of reality adjusts gradually as we become accustomed to the presence, power, and utility of the tools. Unlike those other examples, however, note that mass media attune more directly with our cognitive capabilities and senses. Mass media engineer humans within these lived-in environments by altering the range of stimuli that potentially affect the beliefs, preferences and actions of humans within those spaces. Print, radio, and television are also well-studied examples of such techno-social engineering. In the United States, the power of mass media never quite reached the levels depicted in dystopian science fiction (though some might dispute this claim). But it is indisputable that mass media have had a significant influence on American culture, politics, economy, and society.²⁵

Mass media techno-social engineering encompasses a few interrelated factors: scale, evaluated in terms of audience size, markets, and/or geographic coverage; scope, evaluated in terms of the range and types of content and messages; influence, evaluated in terms of power to persuade, shape beliefs, or otherwise engineer audience members (i.e. do more than simply entertain or satisfy existing preferences); and architectural extension, evaluated in terms of the degree to which the media fit within and bridge different environments. The factors are interdependent, and media scholars have studied scale, scope, and influence extensively. The final factor, however, is one we introduce.

Media intervention into our lived-in environments is often architectural in the sense that it becomes an integral part of the environment, shaping our perception and experience of it. For example, a living room, dining room, or bar with a television is a different environment than one without. The extensibility of mass media depends on how well the media fit within our environments as a contextually appropriate (background) architecture. This should not be surprising as the media, like our other tools, also are extensions of ourselves. Media intervention into our lived-in environments may be abrupt or gradual, contentious or harmonious; it may occur within specific isolated environments, or it may bridge environments.

Radio and television extend architecturally into different spaces, and have done so quite differently. Compare, for example, radio with television in terms of how the media extended into the different environments we listed in the previous paragraph. Both radio and television broadcasters make content available in each of the listed environments, but consumption depends upon reception; what is contextually appropriate is contingent, changes over time, and varies among communities. Radio has long tended to be acceptable in many environments, provided it remains unobtrusive and a source of background ambiance. Television, on the other hand, was most appropriate in select environments, such as the living room and taverns, but inappropriate in others, such as automobiles. Like other creep phenomena, the television medium crept into other rooms of the house (bedrooms, basements) and beyond as its cost decreased. This extension also shifted how people consume the content, for example, by making private individual consumption more easily available. As we discuss below, the Internet fundamentally changed mass media. Today, on the back of the Internet, television is watched on handheld screens in virtually all environments.

Through these combined factors, one begins to see the precursors of the smart environments we’ll discuss in the next chapter. Specifically, the architectural extension of mass media into lived-in environments coupled with expanding scale, scope, and, most importantly, influence conveys power, specifically, to practice techno-social engineering of humans. Early mass media were crude and of debatable effectiveness as techno-social engineering tools. But they could and would improve over time with better data and data analytics. Edward Bernays, often referred to as the “father of public relations,” presciently suggested as much in 1947 when he wrote that “engineered consent,” which is akin to influence, involves the “use of an engineering approach – that is, action based only on thorough knowledge of the situation and on the application of scientific principles and tried practices to the task of getting people to support ideas and programs.”²⁶ Though not explicit, his views evoke Taylor’s approach to scientific management of humans in the workplace, albeit with different undertones. Political theorist and linguist Noam Chomsky and Edward Herman, professor emeritus of finance at the Wharton Business School, went in a slightly different direction in their book Manufacturing Consent: The Political Economy of the Mass Media (1988),²⁷ and forcefully made the case that mass media served as a powerful propaganda tool. Their argument was not that propaganda was novel; it has been around for millennia. Rather, their argument was that the scale, scope, and influence of profit-driven mass media afforded elites more powerful access to and control over the minds of mainstream audiences.²⁸ Others disagreed and still do, as the ongoing debate about media bias and filter bubbles demonstrates.

The debate about power also highlights another important characteristic of mass media as tools for techno-social engineering: the attenuation and distance between the audience within the environment and those exerting influence through the media. This is a dimension of power. The tendency for mass media to support asymmetrical communication also concentrates power to influence, or more broadly, engineer humans. We will not dwell on this point here, however, as it is the topic of extensive scholarship and debate. Again, the simple point suffices: Mass media concentrate power in corporations, elites, and celebrities. This fact has long been a subject of conflict and attempts at resolution through political intervention or market competition have not generally succeeded. Perhaps the single most effective force to decentralize, democratize, and disrupt traditional mass media is the Internet.

The Internet and Networked Digital Media

Over the past few decades, we have witnessed the near-ubiquitous deployment of various information, computation, and communications technologies. As Weizenbaum predicted, the computer presaged our societal infatuation with the seemingly limitless power of computation, digitalization, data, virtualization, automation, artificial intelligence, and related technologies and techniques. And now the ring that binds them all is the Internet, our global all-purpose media network.²⁹

The Internet has grown in just a few decades to completely alter every aspect of our society, economy, and community through its transformation and enhancement of connections and communication across its widespread network.³⁰ Consider, for example, how the Internet provides and shapes opportunities of individuals, firms, households, and other organizations to interact with each other and participate in various social systems. The scale and scope of possible and actual social interactions alone is staggering. The Internet has seeped into our daily lives and environments more deeply than any conventional mass media, and it has correspondingly reconfigured and transformed them even more so. So many formative actions and interactions that humans undergo have been shaped by these changes in the physical, social, and cultural environments.

The Internet is an open, general-purpose communication infrastructure. Although the Internet began primarily as a conduit for solely textual communications, it rapidly expanded to include images, sound, video, and all sorts of content and communications. Everything that occurs on the Internet entails the communication of data between computers at the “ends” of interconnected networks. The computers may be desktops, laptops, smartphones, or various other devices. Digital data are sent in packets that are automatically routed across and among various networks, including telecom, cable, satellite, and other physical infrastructure. The data packets are put back together at the ends and translated into higher-layer communications that can be interpreted and used on any connected device. Though more complicated technical models exist, the following five-layer model provides a useful illustration.

The Internet evolved with the so-called “end-to-end” design principle as its central tenet.³¹ To preserve its robustness and evolvability and to allow applications to be easily layered on top of it, the broad version of this design principle recommends that the lower layers of the network be as general as possible, while all application-specific functionality should be concentrated in higher layers at end hosts. End-to-end design is implemented in the logical infrastructure through the Internet Protocol (IP), which provides a general technology-and-application-independent interface to the lower layers of the network.³²

As a media platform, the Internet extends the human capability to communicate in nearly limitless forms, languages, and content types. Software code virtualizes pre-existing mass and interpersonal media, and so those media remain available and relevant. But in contrast with the asymmetrical nature of mass media and the limited scope of interpersonal media such as the telephone, the Internet enables nearly instantaneous, many-to-many communication around the world. Software code itself has become an incredibly important platform for applications, and, within the applications layer itself, new media platforms have emerged: interpersonal media platforms such as email and text messaging; social media platforms such as Facebook and Twitter; gaming platforms such as the massive multiplayer online games that create virtual worlds regularly inhabited by hundreds of millions daily.

The Internet is socially valuable primarily because of the wide variety of productive activities it facilitates. End-users generate value and realize benefits through their activities, which involve running applications on their computers; generating, consuming, and using content; and creating and engaging in various social, economic, or other relations with other users. End-users also create demand for Internet infrastructure through their demand for applications, content, and relations. Keep in mind that activities on the Internet always involve interactions among end-users; that the interactions may be commercial, educational, social, political, and so on; and that end-users may be individuals, corporations, government actors, or other entities.

The Internet has so pervasively reached into and reconfigured our lived-in environments that for many people, it is difficult to remember or imagine being disconnected. The reach was less pervasive when Internet connections were primarily through personal computers on desktops, but the rapid diffusion of mobile devices capable of connecting to the Internet has dramatically extended the reach of the medium. It’s difficult to evaluate the Internet as a tool for techno-social engineering humans, in part because it is so omnipresent. Consider the Internet through the lens of the interrelated factors we noted previously. Scale: billions of people, worldwide, and substantial though not complete market penetration. Scope: the range and types of content and messages is virtually unlimited, and the same can be said for both applications and social relations. Influence: hard to measure or evaluate generically; the power to persuade, shape beliefs, or otherwise engineer audience members operates at the application, content, and social layers. Architectural extension: the Internet is contextually appropriate in almost all environments, and it bridges most.

To evaluate the Internet as a tool for techno-social engineering of humans, we need to consider an additional factor: the scale and scope of data collection. Recall that Taylorism depends fundamentally on data. Scientific management of human beings in the workplace and beyond requires usable data about task performance, inputs, outputs, behavior, incentives, and so on. One limit of how far Taylorism could reach is the scale and scope of data available for management. Perhaps more than any other technology in human history, the Internet has expanded the pool of data capable of being cheaply collected and used. According to IBM:

Again: everything involving the Internet involves the generation and exchange of digital data – every activity, every communication. Data – strings of zeros and ones – are what economists refer to as a pure public good, meaning the resource can be copied at zero marginal cost. Computing devices at the ends, as well as routers and other devices within and between networks, easily can make copies and collect such data. The dramatic increase in the supply of data created a corresponding demand for improved data storage and analytics technologies, and, in a few decades, major technological advances have coalesced in new fields such as big data³⁴ and reawakened seemingly dormant fields such as artificial intelligence and machine learning. We discuss these developments below, but, for now, our purpose is to highlight not only how the Internet reconfigured our lived-in environments, but also how it simultaneously drew us out of those environments, even if only virtually or metaphorically through our external communications, thereby extracting something of, or at least about, us.³⁵

This data-gleaning feature of the Internet affords various public and private actors with incredible power to engage in surveillance and to use and act upon the specific information collected. Surveillance is simply easy and cheap when you have networked computing devices. It occurs throughout the Internet ecosystem, at the various layers pictured above. There have been many efforts to establish constraints grounded in various conceptions of privacy and implemented in law, norms, and technology, but with limited success. We acknowledge that it is hard to evaluate success or failure, because we don’t have an established normative baseline for privacy nor do we know the full extent of ongoing surveillance practices. Nonetheless, we know that the dominant business models in our networked information economy are surveillance-dependent, as companies clamor to serve targeted advertising and provide personalized services. We also know that governments around the world spy on their and each others’ citizens. Finally, while there are various exceptions, the dominant mindset shaping privacy constraints is rooted in the idea of notice-and-consent, which, as we explain below and in subsequent chapters, is doomed to fail.

Shoshana Zuboff, a professor at Harvard Business School, coined the highly relevant phrase “surveillance capitalism.”³⁶ Per Zuboff, capitalism has entered a new phase that centers on extracting data rather than producing goods, and it’s exemplified by Google in much the same way that Ford once was synonymous with mass production. Google’s business model, Zuboff argues, generates profit by extracting, analyzing, and selling data that the company is constantly collecting, and creating value by offering customized and personalized services that are perpetually being refined through experimentation that, in part, involves crafting ever-more powerful ways to predict and alter behavior. While Zuboff concedes that it’s too early to tell whether surveillance capitalism “will be the hegemonic logic of accumulation in our time” or an “evolutionary dead-end,” she expresses deep reservations about the consolidation of power that’s occurring in the private sector and its capacity to further erode our privacy and diminish our agency.³⁷

While advertising is often advanced as a public explanation and even (partial) justification for surveillance capitalism, it remains hard to verify.³⁸ First, it isn’t clear how well data-driven, particularly behavioral-data-driven, advertising works. Second, it isn’t clear whether or how much or how often data supposedly collected to support advertising is actually used for advertising. Third, even if and when data is used for advertising, it remains unclear whether or how much and how often data is used for other purposes and what those might be in different contexts. Finally, and perhaps most insidiously, advertising may often be used as a surveillance tool. People may be conditioned to accept advertising sprinkled throughout our digital culture and, as a result, many advertisements may serve as cover for ongoing surveillance efforts that serve a range of goals.

This issue of conditioning is taken up by Mark Bartholomew, a law professor at the University at Buffalo, in Adcreep: The Case Against Modern Marketing. Bartholomew suggests that our willingness to allow advertising to become ubiquitous and colonize our public and private spaces is based, in part, on adaptive preferences legitimizing practices that once were disdained.

We emphasize two ways in which surveillance relates to techno-social engineering. First, surveillance itself can exhibit disciplinary power and thus constitute techno-social engineering. Taylor exploited this, and his workers complained of it. One of the fathers of modern social theory, Michel Foucault, examined this relationship extensively across a variety of different contexts. Critically, the reach of the Internet extends the disciplinary power of surveillance dramatically. Second, the data collected can be used as an input for a host of other techno-social engineering tools. For example, at the lowest layers of the network, infrastructure providers, such as providers of broadband Internet access services, might use data to shape traffic in a manner that prioritizes certain user activities over others. Crudely, networks might prioritize applications or applications providers based on profit. More fine-grained, data-intensive price discrimination, however, is the holy grail that would allow them to maximize their returns by extracting as much surplus as possible. This has been the subject of the network neutrality debate, which we return to shortly.

Many of the most powerful media companies that regularly engage in data-driven techno-social engineering of humans operate applications-layer platforms – social networks, search engines, marketplaces, even online games. The next section discusses the Facebook Emotional Contagion Experiment, which is just one example that caught headlines a few years ago and remains salient in technology policy discourse.

Facebook’s Emotional Engineering Experiment

On June 17, 2014, the Proceedings of the National Academy of Sciences (PNAS) published an article titled “Experimental Evidence of Massive-Scale Emotional Contagion Through Social Networks.”⁴⁰ The short article reported on a remarkable experiment that demonstrated that emotional states can be transferred to others by emotional contagion. Researchers at Facebook and Cornell University conducted the experiment and “manipulated the extent to which people (N = 689,003) were exposed to emotional expressions in their News Feed.”⁴¹ Unbeknownst to a few hundred thousand people, Facebook deliberately reduced their exposure to their friends’ positive or negative posts, depending on which conditions Facebook applied. In other words, Facebook deliberately exposed people to the test contagion and then watched to see what would happen. It turns out that the results of the experiment showed that emotional contagion exists and can be deployed by Facebook. People exposed to more positive posts tended to post more positive posts relative to the control groups, with similar results for exposure to negative posts. Moreover, people “exposed to fewer emotional posts (of either valence) in their News Feed were less expressive overall on the following days,”⁴² which the authors described as a withdrawal effect. The authors concluded:

Not surprisingly, a firestorm followed publication of the study. Bloggers, media pundits, researchers, Facebook users, and others debated the ethics of the research.⁴⁴ Most of them focused on whether the researchers should have obtained informed consent from the research subjects and whether the Institutional Review Board (IRB) at Cornell should have played a greater role in regulating, supervising, or monitoring the research. These are very important ethical issues. A few months later, the New York Times reported on some progress: researchers studying us on social networks and other digital media are now grappling with ethics and may develop guidelines to govern how they experiment on us.⁴⁵

But we might not want to leave it to the engineers and tool users. We subjects should grapple with the ethics as well. To get a sense of where you stand, consider a few questions:

1. 1. Is deliberate emotional engineering by Facebook a problem of process (no informed consent for the subjects) or substance (emotional engineering)?

2. 2. If it is a problem of inadequate process: Is IRB review a solution?⁴⁶ What about informed consent? What does that mean to you? Pretend you’re negotiating a one-to-one contract with Facebook. What exactly would you agree to? Would clicking “I agree” when you sign up for the service be enough?

3. 3. If it is a problem of substance, can you explain the problem without reliance on adjectives like creepy? Can you articulate what exactly is wrong with emotional engineering by Facebook?

4. 4. Is it acceptable for Facebook to induce or suppress the emotional contagion of your friends?

5. 5. Suppose Facebook tests, develops, and optimizes its emotional engineering capability to help people make better decisions? Would it be acceptable for Facebook to induce or suppress impulsive purchases (or at least, clicks)?

6. 6. Suppose Facebook optimizes its emotional engineering capability specifically to minimize emotional interference with instrumentally rational decision-making. Would this nudge people to make better decisions? Would people nudged in this fashion act like machines? Would they be (or could they be) any less human?

7. 7. Suppose Facebook optimizes its emotional engineering capability and lets users choose the settings – dial up some happiness! Would you use it?

These are difficult questions. The lack of informed consent and the role of the IRB are important issues, but they are the tip of the iceberg. The tip is all that gets attention until too late. The deeper issues (reflected in questions 3–7) are substantive, have less to do with the research process or this specific experiment, and more to do with the technological capacity for techno-social engineering that Facebook is testing. To be clear, Facebook is testing a tool, a powerful one. Why? What are the predictable consequences? How about the unpredictable or unintended consequences?

The title of the Facebook study caught our attention: massive-scale emotional contagion through social networks. The type of response (human emotion) being engineered struck a chord but so did the scale, scope, and power of the tool being tested. With respect to the first concern – human emotion being engineered – we must acknowledge that many things alter our moods every day. Advertisers and politicians (and their various consultants) are expert manipulators, and so are the rest of us. We try to influence each other regularly, for better or worse. We nudge each other. That’s a big part of what socializing and communicating entails. Emotional contagion is not the only social contagion, but it can be a powerful nudge. Many technologies play an integral role in shaping our beliefs, emotions, and well-being, sometimes, but not always, in ways we know about and at least partially understand.

But systematic techno-social engineering of human emotions through platforms, like Facebook, that reach into, reconfigure, and, in some ways, constitute the environments we live in daily may be much more challenging to know about and evaluate, and it may become more pervasive. Such engineering may be much harder to know about and understand independent of the platforms’ influence on emotional and other social contagions. A focus on process alone will never be sufficient. As privacy scholars have long recognized,⁴⁷ informed consent can be manufactured, in the sense that technological platforms can shape one’s beliefs and preferences with respect to that for which consent is sought.⁴⁸ Aside from the emotional contagion experiment, Facebook is a rather straightforward example, at least when one focuses on privacy. The beliefs and preferences of hundreds of millions have been shaped over the past decade. Facebook set out to accomplish this objective – at least, to encourage broad sharing of content – and largely has been successful.⁴⁹ Although public outcry about the emotional contagion experiment might lead one to conclude that Facebook would not be able to obtain consent from users for emotional engineering because their existing preferences may conflict, such a conclusion seems somewhat far-fetched. Facebook has not, to our knowledge, abandoned the technology or practice, nor have Facebook users revolted and ceased to use the service. Further, there is plenty of time for Facebook to both develop its technology and gradually shape its users’ beliefs and preferences regarding the technology. Only time will tell.

Keep in mind that regardless of whether Facebook is engaged in a formal experiment, it persistently engages in both surveillance and techno-social engineering. We need to engage the ethics, including both process and substance, and we need to develop better tools for identifying and evaluating such techno-social engineering. After all, we only know about Facebook’s experiment because it published the results.

To this point, we’ve only scratched the surface and focused on one actual example of techno-social engineering by Facebook. As we suggested, this may be indicative of the path we are on and what the future may hold. While we hesitate to prognosticate about the future, consider a fictional extension of the Facebook emotional contagion experiment.

Suppose Facebook figures out how to control and deploy emotional contagions and thus optimizes its emotional engineering technology. Now assume it creeps. Suppose Facebook gradually extends the scope of content, contagions, and emotions, and suppose Facebook gradually extends its reach. Finally, suppose Facebook expands beyond its social network interface on the Internet to other interfaces available through the Internet of Things (described in more detail in the next chapter). Thus, suppose Facebook extends its optimized emotional engineering capability to the environments within which we live our lives. For example, suppose Facebook deploys its emotional engineering technology in your smart home, automobile, and workplace through networked sensors and communications devices.

For our fictional extension, we can imagine two different worlds, one in which we still have a choice about whether to log in, and one in which we don’t. Yet it is not clear whether this would even be a meaningful distinction, whether choice in the first possible world would be authentic or illusory. Assume that is our imagined world: Would you consent? Does your answer depend on whether you are in control and whether you could choose the settings? It might be the case that your first decision to consent could be authentic; perhaps you’d be able to deliberate and decide for yourself. But one cannot help but wonder whether, thereafter, consent would itself be subject to engineering. (If the mechanism for consent is a simple click-to-contract-style interface, we may already have been conditioned to automatically accept.)

One question we genuinely struggle with concerns who is doing the emoting and whether it even matters. Suppose you live in an environment within which Facebook successfully programs your emotions. Perhaps you consented and even chose the setting, or perhaps your parents did on your behalf long ago. Facebook provides a comprehensive set of (emotionally contagious) stimuli that trigger a predictable, predetermined set of emotional responses. Who is emoting? You? Facebook? Your parents? Does it matter?

We could ask the same questions about a novel. When you read a novel, who is emoting? You? The author? The publisher? We doubt anyone believes that when you read a novel and become happy or sad anyone besides you is emoting. We might say that the author is communicating and perhaps jointly sharing emotions. Is the author engaged in techno-social engineering? Yes, in a sense. Authorship entails the informed use of language and other tools to communicate, entertain, and stimulate emotional reactions. The novel is a techno-social tool designed to serve those purposes. It provides a set of stimuli and triggers emotional reactions. Generally, this is something we encourage and celebrate.

How, then, is the hypothetical emotional engineering by Facebook any different? We believe it is a combination of factors, the most important of which seem to be the following: the scale and scope of the techno-social engineering, the marriage of deterministic engineering and engineered determinism, and the simultaneously environmental and acutely personalized nature of the techno-social engineering. These differentiating factors are complex, hard to mark and evaluate. For the objects being engineered and society generally, the difficulty is in knowing when a line is crossed, if one can even be identified. The third part of this book develops some tests that might help.

The Facebook emotional contagion experiment and our hypothetical extension highlight steps along a path. We may doubt we’ll ever get to the endpoint, or even very far down the path. But can you be sure? How might humans and society change along the way?

Introduction

In examining the scale and scope of techno-social engineering of humans, we can no longer limit our attention to the isolated examples of the factory floor or public school. We must extend our analysis to almost every other space, including the home and our public spaces. Mass media have reached into those spaces, but so far only incompletely and discontinuously. The Internet dramatically increased their reach, interconnection, and continuity.

Yet in the present – and this may be wishful thinking – the various environments within which we live our lives remain separate, even if potentially interconnected and interdependent. We have not been and are not always on. Put it this way: Radio and television broadcasters may have bombarded all our lived-in environments with analog signals, but we still need to flip the switch and tune in, and we could easily tune out. The Internet may be even more readily accessible, as we carry our smartphones and related devices with us throughout our lives. And we may in fact exercise our capability to access the Internet more regularly in our daily lives; our default may even have flipped, such that we are by default tuned in. Nonetheless, though we may not always appreciate it, we retain the capability or freedom to be off.

The frightening thought is that if we proceed down the path we’re currently on that freedom will disappear. The practical, situated, and reasonably exercisable freedom to be off, to be free from systemic, environmentally architected human engineering, is the – or at least, one of the – fundamental constitutional issues we, as a society, need to confront. Constitutionalists have always had to ask, grapple with, and answer the foundational and ultimately intergenerational normative question of what sort of society we want to build. We all must be constitutionalists and ask ourselves this question. In the twenty-first century, this question is unavoidably about the scale and scope of techno-social engineering of humans and the actual freedom to be free from such engineering – at least for some meaningful portion of our lives.¹ We return to this theme at the end of the book.

This chapter looks from the present to the near future and explains why interconnected sensor networks, the Internet of Things, and big data enabled automation of systems around, about, on, and in humans promise to expand the scale and scope of techno-social engineering significantly. The reason lies in the power of intelligent, often automated systems and the reconstruction of our everyday environments and, as we shall see, us. It’s the even more fine-grained, hyper-personalized, ubiquitous, continuous, and environmental aspects of the resulting techno-social engineering that make the scale and scope unprecedented.

We begin with a familiar example from the present: the smartphone. The smartphone is one prominent case where adding intelligence (smartness) expands the range of applications and uses of a device, making it more general-purpose and providing more affordances for various actors. Smartphones – or smart mobile devices more generally – are jam-packed with powerful sensors, processors, memory, software programs, and well-designed interfaces. Thousands of components are integrated into light wearable devices. As a media device, the smartphone piggybacks on and extends the scale, scope, influence, and architectural extension of the Internet. It contributed to and likely accelerated the convergence of interpersonal and mass media, and, more importantly, extended connectivity to most environments, bringing most smartphone users closer to being always on. The corresponding techno-social engineering is intense and varied.

Again, as with the Internet, it is worth considering the affordances of smartphones. On the demand side – that is, for users – smartphones extend and enhance many of the same affordances as the Internet and networked personal computers. An important additional affordance is mobility. Smartphones travel with users, and this disintegrates constraints in both time and space, meaning you can be online anytime and anywhere. This affordance cuts both ways. Various others – friends, family, co-workers, employers, advertisers, service providers, etc. – can reach you anytime and anywhere. On the supply side, the smartphone software ecosystem significantly lowers the costs of developing and deploying software applications, and this has led to vibrant communities of app developers and a corresponding proliferation of apps, many of which leverage the expanded versatility of smartphones and mobility of users to solve new problems and meet new demands. Just think for a moment about the range of applications, entertainment, news, and other features enabled by the powerful, networked computer you can carry in your pocket. The smartphone invites others into one’s mind and affords them incredible – and incredibly personalized – surveillance, nudging, and control capabilities. As such, we’d like to be able to evaluate:

Each of these questions demands considerable research and attention. Take a moment and consider your own experience with the technology. How have smartphones affected your life, your experiences and interactions with others? According to technological entrepreneur Elon Musk, “The thing that people, I think, don’t appreciate right now is that they are already a cyborg … If you leave your phone behind, it’s like missing limb syndrome. I think people – they’re already kind of merged with their phone and their laptop and their applications and everything.” If you’ve merged with your smartphone and digital technology as Musk suggests, can you ever exercise the freedom to be off?

Smart Techno-Social Environments

Here are (some of) the basic components of tomorrow’s smart techno-social environments:

• Networked sensors

• Data

• Intelligence-generating systems, including artificial intelligence, machine learning, algorithms, and others

• Automation/control actuators

Each of these comprises a broad set of techno-social engineering tools. (Computer scientists and engineers will recognize these as subsets of broader system design categories. Obviously, each of these requires various supporting technical systems, e.g. for power, data storage, and so on.) While some of these tools sometimes work independently in particular contexts, these tools often are and will be components of “smart” interconnected systems that architect, manage, and possibly even constitute our built lived-in and experienced environments.

Though not appreciated by most people, even in the technology community, it is critical to understand that the potential demand for and thus value of interconnected sensor networks, the Internet of Things, and big data depends on automation of systems around, about, on, and in human beings. Put another way, interconnected sensor networks, the Internet of Things, and big data are not in and of themselves socially valuable. Demand for such technologies is derived demand. Many in the business and technology fields assume that these technologies are the next best thing, without really knowing why.² There is a common perception of the technological power and potential, almost inevitability, of an Internet of Things, from smart toothbrushes to smart toilets, but actual consumer demand remains uncertain and likely will continue to be so for quite some time; it likely needs to be stoked, if not outright created (another job for the marketing and advertising community that undergirds much of the modern information economy).

Let’s focus briefly on the Internet of Things, which is an umbrella term often used to capture the basic components of smart techno-social environments. To begin, we must acknowledge that the Internet of Things is clever rhetoric. The Internet, as we know it, is an infrastructure that connects computing devices and ultimately people. Almost everything that occurs on the Internet involves the communication of information between people – it is social and relational, and it involves the creation and sharing of information.³ So what is the clever rhetorical move? It is to replace people with things. The Internet of Things is a metaphor, but, frankly, metaphors matter. The Internet of Things metaphor reveals an explicit and implicit shift in framing.⁴ While people might look at this as simply the next step in the evolution of the Internet and adjacent computing technologies and systems, the regulatory implications could be dramatic, and we don’t just mean government regulation. We also mean regulation by private companies, architecture, and technology because of the ways in which the environment that we live within and interact with changes. Instead of people being in the foreground, the Internet of Things pushes people to the background.

When folks talk about the Internet of Things, the focus shifts subtly from humans actively communicating with each other to devices gathering and exchanging data and automating various technological and business processes to make the lives of human beings easier, more efficient, and happier. The Internet simply becomes a means for ubiquitously distributed sensors – mobile and stationary devices, mere things – to gather, process, exchange, and act upon data. The things are primary; they are technological and perceived to be neutral; they require investment and involve innovation, and will allow service providers – private and public – to more cheaply and efficiently provide us with what we supposedly want and need. But they also will allow those service providers to engage in techno-social engineering of humans – of, at least, our beliefs, preferences, and emotions – if the incremental steps we have seen in recent years are any indication.

Beyond mere rhetoric, the Internet of Things is a major work-in-progress. The incredible hype and investment about a trajectory that’s expected to lead to an “[I]nternet of everything” has generated policy discussions, concerned mostly with paving the way for investment and deployment but also with identifying privacy, security, and other consumer protection issues.

In its Green Paper published in January 2017, the Department of Commerce Internet Policy Taskforce and Digital Economy Leadership Team used the Internet of Things as an umbrella term “to reference the technological development in which a greatly increasing number of devices are connected to one another and/or to the Internet.” The DOC explained that commenters who responded to the DOC’s Request for Comments offered a wide variety of definitions and emphasized different features and applications.

We provide this ridiculously long quote to give you a sense of how the Internet of Things potentially encompasses all the components of tomorrow’s smart techno-social environments: networked sensors capable of collecting and transmitting data to enable smartness (intelligence generation via machine learning and other techniques) and consequently action. Notably, the final value-creating step – action – is not always specified, much less linked to automation. This seems a convenient way to avoid the elephant in the room: Who is smarter? Who acts on the new intelligence, and how? Who gains what power?⁶

As our discussion of the extended mind emphasized, we must ask: First, who will be doing what thinking in these smart techno-social environments? Second, what are the impacts of smart techno-social environments on the development of human capabilities?

This may seem too abstract. We encourage you to pick a few examples – toothbrushes, televisions, homes, cars, etc. – of your own and consider our questions. Let’s consider who learns what from a smart toaster.

Technology writer Mark Wilson highlighted the deskilling potential of smart devices in his critical review of the June, a $1,500 smart toaster. At first glance, it seems ridiculous that anyone would pay this kind of money for a commonplace kitchen item that in “dumb” form can be purchased for a fraction of the price. But the June is built on the back of $30 million in venture capital; it runs on AI and deep learning and contains many sensors.

As of now, the June appears to be glitchy. But we can expect that, over time, some version of it will improve and become available at a significantly cheaper rate. While we can’t predict the implications of this happening – especially since the effects will be influenced by other social trends – we’re sympathetic to Wilson’s concerns about the logic driving the developmental pathway. Above all else, it privileges machine learning: “When you cook salmon wrong, you learn about cooking it right. When the June cooks salmon wrong, its findings are uploaded, aggregated, and averaged into a June database that you hope will allow all June ovens to get it right the next time. Good thing the firmware updates are installed automatically.”

Placing a higher value on machine learning than human learning is the hallmark of today’s Internet of Things, convenience-oriented, devices. The big existential question to consider, therefore, is whether engineering an appetite for pervasive smart kitchens will disincentivize people from cultivating culinary skills, including the personalized touches many of us associate with the favorite cooks in our lives.

The road leading to this possibility was paved long before consumer technology became an Internet of Things staple. As philosopher of technology Albert Borgmann repeatedly argues, widespread use of microwave ovens and a manufactured globalized taste for fast food helped set all of this in motion. That’s why when Borgmann considers the possibility of living in “smart” homes filled with “smart” appliances, he doesn’t envision future liberation. Instead, he cautions that “we will slide from housekeeping to being kept by our house.”

In some cases, improving intelligence correlates directly with improving the efficiency of a well-delineated functional task. Consider a few other examples from the home:

• A coffeemaker can be programmed to brew coffee automatically at a specific time.

• A smart coffeemaker may rely on a timer or sensor to determine whether a pot of coffee has been sitting on a hot burner for too long, in which case it may turn off the burner, reduce the temperature, or generate a signal such as a flashing light or beeping sound.

• A thermostat can adjust heating and cooling systems within a home based on a temperature sensor. It is smart in the sense that it can automatically make adjustments without human intervention.

• A smarter thermostat can integrate additional sensors, data processing, and control mechanisms and thereby adjust heating and cooling systems within a home based on more variables, such as temperature, time of day, and occupancy levels.

• A smart light switch can adjust lighting levels within a home based on time of day or motion.

Some of these examples involve sensors and programmed systems that can process and act upon data from the sensors. The first example doesn’t even need a sensor to be smart. (Some might conclude that the programmable coffeemaker therefore isn’t smart, but we think such a narrow conception of smartness would be underinclusive.) None of these examples requires (i) networking beyond the home, (ii) big data, or (iii) intelligence-generating systems beyond the home. In fact, each of these technologies can accomplish its core improved functionality locally and without integrating or interconnecting with each other or other smart devices.

Yet smarter Internet of Things versions of these technologies often will communicate beyond the home. There are various reasons. Some may do so by mistake or poor design. Some may do so deliberately because the device manufacturer would like to keep future options open. Some may do so because it helps with marketing, in the sense that consumers believe Internet-connected devices are better. Some device manufacturers may genuinely envision additional functionality, and some may assume innovation will follow. None of the reasons suggests technical or economic necessity.

Homeowners should be wary of smart home technologies that extend beyond the home. After all, whose intelligence is really being extended? Is it the homeowners’, the device manufacturers’, or other third parties’? The situation is quite like websites with hidden side-agreements. When supposedly smart devices collect data from within the home and communicate that data outside the home, the homeowner may be treated like a product, or, as the Internet of Things metaphor goes, like a person reduced to a thing.

We advocate two simple default rules/design principles for everyday things (not people!). First, don’t connect, communicate, or interoperate. Second, only engineer intelligence as needed; make things only as smart as they need to be to perform a well-delineated functional task. (This is akin to a data minimization principle that has been proposed in the privacy-by-design literature.) Our proposal amounts to heresy in most engineering, design, and economics circles. We’ve flipped these communities’ existing defaults, and so many people will conclude that we’ve gone too far. But let’s be clear about two critical limiting principles. First, default rules are not absolutes; exceptions should be expected. Exceptions to such rules, however, require justification. Second, both rules and exceptions might need to be tailored to specific contexts. In some contexts, such as transportation, justifying connectivity, communications, interoperability, and engineered intelligence is rather easy, even if, as we shall see below, governance and implementation of smart transportation systems is ethically, politically, economically, and technically difficult.

Let’s consider an example that would trigger an exception from the default rules we’ve proposed. In some contexts, being smart depends on accurate sensor-based intelligence coupled with (i) knowledge about what to do with the intelligence and (ii) the capability to act. Suppose someone installs a device with a sensor capable of detecting water seeping in a basement. If that is all the device can do, it is useless. It must be able to communicate to be functionally useful. Thus, for example, the device may ping the homeowners’ smartphones to let them know about a potential problem. Does that make the homeowners smarter? It obviously does somewhat. They received new information. If they already know how to deal with the problem, then they can do so. If the homeowners lack such knowledge, what good is the notification? It would enable them to reflect upon and determine what to do next: consult friends, look online for do-it-yourself solutions, or contact an expert to assist them. It might be more efficient if the sensor immediately contacted a trusted service provider to handle the problem. That might be a perfectly reasonable and justifiable feature of the smart device, if the homeowners actively chose that option. This final point reflects a third default rule and design principle in favor of so-called active choosing.

Some might retort that we’re making a mountain out of a molehill. Many isolated examples of smart devices, such as a smart toaster, light bulb, or toothbrush, seem innocuous and almost trivial. As with nudging, electronic contracting, and many other examples we’ve discussed in this book, an incrementalist view can be misleading and lull one into habitual complacency. Smart technology will creep along various dimensions.

An individual smart device may creep in terms of data collection and/or function. For example, the specified, and so we shall assume intended, purpose of a smart TV with an always-on microphone may be to capture aural input for voice-activated commands. Communicating such data outside the home may be necessary for machine learning and other techniques to improve the voice recognition technology. But we can and should expect data collection and function creep. The always-on microphone can pick up much more data than actual commands. Apparently, the fine print of Samsung’s Smart TV privacy disclosure once stated: “Please be aware that if your spoken words include personal or other sensitive information, that information will be among the data captured and transmitted to a third party through your use of Voice Recognition.”⁷ When treasure troves of data such as this are collected and shared with third parties, it is hard to imagine companies will abide by purpose and use restrictions. In other words, using the data solely to improve voice recognition technology seems highly unlikely over the medium to long run.⁸

Suppose one deploys a host of different smart devices in the home. Standing alone, each device might not “transform” the home environment. Together, however, they might. The aggregate effects might resemble a tragedy of the commons in the sense that incrementally rational decisions lead to a net tragedy. Obviously, tragedy is not inevitable. Tragedy occurring depends upon various contextual details, such as whether the devices interconnect locally or beyond the home and how defaults are set for security, privacy, and user engagement (e.g. in setting parameters or programming options), among other things. The point is that creep phenomenon applies here as well.

At this stage of smart home development, we cannot say who will be doing what thinking or what will be the impacts on the development of human capabilities. With respect to the first question, many smart devices for the home environment involve outsourced intelligence and raise the possibility of a corresponding invitation for intrusion and/or control. In the third part of this book, we turn our attention to techno-social engineering tests that might provide a useful framework for identifying and evaluating when outsourced intelligence impacts basic human capabilities.

How Smart Should Our Shared Infrastructure Be?

Infrastructure systems are incredibly complex, and intelligence and control technologies exist in many forms and layers. The somewhat oversimplified layered model of the Internet presented in the previous chapter can be used to describe power, transportation, and other infrastructure systems.

Often overlooked and pervasive social policy and technological design questions concern where and how intelligence and control technologies are deployed within infrastructure systems. Who gets to decide how such technologies are used?

One topic where this issue has become the subject of intense debate is the matter of how designers of self-driving cars will solve the trolley problem.⁹ In its classic form, the trolley problem is a thought experiment that asks us to consider whether we would save the lives of several people from being hit by a runaway trolley if we had to pull a lever – an action that would send the vehicle to another track and kill someone there. Among other things, the scenario helps us think about whether there’s a meaningful ethical difference between actively killing and passively allowing people to die. The variation for self-driving cars ponders what autonomous vehicles should be programmed to do if, say, they’re on a collision course with a school bus that’s carrying lots of innocent children and which can only be avoided by swerving and killing the single passenger in the car. Should a simple utilitarian calculation be made? If so, the choice is clear: count the lives of the children as being more valuable since there are so many of them. Or perhaps the passenger of the car has the right to insist that her vehicle prioritizes self-preservation. That attitude can seem selfish, but it’s not an untenable position to hold.

Consider a variation of this problem. Imagine that you’re in a self-driving car going down a road when suddenly, the large propane tanks being hauled by the truck in front of you start falling out and flying in your direction. A split-second decision needs to be made, and you are incapable of running through all possible decision scenarios and fully computing outcomes and trade-offs. The smart system driving your car, however, can do so. How should it handle the question of who deserves moral priority? Consider the following possibilities:

1. 1. Your car should stay in its lane and absorbs the damage, thereby making it likely that you’ll die.

2. 2. Your car should save your life by swerving into the left lane and hitting the car there, sending the passengers to their death – passengers known, according to their big data profile, to have several small children.

3. 3. Your car should save your life by swerving into the right lane and hit the car there, sending the lone passenger to her death – a passenger known, according to her big data profile, to be a scientist who is coming close to finding a cure for cancer.

4. 4. Your car should save the lives worth the most, measured according to amount of money paid into a specific form of life assurance insurance. (Assume that each person in a vehicle could purchase insurance against these types of rare but inevitable accidents.¹⁰ Then, smart cars would prioritize based on their ability and willingness to pay.)

5. 5. Your car should (i) save your life and (ii) embrace a neutrality principle in deciding among the means for doing so, perhaps by flipping a simulated coin and swerving to the right if heads comes up and swerving to the left if tails comes up.

6. 6. Your car should (i) not prioritize your life and (ii) should embrace a neutrality principle and randomly choose among the three options.

7. 7. Your car should execute whatever option most closely matches the moral choices you would have made if you were capable of doing so. (Assume that when you first purchased your car, you took a self-driving car morality test consisting of a battery of scenarios like this one and that the results “programmed” your vehicle.)

We’ve presented a simplified hypothetical with limited options. One point these thought experiments make is that there’s no value-free way to determine what the autonomous car should do. The choice of whether to save one person or many isn’t comprehensible as a purely computational problem. Determining which value system to embrace is no more of a mathematical operation than favoring Hammurabi’s Code over the Declaration of Independence (or vice versa). We will program autonomous cars to deliberately follow paths that preserve and end life, and this only makes the matter of finding morally acceptable protocols even more pressing. Indeed, once it’s acknowledged that some form of ethics needs to be baked into the programming, the following sorts of questions arise. Who should decide how autonomous vehicles will perform when difficult situations arise? Should it be politicians? Automotive executives? Or should people be allowed to customize the moral dashboard of their cars so that their vehicles execute moral decisions that are in line with their own preferences? The alternative, after all, is to put some people in a situation where they’re pressured to abdicate control over highly valued decision-making. While creating design specifications that respect pluralistic values can seem ideal, it’s no panacea. It’s been suggested that a key to making smart transportation systems efficient is to create constraints that impose uniform behavior. If that’s true, there’s an important tension to resolve over whether efficiency or personal choice should matter more.

Once these issues are resolved, difficult questions remain. What’s the best way to convey to consumers what autonomous vehicles are programmed to do? Philosopher of technology Mark Coeckelbergh expresses concern that if humans don’t know how the vehicles they travel in make ethical decisions, they’ll outsource aspects of their agency that are needed to engage in responsible behavior.¹¹ Even knowledge about how smart cars have been programmed doesn’t alleviate moral concerns about ethical outsourcing. What matters is practical agency.

Ethical issues don’t end here. Like the Internet, transportation systems are layered, and in addition to issues surrounding smart cars, we will need to confront a host of ethical and governance issues at the infrastructure layers.¹²

For example, business theorist Shoshana Zuboff insists that “automotive telematics” – which is to say, the surveillance and control capabilities of the automotive industry – are presenting problems that are poised to become more pressing when self-driving cars become the dominant business model. She writes:

Another set of related considerations is highlighted by the following thought experiment about the future of smart transportation management.

Imagine: it’s 2025 and you’re an engineer managing traffic in and out of a major city. You watch the roads fill up at rush hour, as people in autonomous cars, trucks, and buses buzz alongside pedestrians and cyclists guided by Internet-connected eyewear. Your job is to plan efficient, safe, and environmentally friendly routes.

Since this is the future, all your decisions are guided by data. Algorithms predict with astonishing accuracy what will happen when the weather changes, plans alter last minute, and emergencies require people to leave their homes and jobs in a hurry. Still, even with all this information, your job isn’t easy. Every time you try to minimize congestion, you face the same problem: can you do better than first-come, first-served?

Today, police cars, ambulances, and buses sometimes get special treatment on the road because of the contributions they make to public welfare. But these narrow exceptions aside, our roads are managed without prioritization. First-come, first-served is the default.

In the future, however, we will be able to make finer discriminations about who individual drivers are, what destinations they’ve set, and what they’re expected to do when they arrive. Armed with this information, would you place some folks in the fast lane and stick others in slower ones? Perhaps the woman on her way to a business meeting should get priority over the woman who is attending her son’s soccer game. Or should it be the other way around? The decisions don’t end there. Suppose only one of the drivers is going to make her event on time and the other will arrive too late even if she speeds. Presumably, you should determine who gets to go and inform the other person to stay home to minimize her impact on others. Over time, these sorts of decisions can be expected to occur frequently.

Bottom line: Traffic engineers will assume the role of social planners. Decisions made in single instances (e.g. prioritize x over y) according to decision-making protocols and embedded values aggregate and over time become social patterns.

In theory, fine-grained control over traffic seems like the perfect use of big data and artificial intelligence. That’s why folks are enthusiastically lobbying governments to invest in smart grids and transportation systems. In practice, however, things are more complicated. Contentious debates about power and privilege should arise.

To appreciate why, imagine being a traveler in 2025, instead of an engineer. Envision yourself in an autonomous car that’s made it halfway to your destination. Say you’re a parent en route to your child’s championship soccer game, the match that your kid has been obsessing over for weeks. Suddenly, your self-driving car turns around. Confused and upset you say, “Siri, why am I being re-routed? Why is this happening to me?” In response, the digital assistant laconically replies, “Sorry, but there’s priority traffic heading downtown.” What you don’t know for sure but deeply suspect is that the smart traffic-management software is programmed to assign a comparatively low value to “mundane” social outings like amateur sports. Business deals, like the one your neighbor is heading to, count as more important to society because they contribute directly to economic growth metrics.

At first glance, this scenario might seem like bad science fiction. In fact, it’s an old and very real problem. For some time now, the related issues of control and intelligent infrastructure have fueled the network neutrality debate.¹⁴

The Internet both provided the blueprint and unleashed the wave of smart technologies that promises to transform most of the environments within which human beings live and develop, ranging from smart homes and smart offices to smart cities. The smart technology transformation is likely to work its magic first, however, on the basic infrastructures that operate unnoticed by most of us in the background. Smart grids and smart transportation infrastructure will literally pave the way for the smart technology revolution.

The history of the Internet also provides a decent map of future governance dilemmas society will face. The network neutrality debate will not only persist for the Internet but also will return in modified form for other infrastructural systems. At its core, the network neutrality debate is about whether and how private owners of the Internet’s underlying communications infrastructure could use the intelligence they gathered. In other words, the network neutrality debate was and still is about how smart the Internet infrastructure should be.¹⁵ The core Open Internet rules aim to prevent broadband Internet service providers from using intelligence about the identity of end-users and uses (essentially, who’s doing what online) to exercise control through various means (blocking, throttling, pricing). Constraining the networks in this fashion enables and even empowers end-users to be active and productive rather than merely passive and consumptive. As we extend networked intelligence onto other infrastructures – e.g. transportation and electricity – and into other spaces – e.g. cities, workplaces, and homes – society will need to grapple with how to govern intelligence and intelligence-enabled control.

Think again about transportation. In the eyes or algorithms of traffic engineers, vehicles on roads are just like data packets online. Both draw on network capacity, can create congestion during transit, and generate value upon delivery. Yes, there are many important differences between vehicles and data packets and between roads and the Internet. The analogy draws attention to functional similarities and helps us to see relationships between the underlying infrastructures and society.

Functionally, traffic management depends upon intelligence and control. Typically, managers must know something about supply, demand, actual and expected traffic flows, interactions among traffic flows, and so on. And they must be able to exert control over the traffic and users. Control is an essential feature of traffic management, and it can take many forms, ranging from pricing to norms and technological constraints. Such different possibilities mean control can alleviate governance dilemmas but also give rise to them, too.

Infrastructure matters to society for a lot of reasons. Many economists, sociologists, and historians focus on how infrastructure investment played a major role in shaping the modern economy. Changes to transportation are inextricably linked to expanding cities, the rise and sprawl of the suburbs, and the transformation of rural areas. At the same time, infrastructure has shaped the human condition by enabling us to exercise and develop fundamental human capabilities. Consider free will and autonomy, concepts that play a central role in ethical and political visions of responsibility and entitlement. It’s one thing to theorize the importance of people making informed and uncoerced decisions. It’s quite another to create reliable pathways for citizens to realize their potential for free thought and action. That requires opportunities to be mobile, communicate, and socialize.

To democratize these existential benefits and many others,¹⁶ both roads and the Internet historically have been managed as a commons. Guided by the logic that priority shouldn’t be granted to anyone or any purpose, egalitarian policies have regulated infrastructure access and use. Sure, in an emergency, police can break speed limits and run red lights. But this narrow exception and others concern situations where our collective social welfare is at stake. They’re not instances of fast lanes going to the highest bidders.

Different governance regimes can protect neutrality. The end-to-end architecture of the Internet can safeguard this goal. So can bolstering public ownership of most roads with regulations that promote public goods. Esteem for the commons goes a long way towards explaining why traffic engineering aims to mitigate congestion and not maximize market values or anyone’s profits.¹⁷ But this ambition can change.

Forward-looking and quickly-acting industries are actively involved in the design and regulation of smart transportation infrastructure. It would be naïve to expect that shareholder expectations won’t influence the much-touted policy goals of using smart grids to enhance safety, minimize negative environmental impacts, and create efficient routing. These expectations are part of the package deal of using proprietary products and services to mediate how smart vehicles communicate to other smart vehicles and interact with smart infrastructure. As with the Internet, all layers of the emerging smart transportation system present opportunities for surveillance and control.

Consider a hypothetical: Suppose automated and street legal trucks go mainstream before autonomous cars do. If this occurs, the interests of trucking companies might be advanced through systematic design. To maximize the benefits of keeping fleets of trucks close to each other, other vehicles would have to be encouraged to not get in their way. How vigorous might the promotion of fleet flocking be? We don’t know for sure. You might think it’s an engineering question to be left to the technologists, but it’s really a question of political economy. If we adjust the hypothetical by suggesting a different path forward where a different industry sets priorities (say, autonomous vehicles, app-plus-data intelligence services, or even, dare we say it, networked bicycles and pedestrians), the same basic question arises: Who decides how priorities will be determined and on what basis?

The network neutrality debate taught us that smarter technology isn’t always better technology. As we collectively decide how smart new forms of infrastructure should be, we should keep in mind that sometimes smart systems can be too smart for our own good. We need to be very careful here, however, because what we really mean by smart is powerful. Keep in mind that traffic management depends upon intelligence and control, and what we are concerned with is the specific use of intelligence to exert control and set priorities. Such infrastructural power can lead to significant forms of techno-social engineering which are difficult to identify, much less resist. Prioritization of traffic (infrastructure use) based on willingness and ability to pay (market value) rather than some other measure of social value affects the distribution of infrastructural affordances. Simply put, prioritization determines who is capable of doing what and often with whom. It is nothing short of social planning.

What does this mean for our future smart transportation system? We cannot hope to do justice to this question in this chapter. But we will offer a short answer. There is a strong case to be made for network-neutrality-style rules at the infrastructure layers of any smart transportation system. As we’ve explained, such rules sustain an underdetermined environment and thus serve as a defense against engineered determinism, a topic we dig into in Chapter 12. Transportation systems are incredibly complex, and intelligence can and should be used to manage certain costs – such as congestion – and risks – such as accidents. This can be done in a manner that does not discriminate or prioritize based on the identity of drivers or some proxy assessment of or proxy for their value (e.g., market valuation), but again, it is a social choice involving complex trade-offs.¹⁸

Further, we should not fool ourselves into thinking that such costs and risks can or should be eliminated or even driven as low as technically possible because there are trade-offs to doing so.¹⁹ Tolerating some congestion, some friction, some inefficiency, even some transaction costs may be necessary to sustain an underdetermined environment conducive to human flourishing. We revisit this issue in the final chapter.

We are optimistic about the potential of a smart transportation system that intelligently saves lives, manages congestion, reduces environmental costs, and identifies maintenance needs. Autonomous or self-driving cars increase human agency by providing people with more time and attention to devote to other pursuits, whether productive, consumptive, contemplative, or social … who knows? While people debate exactly how many lives autonomous cars can be expected to save, it’s widely thought to be large enough that it would be immoral not to make them street legal as soon as possible.²⁰ We’re highly sympathetic to this view so long as enough attention is given to the difficult issues concerning governance, technical design, and social choices about normative priorities, as reflected in the trolley problem discussion as well as the traffic management thought experiment. These issues return us to the basic and familiar who decides theme. This question needs to be front and center as industry and governments pave the way for the emergent smart transportation systems. We cannot afford to leave it to technologists, politicians, or some abstract and ultimately meaningless conception of a free market.

We hope this book persuades the reader to consider humanity’s techno-social dilemma seriously. We are building our world and ourselves in the process. We cannot ignore the political economic realities or simply defer to idealized conceptions of free markets. “Free market” is merely a slogan, yet it has had quasi-religious power in shaping beliefs, preferences, and political debates and outcomes. Putting aside how all markets exist within social governance structures, we must emphasize that even robustly competitive markets routinely fail in many ways. The most ardent neoclassical economist will admit that competitive markets do not assure us of an environment that maximizes social welfare or human flourishing. As one of us explained in the context of the network neutrality debate:

Despite frequent claims to the contrary, we cannot count on markets alone to self-regulate or provide the necessary discipline on technologists designing techno-social environments. The trolley car dilemma and traffic management examples illustrate what we should expect.

Smart Media/Mediation

We began the previous chapter with mass media, and we’ll end this one with smart media. Essentially, the term “smart media” refers to the various digital media platforms and systems enabled by the Internet that operate at the applications, content, and social layers of the five-layer model we presented earlier. Many of the most powerful media companies that regularly engage in data-driven techno-social engineering of humans operate applications-layer platforms – social networks, search engines, marketplaces, even online games. We’ve discussed many examples in this book and there are more to come.

The dominant data-and-advertising-driven business model (and corresponding incentives) shapes smart media platform design, algorithmic policing, and users. The smart media industry recognizes how the business model affects platform design, often casting the optimization problem in terms of maximizing user engagement. While this term can have nuanced meaning, we think it really boils down to optimization for clicks – actions that generate revenue or data. Many have lamented the quick click culture engendered online, suggesting that it has reduced attention spans and cheapened social discourse. There are many counter-examples. Our simple hypothesis is that smart media platforms optimized for clicks engineer humans to behave like simple machines by treating them as resources. Second, smart media systems run into the same normative issues as other smart systems in terms of social planning and techno-social engineering. Smart media shape beliefs, preferences, knowledge, relationships, democracy, and so on. As such, there seem to be decent reasons to turn to human experts to evaluate quality and shape discourse. Conventional mass media recognized the need for expertise, judgment, and even ethics, particularly in certain areas such as news. Yet various social/cultural/technological developments seem to have pushed in the opposite direction.

The smartness of smart media systems is not grounded in human expertise or judgment concerning the media content; rather, it’s grounded in data, intelligence-generating systems, popularity, celebrity, and the apparent wisdom of crowds. In some domains, these sources of intelligence are likely to be better, but it’s a dangerous social gamble to rely on smart media systems across all knowledge domains.

To limit our discussion, we’ll stick with familiar giants, those household names that have the largest market valuations and social impact: Google and Facebook. In addition, we’ll focus on the problem of filtering objectionable content. However, these really are tips of the proverbial iceberg, and they are exemplary. There are so many different digital media platforms and content evaluation problems to examine. We don’t have the space to discuss the contours of fake news, cyberbullying, hate speech, pornography, or copyright infringement. These examples, like many others, seem to require smart media solutions, meaning intelligence-enabled control, whether through filtering, blocking, prioritization, outing, or other forms of discipline. Who knows whether in fact such control is socially desirable? It’s too important a question to be ignored or for the answer to be assumed. Yet the massive growth in the scale and scope of content, data, relationships, transactions, and so on, generated every minute, across multiple jurisdictions, pushes quite strongly towards smart technological mediation. Whether intelligence-enabled control in smart media systems is, will, or should be exercised discriminately as an exceptional, targeted intervention or regularly as part of a broader program of techno-social engineering remains an open question. We return to some of these issues in the final chapter – by focusing on the fake news problem and possible solutions. For now, we’ll focus on recent examples that highlight the basic problems.

In April of 2017, a shocking video was posted to Facebook that showed Robert Godwin Sr., a retired grandfather, being shot in cold blood by a man who filmed the murder he committed on a smartphone. Facebook didn’t take the video down until approximately two hours after it appeared and critics chastised the company for acting too slowly. Some went further and proclaimed that the tragedy – or more precisely, a tragedy like it – was inevitable thanks to techno-social engineering. Washington Post columnist Kathleen Parker wrote:

Parker’s concerns overlap directly with issues we discuss throughout the book. At the same time, the tragedy is part of a larger problem for smart media that we haven’t addressed yet. Some of the problem revolves around questions concerning whether Facebook and other similar online companies should be subject to content regulations just like “traditional” media broadcasters. And some of the problem revolves around questions concerning whether the artificial intelligence that plays an integral role in content policing is up to the challenge.

To get clearer on these issues, let’s think about some of the design features and value-laden considerations that are associated with Facebook and related online platforms. Many of these platforms want to provide users with a great amount of discretion over what content they can choose to post. Such dedication to diversity is an expression of a commitment to an ideal that many believe is central to the very fabric of democracy: promoting and protecting free expression. Spokespeople for Facebook, YouTube, Twitter, Instagram, and the like regularly portray their services as conduits for communication akin to phone companies that leave the scope and quality of conversations up to the users themselves. This is an idealized view which obscures how platform design mediates user communication and downplays how mediation is a form of techno-social engineering.

Even as these companies tout the value of free expression, they also affirm that their services impose limits and shouldn’t be confused with environments where anything goes. Explicit community standards are established to prevent information that violates widely accepted norms from being shared, remaining online, and travelling to inappropriate feeds. These standards are not just about enforcing decency, but are also a business tool that is partly implemented to minimize the likelihood that users will be so offended by certain shared content that they will cancel their accounts.

Consider Facebook’s Community Standards document.²² It states:

In principle, policies like the one Facebook uses are supposed to be minimally prohibitive, much like the legal proscription against falsely shouting fire in a crowded movie theater doesn’t diminish First Amendment protections of free speech. In practice, however, things are not so simple, such as Facebook’s reliance on a “real name standard.” The standard is said to impact vulnerable populations the most – people who wish to discuss sensitive topics in safe spaces, people who want to criticize power norms without retaliation, and victims of crimes like domestic abuse.²³

Additionally, smart media companies typically provide a mechanism for users to customize (at least somewhat) their experience of what they want to read and see on a platform. Personal customization through filters is supposed to enhance user control. But, in practice, filters can be techno-social engineering tools that modify user expectations and influence their preferences.

Shortly before the Facebook incident, YouTube came under fire for how its “restricted mode” filter was working. The filter is “an optional setting that you can use to help screen out potentially mature content that you may prefer not to see or don’t want others in your family to see.”²⁴ Unfortunately, some LGBTQ vloggers (i.e. people who create video-based blogs) discovered that restricted mode was rendering their content invisible, as did other minorities. Not only did this exclusion raise social justice questions, but it also had financial implications for the people who were losing page views. Technology writer Fruzsina Eordogh notes:

YouTube probably wasn’t intentionally trying to engage in unfair discrimination. At present, restricted mode, like all policing algorithms in the smart online media landscape, are vulnerable to four fundamental problems. First, algorithms can have a difficult time correctly identifying content that has context-specific meaning. This shouldn’t be surprising, as humans can be terrible at it too – a problem that was vividly illustrated when Facebook employees removed a famous photo of a nude girl running from a napalm attack during the Vietnam War.²⁶ Second, values change over time in a pluralistic democratic society, ideally in ways that diminish prejudice and shatter harmful taboos. This makes the designers of policing algorithms responsible for understanding social change and ensuring that their software adequately reflects the times. Neither are easy tasks, and this brings us to the third problem. Companies that rely on policing algorithms need to ensure that explicit and implicit programmer biases are mitigated against when necessary. Unfortunately, problems can go undetected until real world debacles occur. And, finally, there’s the issue that critics focused on when Facebook was slow to remove the gruesome murder post. How quickly can humans at a technology company respond to situations where their policing programs make the wrong call or fail to detect a problem?

Note that a fifth and perhaps more fundamental problem may be the data-and-advertising-driven business model itself.

Digital Companions

Automating Relationships

Algorithmic Bloodhounds

The Quantified Self (QS) movement is becoming quite popular.³⁹ Broadly speaking, QS draws on “body-hacking” and “somatic surveillance,” practices that, as their names suggest, subject our personal activities and choices to data-driven scrutiny. Typical endeavors include tracking and analyzing exercise regimes, identifying sleep patterns, and pinpointing bad habits that subvert goals. Recently democratized consumer technologies, especially smartphones that run all kinds of QS apps, enable users themselves to obtain and store diagnostic data and perform the requisite calculations.

In The Formula: How Algorithms Solve All Our Problems – And Create More, technology writer Luke Dormehl gives an interesting example of how far people are willing to go with QS to follow the ancient Socratic injunction “Know thyself!”

Dormehl sees the type of activity “Angela” engages in as having the potential to transform our very understanding of what it means to be human. Since we’re entering an age where algorithms increasingly tell us who we are, what we want, and how we’ll come to behave in the future, he says our time is marked by a “crisis of self” where algorithmic ideology is challenging the Enlightenment conception that, at bottom, we’re “autonomous individuals.”⁴¹ What does it mean to be me? Perhaps despite the stories I tell myself and others about my unique experiences, preferences, and desires, I’m not a special, singular snowflake, but a mere series of “categorizable node(s) in an aggregate mass.”⁴²

What makes “Angela’s” case especially interesting is that she apparently lacked confidence in her own abilities – her capacity to introspect and have edifying conversations about her well-being – to determine if work was making her miserable. She believed she needed more than her own mind and friends could deliver. She wanted cold, objective, quantified data to identify the source of an emotional problem. She thought that, without consulting the data, she couldn’t make a responsible life-changing decision.⁴³

In our private lives, QS practices won’t be limited to self-tracking. It’s likely that over time they’ll include a range of interpersonal applications and become integrated into routine, domestic techno-social engineering practices. Today, we tell our mobile-phone-carrying kids that being responsible means texting or calling when they get to their friends’ houses. But what will happen tomorrow, especially when we’re dealing with cases where more personal information is communicated peer-to-peer by machines without people in the loop? Will we expect automatic updates from our kids’ phones announcing their locations whether or not they intend for us to have this information? How, in general, can we determine when those practices cross a line and become dehumanizing?

To get clearer about where your sympathies lie, consider the following extreme scenario. If it seems like an instance where dehumanization occurs, techno-social engineering might be a process that brings it about.

Creep

As the preceding examples demonstrate, sociality is a basic human capability that’s highly susceptible to techno-social engineering. Hopefully, we provided diverse examples for your consideration. We intended for the variety to offer a hint about the topic we’re covering here: techno-social engineering creep.

In previous chapters, we’ve discussed the concept and explained how it conceptually relates to humanity’s techno-social dilemma and the slippery-slope arguments. We’ve also suggested that techno-social engineering creep is related to other more familiar creep phenomena, such as surveillance creep, outsourcing creep, and boilerplate creep.

This chapter has provided a series of examples where techno-social engineering creep exists or has the potential to gain traction. Techno-social engineering of human sociality can occur along multiple dimensions. To identify and evaluate techno-social engineering creep, it’s important to analyze those different dimensions. Suppose, for example, we focus on a particularly important relationship, such as the relationship between spouses or intimate relationships generally. We might examine techno-social engineering of a specific aspect or subset of that relationship, as in the menstruation tracking app example, or techno-social engineering of the relationship itself in a more totalizing fashion, as in the Android Ash example. The shift from narrow focus to a more totalizing one is one dimension along which we might observe techno-social engineering creep. As we’ve suggested, a couple might decide to extend their use of available techno-social engineering tools from the menstruation tracking app to apps that rely on other biological signals to manage other aspects of their relationship. The techno-social engineering creep could easily extend to non-biological signals as well, as a few of the other examples demonstrated. It’s hard to say when, if ever, such extensions would go too far and undermine or eviscerate the spousal relationship. But at the extreme, one might wonder whether both spouses come to resemble Android Ash, as far as their relationships are fully mediated and optimized according to the efficiency logic we discussed.

Another dimension along which techno-social engineering creep can proceed is relationship type – that is, the techno-social engineering tool can be extended from a certain type of relationship to others. Again, the menstruation tracking app, as well as any other tracking app, could be used to mediate non-spousal relationships. The technological tools are not limited to any particular type of relationship. Yet their initial adoption might be more easily justified in the context of a particular relationship.

Another dimension that’s worth focusing on is interaction type. For example, a techno-social engineering tool might optimize a particular type of interaction (e.g. a sales pitch, wedding speech, or pick-up line) or mode of communication (e.g. phone, email, text) and then gradually extend to other interaction types or modes of communication.

We’re listing these forms of techno-social engineering creep to give you a clear sense of how sociality maps onto the larger theoretical framework we’ve been developing. That framework will be expanded upon considerably in the next few chapters where we propose novel techno-social engineering tests, discuss the practical value of wagering that free will exists, and elaborate upon what the problem of engineered determinism entails.