In this chapter, we lay out the basic frame for studying innovation management. To do so, we are going to try to understand why innovation is important, for society, for companies, and for individuals, and to do that we take our point of departure in the “urtext” of innovation research, namely, Schumpeter’s work on Capitalism, Socialism and Democracy and especially the notion of Creative Destruction. To follow that up, we are going to untangle how innovation management fits within a broader context of capitalism as an economic system, within a particular ideology, and within the operations of the modern corporation.

1.1 Schumpeter: Creative Destruction

The signs of Creative Destruction are all around us. As consumers, we are incessantly presented with new products, services, and solutions. Things that were once staples of everyday life disappear. We regularly find out that we need things that were until recently nonexistent, even unimaginable. What we recognize as the largest companies, those that most dramatically shape our consuming lives by providing us with all this novelty, are mostly new ones. Just a generation ago, many of them either had not been founded or were in their absolute infancy. The technologies that these companies use similarly did not exist a generation ago. The same can be said of the technologies that shape our working lives. More and more of us work or will work in jobs that a generation ago made up a small share of the overall workforce or simply were not invented. We will also fulfill those jobs under organizational arrangements that are new. If you live in a Western city, you will see what were once factories being turned into fashionable housing and offices for a new kind of work and for a new kind of production. This churn, this constant change, is Creative Destruction unfolding.

Schumpeter’s book Capitalism, Socialism and Democracy was written in 1942, amid the horrors of a World War and at a time when many were deeply disillusioned with capitalism as an economic system. While many today share the disillusionment with capitalism, the real difference between then and now is that in the 1930s and 1940s, socialism really was an alternative, with a substantial share of the world’s population living in socialist economies. At the time, many looked to the Soviet Union and saw a well-functioning economic model, while in the West they would see something not quite undesirable. They would often not see the free markets and perfect competition on which capitalism is predicated, but monopolies, and think (with Adam Smith in mind) that this had to be economically inefficient and (with Karl Marx) that the tendency toward monopolies was inherent in capitalism, as was exploitation of the working class. If you looked, you might see the same today. They would also see vast inequality, with incredible wealth held by what we would call the 1 percent and the 99 percent have just suffered through a desperate depression.

Schumpeter’s book was an effort to understand socialism and capitalism and their respective relationships with democracy. In what is perhaps the most concise characterization of capitalism ever written, this would be Schumpeter’s argument: “Creative Destruction is the essential fact about capitalism. It is what capitalism consists in and what every capitalist [firm] has got to live in” (p. 83). To understand that we need to understand how Schumpeter thought about capitalism. “The essential point to grasp,” he writes, “is that in dealing with capitalism we are dealing with an evolutionary process … Capitalism, then, is by nature a form or method of economic change and not only never is but never can be stationary” (p. 82). Capitalism is inherent, unstable, and dynamic. To live in a capitalist economy is to live in circumstances that, for better and worse, constantly change.

There is a poetic streak, I think, to Schumpeter’s account of this constant change. Creative Destruction is a very evocative and quite intuitive way to capture something that is quite profound: that capitalism implies a constant “industrial mutation” that “incessantly revolutionize the economic structure from within, destroying the old one, incessantly creating a new one” (p. 83, italics in original). To him, there is a storm, a “perennial gale of Creative Destruction” blowing through capitalist society, like the Hindu god Shiva dancing through the world, tearing down what is and letting the new emerge from the ashes.

While we may not appreciate the causes of this process or the profundity of its implications, the threat of Creative Destruction is keenly felt by businesses. Schumpeter talks about Creative Destruction as an “ever-present threat” (p. 85), one that “disciplines before [new competition] attacks” (p. 85). It creates a paranoid anxiety for firms that they must compete not just in the short term on price and quality by constantly marginally improving. This is the kind of competition that firms are always in. In the slightly longer term, they also have to deal with “competition which commands a decisive cost or quality advantage, and which strikes not at the margins of the profits and the outputs of the existing firms but at their foundations and their very lives. This kind of competition is as much more effective than the other as a bombardment is in comparison with forcing a door” (pp. 84–85). The threat of such Creative Destruction constantly looms, even over monopolies, and makes them act as if they were in a state of cutthroat competition, because even if they are not currently competing they still compete against potential future entrants to their industry that will seek to overtake their position.

For Schumpeter, innovation is the fundamental impulse spurring Creative Destruction, the force that propels change in the economy. He employs a relatively broad definition of innovation, covering “new consumer’ goods, new methods of production or transportation, the new markets, the new forms of industrial organization” (p. 83). Today, we tend to use that as the definitional starting point for distinguishing between different forms of innovation, rather than use the umbrella term, but do bear in mind that innovation can take other forms than those presented here. Product innovation is what people typically think about when they think of innovation, because it refers to either new physical products or (increasingly) services. These tend to be highly visible. Process innovation covers new techniques or tools for creating products or services. These tend to be less visible and attention-grabbing because they often are sold in business-to-business markets but are indisputably important. Market creation refers to the opening up of new markets, whether by extending the geography in which a solution is marketed or in extending technologies into new fields. Innovations in transportation get relatively little attention in most modern innovation research but is nonetheless profound in its consequences. Consider, for instance, the implications that the humble forty-foot container has had on global trade and as a consequence on what you consume (Lewinson, Reference Lewinson2008). New forms of organization refer (somewhat counterintuitively to many contemporary readers) not to new forms of organization within industry, but new forms of organization of industry. This might include new forms of networks or interfirm collaboration, but Schumpeter would have imagined things like cartels and other forms of what might today be called anticompetitive behavior.Footnote ¹

These subclassifications are worth noting in part for their own sake – as we will return to, they each have their own set of causes and a vastly different set of effects on organizations. Contrary to what many would have you believe, being good at one does not imply being good at the other, as we will return to in the essays that follow. It is also worth noting that we can make several very meaningful additions to the list. These might include innovations in business models (e.g., new ways to create and capture value) or innovations in techniques (e.g., new ways to use products or services). Similarly, new forms of organizations within industry are also worth considering. The ways that firms organize (legally through constructions like corporations or practically around ideas like the divisionalized form) are often taken for granted but are in fact innovations that were invented and brought into being (see Section 11.1 for an elaboration).

The other important thing to understand about these five forms of innovation is that they in interaction create what Schumpeter would surely see (and what you should see) as the miracle of capitalism. They allow for the long-run expansion of output and the driving down of prices, putting goods and services that were once only accessible to the hyper-rich into the hands of all of us. Innovation is what allows those of us who are fortunate enough to be born into the affluent societies of the Western world to enjoy standards of living that were frankly unimaginable just 200 years ago. To be clear, capitalism has incredible and far-reaching downsides. It creates gaping holes in our souls as both workers and consumers (as we will return to) and is a driving force in creating ecological damage on a potentially catastrophic, existentially threatening, scale. But be clear also that the kind of growth in material wealth that capitalism has created is a miracle that is unprecedented in world history, as Deirdre McCloskey’s work will make eminently clear (in Section 1.2). Irrespective of what you think about the future, there is no way that innovation is not a part of it, either as an engine of progress or as the thing that gets us out of the mess that we, as a species, are in.

Going forward, we are going to see Creative Destruction come up repeatedly as a mainstay of innovation management theory. We will see it as the constant threat that Schumpeter describes. In this capacity, all the various ways that established firms and industries could be destroyed by innovation weigh heavily (and should weigh heavily) on the minds of incumbent firms. We will see the dilemma that it poses between competing at the margins and at the center of profits as constitutive of one of the key challenges of innovation management: of balancing exploration and exploitation both at the level of the firm allocating its resources between different activities and at the level of how a technology should be developed. We will also see it as an opportunity. Just as innovation can destroy, it can also elevate new entrants and new ideas to prominence. Attackers can have advantages over incumbents, and Creative Destruction is something that people can actively try to bring about. We will see the sense of turmoil that Schumpeter describes as a stable of industrial change but also as something that firms alternatively try to bring to an end or to leverage. We will see the tension between the immaculate wealth creation and the far-reaching and painful destruction as a key challenge facing society. And we will see how the constant flux, the constant search for new and better alternatives is currently reshaping the way both that innovation happens and that firms organize. We live, for better and for worse, in fantastically interesting times and understanding innovation and its implications is a central part of understanding them.

Inevitably, we take our point of departure in Schumpeter.

1.2 McCloskey: The Origins of Capitalism as We Know It, or Why Are We Rich Today?

Contrary to what many experience today, human history is – in material terms – mostly a story of non-change. Then, germinally around the end of the eighteenth century and forcefully through the Industrial Revolution and up to today, everything changed. McCloskey makes this evocatively clear:

We are fortunate to live on that blade. As McCloskey makes clear, for all the very real ailments of modern capitalism and all of its incredibly real, incredibly destructive effects, the world we live in is undisputedly much better for a greater share of more people than has been the case at any other point in human history. This is the case because of economic growth.

In his book The Lever of Riches, economic historian Joel Mokyr argues that economic growth in a society can come from several sources. We can grow richer through “Smithian growth” (named for Adam Smith, who needs no introduction), which comes from increased trade. Increased trade allows for larger markets and for increased specialization and with that comes increased productivity, which makes society richer. We can also have “Solovian growth” (named for Robert Solow, an American economist working in the second half of the twentieth century). This comes from increased investment in what economist call capital goods, that is, more sophisticated production equipment. Having more production equipment allows workers to be more productive and society to produce more goods with the same scarce inputs. To this one might add “Marxian growth” (after Karl Marx), which would come from increasingly brutal exploitation of workers or of natural resources. Just as having more production equipment can make workers more efficient, pushing them harder can increase productivity, but only up to a certain back-breaking point. Extracting more resources from the planet or society can do something similar (see Section 11.2).

These things surely matter in explaining why we are richer today than we were in the past, but they cannot as McCloskey argues explain the blade of the hockey stick. That change is too momentous to be explained by trade, capital accumulation, exploitation, or any other single concept in the economist’s theoretical toolbox. Except, that is, for one kind of growth, the kind that Mokyr calls Schumpeterian growth: growth in human knowledge and ideas and innovation. Innovation more than anything else explains the material wealth of the modern world.Footnote ² That is why we need to understand it. Not just because it is essential for firms to compete effectively, but because innovation is what makes society better off. Yes, innovation may be overhyped and glorified and lionized and romanticized, but it is also fundamentally the reason we no longer live in caves and hunt with spears (spears were an innovation, of course).

The question for McCloskey, then, is what brought about all this innovation? Why did the Industrial Revolution happen, and why does it continue to happen? Why did we get on the hockey stick’s blade when we did not, say, 200 years before (in which case we would be unfathomably well-off today) or 200 years later (in which case you probably would not be reading this book, but toiling away in a field somewhere)?

Her answer is that things began to change because ideology had been changing.Footnote ³ Following the Reformation, new ideas began to emerge about the urban middle class of hiring, owning, professional, or educated persons (i.e., the “bourgeoisie”), first in the Low Countries and later in Britain and ultimately in the United States. To be clear, it was not the Reformation and the attendant emergence of what German sociologist Max Weber (Reference Weber1930) calls the “Protestant work ethic” that created the modern “the Spirit of Capitalism” (because, if that were the case, why do we see capitalism today in distinctly non-Protestant countries?). It was that with the reformation came a new way of thinking and talking about the bourgeois class. This thinking, in an evolved version, is still with us today and profoundly shapes how people think and talk about innovation in society and organizations.

Where before this group of merchants had been viewed with some contempt, they began to be viewed more favorably, as dignified and not repugnant. Up to this point, the best and brightest had gone to work in the courts, the church, or the military, because those were “honorable” professions. Alas, they are also mostly conservative ones. But with changing attitudes toward merchants, “[i]t became honorable … to invent a machine for making screws or to venture in trade to Cathay” (p. 12). “Ordinary conversations about innovation and markets became more approving” (p. 7), “general opinion shifted in favor [italics added] of the bourgeoisie and especially in favor of its marketing and innovation” (p. 7). This process had been going on for a while (that is the way it is with the ebb and flow of the tides of history) before the Industrial Revolution took off. What at the end of the eighteenth century emerged from that process was that ideology changed in such a way as to make Creative Destruction the socially desirable thing that it had not been before. The result of that change in ideology has been a “novel and immense and sustained, almost lunatic, regime of innovation” (p. 19). With this regime arose what Schumpeter in his day and we in ours recognize as capitalism, that dynamic economic system of which Creative Destruction is the essential fact.

To be clear, what was new was neither capitalism nor innovation. Capitalism existed long before the early nineteenth century, but this was capitalism in the sense of commercial activity, with trade, with prices emergently determined by supply and demand, with financial credit, with money and accumulation of wealth. None of those things, as McCloskey makes eminently clear in her subsequent chapters, were new. Innovation also was not new. In Mokyr’s book, he describes Western technological progress from Classic Antiquity through to the Middle Ages and the Renaissance before arriving at the Industrial Revolution’s “Years of Miracles.” You could similarly describe an important history of innovation in ancient China, in the empires of South America up to the brutality of European conquest, in the Arab world and at some level in all human society, ever. Innovation has been with us since our ancestors on the Savannah. McCloskey notes technological change already in the Upper Paleolithic and among the proto-Australians (innovations in boats is what got them to Australia, after all), to give just a few of her examples. What was new was the scale of innovation and with it a new competitive process at the heart of capitalism. Sometimes, a change in degree is a change in kind and that is exactly the case here. A change in ideology spurred the gale of Creative Destruction to the historically unprecedented heights that we see today.

This image of innovation as ideologically driven is something that I encourage you to take into your further reading. It is also something that we should have some concerns about. There can be little doubt that we are living in a time obsessed with innovation. Just as you can see the signs of Creative Destruction all around you, you will also hear what some call “the gospel of innovation” and the “industrial religion of the twentieth century” (Salter & Alexy, Reference Salter, Alexy, Dogdson, Gann and Philips2014) preached and hyped by practitioners and scholars in business and management and technology, sometimes with good intentions and other times for personal profit in some form (Kärreman et al, Reference Kärreman, Spicer and Hartmann2021). Change, they say, is the only constant and innovation the only answer. The ideological pendulum has swung very, very far from when change was a dirty word and innovation not yet coined.

Despite this swing, three things are important to understand. One is that Creative Destruction and innovation-based capitalism are not something that firms enjoy or necessarily thrive on, despite all the hullaballoo. Entrepreneurs and innovators may thrive, but established and powerful firms often do not. As Schumpeter saw, Creative Destruction creates an environment of paranoid anxiety and, as we will see in our coming readings, myriad difficult challenges for firms wanting to maintain their position. Most established firms, for sure, would prefer not having to innovate, but do so because others do it and they therefore have to. For those established firms, innovation is a race to the bottom that they sometimes win and sometimes lose. Fortunately, that race is what makes consumers better off. All that struggle to expand output and reduce prices puts more, better, and cheaper goods into the hand of a greater share of consumers.

Another is the possibility that ideology can to some extent substitute for actual competitive pressure, in the Schumpeterian sense that a pervasive ideology of innovation can make firms more paranoid and in the sense that an ideology can make it necessary for firms to innovate in order to stay legitimate in the eyes of relevant stakeholders. A strong ideology can force firms to engage in innovation as a ceremonial, rhetorical practice. This line of thinking would bring you into a somewhat different understanding of the role of innovation in firms, which is undoubtedly a big part of the picture and the subject of much of what is called institutional theory. It is something that falls somewhat outside of the scope of this particular book but definitely is a stable of courses on organizational theory.

The third thing to understand is that the pendulum can swing again. Just as a change in ideology that nobody planned made the modern world possible, a change in ideology that nobody plans or expects can happen again and undo it. And that certainly would be tragic and dystopian. The world we live in may, in Delong’s formulation, be “marvelous and terrible, but by the standards of all the rest of human history, much more marvelous than terrible” (Reference Delong2022, p. 1).

1.3 Chandler: The Rise of Managerial Capitalism and the Idea of Corporate R&D

It is an interesting fact of Schumpeter’s oeuvre that he fundamentally changed his mind about where innovation comes from in the economy and, by implication, what type of organization is fundamentally responsible for economic growth. Living in what was then Czernowitz in what is now Ukraine, the young Schumpeter published The Theory of Economic Development in 1911. In that book, he “flatly asserted … that individual entrepreneurship held the key to economic growth in any country” (McCraw, Reference McCraw2007, p. 149). When he published Capitalism, Socialism and Democracy in 1942, he was a professor at Harvard University, having lived in the United States since 1932. In that book, he describes big business – large corporations, operating in oligopolistic markets – as the real drivers of innovation and economic development.

That is a profound change, and one that reflects the changing nature of capitalism and the changing nature of innovation in the interim period and the fact that a particular form of change happened first in the United States and only later spread to Europe. By the 1940s, a still-new form of organization had come to dominate American capitalism: the “modern,” vertically integrated, multiunit business. This new organizational form brought with it “Managerial Capitalism,” a way of organizing economic activities that simply did not exist a century before. It had also brought about the notion of “industrial research” and the corporate Research & Development laboratory. Originating in the German chemical industry, industrial research was developed and elaborated in the United States in the early 1900s and became, in the interwar period, the de facto way for innovation to be organized in American industry and, later on, most of the world. Profound change indeed, and a suitable occasion for a change of mind.

The emergence of the modern business – large, divisionalized, and vertically integrated – and the attendant emergence of Managerial Capitalism and, to a lesser extent, industrial research are the topics of Chandler’s book. The traditional unit of production in the American economy was the “single-unit business enterprise. In such an enterprise, an individual or a small number of owners operated a shop, factory, bank, or transportation line out of a single office … handled only a single economic function, dealt in a single product line, and operated in one geographic area” (p. 3). Prior to 1840, this was the only type of business that really existed and had existed for a long time. Its operations would have been immediately intelligible to an Italian Renaissance merchant, living four centuries earlier. The modern enterprise that Chandler describes “contains many distinct operating units and it is managed by a hierarchy of salaried executives” (p. 1). “The activities of these units and the transactions between them [are] internalized” (p. 2), meaning that many activities, disparate in nature and geography, happen within the same firm.

More specifically, this type of firm was vertically integrated. When a firm vertically integrates, it integrates activities that would otherwise be done by suppliers or customers into its own operations. A vertically integrated firm might consist for example of distinct divisions operating a shipping line, but also have divisions building the ships that sail for the line and divisions mining and refining steel to build those ships. It might have a division to operate the ports in which those ships dock, and another to extract and process oil to fuel them. It might also have a ground logistics division, moving over land the containers shipped by the shipping line, and a division comprised of wholesale and retail activities receiving shipments through that ground logistics company. In a vertically integrated company, goods are not (only) sold through the market the way a non-integrated shipping line might buy fuel from an oil company. Activities that might otherwise happen through the market are internalized and organized into distinct operating divisions. “By World War I this type of firm had become the dominant business institution in many sectors of the American economy” (p. 3). Such a vertically integrated firm would be comprehensible for us today, accustomed as we are to its prevalence. For the Italian Renaissance merchant, it would have been a strange beast indeed.

With the rise of this type of firm had come the entirely new idea of middle management and, more broadly, managerial capitalism. The management that happened within traditional firms was done directly, by owners who typically partook in the production process. The firm would fulfill one economic function, and then exchange the products of that function in the market. The activities of the large, divisionalized and multiunit modern business were “monitored and coordinated by salaried managers rather than market mechanisms” (p. 3), and the work of monitoring and coordinating was a new form of work, requiring new administrative innovations and ideas and a new class of workers to do it. Consequently, the rise of the modern enterprise led to a demand for middle managers, workers specialized in management activities, having received specialist training in the “science” of management and business. Middle managers would manage subordinate managers (not operating workers) and report to managers (not company owners) and considered themselves to belong to a new and distinct managerial class. Being so used to this form of organization and to the idea of managers and of getting managed, it can seem strange to contemplate that “[A]s late as 1840 there were no middle managers in the United States” (p. 3). By 1940, most of American economic life unfolded in ways planned and organized by managers.

Why did this happen? At the heart of the process that Chandler proposes is an interplay of the technological, the economic, and the sociological that created what Rosenberg would describe as “pressures toward bigness” (Reference Chandler1977). First, “modern multiunit business enterprise replaced small traditional enterprise when administrative coordination permitted greater productivity, lower costs, and higher profits than coordination by market mechanisms” (p. 6). In other words, when the costs of coordinating activities internally fell below those of coordinating through the market, the road was paved for modern firms to supplant the traditional. The reduction in those costs depended on technological advances in the form of cheaper and faster communication and the advent of new managerial techniques. Second, “the advantages of internalizing the activities of many business units within a single enterprise could not be realized until a managerial hierarchy had been created” (p. 7). The availability of a cadre of workers able to undertake the economic function of coordinating and to use the new techniques available for managing was a prerequisite for the emergence of the modern firm. Third, “modern business enterprise appeared for the first time in history when the volume of economic activities reached level that made administrative coordination more efficient and more profitable than market coordination” (p. 8). This increase in volume was driven by new technology. New production techniques made possible a vast expansion of output, and railroads made possible the expansion of markets to absorb it. Once it had emerged, the new managerial class perpetuated itself. It became a source of power, became increasingly technically sophisticated, distanced itself from ownership, shaped company policy in the direction of stability, and ultimately reshaped the economy.

It is with the emergence of this new form of organization and of managerial capitalism that we see the emergence of corporate research and development and, more specifically, the “Corporate R&D Lab.” Before corporate R&D, “science and technology were products firms purchased” (Dennis, Reference Dennis1987, p. 482) and corporate laboratories existed merely to conduct tests and ensure the reliability of purchased technology. With vertical integration, a more expansive role emerged. Just as the growing multiunit business vertically integrated to secure control of raw materials, it might also secure control over the ideas and technology that enabled its operations by internalizing the development of relevant scientific insights and relevant technological advances. Beginning in the early 1900s with General Electric, AT&T, and Kodak, major corporations began to organize their own industrial research laboratories. They took their inspiration from the chemical labs of the German dye industry (Beer, Reference Beer1959) that had first developed this organizational innovation, and by the middle of the century, it had become an entirely mainstream part of large, vertically integrated firms.

With the corporate R&D lab, innovation became something that firms not just did themselves but also something that was subject to professional management and needed to integrate with corporate strategy. More specifically, the process of developing innovations and, before that, choosing between and prioritizing innovations to develop was subjected to management. When firms bought innovation, the innovation could be assessed and its fit with corporate strategy could be assessed, albeit with some uncertainty. When firms had to make innovations, decisions about innovations were shifted forward in time and technological uncertainty increasingly taken on by firm, as opposed to independent inventors, because the firm was now undertaking research, not merely assessing its results. Moreover, the organization had to find a place within the corporate bureaucracy for inventors. Where before inventors, their psychology and their scientific ideals were considered largely incommensurable with bureaucratic organizing (Reich, Reference Reich1985; Dennis, Reference Dennis1987), now both organizations and inventors had to accommodate a new, shared reality.

Like the modern corporation, the modern corporate R&D lab was so thoroughly successful that it ultimately spread throughout the industrialized world and became the standard model of business innovation, in the process prompting Schumpeter to revise his understanding of the nature of capitalist innovation: the largest firms with their R&D labs, not heroic entrepreneurs, were the real drivers of innovation. Today, corporate R&D is a central, if not the presumptive, site of business innovation and many challenges of organizing for innovation relate precisely to the management of industrial research and development.