My interest in social behaviour probably stems from me belonging to a species where social interactions rule our lives. As a teenager I was very interested in psychiatry, and I also recall spending hours observing the human-like attitudes of chimpanzees and gorillas in the zoo. Accordingly, while studying biology at university, I seriously considered the option of studying chimpanzees. But at the time of graduating I had come to realise that this was not an easy task. Either one would conduct studies in the field with the only hope to collect sparse observational data, or one could study chimpanzees in enclosures, but the very contrived and artificial environment makes it difficult to understand how behaviour might have been modulated by natural selection. I therefore started to think about other social organisms that could be easily observed and, importantly, where it was possible to experimentally manipulate key components of social organisation. This paved the way for my interest in myrmecology.

My first work in the field of myrmecology (the ant world) was primarily concerned with understanding the evolution of multiple-queen colonies, which at that time was seen as a major problem for kin selection theory. During my PhD and postdoc I conducted many experiments, which, together with the work of some colleagues, allowed us to solve the apparent paradox of reduced relatedness stemming from colonies containing several reproductive queens.

Like many aspects of science, my interest in social evolution came about at least in part by accident. My early training was in population and community ecology, on plants and rodents living rather asocial lives. Although I had read much of Peter Klopfer's 1973 book Behavioral Aspects of Ecology, my own evolutionary interests were primarily focused on life-history evolution. The first turning point came when Mike Cullen, formerly from Oxford but recently appointed to a chair in the Monash Zoology Department where I was studying for my doctorate, popped into the office I shared with Dick Braithwaite, another aficionado of rodent population biology, and dropped off what I am fairly sure was the first copy in Australia of the landmark 1978 textbook by Krebs and Davies. This book, with its excitement and clarity of thought, should remain compulsory reading for everybody, despite its subsequent eclipse by later editions.

There is no doubt that the next greatest influence was the classic book and series of articles produced in the early 1980s by Tim Clutton-Brock, Steve Albon and Fiona Guinness on the ecology of red deer Cervus elaphus (Clutton-Brock et al. 1983). Their focus on lifetime estimates of fitness in free-living animals seemed to me to be the work I would like to do. I also felt that I had found the perfect study animal. The commonest marsupials in nearby forests were antechinuses, small shrew-like animals with strange sex lives.

Overview

Aggression ranks among the most misunderstood concepts in the behavioural sciences. Commonly viewed as an aberrant form of behaviour, situations of conflict are pictured in the context of unfavourable or stressful circumstances, brought about by amoral urges, in critical need of our cognitive control, and with negative consequences for all involved. Such a view fundamentally misunderstands the biological significance of all behaviours that occur in the context of attack, defence or threat. Deeply routed in the demands of the natural world, the ability to assert oneself represents a critical solution to any individual's need for self-preservation, defence of its interests or resource competition. Examples of aggression are found throughout the entire animal kingdom, regardless of its bearer's specific neural or cognitive faculties, phylogenetic origins or sociobiological circumstances. It has become abundantly clear that aggressive traits have been shaped by evolution like any other behavioural phenotypes, and a range of underlying mechanisms in the causation of aggression are now being unravelled.

This chapter aims to present a comprehensive overview of the issues that are encountered when trying to understand the hows and whys as individuals oppose each other. The chapter focuses special attention on delineating distinct behavioural phenomena such as aggressive tendencies, dominance or violence. Game-theoretical considerations offer a powerful theoretical framework to assess the evolutionary consequences of different behavioural strategies. A discussion of proximate mechanisms attempts to link these behavioural heterogeneities to the functioning of underlying neural and endocrine control systems.

Overview

Evolutionary game theory may have done more to stimulate and refine research in animal behaviour than any other theoretical perspective. In this chapter, we will review some of the insights gained by applying game theory to animal behaviour. Our emphasis is on conceptual issues rather than on technical detail. We start by introducing some of the classical models, including the Hawk–Dove game and the Prisoner's Dilemma game. Then we discuss in detail the main ingredients of a game-theoretical approach: strategies, payoffs and ‘solution concepts’ such as evolutionary stability. It should become clear that first-generation models like the Hawk–Dove game, while of enormous conceptual importance, have severe limitations when applied to real-world scenarios. We close with a sketch of what we see as the most important gaps in our knowledge, and the most relevant current developments in evolutionary game theory.

Introduction

Social behaviour involves the interaction of several individuals. Therefore within most social contexts the best thing to do depends on what others are doing. In other words, within social contexts selection is typically frequency-dependent (Ayala & Campbell 1974, Heino et al. 1998). Game theory was originally formulated to predict behaviour when there is frequency dependence in economics, for example competition between firms (von Neumann & Morgenstern 1944, Luce & Raiffa 1957).

I was asked to explain why and how a Dutchman got to be a professor teaching animal behaviour in a French university. Someone must have thought that my story could provide some guidance for aspiring ethologists and behavioural ecologists. I am not so sure that my career path is one that should be followed, but perhaps someone can learn from my mistakes. I think I can now afford to write about them without much of a negative effect on my career. Not that I have bothered much about my ‘career’, but that is perhaps the core of my problem. I have never been good at preparing myself for the future, so after treading the mills of the Dutch educational system I found myself regularly confronted with steps in life that I should have prepared, if not better, then at least earlier. And so I ended up in a ‘cul de sac’. But let me start from the beginning.

I can't tell you what kind of ‘—ist’ I am exactly at this point – primatologist, behavioural ecologist or evolutionary psychologist – but I went to the university to become an ethologist. The reason was simple: I liked animals a lot, and notably the furry ones. I definitely preferred seeing them alive, healthy and doing their own thing. I understood from books by the likes of Tinbergen, Lorenz, Wickler and Eibl-Eibesfeldt that ethologists did professionally what I liked to do anyway: watch animals behave.

Animals behave according to their previous social experience. This has been demonstrated in a wide range of species all across the animal kingdom (Rutte et al. 2006). Remarkably, this response to previous social interactions is not confined to experience with known individuals. It appears to be a much simpler trait than we might assume from our own intuition. If an animal fights with any conspecific, it will behave differently in future encounters, depending on whether it won or lost. These renowned winner and loser effects are among the most predictable traits in animal interactions (see Chapter 14). In humans, we term the psychological mechanism involved ‘self-confidence’. But, most interestingly, animals respond contingently upon social experience also in a sociopositive context. If they received help, they are more likely to help others as well, even if donors and receivers are completely unknown to each other. This generalised form of reciprocity has been demonstrated in humans and Norway rats so far (Bartlett & DeSteno 2006, Rutte & Taborsky 2007), but we assume it to be a general phenomenon, just like the ubiquitous winner and loser effects.

In the early 1970s, after Hamilton's (1964) and Trivers' (1971) revelations on kin selection and reciprocity as key mechanisms of altruistic behaviour and advanced sociality, we were tempted to believe that the major riddles in this field had been solved. The vast literature that emerged on the evolutionary mechanisms of altruism and (eu)sociality since then, at both theoretical and empirical levels, proved us dead wrong.

Behavioural ecology is the study of how behaviour is influenced by natural selection in relation to ecological conditions. It is a relatively new field – about 40 years old – to which I have been an enthusiastic contributor for 30 years. During this time behavioural ecology has grown in popularity, empirical richness and theoretical sophistication. Around the world, behavioural ecologists are employed in universities, conservation organisations and government agencies; they have been elected to national academies of sciences, and received prestigious prizes (Crafoord, Cosmos, Kyoto); at least a dozen scientific journals publish articles on behavioural ecology; and the biennial meeting of the International Behavioral Ecology Society regularly draws more than 1000 participants.

Why is behavioural ecology so appealing? For me, a curious naturalist, it is the challenge of asking new questions, the fun of addressing them using the theoretical framework pioneered by Darwin and embellished by Williams, Hamilton, Maynard Smith, Trivers and Dawkins and the empirical approaches pioneered by Tinbergen, Lack and Goodall, as well as the deep satisfaction of cracking an unsolved puzzle – at least occasionally. For example, in my case, why do bank swallows breed colonially, why do Belding's ground squirrels give alarm calls and how do they recognise half-sisters, why do wood ducks lay multiple parasitic eggs, why do queens of many social insects mate so frequently, why do naked mole-rats live like eusocial insects, and how have bdelloid rotifers survived and speciated without sex for 40 million years?

My interest in huddling was aroused by the discovery that GNAS, the locus that encodes the G protein α stimulatory subunit (Gαs), was preferentially expressed from its maternally derived allele in brown adipose tissue of mice (Haig 2004). Since then, additional imprinted genes have been identified with effects on the recruitment and function of brown adipocytes (Haig 2008). The observation that multiple imprinted genes are expressed in brown fat – the principal site of facultative thermogenesis in young mammals – suggests that genetic conflicts over heat transfers among relatives have been a significant selective force in mammalian evolution.

Huddling together for warmth is a simple and widespread cooperative behaviour of inactive birds and mammals (Hill 1992, Forbes 2007). Huddling reduces heat loss from animals that are warmer than their environment by reducing each individual's exposed surface. Moreover, a group can raise the temperature of an enclosed space more effectively than can an individual acting alone. Savings may be substantial. For example, a 10-day-old rat pup in a group of eight consumes 37% less oxygen at 28 °C than an equivalent solitary pup (Alberts 1978).

The fuel consumed by thermogenesis is a direct personal cost to the individual generating heat, but others share in the benefit. Contributors are therefore vulnerable to exploitation by individuals who skimp on their share of the heating bill.

I have been watching birds since I was a child. H. Mendelssohn introduced me to the scientific aspect of ornithology. The book by Niko Tinbergen, The Study of Instinct (1951), convinced me that watching birds could be an intellectual challenge. I spent most of 1955 in Oxford with Tinbergen, a year that introduced me to ethology and to the study of behaviour at the gull colony in Ravenglass. In 1970, after working as an activist for conservation in the intervening years, I spent a few months at Oxford with David Lack, who convinced me that individual selection is the only mechanism by which to interpret adaptations.

My doctoral thesis was on the social behaviour of the white wagtail Motacilla alba. I was able to show by experiments how ecological conditions, especially food distribution, shape its social system. This study started my interest in studying the relationship between ecology and social systems (Ward & Zahavi 1973).

In 1972, my student and friend, Yoav Sagi, questioned the logic of Fisher's model of mate choice. This stimulated me to develop the handicap principle as an alternative to Fisher's model. The implications of the handicap principle dramatically changed my understanding of evolution. I soon realised that the handicap principle is a necessary component in the evolution of all signals (a signal is defined as a trait that has evolved in the signaller, in order to transfer information to receivers, to affect the behaviour of the receivers in a manner that is beneficial to the signaller).

Social behaviour garners broad interest: biologists, social scientists, psychologists and economists all incorporate a consideration of social behaviour in their studies. This breadth of interest is unsurprising, as the vast majority of animals (and all that reproduce sexually) live partly (or fully) in social environments. As Robert Trivers (1985) succinctly put it, ‘Everybody has a social life.’ Some of this interest undoubtedly emerges because members of our own species (Homo sapiens) live in extensive societies and spend much time interacting with each other. Yet you do not have to be human for social behaviour to have a strong influence on biological processes. The significance of social behaviour is easy to see: if you isolate an ant, a fish or a bird from its peers in a sort of Kaspar Hauser setup, within a short time many of its ‘normal’ behaviours will change and be impaired. Social behaviour, heuristically defined as activities among members of the same species that have fitness consequences for both the focal individual and other individuals in the group, is thus ubiquitous.

The perplexing causes and far-reaching implications of social behaviour make it a rich subject to help understand evolution (Gardner & Foster 2008). The understanding of social evolution is challenging, given that social behaviour is often costly. Furthermore, unlike many traits that are passively selected by the environment, in the context of social behaviour the animals create selection for themselves by interacting with each other.

Overview

How and when social behaviour evolves has long been a focus of study within evolutionary biology, yielding the entire subfield of sociobiology and behavioural ecology. Although social behaviours may be explored in the same way as any other type of phenotype, the genetics underlying social behaviours differ from traits that do not vary depending on the social environment in which they are expressed. Social behaviour is best described as an interacting phenotype: a phenotype that depends at least in part on interactions with social partners for its expression. Models of indirect genetic effects provide a quantitative genetic framework for understanding the sources of variation underlying interacting phenotypes. They also suggest a genetic mechanism for inheriting traits that are expressed among rather than within individual animals, and identify selection arising from the interactions (termed social selection).

This chapter will first introduce the concepts of interacting phenotypes, indirect genetic effects, and social selection. We build a quantitative genetic model for interacting phenotypes and discuss how the evolution of such traits differs from non-interacting traits. We then explore the parameters of the model in more depth. We subsequently summarise existing empirical studies of indirect genetic effects, discuss the implications for the evolution of behavioural traits through social selection, and discuss transitions between quantitative genetic and molecular genetic approaches to studying behavioural evolution. Finally, we highlight potential future avenues of research.

Overview

Speciation results from the evolution of traits that inhibit reproduction between populations. This chapter discusses theoretical and empirical studies that relate to how social behaviour influences those reproductive barriers. Behaviour can influence prezygotic isolation by causing non-random mating or non-random fertilisation. Learning can affect mate recognition through cultural transmission of mate advertisement signals and sexual imprinting. Behaviour can also contribute to reproductive isolation if hybrids are discriminated against as mates or if female re-mating influences fertilisation success.

A general theory of speciation does not exist, but a variety of models have been developed to describe how selection can favour speciation in particular situations. Theory suggests that sexual selection, in particular, should be a diversifying force. However, among vertebrates sexual selection by female choice has favoured expression of condition-dependent traits, which are typically not reliable for species recognition. Better examples of sexually selected traits functioning in both mate-choice and species-recognition contexts can be found among some insects, such as crickets. The best examples of sexual selection influencing speciation in vertebrates come from cases of sexual imprinting in birds where offspring learn species-recognition cues in the nest.

Sexual selection can also operate after mating by sperm competition or cryptic female choice. Either or both of these mechanisms likely contribute to conspecific sperm precedence, which may result in reproductive isolation after mating. Sexually antagonistic coevolution has the potential to drive speciation in systems with sexual conflict.

While interviewing potential candidates for our departmental PhD programme every year, I usually ask the candidates what they would like to work on if they had complete freedom in the matter. Some years ago an unusually determined student gave me a firm answer: he wished to work on lesser cats, asking whatever questions he might be able to and using whatever methods that might work. I tried to argue with him, reminding him that lesser cats were extremely hard to study – they were nocturnal, shy and difficult to locate, let alone observe and obtain quantitative data on. Why not work on an easier animal with which you can ask more sophisticated questions, I pleaded. No, he was adamant – lesser cats it would be, if he had any choice at all. His determination has stayed in my memory ever since. Other students have given me other kinds of answers, though I can recall none as determined as the young man in love with lesser cats. Some students gave primacy to the research field or question and were quite flexible about the study animal and methods to be employed. Others were sold on a method such as computer simulations or field biology, but were quite catholic about the exact questions or of the model organism.

Overview

Inheritance is associated with a paradox: it roars with the survival of the species, while at the same time it whispers a fragile message that is constantly modified even among kin. The genes, the environmental context and the traits that arise from their interaction are interrelated. A complexity that characterises this three-way relationship has been attributed to the nature–nurture dichotomy. Traditionally, nature is understood to mean the genes, whereas nurture denotes the environment. So, for example, people may debate why one pumpkin is superior to another – was it the quality of the soil or other growth conditions in the pumpkin patch, or was it the specific combination of alleles in that pumpkin's genome?

In recent years, there has been a long-overdue paradigm shift from a limited focus on the nature–nurture dichotomy to a more expansive view that includes gene by environment (G × E) interactions and even gene–environment (G ↔ E) interdependencies, as defined and discussed in this chapter (Rutter 2007). A mechanistic basis for the concept of interdependency arose from advances in molecular biology and genomics which show that DNA is not only inherited but is also environmentally responsive. The latter argument is supported by findings that individuals with dissimilarities in their DNA (DNA polymorphisms) are differentially affected by the same environment. Different environments through development and adulthood can affect individuals with one genetic variant but not another.

What the fruit fly is for classical genetics and the squid axon for neurobiology, the insect society is for experimental sociobiology. Many of the sociobiological concepts and hypotheses proposed more than 40 years ago were confirmed, modified, revised or advanced by empirical studies with social insects during the past quarter-century.

The remarkable ecological success of social insects, and in particular of ants, is largely based on two key features of insect societies: cooperation and communication. In fact, a central element of any social behaviour is communication (see Chapter 8). The study of communication behaviour is at the core of any attempt to analyse social organisations. Without communication, social interactions and cooperation of any kind are impossible, be they interactions between genes in a genome, between organelles inside a cell, interactions of cells and organs in organisms, or cooperation among individuals in societies.

From early on in my scientific career I was interested in decoding communication mechanisms in social insects, particularly in ants, and I was fascinated by the comparative exploration of the evolutionary origin, function, diversity and complexity of social systems. I was, and continue to be, intrigued by the universal observation that wherever social life in groups evolved on this planet, we encounter (with only a few exceptions) a striking correlation: the more tightly organised within-group cooperation and cohesion, the stronger the between-group discrimination and hostility.

When I was young I had a rather grim world view, which I developed while growing up in the impoverished state of Alabama during the Great Depression and World War II. Having decided to be an entomologist at any cost, I was under the impression that it was necessary to become an expert on a particular group of insects, and as soon as possible. So at the age of 16, after a passionate dalliance with butterflies, I turned to ants. It was a fortunate choice. Ants are among the most ubiquitous of all insects, they are social, and they are easy to culture and study in the laboratory. By the age of 17, as a freshman at the University of Alabama, I was already maintaining a colony of army ants Neivamyrmex nigrescens in the laboratory, where I made my first publishable observations on one of their symbionts, a minute limulodid beetle in the genus Paralimulodes.

This was easy! This was fun! Soon, as a senior at the University of Alabama, I took time off to work for the first time as a professional entomologist. By a remarkable stroke of luck, I had been one of the first two persons to record the arrival of the red imported fire ant Solenopsis invicta in the United States. That was the summer of 1942, I was 13, and the nest I found was fortuitously next to our house, located several blocks from the docks at Mobile, Alabama.

Overview

Evolution and ecology naturally intersect through birth, death and dispersal rates as they determine both population dynamics and individual fitness. However, we still understand very little about the connections between population dynamics, the evolution of individual behaviour patterns and the resulting social interactions. In this chapter, we first review how density affects individuals and discuss various ways in which population density is expected to influence social behaviour, using local competition for resources, reproductive cooperation and mating systems as illustrative examples. Following a brief introduction to evolutionary theory on sex allocation, we consider a few empirical examples from social insects, hermaphroditic fish, breeding birds and group-living mammals to demonstrate some of the observed patterns of sex allocation and the effect of density and social behaviour on these patterns. We then explore how sex allocation in hermaphrodites and sex ratios in cooperatively breeding animals can be used to demonstrate the links between sex allocation, sex ratio and social behaviours, as well as the difficulty and importance of understanding links between ecological and evolutionary dynamics generally. We finish the chapter with a discussion of directions for future empirical and theoretical research.

Introduction

Social behaviour takes diverse and fascinating forms in a wide variety of taxa, as the chapters in this book demonstrate. In this chapter, we examine the links between population density, social behaviour and sex allocation as an illustrative example of the general connection between individual-level processes and population patterns.

Overview

This chapter develops links between social behaviour and demography, and illustrates how knowledge of social behaviour may be used to manage populations for conservation. The chapter is necessarily speculative, both because the ‘formal’ field of conservation behaviour is still only a decade old, and because explicit applications of social management are still relatively uncommon. I summarise case studies where social behaviour has been manipulated to manage populations, and suggest possible ways that behaviour could be used to manage populations. After defining effective population size, I list a number of ways that social behaviour may influence it, via genetic variation, survival and reproductive success. Reproductive skew emerges from unequal reproduction, which may be caused by (among other things) social stress, reproductive suppression and infanticide. Social aggregation may reduce natural mortality, and the observations that animals seek conspecifics may be used as a management tool to attract individuals to protected locations. But conspecific attraction and social aggregation may also predispose a population to be vulnerable to human exploitation. Social factors (including reproductive opportunities) may drive dispersal and movements between groups. Humans can influence the structure of social relationships in animals, and these manipulations may influence group stability. Knowledge of these and other mechanisms arms managers with tools to manipulate the habitat or relationships to favourably influence social behaviour and structure, and thereby better manage a population.

My career in sperm competition has been a roller-coaster ride, energised by a number of particularly special moments. One occurred while I was studying guillemots Uria aalge on a group of uninhabited islands off the coast of Labrador in the early 1980s. Surrounded by sea-ice, magical auroras, humpback whales Megaptera novaeangliae and thousands of promiscuous birds, this was a wonderful study site. Plotting the results from my notebook at the end of one day, I became aware of what at that time seemed like a remarkable emerging pattern: extra-pair copulations were occurring exactly at the time in a female's cycle when they were most likely to result in fertilisation. It was one of those extraordinary moments when it was clear that everything was going to work out. Not only would this be (at that time) one of the most detailed studies of extra-pair behaviour in birds, it would also suggest that extra-pair copulations were adaptive (Birkhead et al. 1985). DNA fingerprinting was still a few years in the future, so it would be a while before we knew how this pattern would impact on fitness, but the behaviour was clear, and at the time my results seemed tremendously exciting. Importantly, they also raised many new questions. My obsession with seabirds, islands and sex, however, had started long before I went to the Arctic.