RANDEEP SINGH HOTHI: One question that emerges from our work concerns the relation between the study of signs and the theory of action. I’m noticing specifically the articulation of social semiotic analysis with phenomenology and co-operative action specifically, but also with further implications for psychology, cognitive science, and kinesiology amongst other fields and disciplines, as well. How might a study of improvisation demand that social semiotic analysis be supplemented?

SARAH HILLEWAERT: I’m trying to bring this phenomenological approach of embodied experience into conversation with semiotics, to think about how these fields speak to one another. When I look at the semiotic aspects of both climbing and social interaction, I’m asking how people anticipate qualia, how they anticipate what something will feel like, and how those anticipations inform what happens next, whether it’s grabbing a hold or extending a handshake. What’s interesting with respect to improvisation is that these anticipations can fail. In that moment of failure, there’s a need to improvise, not spontaneously in the sense of random action, but informed by embodied memory and a felt sense of what may work. What’s fascinating in both examples I work with is how time comes in. Time and risk really shape how improvisation unfolds. I brought in this question of time and risk to think through what they can tell us about meaning-making, or the emergence of meaning in action. And I was thinking about this tension, the contradiction that risk makes improvisation necessary, but that it also undermines its potential success. Because the pressure of risk means there’s no time to reflect, you just have to act. So, I used improvisation to think about what I ended up calling the edge of semiosis, that moment where meaning may emerge, but may as easily fail.

MATTHEW BRUCE INGRAM: I have found that Goodwin’s framework for co-operation helps me think through improvisation. An awesome paper in Signs and Society (Dondero Reference Dondero2024) considers this work in diagrammatic thinking. It makes me ask how new emergent knowledge comes to bear on early stages of rehearsal that I examine, where there’s shared or limited shared lexicon, or limited shared knowledge of how interactants think in the collaboration. And one of the things that helped me with studying situated improvisation was thinking about gestures, thinking about the multimodal package that happens in the creative process.

SARAH HILLEWAERT: But what is interesting is that this does not happen in a vacuum. A climber does not randomly start to climb. Yes, the meaning is derived when they touch the hold. But I talk about a kind of previewing we do, both in social interaction and in climbing. Climbers look at a climbing wall before they start, and anticipate how certain holds will feel, based only on visual cues and past experiences. So, they anticipate the qualities, the qualia, and what their body might do in response to the anticipated qualities of a hold and their interaction with it. This is where, for me, semiotics come in. It’s a semiotic landscape. I talk about improvisation when that anticipation fails. So yes, the moment of actual touch is when you may realize that what you anticipated is not there. So, the handshake I had anticipated, or the route I had anticipated to climb is not gonna work. In this moment, I need to improvise. So for me, it is about semiotics, because it’s about this anticipation of what a rock as a sign would have signified for the body, but in the moment of actual touch, phenomenologically, then, there needs to be an improvisation that is built on embodied memory.

JÜRGEN STREECK: I think a distinction would clarify things a little bit. When we talk about time and temporality in climbing, we focus on the moment of the grip. Looking at Jackson Pollock’s drippings and pourings, it would be the moment when the paint comes out of the bucket and splashes onto the canvas. In freestyle, when the rapper prospectively finds the word that rhymes with the one that he’s or she’s currently uttering. But at this “micro-level” we cannot identify the other process that, in my view, is so interesting about improvisation, which, as I’ve said before, is the emergence and stabilization of new forms. This process takes place at least over the duration of the interaction, and possibly beyond.

NORMA MENDOZA-DENTON: Well, for me, part of it is that a semiotic system is symbolic, indexical, what have you, but it relies on categories. And the categories are the stuff that I was dealing with in my paper. I’m still continuing to think about places where you’re not sure what category can apply for instance. I wound up looking at response cries in video games. And response cries are that language-like thing that is also not language. It turns out that in different languages, response cries can be more lexical, or less lexical. A response cry of surprise in Mexican Spanish would be, but that’s not lexical, whereas, in the literature, apparently response cries in Japanese are lexical. They’ll say something like, “oh no, terrible.” The F word is a lexical response cry. But, “ouch” is a sublexical, I think. Non-lexical. It depends how you say it. I think if you say ouchy, that’s lexical. But, but if you utter, “ouch,” that I don’t know, it’s hard to say. So is this going to be indexical, or is it not going to be indexical? When is a body movement going to engage a semiotic system such that another person would recognize it? And when is it going to be taken as incidental to being part of the body? And I think Jürgen has written some about this, too. In the whole, I think, many people in gesture studies are preoccupied with this. When is it gestural, when is it part of the trajectory?

JÜRGEN STREECK: A lot of people recognize that there is no cutoff point, or it’s not that people only understand gestures that they’ve seen before or that are already lexicalized, but new lexical forms appear all the time and are understood even if they haven’t been seen before because of the contextual circumstances. Primate researchers also observe more and more often how in groups of wild living chimpanzees, for example, new gestures are negotiated. I think they use the term negotiation to describe that process. The gestures become intelligible as they are being produced, and then, through their ties to their original context can project meaning onto new contexts.

NORMA MENDOZA-DENTON: Yeah. It’s the improvisational engine of interpretation.

JÜRGEN STREECK: Excellent. Right. Even though I would not look at climbing as a semiotic process, I like Sarah’s use of the term “lexicalization.” When a climber finds the right grip and a speaker the right hand-shape and motion to conceptualize, say, “having” or “seizing,” they do something very similar and, I would argue, connected in our cultural evolution. The vocabulary of gestures is derived from actions with things. In the wild, we can occasionally observe how gestures are abstracted from practical, instrumental actions and how they then take on a life of their own, now as “ready-mades” or lexical units that can be re-enacted as needed. For the most part, these lexicalizations are transitory, don’t travel beyond the borders of the situation. But those that so many people “use,” that is, keep re-enacting today, have probably originated in this fashion. Primate researchers such as Simone Pika have noticed remarkable differences in the gesture repertoire of different groups and describe the rise of new gestures as the result of “negotiations” (Prieur, Liebal, and Pika 2024). In other words, gestures are initially produced and understood within specific contexts and then get re-used, now “filled” with social meaning, and become a unit in the group’s gesture lexicon.

SARAH HILLEWAERT: So one of the comments that I got, and I appreciate Jürgen from your reflection upon what I’m trying to do, is: what is semiotic about this piece? From the beginning, I didn’t think about improvisation as spontaneity, as that of jazz musicians or breakdancers. I immediately started thinking about improvisation in everyday interactions. When do we not improvise? When do the rules come at play? And when I thought about that more, it’s often when things fail, when rules break down, when there’s a gap in what we anticipate is gonna happen, and how it actually plays out, then we have to improvise for whatever we’re doing to succeed.

KEITH SAWYER: Yes, the various features of process, where you don’t know where it’s going. And you have to act without intention, in a sense. So then, in the case of, I guess I would say, artistic improvisation, whether it’s jazz or improv theater, that there’s this sense of acting without intention, that you take action without knowing what it might mean. It doesn’t mean that you don’t know at all what you’re doing, or that you’re confused. It’s actually an ambition of improvisational actors to perform in a style of ambiguity, where you take actions that can be interpreted in multiple ways. When doing this, you learn through experience to trust that meaning will emerge from the process and the people that you’re interacting with. You trust that the process will later tell everyone what your action must have meant. And that’s what I mean by retrospective interpretation. The meaning of acts isn’t determined by the people who enact them, but it’s determined by the ensuing flow of the interaction. So yeah, that’s what I mean by retrospective interpretation. And I find that to be the case in all forms of creativity. Should I say all forms of social action, or all forms of intentional behavior? Maybe. I might say that all social life occurs in a space of ambiguity where meaning is collectively determined.

RANDEEP SINGH HOTHI: The implication here might be that improvisation is inherent in action, because abduction is constitutive of it whatsoever. One is always taking a lark or a jump into the unknown in every action, since no rule can completely account for the specificity demanded in the course of performance itself. Relatedly, embodied know-how might be considered for its responsivity, creativity, and even decision-making, even its taking signs as signals in the Batesonian ([1955] 2000, 178) sense. This might challenge us to consider the locus of the interpretant, which cannot be equated with subjectivity but which an “interpretant-driven” way of doing semiotics tends to center nevertheless. I think that’s what we might be implying as well. And as we’re thinking about the ways that the body might make adjustments before the mind knows what it’s doing, which can then become socially recognizable and enregistered, lexicalized, and so forth, I am also reminded of Norma’s thinking with the new materialism and affect theory, which claims to consider pre-linguistic social life.

NORMA MENDOZA-DENTON: I think that Massumi, in Parables for the Virtual, says that “the skin is faster than the word” (Massumi [2002] Reference Massumi2021, 28), where the idea is that galvanic skin conductance and other autonomic processes register intensities before we can even begin to form language. For him, a lot of this is maybe sublinguistic, pre-linguistic. Intensity would be classified as affective, reactions that may not necessarily rise to the level of language, per se.

JÜRGEN STREECK: What struck me when Sarah talked about that process of finding a new grip, improvising after the first grip has failed, as a semiotic process, is that I can see how, or rather imagine that, this situation can be accommodated to a Peircean semiotic account. But I like to distinguish between sign-mediated activities and the sense-making that goes on underneath, or often prior to them. There is our experience in the world, which may or may not be sign-mediated. And then there is the display of that experience for the benefit of others. I don’t see finding a new grip as a semiotic event. It is possible, however, that act and display are the same. The car-mechanic whom I have studied often communicated to someone how something is done simply by doing it in their presence. But there is no other for the climber’s grip, just the rock. Narration and display may come later.

SARAH HILLEWAERT: So that’s what I wanted to tap into with lexicalization. This momentary action can fail or succeed, and I’ve been trying to think about what Randeep said. Yes, in the moment, that improvised response is shaped by risk, time, and embodied memory. I want to think more about the interpretive question there, in that specific moment of improvisation. Even though these momentary responses may feel immediate, like the body moves before the mind catches up, they still involve interpretation. They’re rapid, embodied readings of sensory cues and the broader social environment, where meaning emerges through action. These responses may not draw on fully stabilized signs, but they do function semiotically, as it involves recognition, interpretation, and response. And once such response succeeds, it can get taken up. If we’re thinking about climbing, if an improvised grip works, a climber may use it more often. Others who observed and practice the same move may take it up, and this move may become part of a lexicon. I don’t think it’s enregistered yet in that moment of uptake. By considering an improvised gesture that gets repeated and recognized, I wanted to get to that moment before enregisterment. Improvised meaning can stabilize, can sediment, and can lexicalize. And there, we see a much clearer interpretant about the significance of what that gesture did in that moment.

JÜRGEN STREECK: Well, I think that would be great if we could make that leap because this situated production of new forms, and repetition of new forms, in my case gestures, compares to, or relates to, or is the same process as what goes on in grammaticalization, where, of course, grammaticalization happens at the communal level. But ultimately individual speakers are involved. Whether there’s one coherent way about these different processes, or parallel processes, is what I wonder about: namely, whether the same logic governs what unfolds in the moment and what unfolds in the lexicon or grammar over time. Through repetition, a form gets established or entrenched. And it seems that, no matter what genre of improvisation we look at, we find this process, that, once a fitting grip or gesture has been come upon, it may get re-used. In aesthetic and competitive genres such as oral poetry, rap, and jazz, the normative focus is on the ingenuity of the new form. And the poetic methods that Jakobson ([1981]2010) has described—parallelism, repetition—cut across all of the genres and even includes self-aware gesturing, gesturing that recognizes what it has done and then does it again, in a new context, generating more sense.

RANDEEP SINGH HOTHI: Here, improvisational processes can become subject to what might be called retroaction, the social enregisterment of matters in a post hoc (re-)construal. This is actually a crucial and central part of collaborative social action for Keith, because doing action together can require cleanup work, dealing with loose ends, assigning them their proper place, and repairing any problems. And, Norma’s discussion of clams also picks up this theme of repair in that, in jazz, if you make a sour note, or “clam,” you might then need to somehow repeat it in such a way that it becomes part of a recurrent cluster of sounds.

MATTHEW BRUCE INGRAM: Yes, Keith, I appreciated re-reading your book Group Creativity (Sawyer Reference Sawyer2003), and thinking about that emergence part, thinking about the way that flow is created, because ultimately, in the dance material that I looked at, if you look at all the routines, they all come to one stabilized process at some stage. They go from very early on in the co-creation of a shared language to something having to do with their embodied knowledge, all the way to a stabilized, formalized rehearsal process where they no longer need to do marking, because marking is schematic and quick hand for these actions that they’re building and they’ve been testing out. So I’ve been thinking through that a little bit more, and thinking a little bit more about the semiotic approach. My collaborator Ian also highlights how these stabilized things look and come into being and what you can easily take for granted when you look at an interaction without something like that.

KEITH SAWYER: Yeah, I always find myself struggling with the conception of improvisation. It’s hard to define it because it’s always in flux between structure and freedom. In my writings I swing between focusing on structures and constraints, on the one hand, and then on the other hand, focusing on the uncertainty that leads to failures and emergence. And then always coming back again to the role of constraints and structures. In my case of studying improvisational theater, performers don’t know what they’re doing when they start the performance, but within a few seconds, a dramatic frame has emerged. Very quickly, you already know something about the roles of the performers, and something about their relationship, you might already know something about events that they’ve talked about as having occurred in the recent past. I think of all of those features of the dramatic frame as constraints, so they are structuring elements, even though these structures did not precede the performance. From their dialogue, interactional frame elements have emerged. The actions that take place throughout the performance are guided by those constraints that have emerged. That’s why I’m often saying that all improvisation takes place at the balance between structure and freedom. Practice-oriented theorists might be critical in saying there is no such thing as structure except as people demonstrably orient towards it in their talk, as the conversation analysts would say, so that structure is actually networks and webs of process. So I’m not sure I agree with that, I’m willing to acknowledge that there is something called structure that is autonomous from interpersonal interaction, but I understand where that’s coming from, because the essence of improvisation is process, ultimately.

MATTHEW BRUCE INGRAM: This question opens up a good deal of doors. I think one of the things that I struggle with when I look at the dancers and the arborists working together is that what improvisation helps them do is find out what they don’t know. They don’t know about what the truck can do, what they don’t know about an artistic way of thinking. So, I was thinking about this because my mind went to looking at the data, and thinking about it semiotically, and thinking about the gestures, and thinking about the movements. But to understand this, you have to take 20 steps back and think about the previous ethnographic work they’ve done. They engage in understanding what the workaday movements are. All of that is basically made a shareable object and knowledge in these situations, then it’s manipulated. And there’s a lot that can be taken for granted. So the reason I went to the Goodwin (2017) work is because it didn’t make sense for me to isolate one thing when they’re making sense of this whole landscape. It’s because there’s all these indexical entailments that are possible, and it’s possible because they understand where the truck’s been moving. They understand how the arborists have talked about what the truck can do. I guess for me, the core of what I learned from this is that the improvisation helps them problem find, and helps them understand what epistemic pieces are missing. And it’s not, I want to emphasize, it’s not a perfect fit either. It’s not like they learn exactly how to be an arborist, or the arborists learn exactly how to be a dancer. It’s that they make it accessible enough.

NORMA MENDOZA-DENTON: But, also thinking about Matthew’s work, the deictics and fields of action that we can impute to non-humans like your projection of what these kinds of machines are capable of also constrains your idea of what this could possibly index.

MATTHEW BRUCE INGRAM: And Keith mentioned this, because he talks about downward causation, but one thing I always struggled with is that on the one end, there is an artistic framework that is the highest level that constrains where this is gonna head and, of course, at the same time, there’s the competing aspect of the arborist, what they actually do in everyday life, that needs to somehow translate into that so they have to bridge those two things together. I don’t know if the artistic framework of the choreographers has what Keith would call downward causation on what’s possible and what things can be evoked. It’s collaborative dance making, for sure, but there is that tension between what the truck does and what’s artistic and aesthetic.

NORMA MENDOZA-DENTON: That says something about the norms, but not so much about the nature of the practice. So what I mean to say is that what constraints there are may be external to the nature of the improvisational activity that’s going on.

JÜRGEN STREECK: Well, I think part of this may simply be the constraints imposed by the institution, improv theater, or dance rehearsal, or dance performance. In this recording by Ornette Coleman (1961), entitled Free Jazz and recorded in 1960, two jazz quartets meet and start playing. There’s not even a theme, no premeditated rhythm, none of that. It’s an encounter created out of nothing. So there, how long this will take, who will take a solo, when, none of that is constrained. Except, of course eventually the record, I don’t know, is maybe 50 minutes long, so ultimately, there is a time constraint, but, other than that, I think the constraints are set moment by moment. One player plays, and somehow I have to connect with that. And of course, then the audience was definitely at the time of divided opinion on how to appreciate that, or whether to accept that as serious music.

NORMA MENDOZA-DENTON: Yeah. Yeah. I’m thinking of, Mondada’s (Reference Mondada2014) work with the surgeon and surgeon’s assistant, where you have a surgeon, who expects help from the assistant, and they are both looking into microscopes and working in this tiny, tiny space. When the surgeon needs help, they’ll wave the tiny little pliers a little bit, both to signal that there must be a motion now and to point to where. And this is all, of course, all routinized. But you have to think about the possibility that suddenly the assistant needs to take another job in another town, and the next person comes in. Because of the constraints of the task, the constraints of the space, and the constraints of the person who is under sedation, this needs to happen quickly, we can’t talk a lot. The constraints of the situation itself make it so that whoever steps into the surgeon or assistant role has to have some of these tiny movements. It’s what makes sense within the frame. Right?

JÜRGEN STREECK: Yeah, that’s definitely true, right. Yeah, an altogether different constraint framework. In a way, there’s very little that can be done, right, in the way of producing communicative science. Speaking of constraints, and improvisation, I think our time is out.