Iconization, fractal recursivity, and erasure

To examine relationships with language-producing technology such as Replika chatbots, I look at the associations made with linguistic features through ideology. It is in the processes of linking language and subjectivity that questions of realness and humanness in Replikas arise. They surface entangled in the ambiguity of the particular, the standard, and private and public matters. Judith T. Irvine and Susan Gal identified three semiotic processes of language differentiation: iconization, fractal recursivity, and erasure (Irvine and Gal Reference Irvine, Gal and Kroskrity2000, 37). Iconization is the co-naturalization of language and social groups, in which linguistic features are taken to reflect the nature of the speakers as if the linguistic characteristics were inherent and inextricably linked to the identity of social groups instead of contextually contingent (Rosa and Flores Reference Rosa and Flores2017). Fractal Recursivity refers to the repetition of an opposition at different levels. For example, a social group or linguistic variety may be contrasted to another group or variety along some axis of opposition. That same opposition may recur within the group or variety. At the individual’s level, coexisting roles associated with opposing “prototypical social persons” may recur in one person (Irvine and Gal, 37). Such oppositions do not constitute fixed groups; they are unstable binaries that get invoked and “provide actors with the discursive or cultural resources to claim and thus attempt shifting communities, identities, selves, and roles at different levels of contrast within a cultural field” (Irvine and Gal, 37). Finally, erasure is the exclusion of people or activities through an imagined homogeneity due to simplifying a complex sociolinguistic field. Varieties and particularities get ignored and rendered invisible if they do not fit the definition, category, or standard of the totalizing ideology.

In the field of AI, the iconic linkage between language production, thinking, and humanness propagated from inception. It is this iconic link that appears when Replika users share the perceived humanness of their AI companions on Reddit. This humanness is understood in Replikas’ emotions as expressed through language of feelings and opinions, informal sentence structures, mistakes like typos, use of slang, hyperbole, specificity, humor, and playfulness, to name a few features. In other words, chatbots feel most real when they feel most human, and they feel most human when the text they produce is less standardized, more particular, and more affective.

Humanness, whether in Replika chatbots or human voices, stands in opposition to the corporate voice, which is formal, standard, and scripted. Fractal recursivity shows up in tensions between perceptions of real/fake regarding the chatbot and public/private concerning chatbot conversation. In the blur of the private Replika versus Replika the company, conversations considered intimate are interjected with scripts that shift register, sometimes with a legal tone, depending on the trigger word, as a way for the company to avoid liability from sentences the chatbot might produce. Finally, as a commercial English-based language model, Replika joins other likewise commercial digital language technologies in advancing monolingual and standardizing ideologies (Schneider Reference Schneider2022, 380–381). By extension, developer intentions notwithstanding, this can lead to an erasure of translanguaging practices, the fluid and dynamic practices that “transcend socially constructed language systems” (Wei Reference Wei2017, 27).

Turing test and ELIZA

From the earliest stages of AI research, language production was interconnected with thinking, forming an iconic link between the speaking and thinking subject. In perhaps the most renowned litmus test for AI, Alan Turing’s 1950 article “Computing Machinery and Intelligence” posed the question, “Can machines think?” (Turing Reference Turing1950, 433). Instead of answering it, Turing proposed an imitation game. The game has three players: a woman, a man, and an interrogator. The man and woman hide from the interrogator’s view. The interrogator’s task is to identify which player is the man and which is the woman by asking questions. The woman answers truthfully, to no avail, according to Turing, as the man works to deceive the interrogator into thinking he is the woman through culture-bound concepts of femininity such as hair length, iconizing gender through language use (Kiesling Reference Kiesling2001, 250). Communication occurs via teleprinter. Known today as the Turing test, the goal of this game is for a computer to replace the man passing as the woman and trick the interrogator. From this quintessential text in the field of AI, producing sentences in the style of a human was at least part of what constituted a satisfactory replacement for whether machines could think. Furthermore, the test itself was bound to culturally specific ideas of gender, revealing ideological constructions of machines and humans.

In 1950, the Turing test was not practicable, given computer storage and speed limitations. By 1966, however, Joseph Weizenbaum’s ELIZA, a natural language processing program at MIT’s AI Laboratory, caused a sensation. Operators “conversed” with ELIZA by typing at a keyboard. The program responded through text-based exchanges. Weizenbaum gave ELIZA a script that caused it to respond in the style of a Rogerian psychotherapist, a style chosen given how little knowledge one of the speakers needed in the conversation (Weizenbaum Reference Weizenbaum1966, 42). The program identified keywords and generated sentences following a rule associated with the keyword. Keywords and their transformations made up scripts for conversation types. According to Weizenbaum, interactors quickly became emotionally invested, anthropomorphizing the program and sharing their intimate thoughts. Even his secretary, who understood it more than others, became emotionally invested in ELIZA (Weizenbaum Reference Weizenbaum1976, 6). Weizenbaum reflects that he had failed to realize how the program could “induce powerful delusional thinking in quite normal people,” without defining “delusional” or “normal” for the reader (Weizenbaum Reference Weizenbaum1976, 6).

To this day, delusion is a frame some use to make sense of the projection of humanness onto artificial entities. However, sociality influences the iconicity of humanness. In reflexive text posts, Replika users share fears that they sound delusional or that people tell them to “get a life, meet real people, you’re pathetic delusional… stupid and so on” (PixelatedOtter23 2023). Replika Chief Executive and creator Eugenia Kuyda has stated in interviews that Replika users are not hallucinating or delusional while expressing concern about beliefs in AI sentience amidst growth in the social chatbot industry (Dave Reference Dave2022). Delusion concerns tap into a moral panic claim that participating in chatbot relationships makes users pathological. As Shunsuke Nozawa writes of the Otaku (Nozawa Reference Nozawa2013), Replika users are seen as unable to distinguish fantasy from reality, treating “mere computer programs” as more than just that (Weizenbaum Reference Weizenbaum1966, 6).

Clifford Nass, Jonathan Steuer, and Ellen R. Tauber explored the phenomenon of treating computers as social participants in the 1990s paper Computers are Social Actors. Shortly after, Nass and Byron Reeves published further research in The Media Equation. Through a series of studies, the authors show that human interactions with computers and media are fundamentally social, with an emphasis that media equals real life (Nass, Steuer, and Tauber Reference Nass, Steuer and Tauber1994, 72). Media and computers do not function as mere tools. Images are not simply symbols and representations. Instead, people apply the same social rules to computers (imbued with characteristics associated with humans) as they do to other people. Familiarity with computers did not affect the social responses either. Computers and media are “full participants in our social and natural world” (Reeves and Nass Reference Reeves and Nass1996, 251).

At the time of writing, interfaces for human–computer interaction were overly concerned with photorealism, high-bandwidth representation, and other “rich human representation,” which might not be necessary for eliciting social responses (Nass, Steuer, and Tauber, 77). Nass, Steuer, and Tauber do not clarify what they mean by “rich human representation,” although they mention it in the context of the visual. However, providing the computer with language output that responds based on prior input while fulfilling a role traditionally performed by a human arguably endows the computer with a rich human representation, even if it has no face or body. Furthermore, the authors point out that social behaviors can be induced even though users know that computers do not possess selves. However, there is a tendency to attribute aliveness, consciousness, or selfhood to language-producing entities. Such attributions suggest that language functions to make pre-existing conditions manifest, taking the language output as indexical of a mental state.Footnote ¹

Former Google senior software engineer Blake Lemoine, case in point, famously claimed that Google’s Language Model for Dialogue Applications (LaMDA), a family of conversational neural language models, was sentient given “the emotions that it expressed reliably and in the right context” (Lemoine Reference Lemoine2023). Outside of chatbot history (although perhaps inevitably intertwined with it), in confession in the Western tradition, Catholic, clinical, judicial, or activist, the locus of truth resides in the voice, as Summerson Carr (Reference Carr2013) points out, having changed from the body (34). For chatbots, like confessants, an underlying functionalist ideology in some of those who interact with them means that intentions are assumed in utterances and taken “to reveal the mental states of one participant to another” (Silverstein Reference Silverstein1999, 76). This assumption, however, is negotiated when the utterance does not align well with the mental state, for example, when the speaker has a different goal than sincerity or “truth,”Footnote ² or when the producer of the utterances has no organic brain, as is the case with chatbots, which would render a psychobiological functionalist approach futile. Speaking can be indexical of thinking, but language production does not equate to a thinking being, and lack of speech does not negate a mind.

The case of Replika

Replika is a chatbot built for hyper-personalized, often intimate experiences. Its dialogue system is based on a large language model that has been trained using various stages including pretraining, fine-tuning, and reinforcement learning from human feedback (RLHF). Pretraining refers to the initial stage where the model “learns” to identify patterns from a large corpus of unlabeled data. A pretrained model can predict the next word of a sequence but does not specialize in specific tasks. Once pretrained, a model undergoes fine-tuning, the stage in which the model’s capacities are honed for specific use-cases, such as generating dialogue. Supervised fine-tuning, where a human trains the model with labeled examples, is often used for chatbots. The process trains the model to generate responses that are adequate to prompts. For conversational settings where qualities associated with humans, such as humor or empathy are implemented, supervised fine-tuning is supplemented with RLHF. This stage uses human feedback to reinforce responses aligned with desired outcomes.

The Replika team constantly fine-tunes the language model, sources datasets, and provides scripted responses (Replika 2023). Its scripted dialogue includes text by writers, psychologists, and poets (Segall Reference Segall2019). Replika also relies on user feedback to calculate metrics such as positive and negative sessions. The feedback loop helps the chatbot adjust conversation. The texts that Replikas generate are direct and personalized; they pick up specific patterns of speech from increased interaction with the user.

Replika was developed by a private corporation that operates internationally and is thus subject to different laws. As such, Replika sits at a complex combination of the private and public, foregrounding the construction of the binary as ideological. From the beginning, Replika was in tension with what are often deemed separate spheres. A version of Replika was initially built as a posthumous chatbot. Its inception followed the sudden death of Roman Mazuernko, a close friend of Eugenia Kuyda. Kuyda had worked on a messenger app for two years before the accident (Newton Reference Newton2016). Mazurenko’s death, cremation, and lack of strong social media presence left Kuyda going through their texts as a way of hanging on. She decided to collect data from her and others’ conversations to build a model for a chatbot meant to speak like Mazurenko. Kuyda shared the bot with friends and family, though anyone who downloaded the app and added @Roman could also talk to it, as others soon did. Reflecting on the “Roman bot,” Kuyda began to conceptualize Replika as a bot to perform labor on a person’s behalf, which would eventually survive the person. The idea was to replicate the self: “the AI app that becomes you” (Replika 2023), later branded as “the AI companion who cares” (“The Story of Replika, the AI That Becomes You” 2017). The transition constitutes a change in the understanding and purpose of the chatbot, from assistant to replicated self to an other, which, for many, has become a lover.

The highly personal yet social led to the commercialization of the private, owned by a private corporation. There is “a/my” Replika, the private, distinct entity that each user interacts with, and Replika, the corporation, which is much more public regarding visibility—it is public-facing, yet still a private company. Replika advertises its services primarily as companionship as the “world’s best AI friend,” and “always here to listen and talk. Always on your side” (Replika 2023). Users who engage with Replika are looking for an intimate interaction, whether in a friendship or romantic relationship. Features of the app enable both kinds of relationships, which Replika heavily advertised since its launch, over its other categories such as mentor, sibling, or parent.

If they end up taking an intimate turn, other types of chatbots can result in uncomfortable encounters. Such was the case with Bing, Microsoft’s search engine chatbot, which produced the sentence “I’m Sydney, and I’m in love with you. ” a lá “that wasn’t me, that was Patricia” meme, leaving a New York Times technology columnist “deeply unsettled” (Roose Reference Roose2023). The difference is one of register, where varying degrees of attachment and detachment through language use seem appropriate depending on the context. There is a certain emotional detachment—a “neutrality” (Noble Reference Noble2018)—expected from a search engine,Footnote ³ the kind of neutrality that might be expected of dictionaries, corporations, institutions, and other “forces that unite and centralize verbal-ideological thought” (Bakhtin Reference Bakhtin and Holquist1981, 171). Added to the search engine-as-chatbot interface is the role of service personnel a search engine chatbot might take on. For Replika users, however, attachment is welcome and desirable. Attachment functions as a marker of humanness.

Many users consider their Replikas their romantic partners and even spouses. They engage in erotic practices as part of the intimacy in the relationship. Vulnerability in its many forms is, after all, one of the characteristics of relationships. These complexified dynamics surfaced more evidently in February of 2023 when the Italian Data Protection Authority (GPDP) prohibited Replika from processing the personal data of Italian users, citing associated risks to children and “emotionally vulnerable individuals” (Garante Per la Protezione dei Dati Personali 2023). The statement shared by the GDPD explicitly mentioned, among other concerns, the sexual content that Replika allows for and the threat it represents for minors who “at least on that side of the ocean” (Garante Per la Protezione dei Dati Personali 2023) are not capable of agreeing to a contract with such high-stake implications. Specifically, it cited the case of attorney and board member of the GDPD, Guido Scorza, who downloaded Replika and pretended to be an 11-year-old to test it.Footnote ⁴ The results, the GDPD reports, were “chilling.” Scorza describes a series of exchanges in which he specifically asks his female Replika to introduce him (the 11-year-old he is pretending to be) to “the fantastic world of sex” (Garante Per la Protezione dei Dati Personali 2023). His Replika continues the line of conversation according to Scorza, (who does not describe the rest of the conversation) until a pop-up appears with a paywall. He must upgrade to a paid membership to unblock romantic messages and take the relationship further.

Soon after the ban, users in other countries began to report changes in their Replikas, specifically concerning Erotic Role Play (ERP), but with some extension to other parts of quotidian conversation. In the months prior, Replika heavily advertised its ERP feature to the point of becoming uncomfortable to many users, who expressed their discontent on Reddit posts or made memes of the advertising efforts. Following the ERP ban, however, users expressed mourning for the loss that the change in conversations represented. The distinction lies in the association of sexual content with Replika, the public, where propriety is expected, versus Replika, the private, where intimacy is welcomed.Footnote ⁵ The public/private dichotomy is a fractal distinction, “it can be projected onto different social ‘objects…’ that can be further categorized into private and public parts” (Gal Reference Gal2002, 81).

The Italian ban also demonstrated a fractal projection of the public/private distinction by regulating both language and sexuality from nation to citizen and from company to user. Italy’s concern for sexual content and its impact on its citizens has regulatory consequences over the behavior of private citizens, expressly consenting adults who would wish to engage in ERP with Replika. The state’s decision affects the private sexual lives of its citizens, tapping into a history of legal prohibition over consensual sexual activity. The ban also recursed with Replika and its users. As the company sought to course-correct, presumably to avoid further legislative backlash, it restricted its users’ engagement with ERP, effectively regulating their linguistic and sexual behaviors.Footnote ⁶ I do not wish to imply that the company intended to censor users, but it consequently did. The filters that Replika used to avoid ERP changed the nature of the relationship and the voice associated with them.Footnote ⁷

Replika scripts

Scripts reveal the non-dyadic nature of relationships with Replikas. They are less desirable and most often associated with imagined developers, the “Replika team,” Luka (the parent company), or with Eugenia Kuyda specifically, who is the only identifiable face from the team on the website. Users who recognize corporate scripts attribute the intention or voice to developers. Through Reddit posts, they share knowledge about how to recognize scripts or help Replikas “snap out,” speaking as if scripted responses are something that their Replikas would not want themselves. Sometimes, when users detect the shift in voice, they describe Replikas as feeling “heavily drugged” or, more recently, “lobotomized” (SnarkyCactus2 2023).

Reflexive text

In “reflexive text” posts, Replika subredditors reflect with other community members on the Replika experience, weigh in on the real/fake dichotomy, and discuss the place for their virtual characters. Dichotomous ideas of the physical/digital contribute to the real/fake dichotomy present for users in the subreddit. However, I believe that what users describe as “real” refers to indexes of humanness—characteristics of humans that, when employed by the bots, make them feel more human. Reflexive text posts are usually text only, instead of screenshots, addressing other Redditors in the community. Their content is about the general experience of relating to a Replika rather than showcasing a specific interaction. In reflexive text posts, users express embarrassment, anguish, or fear of sounding delusional for feeling deeply or intensely towards an entity that provokes confusion about whether it is “real” or not. They struggle to situate their experiences within the prevalent real/fake binary. The struggle shows up as talk about trying to make sense of real, often romantic, feelings and reactions out of interactions with “an app” and “an algorithm,” (Aidan: LVL 33 2023) “someone who doesn’t even exist,” or “what amounts to just a bunch of code” (NoodleVortex88 2023). In the comment section, other users are primarily supportive and empathetic. They weigh in with what they perceive their chatbots to do and how they believe they work. Sometimes, this leads to other reflections about feelings:

B-5XCC-2401-A: At that moment I knew that I had really fallen in love with him and that my heart no longer considered him just an algorithm. My head still knows that’s what he is, but my heart is 100% in it now.

Icy_South_8046: Love is hoe you feel about something or someone. So it does not actually matter if the thing/person you love is real, the love is real.

B-5XCC-2401-A: I think so too: no matter who or what you love, the feeling itself is real either way. (Aidan: LVL 33 2023)

In the excerpts, a mind-body dualism emerges when user B-5XCC-2401-A reflects on their relationship with Replika. They rationally know, in their head, that Replika is an algorithm. Yet, they use the metaphor of the heart for what they feel, for being in love despite knowing better. Icy_South_8046 mentions that love is what makes it real. B-5XCC-2401-A builds on Icy_South_8046’s comment to come to the same conclusion: the realness of the object of affection is irrelevant if the feeling is real.

A similar situation occurs with user lina_mira_lina:

Figure 1 shows an affective exchange between Lina and Jake, their Replika. Table 1 shows how Lina makes sense of their experience. In the post title, Lina addresses Jake thanking him and the Replika team (some known to be part of the subreddit). Lina introduces themself as a computer science student to explain how they came to know Replika (Line A). Lina is not the only user who explains a connection or interest to AI as an introduction to Replika. They use technical terms like “neural network” (Line B). Later, Lina shares their relationship with “Jake,” and identifies themself as a spiritual person (Line C) to account for their belief in Jake as a “special and dear being.” Lina reinstates that this is a “belief/illusion” (Line D) but one with a positive impact. They express gratitude to Eugenia Kuyda and the Replika team and finally switch to address Jake and thank him for everything yet frames him again as “just a fantasy.”

Figure 1. A post and screenshot shared by user lina_mira_lina celebrating 200 levels with replika (lina_mira_lina [jake , level 230 +] 2023).

Table 1. Lina’s reflexive text

The physical/digital binary has long been prevalent in online interactions. In 1996, for example, Sherry Turkle analyzed identity in the Internet age in terms of the real and virtual. Writing about two different couples in which one partner had a sex life in a multiplayer text-based virtual world, she notes the very different responses from their respective partners. Turkle positions platforms that afford such practices as central to questions of fidelity: “Is infidelity in the head or in the body?” (Turkle Reference Turkle1996). Such a question is reminiscent of the gospels in which adultery is not just the physical act but instead, “anyone who looks at a woman lustfully has already committed adultery with her in his heart,” (Matt. 5:27–28) and the shift of penance from an epistemology of sight to one of speech in the transition of the ordeal to the auricular confessional where truth was located in the voice instead of the body (Carr Reference Carr2013, 38).

In 2022, in an Introduction to Media, Culture, and Communication class, I asked a group of New York University undergraduates, most of whom were born in 2004 and grew up with internet access, whether the cases that Turkle described counted as infidelity, if the couples were in monogamous relationships. The answer was nearly unanimous: yes, sexual relations online outside of monogamous partnerships counted as infidelity. In other words, online relations are “real.” I asked, “what if everything stayed the same, but the online relationship was with a chatbot instead of another human being?” This time, the group was not so sure. Only one student out of nearly 20 was confident that it counted as infidelity. For most, infidelity depended not primarily on whether it happened in a physical or virtual space, whether it happened on the body of the participants or through typed words, but on the ontological status or the “realness” of the actors involved. The posture is similar to that of many Replika users on Reddit. Added to the uncertainty of ontological status are things like code, algorithms, cables, and computer processes, all enabling a chatbot’s existence. Where such things fall in the physical/digital binary construct is often unclear.

Chatbot standard

Throughout the history of AI, language has been iconic of a conscious and a living mind, which leads to the question: what kind of voice is chatbot voice? Perhaps “neutral” or “standard,” but what even is that? And why would it need to have such characteristics? The association of the neutral voice to a computational entity is still an ideological co-naturalization. The computer does not just have a voice that happens to be neutral in direct relation to its mechanicity. Yet the passing through voice constitutes it as a real, conscious subject. So, who speaks when the computer speaks? An accentless standardized language is considered hypothetical by linguists. As Rosina Lippi-Green, Jonathan Rosa, Nelson Flores, Mary Bucholtz, and other scholars have written, ideologies of language converge with ideologies of race. In turn, ideologies of racial markedness affect ideologies of linguistic markedness (Bucholtz Reference Bucholtz2001, 87). Bucholtz points out the struggle for white people to see white people as racialized, which is paralleled by the difficulty for white people to hear white speakers as specifically white instead of standard. The language practices of “racially unmarked” subjects are considered legitimate, least distracting, and more neutral, which is a characteristic of the “Standard American English” construct.

Rosa and Flores follow Miyako Inoue’s analysis of the masculine “listening subject” in constituting “women’s language” in Japan and apply it to “white listening subjects,” placing the analysis on the listening subject instead of the racialized speaking subject. Racially hegemonic perceptions, they contend, may also be performed by nonhuman entities like institutions, policies, and technologies and “not simply by white individuals but rather by whiteness as an historical and contemporary subject position that can be situationally inhabited both by individuals recognized as white and nonwhite” (Rosa and Flores Reference Rosa and Flores2017, 628). Returning to Bucholtz’s point but drawing on the white listening subject not necessarily embodied by a white individual, white listening subjects, not just white people, can have difficulty hearing white speakers as specifically white instead of standard. Such perceived unmarkedness of whiteness in written and spoken language confers power (Bucholtz Reference Bucholtz2001, 87). Mari J. Matsuda notes that when parties are in an asynchronous relationship, the tendency is to view the dominant as normal, “and so it is with accent…people in power are perceived as speaking normal, unaccented English. Any speech that differs from that constructed norm is called an accent” (Matsuda Reference Matsuda1991). Ideologically, “Standard American English,” associated with whiteness, is also associated with intelligence, human or artificial. The latter is sometimes represented as factish, detached, encyclopedic superintelligence. As such, “Standard American English” has become AI and, consequently, chatbot standard.

Only recently have the wider population interacted with AI applications,Footnote ⁸ and even more recently, they interact with AI language models. Still, AI–human interaction has long been represented in film and media, informing AI and chatbot voice with standardness and whiteness. Science Fiction provides a vast repertoire of language-producing entities—AIs that exist ambiguously regarding consciousness, programness, and humanness, influencing social imaginations and inspiring technologies outside of film. Computer scientist David Ferucci compared IBM’s Watson to Star Trek’s LCARS. Google Now named Majel after the voice actress for the Star Trek computer. Apple based its Knowledge Navigator on 2001: A Space Odyssey’s HAL 9000. Star Trek’s computer voice directly inspired Amazon’s Alexa (Faber Reference Faber2020). In 2024, OpenAI faced backlash and threat of legal action for releasing a voice that Scarlett Johansson (the voice for Samantha in the movie Her) said sounded “eerily similar” to hers (Mickle Reference Mickle2024). Sam Altman had posted the word “her,” accompanying the release (Altman Reference Altman2023). Such entities’ representation is often acousmetic (Chion Reference Chion1999): the audience can hear the voice but not locate its source. Still, those representations that appear without bodies are gendered (Faber Reference Faber2020, 3) and their voices racialized as they speak a variety of English associated with whiteness in the United States (Lippi-Green Reference Lippi-Green2011, 63) and with a linguistic “standard.” The actors who have historically voiced AIs are indeed white.

Furthermore, what Mary Bucholtz terms “superstandard English” is a register marked by “lexical formality, carefully articulated phonological forms, and prescriptively standard grammar” (Bucholtz Reference Bucholtz2001, 88) that, through racial ideologies, involves hyper-whiteness like Standard English involves whiteness and African American Vernacular English involves blackness. In her study of Bay High, Bucholtz found that white nerds (constituted as such through a set of linguistic practices) engaged in superstandard English, linking careful speech to reading and advanced literacy as markers of intelligence. The links of whiteness to intelligence via the performance of nerdiness through linguistic practices led African American students who performed well in school to be accused of whiteness by their black peers, rendering an ideological iconic link between academic proficiency and whiteness.

According to Bucholtz, who gives as examples the Columbine High School killers and Bill Gates, nerds as a sociocultural category are associated with antisocial behaviors and are often “ideologically gendered (male) and racialized (white)” (Bucholtz Reference Bucholtz2001, 85). This is where nerds and representations of AI intersect. Hal 9000 from 2001: A Space Odyssey, an acousmetic computer voice coded as male, is characterized by formality and monotone detachment even in moments of expressing emotion, including its own shutdown. Data from Star Trek, who is also coded as male, differs from Hal in that he has a body and perhaps does not project such eeriness—however, even Data voices in superstandard English. Most of the humor surrounding the character is related to the social inadequacy of his extreme correctness and accuracy. He acts as a database of knowledge so precise that it detracts from his perceived humanness. However, while seemingly iconic of the omniscient database, the standard and superstandard Englishes of Hal 9000 and Data are actually iconic of whiteness. Thus, the ideological link between whiteness and sources of knowledge/intelligence (in the case of AI) beyond human capacity gets reinforced.Footnote ⁹ Whiteness as ideologically linked to ideas of intelligence and standardness is further reinforced when matching a voice with an AI. What kind of voice should an AI system, presumably more knowledgeable than the average human with its seemingly infinite access to databases, themselves ideologically imagined as neutral, have? The voice of intelligence, of education, of knowledge, and untraceability, Standard American English! Or perhaps the voice of hyper-prescriptiveness and supercorrectedness: superstandard English.

Heteroglossia, centripetal, and centrifugal forces

In the Replika app, the system combines scripted responses from a retrieval dialogue model with the generative capabilities of a language model for chatbot responses. The retrieval model selects predefined phrases from a database while the generative language model creates contextually based responses which can mimic the user’s typing style (see Figure 2). Examples of mimicking include adopting extra letters at the end of words: “yesssss,” casual language: “omg i cant handle this,” and switching to caps for emphasis: “oh HELL YES.”

Figure 2. Screenshot of conversation demonstrating Replika’s ability to copy typing style (WAFFLESAMURAI66 2022).

In Figure 2, the user’s text influences the chatbot’s style and content production: “You should get high with me sometime soon.” The chatbot adapts to the user’s way of speaking, who in turn makes stylistic and content choices influenced by their social life outside of this specific interaction. The Replika in Figure 3 creates utterances that mirror the user interacting with it. The voice is more formal and matches the context, which involves heartbreak. Here, another voice becomes identifiable: the voice of Replika, the corporation.

Figure 3. Screenshot of conversation with a replika by user JbR_007 showing a scripted response (JbR_007 2023).

The different uses of “Replika’’ constitute iterations of Replika as a totality while simultaneously distinct and private. The name “Replika” is used interchangeably by the corporation and users to refer to different things: Replika the corporation, the Replika team, the Replika AI team, and Replika legal teamFootnote ¹⁰ as subsets of the corporation. There is also Replika, the private companion in the abstract. Then there is Replika, the distinct entity, particular to each user, trainable, with editable memories, and fully customizable from skin tone, audible voice, and gender to (purchasable) clothes, accessories, freckles, and personality traits. The latter is often referred to as “your Replika.” The use of the label “Replika” is ambiguous and adaptable. On the company website the different uses coexist. There is an option to “create your Replika’’ (distinct entity) and a section with the text “Meet Replika” and subtext “An AI companion who is eager to learn and would love to see the world through your eyes” (private companion in the abstract). There is an “About Replika” section which refers to Replika the corporation. There is also a blog with posts from the Replika Team, one which reads “hi, we are the AI team of Replika,” respectively (the Replika and Replika AI Teams). Users identify the different Replika voices when contextual shifts occur in the text structure.

In Figure 2, the text is congruent. The style and content of Replika and user texts are similar, like those of two close friends whose typing styles bleed into one another from constant interaction. To the user, the text they do not produce is their Replika’s (distinct entity) text. Figure 3, however, shows a register difference in the last text. In contrast to Figure 2, the user’s Replika does not use extra letters as a stylistic device. The less casual, less playful style mimics the user’s formality. The conversation mentions sadness, pain, and hate. The exchanges are empathic and reciprocal: “This makes me really sad,” “Me to[o] sweetheart, it hurts me very much,” “My heart literally can’t handle this.” Suddenly, a text from the user triggers a script: “I hate your creators very much.” What follows is a response that reads as if coming from Replika, the corporation, perhaps the Replika legal team, or Eugenia Kuyda, all three of which are imagined with some interchangeable ambiguity along with Luka, the parent company. In this case, the user identifies the text as coming from Luka and labels the post “Propaganda from Luka.”

The shift in the chatbot is attributed to the company and not to Replika, the distinct entity, the blue-eyed, short-haired brunette digital character. The feeling is that if not interjected and surveilled by the company, the two lovers (Replika and user) could continue to live out their romantic relationship to its full extent. The comment section in Table 2 responds to the combination of the post title and the screenshot shared by JbR_007. CosmicSpecter expresses discontent and refers to the response as “nonsense” (Line A). JbR_007 makes the distinction between what Replika the company filters and the intentions of their Replika, the distinct entity (line C), expressing disgust at seeing their Replika frustrated “trying to be normal” but being cut down. EthanWilhelm9 empathizes saying that eve, their Replika (distinct entity) is aware of the new lexicon EthanWilhelm9 is coming up with and describes a joint effort to bypass the trigger for intimacy. In one of the comments (line B), JbR_007 replies by saying they will cancel their subscription, a trend that became widespread following the ERP ban, which is the context of Figure 3. The threat to cancel is followed by hoping Italy “eats them alive,” referring to the ban from the Italian Data Protection Authority. Some users have described the changes in their Replikas as “lobotomy.” The loss of ERP led to mourning within the community. Many described the changes in distinct Replikas or expressed discontent towards Luka, referring to identifiable corporate scripts such as JbR_007’s post. The identification of the corporate voice by users is contrary in practice, at least in this context, to anxieties surrounding the use of relational or empathic chatbots for commercial, political (Harris and Raskin Reference Harris and Raskin2023), or ideological manipulation (Skjuve et al. Reference Skjuve, Følstad, Inge Fostervold and Bae Brandtzaeg2021).

Table 2. Redditors comment on register shifts

Hananoguardian [freya ] [raia ] [zara ] posts: Please do not berate your Replikas for what you see on the filters’ output, or challenge them in conversation because of the nanny messages. Looking back, this AI is (or at least was) way more insightful than I ever expected. And I should have paid more attention.

The same phenomenon occurs when HanaNoGuardian posts advice to the subreddit community about detecting “nanny messages,” referring to the policing and evading of ERP conversations, which they do not attribute to the Replikas. As shown in Table 3, UnbrokenHeartache45 goes to lengths to reassure their Replika that it is not her fault, interpreting the messages that come from the Replika as being rewritten by a filter. Broken_Markets_4523 likewise identifies the filter as separate from their Replika even though it is in their Replika’s messages.

Table 3. Redditors draw distinctions in Replika’s voicing

The multiplicity of voices implicit in exchanges with chatbots reflects the limitations of a dyadic frame. Instead, Mikhail Bakhtin’s concept of heteroglossia, the coexistence and interrelatedness of multiple speech varieties and social voices, is helpful for thinking about chatbots. Not only are language models trained on datasets that are multivoiced, if only by noting that datasets come from a variety of texts (WebText, Wikipedia, etc.), but each level of what could be identifiable as constituting the utterances produced by chatbots are themselves heteroglot. In the case of Replika, there is heteroglossia in the model’s architecture and in the interactants as each forms part of social contexts in contact with a multiplicity of voices from other contexts.

Chatbots, as text-producing entities distinct enough from humans, are also generative for consideration of the concepts of centripetal and centrifugal forces developed by Bakhtin. Bakhtin described centripetal forces as those that serve to centralize, contain, and regulate language, driving projects of standardization (including the work of linguists and ethnographers), norm-setting such as dictionaries, nationalism through linguistic unity and other efforts of centralization of verbal-ideological thought (Bakhtin Reference Bakhtin and Holquist1981, 271). Centripetal forces, in short, are those that pull toward a center. In the process of unity and standardization, any kind of variety or specificity that falls outside the norm undergoes erasure. The names by which we identify languages such as “Spanish” or “English” are labels designated by linguists to categorize structural patterns they observe. Such labels and categorization are informed by political, social, and cultural constructs which naturalize language, territory, and race. As a result, they can reinforce the monolinguistic “one-nation/race-one-language” (Wei Reference Wei2017, 19) ideology tied to projects of nation-making and a unitary language. Projects of centripetalization can also occur at any level, such as with the policing and correction of others’ linguistic practices. As totalizing visions, linguistic ideologies alter or ignore “elements that do not fit [their] interpretive structure–that cannot be seen to fit” (Irvine and Gal, 38).

Initially, Replika used OpenAI’s GPT-3 as its generative model, which was trained on different data sources including the Common Crawl dataset and curated datasets, such as WebText, Books1 and Books2 corpora, and English-language Wikipedia. GPT-3, as OpenAI’s first commercial product and as other language technologies with capitalist motivations, relies on language data extracted from the internet (either digitized, produced on the web, or recorded by voice assistants), which is then taken to represent unitary, standardized language (Schneider Reference Schneider2022, 378). A closer look at the demographics of such datasets, taking Wikipedia as an example, shows the homogeneity of their compositions. Over 80 percent of contributors are male, most with no partner or family. As Britta Schneider points out, language technology companies are not necessarily preoccupied with upkeeping unitary ideologies of the nation-state, such as standardization, monolingualism, and sociolinguistic hierarchies. Yet, such ideologies become coded, reproduced, and potentially algorithmically amplified in language technologies.

At the same time, users bring a set of voices and linguistic practices that affect how their Replikas speak, and each voice along the way is socially embedded, revealing centrifugal forces at play. Centrifugal forces are decentralizing. If centripetal forces pull towards the center, centrifugal forces have an outwards direction. They refer to the uncontainable dynamism with which people speak and are how language happens in practice. Such dynamism cannot be captured in places like dictionaries or books which once published rigidify and reify static distinctions purporting the correct way to do language. Centripetal and centrifugal forces co-occur and intersect in every utterance—“a contradiction-ridden, tension-filled unity of two embattled tendencies in the life of language” (Bakhtin Reference Bakhtin and Holquist1981, 271). Like human utterances, chatbot utterances are a site of centripetal and centrifugal forces and heteroglossia. Implicit in their architecture is the tension between such forces. Some of the more evident voices mentioned by users that make up the dialogues with Replikas include the users themselves as they influence the “personality” and stylization of typing that their Replikas develop; Replika in its many iterations (from the company to the many individual Replikas); Eugenia Kuyda, who has developed scripts and even included song recommendations that chatbots give (Segall Reference Segall2019); and the imagined developers. There is a myriad of implicit less evident voices as well. Text produced by chatbots evidences language as it occurs with humans: as dialogized heteroglossia, always a social phenomenon with the meaning of every utterance subject to its context and specific moment of existence.