1.1 Sensory Experience: Blending Inner Lives with Outer Descriptions

Sticking a needle in your eye is usually not a bright idea. Sticking it repeatedly to various places along your eyeball’s curvature seems even less advisable. But leisure has a way of giving some people odd bouts of inspiration. Concluding his prodding and probing with a bodkin – a long sewing needle with a blunt tip – a success, Isaac Newton not only kept both of his eyes but left us with some helpful diagrams (Figure 1). His attempts were not in vain. Newton reported seeing colored circles, documenting how his perception of color changed with slight moving pressure on the eyeball (Gleick, Reference Gleick2003). We can learn two things from this episode. First, don’t stick a needle in your eye, if only to see what may happen. Someone else probably has done so already – and can give you instructions. Second, Newton’s attempt to bridge the gap between our phenomenal inner lives and their public description is part of a long tradition of self-experimentation to understand how the mind works and how our senses give us both objective knowledge and subjective experience of the world.

Figure 1

Newton’s eyeball self-experimentation.

By applying pressure with a bodkin to probe the area between his eyeball and eye socket bone, Newton induced the appearance of colored circles, including white, dark, and vivid hues. Newton observed that colors were most pronounced when he continued to move or apply pressure with the bodkin but began to fade when he remained still.

Source: Newton (2003).

This epistemological gap between objective knowledge and subjective experience has widened, not been bridged, in the twentieth century. In contemporary philosophy of mind, a dichotomy persists. One camp posits that the subjective essence of sensory experiences – “qualia” or “what it’s like” to perceive the redness of an apple or the aroma of bacon (Nagel, Reference Nagel1974; Chalmers, Reference Chalmers1995; Tye, Reference Verhagen, Wesson, Netoff, White and Wachowiak2025) – eludes complete explanation by physical mechanisms. These philosophers believe that the sensory qualia at stake constitute a mystical presence in mental experience that physicalist science simply cannot explain. For instance, the molecule cis-3-hexenol emits a scent of fresh-cut green grass. Nevertheless, chemical notation alone is insufficient in describing the distinct psychological impact one undergoes when inhaling deeply while walking through a recently mowed field. This represents the distinction between knowledge and experience, between knowing what it is and how it feels, and it marks an epistemological barrier dividing first-person descriptions (i.e., your purported immediate and private phenomenological access to sensations) from third-person explanations (e.g., neuroimaging data of your brain).

In direct opposition, another camp disputes the notion that psychological experiences cannot be elucidated through physical causes and mechanisms, where “physical” encompasses various elements: the microstructure of external stimuli, patterns of neural activity in the brain, interactions between the brain and body, such as the gut–brain axis, among others. These philosophers argue that the emphasis on the irreducible nature of sensory “qualia” is a misstep, viewing it not as an intractable philosophical problem but rather a conceptual confusion (Churchland, Reference Churchland1986, Churchland, Reference Churchland1995, Reference Churchland2013) or an introspective illusion (Dennett, Reference Dennett1988, Reference Dennett1991; Frankish, Reference Frankish2017). They have raised the point that empirical findings may challenge the applicability of longstanding philosophical notions that have traditionally guided our comprehension of the mind. The idea here is to jettison our venerable philosophical heritage and replace it with scientific modeling, positing that combining neuroscience with philosophical reasoning might just give us a clue about how brains make minds.

This Element investigates current scientific accounts of our senses and how these explanations intersect with philosophical inquiry. How exactly is science supposed to shed light on mental phenomena, such as the subjective quality of perceptual experience?

Consider cilantro, or coriander. Its aroma tends to divide civilized company over supper. You’ve got the lovers, those who think cilantro is the fairy dust of herbs with its fresh and green aroma, and then there are the haters who’d rather eat soap. Actually, they feel they are tasting soap. What is happening here? People who detest cilantro are mutants (no offense). It turns out that individuals who cannot stand cilantro aren’t just picky eaters; they are genetically predisposed to find it appalling, thanks to a genetic mutation near the OR6A2 olfactory receptor gene (Eriksson et al., Reference Eriksson, Shirley and Chuong B.2012). Such a slight genetic alteration affects the OR6A2 receptor response to aldehydes, the chemical culprits behind cilantro’s divisive aroma. Dispute over cilantro’s “true quality” thus misses the mark, revealing a confusion between the neurobiological underpinnings of perception and the subjective narratives we construct around their sensory effects. Picture this: a research funding competition where two applications stand out. One is from a neurogeneticist who intends to use CRISPR to knock out the genetic variant near OR6A2, shifting the populace’s perception of cilantro to uniformly perceive its aroma as fresh and green. Another neurogeneticist has the opposite ambition by planning to genetically engineer the masses to universally find cilantro’s aroma soapy and pungent. (To furnish our thought-experiment, a nod to Dennett (Reference Dennett1988), with realism, neither proposal receives funding.)

Cilantro offers two critical lessons. Firstly, sensory quality is relational, not relative. Relational indicates that different parts of a system interact with each other in a way that is both contextually dependent and causally significant. Relative means that we cannot find a generalizable cause and explanation for a phenomenon, which is rendered ineffable instead. Relational states that the sensory qualities we experience are not engraved in the physical structures of stimuli but are formed in causal relation to the constitution of the organism perceiving them. Secondly, the expression of sensory quality is tied to physiological factors. Physiological factors, such as genetics and receptor mechanisms, account for the variations in perception across different individuals and even the fluctuating responses of a single individual to the same stimulus (Barwich, Reference Barwich2019, Reference Barwich2020, Reference Barwich2022a). Such insights lead us to challenge a deeply ingrained yet seldom questioned assumption in both philosophical discourse and scientific modeling of perception: the reliance on dyadic stimulus–response (SR) models as the alpha and omega of understanding the senses. Orthodox SR-models assume a direct correlation between the physical characteristics of stimuli and the resulting perceptual experiences. This assumption suggests that specific physical attributes of stimuli, their microstructure, can be straightforwardly linked to a specific quality they elicit.

Focusing the discussion on the SR-model quickly leads to an impasse. Naive physicalists contend that sensory qualities can indeed be mapped onto microstructures (e.g., for olfaction, we find this view held by scientists; see Keller et al., Reference Keller, Gerkin and Guan2017, as well as philosophers, see Young, Reference Young2016, Reference Young, Shottenkirk, Curado and Gouveia2019). Without this assumption, the precise sensory targeting achieved in product development, specifically foods and beverages (think of the reproducible flavor profiles of potato chips), would be nothing short of miraculous. Qualia proponents, however, may offer a twofold rebuttal. Firstly, there may still be causally fundamental individual variations in how these products are qualitatively experienced and appraised (as in: causally more fundamental than physical stimulus structure); and secondly, even if we accept a direct causal relationship between specific stimuli and sensory experiences, this SR-model skirts around the “hard problem” of consciousness (Chalmers, Reference Chalmers1995): Why the combination of particular molecules (like methanethiol, which smells of rotten cabbage; methional, reminiscent of potato; and 2-ethyl-3,5-dimethylpyrazine, evoking toast aroma) conjures the mental sensation of eating potato chips (Rochelle et al., Reference Rochelle, Julie Prévost and Acree2017)?

However, this apparent standoff easily leads to conflating two distinct theoretical questions. The adequacy of SR-models as suggestive of perceptual mechanisms (Keller and Vosshall, Reference Andreas and Vosshall2016) is independent of whether conscious experience has irreducible qualitative properties. One could reject SR-models while holding a more sophisticated physicalist stance, for instance, arguing that perception is created from complex organism–environment relations that nonetheless admit full physical explanation. Conversely, one could accept reliable stimulus–response mappings while insisting these leave unexplained the subjective perspectivalism of experience. The physicalist appeal to regularities in causal connections between distal stimuli and particular sensory responses contrasts with the qualia advocate’s position, which does not necessarily dispute a causal mapping of distal stimuli to sensory effects but seeks an account of the subjective perspectivalism of these effects, meaning the experiential essence or “what it is like” to perceive potato chips, including the subtle difference in feel when tasting the same chip molecules in different settings, such as cinema versus airport lounge. Recognizing this orthogonality clarifies that the mechanistic question of how perception works, and the ontological question of what consciousness fundamentally is, operate on separate, albeit connected, theoretical planes, even when both invoke the same empirical phenomena.

An escape from this philosophical deadlock suggests itself, a third route that combines the best of both worlds: the hardcore physical underpinnings of sensory phenomena on the one hand and the undeniably personal flair of subjective experience on the other. This solution requires a paradigm shift, moving our attention from the pure physical characteristics of external stimuli (be they chemical compounds or electromagnetic wavelengths) to the internal, biological and causally selective processes of the organism or perceiver interacting with these external features. This approach reconciles a physical grounding of sensory effects with their subjective expression; it furthermore opens new avenues for analyzing the interaction between our biology and our environment in shaping sensory experience.

A key part of this biological approach is figuring out how organismic processes impact perception in an active way. In this fashion, this Element uses a process biology framework to look at the nature of sensory perception (Sattler, Reference Sattler1986; Dupré, Reference Dupré2021). Process biology emphasizes that organisms are not static entities with immutable characteristics; they are dynamic systems formed by continuous biological processes (e.g., metabolic, developmental, and interactive) that perpetually shape and reshape organismal capabilities through their interactions with environments. Instead of viewing perception as a straightforward input–output system between designated individuals and objects, this paradigm emphasizes the development, maintenance, and modification of perceptual capacities and skills through dynamic biological processes that evolve over time in connection to environmental settings.

Olfaction, with its genetic potpourri, lays bare the reason why we are not all smelling things in the same way (Barwich, Reference Barwich2022a). But smell isn’t the odd one out. The insights we just gleaned from olfactory receptor genetics extend to all senses, vision included. Human color vision appears more uniform in its overall qualitative experience due to its more homogeneous genetic foundation among individuals. Still, genetic variance likewise shapes human color perception. Take, for instance, the color red at a ~700nm wavelength; a universal hue, one could assume. However, a slight genetic modification in the L-cone gene transforms “red” into “ochre” in conditions such as protanomaly or protanopia. Upon closer investigation, it becomes evident that the sensory qualities we perceive are neither attributes of external objects nor purely mental phantoms. Rather, they arise within the dynamic interaction between the external world and our biological makeup. While the specifics of each sensory system may vary – whether we are sniffing, seeing, hearing, tasting, or touching, alongside many other sensory processes we engage with – the underlying biological processes hold the keys to a better understanding of sensory experience.

My emphasis on biological processes does not exclude other central factors. Developmental trajectories, cultural contexts, and linguistic frameworks all shape how our biological mechanisms operate, becoming inscribed in our neural circuitry as our embodied brains navigate their environments. These perspectives have direct practical implications for sensory studies as demonstrated by a seminal early twentieth-century discovery.

In 1907, Kikunae Ikeda, a chemist in Japan, identified umami as a distinct “fifth taste” while his mind was immersed in enjoying the flavor of miso soup, challenging the conventional quartet of salty, sweet, sour, and bitter (Ninomiya, Reference Ninomiya2015). Western scientists remained skeptical for decades until the 1980s discovery of glutamate receptors finally provided the biological evidence needed to accept umami as a legitimate taste category.

Umami means “delicious savory taste.” Curiously, it took a chemist, not a philosopher, to articulate the quality of deliciousness.

1.2 Shifting the Philosophical Paradigm: Beyond the WEIRD Armchair

Contemporary philosophical interpretations of mental phenomena, sensory experiences included, present a strikingly myopic vision. Most analytic philosophers, whose debates have dominated Anglo-American discourse (Dummett, Reference Dummett1993), judge theories about mind and perception primarily on logical consistency within particular semantic frameworks, not empirical adequacy (Williams and Barwich, Reference Barwich2025). But these frameworks are not intrinsic to perception itself; they are historical and cultural products that impact how we interpret sensory experience.

Analytic philosophy is markedly WEIRD, an acronym for “Western, Educated, Industrialized, Rich, and Democratic” societies (Henrich et al., Reference Joseph, Heine and Norenzayan2010). This geographic and linguistic focus has become a recognized source of research bias. About 80 percent of study participants in medical and psychological experiments have WEIRD backgrounds, shaping our “common sense” of what counts as typical or normal. Philosophy has inherited this bias wholesale, reflected in institutional hiring and publishing patterns (Katzav and Vaesen, Reference Katzav and Vaesen2017).

This restricted focus has fashioned dominant views on the nature of the mind. My aim here is not to abolish the Anglo-American analytical tradition, where philosophical issues are largely treated as propositional problems to be dissected through language analysis. I do, however, object to the convention that this methodology should set the gold standard for philosophical investigation into the human mind and senses. Using WEIRD perspectives as the universal yardstick for philosophical reasoning is like claiming Tetris as the only game in town.

We end up solving problems that exist only in our own theoretical frameworks rather than understanding the sensory roots of human experience. Our earlier cilantro example points toward a different approach. Instead of asking whether cilantro “really” tastes fresh or soapy, a question that assumes universal perceptual categories, we can investigate how genetic, developmental, and cultural factors interact to produce varying sensory experiences. This shift from universal claims to process-based understanding opens new avenues for comprehending both the biological foundations and cultural variations in human perception.

1.2.1 When “Universal” Illusions Aren’t Universal

It is pivotal to distinguish between contemporary views of the mind, with its peculiar analytic concepts of mental phenomena (i.e., notions located in philosophical debate), and the mind as a real phenomenon of investigation. Consider illusions, specifically optical illusions (Figure 2): Are they an inbuilt feature of neural wiring? Philosophers consider them fertile ground for probing the vexing “problem of perception” (Crane and French, Reference Crane and French2015; Macpherson and Batty, Reference Macpherson and Batty2016). The key question is how our “ordinary” or “everyday” sensory experiences stack up against the real world out there. Does our mind’s eye faithfully mirror the external world?

Figure 2

Examples of optical illusions.Footnote ¹

a

Checker-Shadow Illusion (where shadows in squares A and B appear different but are the same color).

b

Hering Illusion (two straight and parallel lines in front of a radial background appear bowed outwards);

c

Ebbinghaus illusion or Titchener circles (inner circles are the same size, but the right appears larger);

d

Kanizsa triangle (contours create the impression of an object);

e

Müller-Lyer illusion (arrows are of the same size but appear to be of different length);

f

Ambiguous images like the Duck-Rabbit (can be perceived as different objects).Footnote ²

Source: Images from Wikimedia Commons.

Different philosophical challenges are associated with cases of perceptual illusions, and these problems vary depending on how one feels about perceptual realism. Naive direct realists consider illusions as problematic, as they seemingly illustrate deep deficiencies in the mind’s ability to accurately represent reality through our senses (Campbell, Reference Campbell2002; Fish, 2010). Some illusory experiences involve misperceiving genuine properties, for example, when the Müller-Lyer lines appear unequal despite being identical in length; others, like the Kanisza triangle’s illusory contours, seem to present features with no corresponding physical counterpart. For naive direct realists, who maintain that perception typically provides immediate, accurate access to mind-independent objects and their properties, illusions thus threaten the foundational assumption that genuine empirical knowledge depends on perceptual fidelity defined as systematic correspondence between experience and objective reality. A possible strategy to address this difficulty is to assert that humans do not misperceive but rather develop incorrect beliefs from accurate sensory perception (Brewer, Reference Brewer2011).

Other, more sophisticated direct realists contend that illusions can bolster, as opposed to weaken, their stance (for ecological and embodied accounts: Gibson, Reference Gibson1979; Noë, Reference Noë2004; disjunctivist accounts in Haddock and Macpherson, Reference Haddock and Macpherson2008). This view reframes illusions not as failures of perception but as instructive cases of degraded or atypical sensory input. These instances help clarify the distinction between mere sensory stimulation and successful, direct engagement with environmental affordances. For example, here the Müller-Lyer illusion does not demonstrate inherent fallibility but instead reveals the contextual conditions necessary for accurate perception to operate correctly. Perception is distorted precisely when its normal, ecologically valid conditions are absent.

Indirect realists and modern representationalists contend that we do not directly experience external objects but rather mental representations or sensory content (Locke, Reference Locke and Nidditch1979 [1689]; Tye, Reference Verhagen, Wesson, Netoff, White and Wachowiak1995). From this perspective, supported by work in visual neuroscience (Gregory, 1997; Eagleman, Reference Eagleman2001), illusions do not come as a surprise; perception is always a constructed interpretation of sensory data, thus making misrepresentation an inherent possibility.

Conversely, anti-realists and constructivists argue that illusions dismantle the very idea of establishing a consistent correspondence between perception and a mind-independent world. For them, illusions are not exceptions to a rule of accurate perception but demonstrations of perception being a constructive process. Accordingly, perceptual “objects” are inferences, hypotheses, or logical constructs derived from sensory input, not mirrors of external features. Anti-realists are a diverse camp, including some phenomenalists (holding that physical objects are merely permanent possibilities of sensation: Berkeley, 1713; Mill, Reference Mill1865), and modern cognitive constructivists (Goodman, Reference Goodman1978; Von Glasersfeld, Reference Weiskrantz1995). In this light, a seemingly “veridical” perception is simply a more useful or stable construct than an illusory one, not a truer representation of reality as it is in itself. For example, the Kanizsa triangle, then, is not a flawed attempt to copy reality but evidence for the visual system’s capacity to impose coherent structure (a “triangle”) onto ambiguous sensory data. Ultimately, the illusory contours are considered real as experiences and are generated by the same processes that create all perception.

The question of whether illusions are innate or developmentally acquired is surprisingly minor in these core philosophical debates, which have prioritized abstract ontological and epistemological analysis over empirical developmental or cross-cultural psychology. If sensory illusions are learned rather than hardwired, the framework used to evaluate perceptual accuracy needs to be reconsidered. Could cultural differences in sensory experience affect our ability to claim objective knowledge? Or are we seriously implying that people from cultures that value optical illusions differently are epistemically handicapped? The idea that our sensory experiences may be as diverse as our cultural backgrounds, or more diverse, should make us rethink our assumptions.

As it turns out, (at least some) effects of optical illusions appear as acquired, not hardwired. In cross-cultural psychology, there is a long tradition that suggests that visual inference is molded by ecological and cultural elements (the “carpentered world hypothesis” in Segall et al., Reference Segall1966). A fascinating intersection of culture and environment with sensory perception came to light in a comparative study between British children and their counterparts from the Himba community in Namibia. Himba youth, who grew up with very limited exposure to Western cultural influences, did not readily perceive the Ebbinghaus illusion (Figure 2 c) as such. The Ebbinghaus illusion effect became apparent to them at a comparatively later developmental stage. Himba children began to recognize it around ages nine to ten, in comparison to UK children who recognized the Ebbinghaus stimulus as an illusion at ages seven to eight (Bremner et al., Reference Bremner, Doherty and Caparos2016). This finding is not an isolated occurrence. Such cross-cultural investigations highlight how our means of making sense of the world and our perceptions of it are noticeably shaped by factors such as age, society, and urban environment. We learn to make sense of two-dimensional signals on the retina based on rules about how our three-dimensional world works. Our environment dictates these regulations.

The “problem of illusion” may not be a single, monolithic problem but a varied set of phenomena whose very existence and force depend on the particular perceptual skills one acquires. Cross-cultural research raises sufficient doubts about the philosophical inclination to declare universal truths about the human psyche, mapping the contours and contents of our inner lives, drawing on biased and limited data.

1.3 Taking the Naturalistic Turn: Integrating Philosophy and Science

WEIRD philosophy shows how narrow cultural assumptions create blind spots in our understanding of perception. How, then, should we study the mind? More than seeing the tension between first- and third-person perspectives as a problem to solve, we can use it as the foundation for a more naturalistic approach to consciousness.

2.1 The Five Senses: A Flat Earth Theory of Mind

As snow settled over Königsberg, the philosopher sat down to his evening meal, a bowl of soup prepared by his faithful servant Lampe. The steam rising from the broth should have carried the rich aromas of simmered poultry, onions, and parsley. But to Immanuel Kant’s dismay, he tasted only a flavorless void. When he complained to Lampe about the ingredient-free soup, the servant protested that he had prepared it exactly as always. This culinary standoff, a figment of this Element’s author but with a true message, lays bare the flawed understanding of our sensory experience. Kant (Reference Kant2006[Reference Kant1798]), who dismissed smell as the most “ungrateful” and “disposable” sense, failed to recognize that our aromatic experience of food depends on the nose more than our tongue. To trivialize the role of the sense of smell is to forget its monumental impact on human history, however. The hunt for new and exotic flavors has fashioned the globalized world as we know it today (Freedman, Reference Freedman2007).

Kant was not alone in this oversight. From Étienne Bonnot de Condillac to Charles Darwin, venerated thinkers overlooked a remarkable detail about human olfaction: It is dual-natured (Rozin, Reference Rozin1982; Shepherd, Reference Shepherd2011; Smith, Reference Smith and Matthen2012, Reference Smith and Matthen2015, Reference Spence2020; Barwich, Reference Barwich2020; Wilson, Reference Wilson2021; Barwich and Smith, Reference Barwich and Smith2022). We engage in olfaction both upon inhaling and exhaling. Orthonasal olfaction acquaints us with ambient odors through inhalation, alerting us to the recent history of our immediate surroundings, such as prior restroom use. Exhalation, however, carries the subtleties of what we commonly call “taste.” And even Kant must have suffered from a cold during which his congested nose had silenced what sensory science calls retronasal olfaction, the hidden process by which aromatic compounds from food in our mouths travel up through the nasopharynx to reach the olfactory receptors in the nasal epithelium during exhalation (Figure 3).

Without retronasal olfaction, flavor experience vanishes. A simple sensory test, which you can conduct yourself, illustrates this phenomenon. Blindfolded subjects with pinched noses struggle to distinguish between grated apple and onion, as both register primarily as sweet-tinged crunchiness. Only when the nasal passage opens does flavor return in a sudden rush, proving that what we experience as taste is actually smell in phenomenological disguise. Once you unclamp the nose, your experience quickly differentiates the apple’s fruity esters from the onion’s sulfurous punch. Still, we feel flavors in the mouth, not the nose, and thus this phenomenon of “oral referral” (Spence, Reference Spence2016) creates a persistent and perfectly ordinary phenomenological illusion.

Figure 3

The dual olfactory pathway. During orthonasal olfaction, odor molecules enter the nasal cavity through the nostrils and reach the olfactory epithelium (OE). During retronasal olfaction, volatile molecules released from food in the mouth travel through the nasopharynx to the OE. In both cases, olfactory sensory neurons in the OE transmit signals to the brain, integrating smell and taste into flavor perception.

Source: Meissner-Bernard and Fleischmann, 2024.

The hidden physiological basis of flavor highlights deeper flaws in how we commonly categorize perceptual experience. Aristotle’s classification of five senses – vision, hearing, touch, taste, and smell – has persisted for millennia not because of its accuracy but because of its deceptively intuitive appeal and, historically, religious symbolism (Kaufmann, Reference Kaufmann1884). The Greek philosopher himself seemed uneasy with his system’s boundaries, noting how taste was invariably contingent on touch. Yet this framework became entrenched, fossilized in textbooks and cultural idioms, while other traditions developed more nuanced understandings. Like a flat map of the spherical Earth, the notion of the five senses presents a simplified model that crumbles under closer scrutiny.

Consider the Anlo-Ewe people of southeastern Ghana (Geurts, Reference Geurts2003), whose concept of seselelame (“feel-feel-at-flesh-inside”) encompasses balance, movement, and internal awareness of intuition and kinesthetic acumen in ways that resist translation into Aristotle’s classic categories. Where Western thought separates touch from proprioception, the Anlo-Ewe recognize their fundamental unity with their notion of seselelame. Modern neurology and cognitive science likewise confirm this insight. Our sense of body position and movement (proprioception) and our awareness of internal states (interoception) influence cognition as powerfully as sight or hearing (de Vignemont, Reference de Vignemont2023). But we find no place for these sensations in the classical five-sense model.

This realization forces us to confront a more fundamental question: What constitutes “a sense”? Brian Keeley (Reference Keeley2002, 2012) proposed five potential criteria: dedicated organs, distinct phenomenological qualities, specific types of physical stimuli, unique neural pathways, and particular evolutionary functions. Nonetheless, each of these criteria breaks down upon examination in the following sections. What emerges from our examination is neither a neat taxonomy nor a definitive count but a recognition of the fundamental nature of senses as integrated perceptual systems.

The metaphor of the “radar-net of mind” in the title of this section thus expresses a reconceptualization of how to think about human perception and sensory experience. Instead of five distinct channels, like radio stations, we should conceive of our senses as a web of sensors that work together like a radar network. In the same way that a radar system uses many observation points that work together and compare data to create a fuller picture, our minds combine sensory data from many systems at the same time.

2.2 Sensory Organs: The Promise and Peril of Carving Nature at Its Joints

At first glance, classifying senses by their associated organs appears eminently reasonable. Eyes, intricate and diverse in anatomical structure across the animal kingdom, serve vision. Comparatively, in human audition, the ear’s cochlea converts sound waves into neural signals. This organ-centric view offers tangible, observable criteria for distinguishing the senses, avoiding the slippery slopes of subjective experience. Aristotle himself anchored his sensory taxonomy in anatomy, noting in De Anima how each sense has its own proper organ (Book II.12). For centuries, this approach provided a seemingly objective foundation for sensory classification.

Biology initially supports this framework. Consider chemical detection in mammals, where preliminary examination helps draw plausible boundaries between even closely related sensory capabilities. The detection of odors and pheromones, for instance, represents two different kinds of chemical sensing in many animals that, across various species, engage with the same physical stimulus (Keeley, Reference Keeley2002): airborne volatile compounds. Pheromones, broadly construed, are a special category of odors and, more specifically understood, are molecular signals that elicit species-specific behaviors in conspecific animals (Wyatt, Reference Young2014). However, the paths to recognizing odors and pheromones diverge, rooted in disparate organs and receptors. The vomeronasal organ (VNO) specializes in detecting pheromones, while the olfactory epithelium (OE) primarily handles odor detection. Specifically, the olfactory epithelium lining the nasal cavity contains specialized neurons expressing odorant receptors (ORs). At the same time, many species possess a separate vomeronasal organ (VNO) housing distinct receptor families (V1Rs and V2Rs) for pheromone detection (Dulac and Axel, Reference Dulac and Axel1995). Figure 4 illustrates their anatomical separation in rodents, with the olfactory epithelium positioned in the main nasal airway and the VNO located at the base of the nasal septum. This physiological segregation, combined with a different genetic basis between odor and VNO receptors, may suggest a clean enough biological division between smell and pheromone detection. But this distinction soon appears less categorical upon closer examination, as the accessory (pheromone-processing) and primary olfactory systems seem to converge downstream (Barwich, Reference Barwich2020).

Figure 4

Olfactory and Pheromone Pathways. Distinct anatomical positioning between the Main Olfactory Epithelium (MOE) and the Vomeronasal Organ (VNO) in rodents. In addition, there are significant differences in the genes that express odor and pheromone receptors.

Source: Chespasos, Wikimedia Commons, 2023.

The significance of sensory organs as a taxonomic criterion becomes more pronounced when we examine causal details such as signal transduction mechanisms. Olfactory receptors couple to G-proteins that activate adenylate cyclase, while vomeronasal receptors primarily utilize phospholipase C pathways (Zufall and Leinders-Zufall, Reference Zufall and Leinders-Zufall2007). These mechanistic differences mirror the anatomical distinctions, creating a satisfying congruence between structure and function that is crucial to our understanding of sensory systems and taxonomy.

This seemingly solid foundation begins to crumble when we take a closer look, though. Human anatomy presents the first anomaly. While we possess the genes for vomeronasal receptors and show fetal development of a VNO-like structure, most adults retain only nonfunctional remnants (Meredith, Reference Meredith2001). Nevertheless, we still detect putative human pheromones like androstadienone through our standard olfactory epithelium (Savic et al., Reference Savic, Berglund, Gulyas and Roland2001). This biological bait-and-switch (where chemical sensing appears to persist despite vestigial organ loss) contests strict organ-based definitions in a way that is both surprising and intriguing.

Touch delivers a greater and more fundamental challenge. What precisely constitutes the organ of touch? The sense of touch has no single anatomical address; it involves multiple types of sensory receptors rather than a single organ, making it challenging to categorize anatomically (Gallace and Spence, Reference Gallace and Spence2014; Fulkerson, Reference Fulkerson2020). The skin contains at least six receptor types: Meissner’s corpuscles, Merkel cells, Pacinian corpuscles, Ruffini endings, separate nociceptors, and thermoreceptors (Abraira and Ginty, Reference Abraira and Ginty2013). These receptors merely form the frontline of a markedly distributed system. Consider how we judge an object’s weight, where cutaneous receptors sense pressure, muscle spindles gauge tension, and Golgi tendon organs monitor force, while proprioceptive nerves track limb position (Proske and Gandevia, Reference Uwe and Gandevia2012). Thus, experiences like “touching a velvet cushion” incorporate far more than cutaneous signals. Proprioceptive nerves in muscles and joints report finger position, thermoreceptors gauge material warmth, and even the auditory cortex processes the faint whisper of fingers moving across fabric (Guest et al., Reference Guest, Catmur, Lloyd and Spence2002). The experience of touching emerges from this collective orchestra, not any single anatomical section.

Philosophers have long noted this tension. Though Aristotle located touch in the flesh, he also acknowledged its perceptual diversity, writing that “the sense of touch is really a group of senses” (De Anima, II.422b). Maurice Merleau-Ponty (Reference Merleau-Ponty1945) later argued that tactile perception inherently incorporates movement and spatial awareness as qualities extending beyond skin boundaries. He captured this vividly when he described how a blind man’s cane becomes an extension of his tactile perception – not through the skin but through the dynamic interplay of pressure, vibration, and spatial awareness. Neurological research further strengthens this insight (Winter et al., Reference Wurtz and Goldberg2022). Stroke patients with damage to the somatosensory cortex demonstrate an impressive ability to identify touched objects through their intact proprioceptive pathways. Notably, procedures involving active movement of the patient were most effective in enhancing sensorimotor performance. For instance, a patient documented by Paillard and Stelmach (Reference Paillard, Stelmach and Paillard1983) was able to recognize the shape of objects (such as a sphere or cube) that were placed in her hand by actively manipulating them, even though she did not have any conscious cutaneous feeling.

This complex and promiscuous signal processing characterizes touch as inherently multimodal, providing multiple types of information, including object shape, texture, and temperature, as well as body position (proprioception) and movement (kinesthesia). What we call touch, therefore, involves more than skin receptors alone, combining passive skin sensations with active exploration that uses proprioceptive and other systems. Additionally, sensations like pain and itching, though processed through the skin, are not typically considered part of touch.Footnote ³ Where ought we draw the line, if at all?

Even vision, our popular paradigm, defies neat categorization. While the retina serves as the anatomical seat of photoreception, nonretinal inputs crucially shape visual experience. For example, the pupillary light reflex depends on the pretectal nuclei. This means the signals that cause the pupils to constrict in response to light are processed by fibers that split off from the main visual pathway. Furthermore, circadian photoentrainment is mediated by a specialized pathway in the retina that includes melanopsin-containing ganglion cells. These cells deliver environmental light signals straight to the suprachiasmatic nucleus, synchronizing the body’s internal rhythms with the external ~24-hour cycle. Similarly, balance and spatial orientation depend on motion perception and vestibular coordination. The inner ear’s vestibular system monitors head acceleration and movement, conveying vital information about body motion in space, and this information is smoothly integrated with visual motion signals to sustain self-motion perception and coordinate reflexes that stabilize the eyes, head, and posture. These integrated systems all contribute to “seeing,” even though they originate outside classical visual anatomy (Hattar et al., Reference Samer, Liao, Takao, Berson and Yau2002).

The deeper we probe, the clearer it becomes that sensory organs are convenient anatomical landmarks, not valid boundaries for perceptual categories.

2.3 Sensory Modalities: When Intuition Deceives Philosophers

The phenomenological approach to defining senses, classifying them by their distinctive qualitative experience, initially appears self-evident. The velvety purple of a ripe fig looks nothing like its honeyed sweetness tastes; the scent of freshly turned soil shares no obvious qualities with the sound of rain falling upon it. These intuitive distinctions form the bedrock of our philosophical intuitions about the senses. Historically, philosophers like John Locke considered sensory modalities “simple ideas” that resist further decomposition (An Essay Concerning Human Understanding, 1689). At this phenomenological level, the five-senses schema seems unassailable.

Again, this clarity dissolves upon closer inspection. In addition to “oral referral” (recall Kant’s flavorless soup), multisensory integration reaches its zenith with “mouthfeel,” that sensory feel where a wine’s tannins seem both tactile (drying the gums) and gustatory (bitter), or where carbonation prickles simultaneously as touch and taste. Gastronomy exploits these overlaps deliberately, for instance, by serving oysters with granite stones to accentuate brininess through texture, or pairing espresso with dark chocolate to amplify bitterness through congruent aromas (Spence and Piqueras-Fiszman, Reference Spence and Piqueras-Fiszman2014; Mouritsen and Styrbæk, Reference Mouritsen and Styrbæk2017).

Philosophers remain divided on how to classify such phenomenological “illusions” and hybrid experiences. For example, Richardson (Reference Richardson2013) argues phenomenology should trump mechanism, and since we experience flavor in the mouth, it belongs to taste. Barwich and Smith (Reference Barwich and Smith2022) counter that this conflates localization with ontology, akin to claiming amputees still have limbs because they feel sensations there. In effect, such debate crystallizes a deeper tension: Should sensory categories reflect how perception feels or how it works? (Siegel, 2010).

This debate extends beyond flavor to other sensory modalities where phenomenology diverges from physiology. The thermal grill illusion provides a compelling example from touch: When alternating warm and cool bars contact the skin simultaneously, people experience a burning sensation despite neither temperature being painful individually. This occurs through neural disinhibition, that is, the cool sensation blocks nerves that normally suppress pain signals from warmth, causing the brain to misinterpret the warm input as dangerous heat. Rather than reflecting actual tissue damage, this burning sensation demonstrates how pain is constituted by neural processing patterns rather than unmediated stimulus expression. Like the flavor illusions discussed earlier, this example reveals how our subjective experience can systematically mislead us about the underlying mechanisms of perception (Craig and Bushnell, Reference Craig and Bushnell1994).

Invariably, the phenomenological approach becomes most fragile in synesthesia, where modalities blend irrevocably (Simner, Reference Simner2019). Notoriously, some individuals taste words or see sounds. Rather than dismissing this as pathology, we might view it as exposing the artificiality of our sensory categories.

Cross-cultural insights further help us grasp this point. The Himba people of Namibia describe colors primarily by texture and luminosity, not hue (Lutz, Reference Lutz2013). This phenomenological framework may baffle Pantone users, but it reflects the Himba’s environmental demands. This cultural variation finds theoretical grounding in Mazviita Chirimuuta’s (Reference Chirimuuta2017) adverbial account of color perception, which argues we do not passively receive colors and instead actively “color for” functional purposes rather than “coloring in” objective properties. Just as the Himba experience color through its ecological utility (e.g., distinguishing ripe fruit), some philosophers suggest all color perception is an action-oriented process (Thompson, Reference Turnbull1995). This view further dissolves traditional boundaries between “visual” and “tactile” qualities (Merleau-Ponty, Reference Merleau-Ponty1945), so much so that when a Himba describes red as “rough,” they are expressing the functional meaning of color instead of committing a category mistake (which some analytic philosophers might be tempted to object).

Ultimately, raw experience proves an unreliable guide to sensory boundaries. Like a stage magician’s misdirection, phenomenology highlights certain features while obscuring underlying mechanisms (Dennett, Reference Dennett1991). This does not render subjective experience meaningless; phenomenological experience may remain our sine qua non of perception without luring us into treating subjective distinctions as ontological fixtures.

2.4 The Distal Stimulus: Where Physics Meets Biology

The electromagnetic radiation we call “light,” the pressure waves we term “sound,” and the volatile compounds we identify as “smells” all share a fundamental characteristic. Their physical properties exist independently of an observer. This objective physical reality of the distal stimulus forms the first link in the perceptual chain.

The stimulus-based approach to sensory classification initially offers compelling advantages by anchoring perception in objective physical reality. Vision detects 380–740 nm electromagnetic waves, hearing senses 20–20,000 Hz air vibrations, and olfaction responds to airborne molecules with several thousands of physicochemical features. Such a physics- and chemistry-based framework offers apparent precision, grounding sensory taxonomy in measurable environmental regularities rather than subjective experience.

The predictive value of this approach becomes evident when examining how stimulus physics shapes perceptual capabilities. Photons travel at light speed along linear paths, enabling the visual system to construct spatially precise “representations” through mechanisms like perspective invariance, that is, our ability to recognize a door’s rectangular form despite trapezoidal retinal projections (Palmer, Reference Palmer1999). Atmospheric wavelength filtering (Rayleigh scattering) creates depth cues by making distant objects, like mountains, appear bluish, demonstrating how the physical properties of light directly inform visual phenomenology (Chirimuuta, Reference Chirimuuta2017). These phenomena, which artists have exploited since the Renaissance, emerge directly from the physical properties of light, enlightening us about the role of physical properties in shaping sensory perception.

Olfaction reveals a diverging yet equally instructive relationship between stimulus and perception. Odorant molecules diffuse through turbulent air, following chaotic paths that preclude precise source localization. This physical constraint shapes the distinctive phenomenological character of olfaction. Rather than spatial mapping, olfaction specializes in temporal pattern analysis, tracking concentration gradients across seconds to minutes (Barwich, Reference Barwich2020; Batty, Reference Batty, Young and Keller2022). Like adaptation to light physics in vision, the qualitative nature and behavioral affordances of smell are causally connected to the fluid dynamics that govern chemical stimuli in their environment.

But the stimulus approach falters when faced with nature’s penchant for repurposing physical signals across sensory systems. Consider androstenone, a steroid molecule serving as a pheromone triggering mating postures in sows (Wyatt, Reference Young2014). Humans perceive it either as musky or urinous, depending on genetic variations in OR7D4 receptors (Keller et al., Reference Andreas, Zhuang, Chi, Vosshall and Matsunami2007). This perceptual dichotomy from a single molecule mirrors the cilantro paradox discussed earlier (Section 1) as, in both cases, stimulus properties fail to predict qualitative experience.

The electromagnetic spectrum reveals even deeper complexities (Figure 5), especially in cross-species comparisons. Humans partition this continuum into “colors” through three cone photoreceptors. Meanwhile, the mantis shrimp’s sixteen photoreceptor types suggest a different spectral organization (Thoen et al., Reference Thoen, How, Chiou and Marshall2014). Infrared radiation, invisible to our eyes, guides snakes to warm-blooded prey via specialized pit organs (Gracheva et al., Reference Gracheva, Ingolia and Kelly2010), while ultraviolet patterns on flowers direct pollinators like a secret visual language (Kevan et al., Reference Kevan, Chittka and Dyer2001). Where we experience only darkness, pit vipers “see” the infrared glow of a mouse’s body heat. The physics remains constant; the perceptual realities differ radically. The photons themselves do not change, only the biological interpreters. This shifts our comprehension of color from being a fixed attribute of light to an engaging physiological dialogue between the principles of physics and our perception.

Figure 5

Milton spectrum of radiation wavelengths, frequencies, and black body emission temperatures. Wavelengths relate to electromagnetic radiation from gamma rays to radio waves in the graphic. Example sources or events for each wavelength category demonstrate their natural and technological uses and impacts. The spectrum also indicates the blackbody temperatures at which distinct wavelengths peak, demonstrating the theoretical temperatures at which things emit radiation.

Source: NASA, Wikimedia Commons, 2003.

These examples illustrate the core weakness of the stimulus criterion. Physical stimuli act as signposts, not boundaries. Physical properties constrain perception, but they cannot principally determine sensory categories because (1) identical stimuli serve different functions across species, and (2) qualitatively distinct experiences can arise from physically similar, even identical, inputs within the same species and even the same individual. This necessitates moving beyond stimulus-based classification to consider how evolutionary history, neural architecture, and functional demands transform environmental signals into perceptual realities.

2.5 Sensory Pathways: The Brain’s Fractured Highways

Contemporary neuroscience often conceptualizes sensory systems through a modular framework, where discrete neural pathways correspond to distinct perceptual modalities. This paradigm traces its pedigree to nineteenth-century neuroanatomy, when researchers such as Broca and Wernicke first mapped functional specialization to cortical regions (Finger, Reference Finger2001). The model persists in modern textbooks, providing pedagogical clarity (Figure 6): Visual information flows from retina → lateral geniculate nucleus → primary visual cortex (V1); auditory signals progress from cochlea → medial geniculate nucleus → auditory cortex; olfactory data travels from epithelium → olfactory bulb → piriform cortex. While this schema provides pedagogical clarity, accumulating evidence reveals its limitations.

Figure 6

Sensory pathways. (a) Vision: Light information travels from the retina through the optic nerve and lateral geniculate nucleus (LGN) to the primary visual cortex (V1) (Source: Miquel Perello Nieto, Wikimedia Commons, 2015). (b) Audition: Sound waves enter the ear, are transduced by the cochlea, relayed through the brainstem and medial geniculate nucleus, and processed in the auditory cortex (Source: Zina Deretsky, National Science Foundation, Wikimedia Commons, 2006). (c) Olfaction: Odorant molecules bind to receptors in the olfactory epithelium, with signals transmitted via the olfactory bulb to the anterior and posterior piriform cortex

(Source: Barwich and Severino, 2023).

The existence of multiple parallel pathways most strikingly demonstrates the complexity of vision. Beyond the canonical geniculostriate pathway (Figure 6a), other subcortical routes play a crucial role in mediating visually guided behavior without conscious perception (Figure 7). On the one hand, the retinotectal pathway, projecting to the superior colliculus, is a prime example. It enables blindsight patients with V1 lesions to accurately localize targets despite no conscious visual awareness (Weiskrantz, Reference Weiskrantz1986). This phylogenetically ancient pathway, which persists across vertebrates, highlights the need for understanding of vision in evolutionary terms (Krauzlis et al., Reference Krauzlis, Moschovakis, Simpson, van Opstal and Sharkey2013). The retinopretectal path, on the other hand, mediates reflexive responses like pupil constriction, operating at latencies (90–100 ms) faster than conscious visual perception (~150 ms) (Wurtz and Goldberg, Reference Wurtz and Goldberg1989). These pathways share a critical feature. They bypass the thalamocortical loops traditionally associated with conscious perception. Their existence challenges the notion that sensory modalities, including functions involved in visual behavior, can be cleanly mapped to discrete anatomical routes.

Figure 7

Neural circuitry of the visual and oculomotor pathways. Anatomical connections between visual processing centers and oculomotor control nuclei. Visual information from the retina travels via the optic nerve to the lateral geniculate body and calcarine cortex, while parallel projections reach the superior colliculus to coordinate eye movements. The oculomotor nuclear complex, including the Edinger-Westphal nucleus and the nucleus of Perlia, regulates parasympathetic and motor outputs through the oculomotor nerve (cranial nerve III), controlling the sphincter pupillae, ciliary, and medial rectus muscles.

Source: Lawrence (1960), Wikimedia Commons.

This decentralization becomes more pronounced in the case of the “hidden senses,” our bodily awareness systems. Proprioception relies on a distributed network spanning dorsal root ganglia, cerebellum, and posterior parietal cortex (Proske and Gandevia, Reference Uwe and Gandevia2012). Unlike vision or audition, proprioception lacks a dedicated “primary” cortex, instead integrating with motor and tactile pathways. Interoception reveals greater divergence still (Figure 8). Signals from viscera travel via the vagus nerve to the solitary nucleus, then synapse in the insula without passing through traditional thalamic sensory relays (Craig, Reference Craig2002). These systems thus defy the classical sensory pathway model twice over, both by their anatomical diffuseness and their mandatory multisensory integration.

Figure 8

Interoceptive Systems. Overview of the mechanisms involved in the internal monitoring of bodily functions, featuring the cardiorespiratory, gastrointestinal, nociceptive, endocrine, and immune systems.

Source: Schapelle, Wikimedia Commons, 2017.

Another challenge to modularity is from olfaction. For example, the piriform cortex challenges modular thinking with its functional versatility. Neuroimaging findings show its activation during both gustatory and somatosensory stimulation (Gottfried, Reference Gottfried2010), indicating that, more than being just a dedicated smell processing center, it appears to be a dynamic interface for flavor construction. This multimodal integration resembles the distributed processing in proprioception. Additionally, olfaction rejects the topographic organization seemingly central to other distal senses. Unlike the retina’s and the basal membrane’s precise spatial mapping (retinotopy and tonotopy), the olfactory epithelium shows no “odotopic” or “chemotopic” organization, and there is furthermore no orderly arrangement of receptor types or odor-specific spatial representation in the piriform cortex (Barwich et al., Reference Barwich, Firestein and Dietrich2025). In effect, and akin to interoception, odor-specific signals emerge from population coding rather than dedicated channels or locations (Barwich and Severino, Reference Barwich and Severino2023).

These findings resonate with contemporary network theories of brain function. Analysis of connectome data suggest that canonical sensory pathways represent only the most statistically prominent connections among dense, overlapping networks (Sporns, Reference Stokes, Matthen and Biggs2011). Evolutionary developmental biology accounts for this apparent disarray. Sensory systems develop through exaptation of existing circuits instead of de novo engineering (Krubitzer and Prescott, Reference Krubitzer and Prescott2018). In a way, the resulting cellular architecture resembles a living city where ancient footpaths persist alongside modern highways, each serving complementary functions.

This perspective connects with well-known observations in sensory neuroscience. Consider congenitally blind individuals where the “visual” cortex is repurposed for tactile processing (Pascual-Leone et al., Reference Pascual-Leone, Amedi, Fregni and Merabet2005), highlighting how neural real estate follows functional demand rather than predetermined design. For example, the superior colliculus merges visual and auditory space maps (Stein and Stanford, Reference Stokes, Matthen and Biggs2008), the insula blends gustatory and visceral signals (Craig, Reference Craig2009), and the auditory cortex processes visual information in the absence of auditory input (Amedi et al., Reference Amedi, Merabet, Bermpohl and Pascual-Leone2007).

The lesson is clear. Clinical and experimental pathway analysis is useful. But it cannot be our main sense-individuation criterion. Brain anatomy reflects evolution rather than rationality. And, as we approach our last criterion, we must reconcile this biological reality with our need for systematic understanding.

2.6 Evolutionary History and Function: The Lure of Adaptationist Thinking

The evolutionary perspective provides a compelling framework for understanding sensory systems when applied with appropriate rigor. Following Theodosius Dobzhansky’s (Reference Dobzhansky1973) famous dictum that “[n]othing in biology makes sense except in the light of evolution,” Keeley (Reference Keeley2002) proposed function and phylogeny as essential criteria for individuating senses.

This view transforms sensory systems from passive conduits of information into dynamic interfaces shaped by ancestral environmental challenges (Gibson, Reference Gibson1966). The theoretical appeal is evident in how evolutionary thinking resolves ambiguities that stump anatomical or phenomenological approaches, offering principled distinctions based on documented selection pressures and phylogenetic trajectories.

However, this approach faces substantial methodological challenges that demand careful consideration. The specter of “just-so stories” looms large in sensory evolutionary studies, where plausible adaptive narratives often outpace empirical evidence (Gould, Reference Gould1978). Gould and Lewontin’s (Reference Gould and Lewontin1979) famous critique of pan-adaptationism remains particularly relevant here, cautioning against assuming perfect evolutionary optimization without concrete evidence of historical selective pressures.

Consider first the well-documented case of color-sweetness associations. Experimental evidence shows participants consistently rate more intensely colored beverages as sweeter, even when sugar content remains constant (Pangborn, Reference Pangborn1960). This might suggest an adaptation for identifying ripe fruits. But this interpretation glosses over observations that fruit coloration varies tremendously across ecosystems (Dominy et al., Reference Dominy, Lucas, Osorio and Yamashita2001), undermining claims of a universal evolutionary mechanism. The effect may instead reflect domain-general learning processes and need not be a result of dedicated sensory adaptation.

More tenuous evolutionary claims can be found in studies of cosmetic product design. Research indicates that people perceive shampoos scented with rose, jasmine, or fruity apple to produce softer hair (Churchill et al., Reference Churchill, Meyners, Griffiths and Bailey2009); but an explanation of this perception lacks any straightforward adaptive rationale. While imaginative stories could be crafted about floral associations or mate selection, these remain speculative without evidence of cross-cultural consistency or genetic bases. Indeed, such explanations may reveal more about the conceptual preferences of scientists than they do about demonstrated evolutionary histories (Alcock, Reference Alcock2021; Nemati, Reference Nemati2024).

These examples expose three pivotal challenges for evolutionary sensory taxonomy. First is the time–depth problem. Human sensory systems integrate adaptations from vastly different evolutionary periods, from ancient vertebrate chemoreception to recent primate trichromacy (Jacobs, Reference Jacobs2009). Disentangling these layers requires comparative genetic evidence that is often incomplete.

Second is the causality problem. Current utility cannot automatically be equated with original function (Gould and Vrba, Reference Gould and Vrba1982). A case in point is the evolutionary history of whale pelvic bones. These vestigial structures, freed from their ancestral role in terrestrial locomotion, were co-opted to anchor reproductive anatomy in marine environments (Zimmer, Reference Zufall and Leinders-Zufall2009). This shows how traits can gain new functions unrelated to their original adaptive purposes, making it challenging, if not misguided, to define senses by a single evolutionary function.

Third is the circularity problem. Defining senses by evolutionary function presupposes the very discrete functional categories we seek to establish (Heffner and Heffner, Reference Heffner, Heffner, Webster and Popper2020). This logical circle emerges because evolutionary analysis requires identifying coherent units of selection. However, determining what constitutes such a unit already commits us to particular sensory boundaries or functions. Consider echolocation in bats: classifying this as “modified hearing” versus “a distinct sense” fundamentally shapes subsequent evolutionary investigation. If we assume it represents auditory adaptation, we emphasize continuities with existing sound processing mechanisms. If we treat it as a novel sensory modality, we focus on its unique computational properties and neural architecture. The evolutionary explanation follows from our prior categorization rather than determining it. Elisabeth Lloyd’s (Reference Lloyd2015) Logic of Research Questions reveals how this problem extends beyond methodology to epistemology itself (Barwich and Lloyd, Reference Barwich and Lloyd2022). When scientists ask, “What is the evolutionary function of smell?” they have already presumed that “smell” constitutes a distinct evolutionary unit of functional explanation. This question-structure biases investigation toward finding unified adaptive stories rather than recognizing the potentially fragmented, multi-layered, or historically contingent nature of what we conventionally group under single sensory labels. Breaking this circle requires reformulating research questions to remain open about the proper units of evolutionary analysis, perhaps asking instead how different environmental challenges shaped information-processing capacities without presupposing which capacities belong together.

Contemporary evolutionary neuroscience addresses these challenges through multiple lines of evidence. Comparative studies across closely related species reveal conserved and divergent sensory adaptations (Nummela et al., Reference Nummela, Thewissen, Bajpai, Hussain and Kumar2013). Analysis of fossilized sensory structures provides direct evidence of historical morphologies (Benoit et al., Reference Benoit, Norton, Manger and Fernandez2020). Computational modeling tests the plausibility of proposed selective pressures (Higley and Heffner, Reference Higley, Heffner, Wenstrup and Zupancic2021). Together, these approaches help constrain speculative storytelling while preserving the heuristic and explanatory power of evolutionary frameworks in sensory studies.

Ultimately, evolutionary analysis functions most effectively not as an independent criterion but as a vital framework for assessing other classification approaches. This perspective grounds our understanding in well-documented phylogenetic patterns and selective pressures, limiting the imposition of artificial categories while honoring biological complexity. These insights harmonize with Carrie Figdor’s (Reference Figdor2022) account of phylogenetic cognition, which posits that psychological categories, including sensory modalities, emerge as contingent outcomes of evolutionary history rather than essentialist natural kinds. This approach proves especially illuminating when examining the remarkable diversity of sensory systems across species (Yong, Reference Young2022). In the end, an evolutionary perspective proves indispensable as we confront the diversity of sensory systems across the animal kingdom, from echolocation to electroreception, which challenge human-centered taxonomic models.

2.7 Beyond the Flat Earth Theory of Mind

The quest for definitive “psychological primitives” or discrete “physiological modules” implicit in many sensory classifications appears increasingly quixotic. This pursuit falters less from lack of scientific evidence but from a fundamental mismatch between our categorical frameworks and biological reality. The interconnected nature of biological systems, the dynamic adaptability of neural circuits, and the context-dependent flexibility of perception collectively resist rigid taxonomies. Nature operates without regard for the clean distinctions philosophers and scientists seek to impose (Dupré, Reference Dupré1993). Accordingly, our inability to establish a universal sensory framework stems not from inadequate data but from unexamined presuppositions. Even with perfect neural maps and complete cellular understanding, we would still face the conceptual challenge of determining where one sense ends and another begins. This is ultimately a theoretical problem inviting philosophical scrutiny.

The conventional mapping of sensory modalities to specialized cortical areas exemplifies this very conceptual trap. While textbooks neatly assign vision to occipital lobes and hearing to temporal lobes, emerging research reveals a more nuanced reality. Consider three critical observations that undermine this modular view.

First, neuroplasticity demonstrates the brain’s remarkable functional flexibility. In blind individuals, the “visual” cortex repurposes itself for auditory and tactile processing, enhancing spatial navigation and Braille reading (Pascual-Leone et al., Reference Pascual-Leone, Amedi, Fregni and Merabet2005). This plasticity suggests that cortical regions process information based on current needs rather than predetermined sensory labels.

Second, cross-modal interactions pervade normal perception. The auditory cortex contributes to visual motion detection (Alais et al., Reference Alais, Burr and Burr2010), while the somatosensory cortex is activated during speech comprehension (Ito et al., Reference Takayuki, Tiede and Ostry2009). Such findings counter the notion of strictly segregated sensory streams.

Third, evolutionary developmental biology shows that neural circuits often serve multiple functions across species, highlighting the role of evolution in sensory processing. Both electrosensation in platypuses and infrared detection in snakes use the trigeminal nerve as their sensory pathway (although platypuses process the signals as touch while snakes integrate them with vision) (Schneider, Reference Schneider, Gracheva and Bagriantsev2016), indicating that sensory systems are evolutionary repurposing of existing circuits rather than dedicated modules, and invoking a sense of admiration for the complexity of sensory processing.

This evidence encourages radical rethinking and questioning of paradigms, models, and taxonomies. We could better interpret perception as a dynamic set of competencies influenced by evolutionary pressures and developmental experiences rather than striving for definitive sensory bounds. From this perspective, the visual cortex is not “for” vision per se but provides spatial processing capacities that can serve multiple sensory inputs. The auditory cortex may specialize in temporal sequence analysis. This perspective helps illustrate why sensory classification is contentious. Perceptual functions can arise from distinct neural substrates across species, much like mathematical equations might have different notations. We must renounce stale essentialist sensory categories and create new frameworks that build on how biology really works, through diversification, ecological dependence, and cross-cultural or cross-niche sensory affordances and experiences.

3.1 Recalibrating Experiential Reality

Some years prior, I chanced upon an adventure. Andreas Mershin, comrade in arms in the academic trenches, and Michael Skuhersky, a doctoral candidate at MIT, were toying with Virtual Reality (VR) ostensibly for psychological research, though really for the thrill of it. In one of their simulations, they had designed what they called a Temporal Stereoscope – a device that fed different temporal versions of the same visual information to each eye. One eye received a delayed replay of events, while the other saw them unfold in real time. The premise was elegant in its simplicity and, as Andreas told me, could be replicated with startling ease:

Andreas dashed towards me with the urgency of a man on a mission, VR goggles in hand. Sliding them on transported me into a disorienting world where familiar objects seemed to refract through time. The things I saw were so peculiar that I later berated myself for not jotting down notes, but it was the ephemeral aftereffect that would prove most significant. After only a brief stint with these goggles, my vision hesitated to return to ordinary sight right away. Something had shifted in my perceptual apparatus, a lingering souvenir from my glance through this temporal looking glass.

Skuhersky had noted similar effects across goggle testers. Some experienced a fusion of their visual fields, their brains somehow alchemically blending the dual temporal inputs into a single, albeit perplexing perceptual object. Others observed two distinct visual streams running in parallel, each maintaining its own temporal signature. But every participant reported the same aftereffect: a brief period where normal vision felt somehow altered, as if the brain’s temporal processing had been recalibrated.

My encounter with the Temporal Stereoscope became a watershed moment, altering my inquiry into sensory perception. It felt like my mind had tripped on the narrative laid out by my own neural circuitry. Here was tangible evidence of how easily our perceptual systems could be modified, and that the deceptively fixed boundaries of human sensation were much more malleable than we typically imagine.

Consider what this malleability might mean for human experience. Our sensorium offers us only a narrow window onto the vast expanse of the universe. We cannot detect the electric fields that surround us, nor can we feel the moon’s gravitational pull consciously as a physical sensation. Scientific technologies have been peeling back some of our observational curtains, but what if we integrated such technologies directly with our organic sensory systems? What if these tools no longer served as extensions of our agency but became perception itself? The world, deceptively familiar to human minds, would create itself anew.

This junction of technology and biology, where electrodes meet neurons, speaks to an evolution not merely of capability but of concept. The idea of modifying the human body is as ancient as the first sharpened stone, yet it has captured our collective imagination anew. Within this context, the cyborg experience of sensory augmentation emerges as a lens through which to examine what it means to be human in an age of technological integration.

Andy Clark (Reference Clark2003) and Cecilia Heyes (Reference Heyes2018) have argued that extending our biological and cognitive capabilities through tools is intrinsic to human nature. Donna Haraway (Reference Haraway1985) more radically suggested that technology’s increasing integration into human life gradually erodes the boundaries between our organic and synthetic identities. This framing positions sensory augmentation devices not as artificial intrusions but as the next logical step in our ongoing cultural evolution, where each progression from the simplest tool to the most complex machinery has redefined our cognitive engagement with the world – and with each other.

This Section adopts a techno-cultural perspective on the senses, examining how technological advancements continually reshape our sensory and cognitive paradigms. Through this lens, we may better understand not only what these technologies might offer us but also how they fundamentally transform the meaning of perception and, ultimately, the condition of being human.

3.2 Sensory Technology: Perception as Habit and Habituation

Research on sensory augmentation reveals how habituation shapes perceptual processing in fundamental ways. Habituation allows the nervous system to filter out constant, unchanging stimuli, freeing cognitive resources for novel or changing events that demand attention. This filtering process operates through repetitive behavioral patterns that become embedded in our cognitive framework, creating established perceptual templates. When encountering new sensory input, the brain compares incoming information against these templates, generating experiences of either familiarity or novelty based on pattern matching. We actively construct perceptual experience through this dynamic interplay between learned expectations and environmental input, as opposed to passively receiving the available sensory data.

Consider your nose. Unless you live in a vacuum, you are surrounded by countless volatile airborne molecules. Yet, you seldom take note of your olfactory experience. This stems from sensory habits and habituation processes working in tandem. People in Western countries seldom pay attention to olfaction, so they remain largely unaware of the plethora of smells around them. Their experience of “olfactory silence” stems from inattention, not an absence of stimuli or capability.

Once you retrain your attention, you begin noticing smells everywhere. This shift occurs through both cognitive and physiological mechanisms. Many ambient odors are easily ignored because odor receptors in your nose habituate quickly following stimulation (Binder et al., 2009; Pellegrino et al., Reference Pellegrino, Sinding, Wijk and Hummel2017). This explains why all households, save yours, appear to possess a characteristic odor. But when you return from vacation, a familiar scent greets you at home before it fades again. Odor receptors in the nasal epithelium cease firing when there is no salient change in environmental chemicals. Simply imagine being constantly aware of all surrounding odors. You would have difficulty concentrating on anything else. Habituation allows mental habits to form by foregrounding and backgrounding sensory information in conscious awareness. How and what we perceive depends fundamentally on these habits.

This relationship between habituation and perception becomes clearer when we examine sensory augmentation technologies. These devices allow us to explore to what extend specific modes of perception arise from fixed sensory anatomy or from learned behaviors that can be modified. By disrupting our normal sensory patterns, augmentation reveals the plasticity underlying what we typically experience as stable perception.

A striking example are invertoscopes, devices that flip the visual field upside down, left to right, or both. Their origin dates to George M. Stratton (Reference Straus1897), who flipped his own retinal image to examine the brain’s adaptability. Stratton wore the goggles for approximately 87 hours, testing the limits of perceptual adaptation on himself. Later, Theodor Erisman and Ivo Kohler expanded this work in the Innsbruck Goggle Experiments in the 1940s and 1950s.

Erisman and Kohler designed augmented goggles to create systematic “disturbances” in perception through vertical and lateral inversion and image distortion. They observed the course of these disturbances, specifically tracking the adaptation period to understand how perceptual processes form and sustain their organization through self-correction. Their subjects, often the experimenters themselves, wore this modified eyewear for extended periods ranging from days to months, learning to perform tasks from grasping teacups to riding motorbikes (Sachse et al., Reference Pierre, Beermann and Martini2017).

Kohler’s detailed observations reveal the profound disorientation and gradual adaptation that occurred. On his first day wearing the glasses, he noted that “the most notable impressions while moving the body are by all means the movements within the visual field.” Walls appeared as rhombs and rhomboids, sheets of paper and books changed shape when viewed from above, cups became elliptic. Room corners appeared overly blunt or pointed, depending on head position. Yet even on this first day, he observed that “the impression of movement that was so very intrusive in the beginning may already have decreased somewhat” (Kohler, Reference Kohler1941, quoted in Sachse et al., Reference Pierre, Beermann and Martini2017, p. 226).

Kohler found that visual space lacked absolute, perspective-invariant values. Objects did not possess fixed positions or forms. Instead, Kohler identified three distinct phases of visual adaptation that highlighted the importance of sensorimotor learning in regaining perceptual stability. During the first phase, spanning days one through three, participants experienced the world as genuinely upside down. Their immediate reactions were misguided, leading to actions taken in the wrong direction. By the second phase, around day five, marked improvement in coordination and perception emerged. Objects previously perceived as upside down started to appear right-side up. Notably, this adjustment coincided with participants using their hands to explore the shapes they saw, effectively reorienting their visual perception through tactile interaction. The act of tracing upside–down shapes with their hands led participants to perceive these shapes as upright, demonstrating that sensorimotor interaction was key to understanding spatiality. From day six onwards, participants consistently perceived their surroundings “correctly” while wearing the reversing spectacles. Their behavioral responses and tasks, such as drawing, were carried out with accuracy comparable to performance without the spectacles.

As we act, our perception shifts in response.

Follow-up studies have confirmed that this adaptation involves distinct cognitive adjustments (Logvinenko, Reference Logvinenko1974; Dolezal, Reference Dolezal1982) as well as neural changes, with different brain areas becoming activated as the process unfolds (Sachse et al., Reference Pierre, Beermann and Martini2017). Research on sensory augmentation highlights the remarkable flexibility of human perception, portraying it as an active, constructive process that integrates cognitive and sensory elements, including memory (Eagleman, Reference Eagleman2020). This interplay of sensorimotor feedback and exploration fundamentally crafts our perceptual experiences.

Beyond spatial awareness, sensory augmentation influences our perception of time. VR gaming, for example, diminishes our awareness of our physical selves, which compresses subjective time, allowing longer real durations to be experienced as shorter segments (Mullen and Davidenko, Reference Mullen and Davidenko2021). These temporal effects, along with findings from spatial augmentation studies, demonstrate the cultivated rather than fixed nature of perceptual stability. They emphasize cognitive aspects such as perspective and object constancy, identifying sensorimotor coupling as the key element in maintaining coherent perceptual content.

The temporary nature of these interventions raises intriguing questions about more permanent modifications. Intervertoscopes, VR, and other wearable devices offer fleeting encounters with altered perception. When removed, their induced effects typically dissipate within seconds to hours, depending on exposure duration. This impermanence contrasts with technologies integrated more directly into the body, such as implants and skin interfaces. With extended use, physiologically integrated devices might become more difficult to remove or deactivate, potentially leading to lasting changes in sensory perception. The extent to which permanent enhancements might rewire sensorimotor processing, reshape body schema, transform phenomenological experience, and redefine our interactions with our surroundings is an open and fertile ground for inquiry.

3.3 Colors Unseen: The Phenomenology of Sensory Substitution

Meet Neil Harbisson, a techno-sensory pioneer and the first individual to be legally recognized as a cyborg. Harbisson lives in a world of grayscale due to achromatopsia, a rare condition that prevents him from seeing colors. Rather than accept this circumstance, he underwent a surgical procedure to have an antenna permanently implanted in his skull, connected to a chip in his brain. This bespoke apparatus translates electromagnetic frequencies into audible sound frequencies using bone conduction, akin to dolphin echolocation. Through this surgically integrated antenna, Harbisson experiences color information as sounds, broadening his sensory world to include wavelengths beyond the conventional human visual spectrum (Figure 9).

Figure 9

Harbisson’s skull-implanted antenna to “hear” colors by converting light into sound. This technology creates new sensory experiences by turning visible and invisible colors (infrared and ultraviolet spectrum) into audible vibrations.

Source: Dalron Murray, H+Pedia.

Is Harbisson “hearing color”? This question strikes at the heart of fundamental assumptions about sensory experience. Do we need eyes for sight or tongues for taste? How closely can artificial senses mimic their natural counterparts? Some philosophers will dismiss these questions as category errors, misappropriations of language, or misunderstandings of experience itself. The notion of hearing colors or seeing through touch is, to them, an affront to common sense, as Leon (Reference Leon1988) proclaimed: “It’s no more convincing than the idea that we’d hear sounds through our eyes, just because we can see an optical transformation of an aural input with an oscilloscope.” By contrast, reflecting on how his antenna has transformed his interaction with the world, Harbisson describes how his antenna has led to novel forms of sensory organization that sometimes resemble visual experience:

While his clothing choices reflect musical rather than visual thinking, Harbisson’s approach to food arrangement suggests he has developed spatial-visual strategies for organizing auditory input. Such divergence in defining sensory boundaries and experience between philosophical skeptics and cyborg practitioners prompts us to explore deeper questions about the nature of sensory experience. Four distinct philosophical positions have emerged to explain how sensory substitution devices like Harbisson’s antenna might operate. Each is based on different assumptions regarding the nature of the senses and their relationship to conscious experience (Deroy and Auvray, Reference Deroy, Auvray, Stokes, Matthen and Biggs2014; Kiverstein et al., Reference Kiverstein, Farina, Clark and Matthen2015).

3.3.1 Sensory Replacement: Hearing as Seeing

The first position argues that Harbisson genuinely perceives colors through sensory substitution. His antenna functions as a replacement sensory organ, with sound taking the place of light as the medium for color perception. This view can be supported through two related but distinct arguments about how our senses function.

On the one hand, the functional equivalence argument suggests that senses should be categorized by their psychological and behavioral roles rather than their biological mechanisms. Under this view, “perception” is simply the process by which sensory information serves specific objectives like spatial navigation or object recognition. While eyes typically help us differentiate distant objects, an individual receiving sensory input through their skin could theoretically achieve the same result.

Paul Bach-y-Rita’s pioneering work with Sensory Substitution Devices (SSDs) in the late 1960s exemplifies this view (Bach-y-Rita et al., Reference Bach-y-Rita, Collins, Saunders, White and Scadden1969; Bach-y-Rita, Reference Bach‐y‐Rita2004). His vibrotactile device transformed camera images into tactile vibrations felt on the user’s back. Bach-y-Rita later miniaturized this concept into the portable BrainPort (Figure 10), which converts visual data into vibrations on the tongue, inspiring a wave of similar subsequent innovations (like PSVA; Vibe; and vOICE, which melds visual with auditory input). At the heart of this invention lies the principle of neuroplasticity (Bach-y-Rita et al., Reference Paul, Tyler and Kaczmarek2003): “The eyes and the ears are just the receptors; the actual processing of sight and sound takes place in the brain” (Bach-y-Rita quoted in Eagleman, Reference Eagleman2020). For Bach-y-Rita, the skin could effectively serve as a new set of eyes because seeing and hearing are fundamentally brain activities.

Figure 10

BrainPort. Device converting visual pictures into tactile sensations on the tongue, allowing users to “feel” the images.

Source: Zilbershtain-Kra, Scholarpedia.

On the other hand, the qualitative equivalence argument offers a different reason for adopting the replacement view, suggesting that sensory substitution devices deliver experiences similar to original senses, albeit in a diluted form. The critical measure here is likeness in mental representation rather than functional outcome (Leon, Reference Leon1988; Block, Reference Block, Hahn and Ramberg2003). Remarkably, and in support of this view, users of vibrotactile devices report experiencing quintessentially visual phenomena like motion parallax and looming after training (Bach-y-Rita et al., Reference Bach-y-Rita, Collins, Saunders, White and Scadden1969). Motion parallax occurs as objects at varying distances create dynamic depth perception, while looming refers to objects appearing larger as they approach. If these experiences, commonly thought of as visual, can occur through touch, perhaps perceptual content is not categorically bound to a specific sense per se. Bach-y-Rita’s devices conjure the presence of distant objects, communicating visual dimensions through touch. Yet, while this view works well for spatial characteristics like length, shape, and motion, it appears challenged by qualities like color that seem inherently tied to specific sensory modalities.

Nonetheless, cross-modal integration of phenomenal content is not unprecedented in normal sensory experience. For example, the McGurk effect demonstrates how vision can override hearing to create perceptions that match neither sensory input (McGurk and MacDonald, Reference McGurk and MacDonald1976). When people hear the sound “ba” while watching someone mouth “ga,” they typically perceive “da” – a sound that corresponds to neither the auditory nor visual input alone. The brain seamlessly blends these inconsistent signals into a unified, if illusory, phenomenally auditory experience. Conceivably, Harbisson’s brain might be learning to integrate sound frequencies with visual and spatial concepts, creating genuinely visual experiences from auditory input through continuous feedback and adaptation.

3.4 Back to the Future: What Is Molyneux’s Problem?

The philosophical debates surrounding Harbisson’s antenna and sensory substitution devices echo a much older question that has puzzled thinkers for over three centuries. In the waning years of the seventeenth century, physician William Molyneux posed a deceptively simple question to philosopher John Locke that strikes at the heart of how we understand cross-modal inferences in perception:

This thought experiment, known as Molyneux’s problem, asks whether someone born blind who suddenly gains sight would immediately recognize through vision objects they previously knew only through touch. The question directly parallels our inquiries about sensory substitution: Can knowledge gained through one sensory modality transfer seamlessly to another? The various philosophical responses to Molyneux’s problem mirror the different interpretations we have seen regarding how sensory substitution devices might work (Ferretti and Glenney, Reference Ferretti and Glenney2020).

Two major arguments support a negative answer to Molyneux’s problem, suggesting that cross-modal recognition requires learning and cannot occur immediately.

The first argument emphasizes the fundamental differences between sensory modalities and their informational content. Bishop Berkeley (Reference Berkeley1975[Reference Berkeley1709]) argued that vision lacks intrinsic spatial qualities, which are instead acquired through tactile experiences. According to this view, the interaction between vision and touch gives visible objects their spatial characteristics over time. A person who has felt a cube’s sharp edges and flat surfaces would not immediately recognize these same features when seen for the first time, because visual and tactile representations of shape are fundamentally different kinds of information. This perspective aligns with observations from invertoscope studies, where subjects’ spatial awareness improved markedly once they began physically interacting with objects while wearing the vision-reversing goggles.

The second argument supporting a negative response centers on the necessity of perceptual learning. Both Molyneux and Locke believed that understanding the relationships between different perceptual attributes requires experience and practice. Étienne Bonnot de Condillac in his Treatise on Sensations (1930[1754]) elaborated this view, arguing that each sense operates independently when processing spatial information. Mastering cross-sensory object recognition depends on developing cognitive skills that connect information across modalities. Condillac predicted that someone gaining sight would initially face confusion even when presented with basic color patches, emphasizing how challenging it would be to adapt to an entirely new sensory channel.

Two alternative arguments favor a positive answer, suggesting that cross-modal recognition could occur immediately upon gaining sight.

The first argument relies on innate mechanisms or built-in schemas for object recognition. This perspective, associated with Kantian philosophy, proposes that connections between senses are present from birth. These connections might exist either as fundamental sensory frameworks or, following Leibniz, as geometrical relationships that enable rational inferences about object identification. Different versions of this view vary significantly in their details. For instance, Thomas Reid suggested that while we can identify two-dimensional shapes through both sight and touch, this ability does not extend to three-dimensional forms (Glenney, Reference Glenney2012).

The second argument takes a markedly different approach by focusing on embodied cognition rather than abstract mental representations. Contemporary philosophers like Alva Noë (Reference Noë2004), among others, argue that our understanding of objects and their spatial relationships is deeply rooted in physical experience and bodily action. From this perspective, the senses are extensions of a body in motion, equipped with joints and the ability to navigate through space. Perceiving objects involves more than just receiving sensory input; it requires understanding how bodily movement enables interaction with the surrounding world (Sheets-Johnstone, Reference Sheets-Johnstone2011; Lyon, Reference Lyon2020). Since both vision and touch involve the same moving exploring body, knowledge gained through one modality should be accessible to the other.

These competing philosophical positions reflect the same fundamental questions we encounter with sensory substitution devices. Are sensory experiences effectively separate channels that require learning to connect, or do they tap into shared underlying mechanisms for understanding the world?

Empirical evidence remains mixed, echoing the ambiguous results we have seen with sensory substitution research. For instance, Held et al. (Reference Held, Ostrovsky and de Gelder2011) studied children with severe visual impairment who underwent cataract surgery, providing a real-world approximation of Molyneux’s scenario. Initially, these children struggled to visually distinguish the shapes and sizes of objects after surgery, supporting the perceptual learning theory. However, they rapidly developed these abilities once they could correlate their new visual experiences with familiar tactile sensations. Intriguingly, research with bees offers a different perspective (Solvi et al., Reference Spence2020). Scientists conditioned bees in darkness to associate sugar rewards with either spheres or cubes using only tactile information. When light was introduced, the bees could immediately identify these shapes visually, suggesting some form of innate cross-modal transfer. This finding supports a positive response to Molyneux’s problem, at least for simple geometric forms in nonhuman animals.

Such mixed empirical results point to a basic dilemma. Molyneux’s thought experiment cannot be perfectly replicated in reality. People who gain sight after lifelong blindness often have complex medical histories, partial vision development, or other confounding factors. Similarly, sensory substitution devices operate under different conditions than the thought experiment posits.

Nevertheless, empirical investigations are far from futile. Different experimental approaches illuminate various cognitive and sensory mechanisms that might otherwise remain hidden. Instead of seeking a single answer to Molyneux’s precise problem, we can use variations of his thought experiment to dissect the multiple processes involved in cross-modal perception. This approach reveals that what initially appears to be a single phenomenon might actually involve several distinct cognitive and sensory capacities.

The enduring relevance of Molyneux’s problem for understanding sensory substitution becomes clear when we consider Harbisson’s experience. Like the hypothetical person gaining sight, Harbisson had to learn to correlate new sensory information with existing knowledge about the world. His reported success in developing meaningful color-sound associations, or functionally equivalent responses to color information, through extensive practice suggests that cross-modal transfer is possible, though it may require the kind of adaptation that both Molyneux and Locke anticipated. Therefore, the question is not simply whether such transfer can occur but under what conditions and through what mechanisms it becomes possible.

3.5 Reality, What a Concept!

The philosophical debates surrounding sensory substitution devices and Molyneux’s problem leave us with more questions than definitive answers, but this uncertainty proves productive rather than problematic. These unresolved questions illuminate fundamental assumptions about perception that typically remain hidden beneath our ingrained sensory habits. By decoupling sensory reception from familiar biological pathways, augmentation technologies create unique opportunities to investigate the principles underlying our experienced reality or construction of reality. They may even reveal entirely new forms of perceptual experience.

The theoretical possibilities we have explored throughout this section are finding concrete expression in the work of contemporary sensory pioneers. Moon Ribas, an avant-garde artist and dancer with heightened proprioceptive awareness, has engineered for herself a seismic sense through permanent sensors implanted in her feet (Alcaraz, Reference Alcaraz2019). These sensors connect to online seismographs, enabling her to detect global tectonic movements as subtle as 1.0 on the Richter scale. This augmentation exemplifies the third philosophical position we examined (in Section 3.2.3): the creation of an entirely novel sensory modality that aligns with neither existing senses nor their technological substitutes.

Ribas’ seismic sense has transformed not only her perceptual capabilities but her entire relationship with the world. As she explains, “having a new sensory input changes the way one thinks about the world and acts within it” (quoted in Alcaraz, Reference Alcaraz2019, 70). This transformation supports the embodied cognition argument from our discussion of Molyneux’s problem, demonstrating how new sensory channels reshape cognitive frameworks rather than simply adding information to existing ones. Harbisson echoes this insight, noting how augmented sensory experiences fundamentally alter reality itself:

Ribas and Harbisson’s often collaborative experiments further illuminate the plasticity of human perception. Ribas spent three months with a device that provided kaleidoscopic vision, amplifying color perception without relying on shape recognition. Over time, she developed enhanced color acuity and began detecting motion through chromatic changes alone (Solon, Reference Spence2013). This adaptation mirrors the three-phase process we observed in invertoscope studies, where sensorimotor exploration proved crucial for perceptual reorganization. Her experience also suggests that the answer to Molyneux’s problem might depend on the specific type of cross-modal transfer being attempted.Footnote ⁴

Harbisson’s Solar Crown, a helmet equipped with heat sensors that track the sun’s position, represents form of temporal sensory augmentation. By training himself to interpret warmth distribution patterns, he aims to develop an intuitive sense of time that draws inspiration from Einstein’s relativity theory (de Jorge Gama, Reference de Jorge Gama2017). This project directly addresses questions about whether augmented senses can provide genuinely new forms of awareness or merely cognitive interpretations of familiar information.

Perhaps most intriguingly, Harbisson and Ribas have developed BlueTOOTH, a device that enables interpersonal communication through dental vibrations (Harbisson, Reference Harbisson2016). This innovation pushes beyond individual sensory augmentation toward new forms of intersubjective experience. More than replacing or extending existing senses, BlueTOOTH creates a communication channel that bypasses traditional sensory categories. Their experiment suggests that one profound implication of sensory augmentation lies in enhancing individual perception by creating new possibilities for shared attention.

Such real-world explorations provide crucial data for evaluating the philosophical positions we have examined. Ribas’s successful adaptation to seismic sensing supports both the neuroplasticity arguments advanced by Bach-y-Rita and the novel modality thesis proposed by critics of traditional sensory substitution models. Her kaleidoscopic vision experiments demonstrate the kind of cross-modal learning that both supporters and critics of Molyneux’s problem would predict, though they required the extended practice period that learning theorists emphasize.

The work of these sensory pioneers, working outside academic convention, indicates that our familiar categories of sight, sound, touch, taste, and smell represent only a fraction of possible perceptual experiences. Their experiments show that human consciousness can accommodate radically new forms of sensory input, but this accommodation requires the kind of sensorimotor exploration and cognitive adaptation that philosophers have debated for centuries. The success of their augmentations suggests that the boundaries between senses are more fluid than traditional philosophy assumes, while their learning curves confirm that such fluidity emerges through active engagement with the world, not passive reception or reproduction of information.

Philosophical investigations into the nature of sensory experience gain new urgency as technological intervention makes previously impossible sensory configurations attainable. Sensory augmentation technologies, rather than resolving ancient debates about the nature of perception, illustrate their ongoing relevance while opening entirely new domains for philosophical inquiry. The reality we construct through our senses may indeed be far more malleable than we feel comfortable with.

4.1 The Parts and Wholes Problem in Mind

The mathematician stood in his cluttered workspace, match in hand, watching years of labor surrender to flame. Walter Pitts had pulled off what many thought was the impossible: a complete mathematical cartography of the mind, a dissertation promising the grand unification of gray matter and algebra. Yet somewhere between hypothesis and proof, his ethereal mathematics had met an unlikely nemesis – the eyes of a frog.

Pitts found himself entangled in his own undoing, co-authoring the very study that thwarted his ambitions. Jerome Lettvin’s “What the Frog’s Eye Tells the Frog’s Brain” (Lettvin et al., Reference Lettvin, Maturana, McCulloch and Pitts1959) became a landmark in neuroscience by unveiling that the eye was no passive receiver but an active processor of visual information (Shepherd, Reference Shepherd2009; Haueis, Reference Haueis2023). Multiple types of ganglion cells in the frog’s retina are specialized in different visual tasks, involving edge detection, dimming detection, moving object detection, and stationary object recognition. This discovery shattered the assumption underlying Pitts’ mathematical framework, which had rested on the premise that the brain performed all the analytical work while sensory organs merely transmitted raw data.

The brilliant yet troubled Pitts famously torched his doctoral thesis (Gefter, Reference Gefter2015). Among his motivations was a stark epiphany. Pursuing a purely mathematical blueprint for mind and brain, divorced from biological reality, had become fundamentally misguided. He had tried to design a bridge without anticipating whether its materials were stone, wood, or rope.

Pitts’s intellectual arson illustrates a profound philosophical dilemma that continues to shape how we understand the relationship between mind and brain. When we try to explain perception and cognition, should we focus on the discrete building blocks, such as the individual sensory inputs, neural response patterns, and mental representations? Or should we emphasize the dynamic properties of the whole system, involving the functional organization that transforms sensation into meaningful experience? This is, effectively, a mereological question, as it targets the relationship between parts and wholes in understanding mental phenomena.

Mereology, the philosophical study of parts and wholes (Cotnoir and Varzi, Reference Cotnoir and Varzi2021), investigates how basic elements relate to complex systems.Footnote ⁵ In cognitive science and neuroscience, mereological thinking frequently guides how we conceptualize the relationship between elementary sensory inputs and our multifaceted perceptual experiences. Should we explain vision by decomposing it into photons, retinal responses, neural firing patterns, and computational operations? Or does vision only make sense within the context of the whole sensory system and its evolutionary function as part of an integrated, behaving organism?

Contemporary scientific research on the senses typically employs decomposition and localization strategies to arrive at causal-mechanistic explanations (Bechtel and Richardson, Reference William and Richardson2010). Decomposition involves analyzing systems by examining individual components, while localization focuses on the specific functions of each part. These strategies delineate pathways from inputs such as photons or volatile molecules to outputs like perceptions of visual objects or odor images. By isolating the properties and behaviors of individual components, scientists seek to gain insight into the traits and actions of the system.

Mereological thinking extends beyond science into the philosophy of mind. Debates about mental states and their physical basis often entail breaking down perceptual experience into discrete components, such as modules, percepts, and representations (Fodor, Reference Fodor1981; Markman and Dietrich, Reference Markman and Dietrich2000). Consider observing an apple: We might divide this experience into distinct visual attributes like shape, color, and size, each of which appears to correspond to different neural processing. Philosophers employ decomposition to figure out how mental states are formed and how they interact causally.

Yet this “mental atoms” approach faces significant challenges. Critics argue that breaking perception into discrete components distorts our understanding by disregarding the dynamic and contextual nature of mental processes (Chemero, Reference Chemero2000; Barwich, Reference Barwich2014, Reference Barwich2018; Buccella and Chemero, Reference Buccella and Chemero2022). The popularity of this atomistic ontology arises in part from philosophical methodology itself (Williams and Barwich, Reference Barwich2025). Formal tools and logical analysis work best with clearly defined, separate elements. But real perceptual experience is more holistic and integrated than such approaches capture.

The tension between atomistic and holistic approaches has played out dramatically in the recent history of research on mind and brain, where computational methods have dominated. This mereological dilemma of whether to prioritize the mind’s stones (molecular and cellular components) or its arch frames (causal connectivity) is illuminated by two major frameworks: Warren McCulloch and Walter Pitts’ logical approach, which emphasized discrete symbols as mental atoms, and David Marr’s computational vision, which focused on functional organization.

The following historical analysis of these dominant computational frameworks will reveal how both approaches ultimately point toward the need for a more embodied and situated framing of perceptual experience to fully explain the nature of the senses. Thus, we encounter the flip side of the methodological challenge with which we began this Element. Whereas Section 1 highlighted how philosophical analysis of the senses has become disconnected from scientific studies, this section examines the conceptual pitfalls that await scientific research when it fails to reflect continuously on its foundations through philosophical and historical inquiry.

4.2 Mental Stones: McCulloch and Pitts’s Logical Calculus

In 1943, Warren McCulloch and Walter Pitts published “A Logical Calculus of the Ideas Immanent in Nervous Activity,” offering a revolutionary model of how nerve cell activity could generate mental processes through binary logic. Their computational network model proposed that neurons operated like digital switches, firing in crisp ON or OFF states that had affinities with emerging computer technologies.Footnote ⁶ This binary framing of mental functions provided the operational framework for models of the mind in cognitive science, in collaboration with computer science and cybernetics (Piccinini, Reference Piccinini2004).

The principal innovation of their model lay not in its analysis of neural communication itself but in its concept of “psychons,” hypothetical mental atoms that could be mapped onto neurons. Psychons denoted basic sensations and ideas like colors or shapes, interlinked through logical operators like AND, OR, and NOT. These mental building blocks could combine to form increasingly sophisticated concepts and thoughts, with neurons acting as the physical substrate for these logical operations.

McCulloch and Pitts illustrated their framework through the heat illusion, in which touching extremely cold things causes a hot sensation before the sensation of cold sets in. Their hypothetical model accounted for this sensory phenomenon with six hypothetical neurons (Figure 11): heat and cold sensors, heat and cold perception neurons, and two intermediary neurons facilitating communication between them. In this model, when the cold sensor fires once, it triggers the heat perception neuron through one pathway. When it fires twice, indicating sustained cold input, it activates the cold perception neuron through a different pathway. This elegant example demonstrated how the logic of neural networks could explain perplexing sensory phenomena like perceptual illusions.

Figure 11

Simplified illustration of McCulloch and Pitts’ hypothetical network explaining the heat illusion. When neuron 1 senses heat (ON), it sends a signal to neuron 3, eliciting the sensation of heat. When neuron 2 senses a chilly input and fires once, it sends a signal to intermediary neuron A, which activates neuron 3 and elicits an illusory sensation of heat. If neuron 2 fires twice, it sends a signal to intermediary neuron B, which turns off intermediary neuron A and activates neuron 4, eliciting a cold sensation.

The model’s appeal was its promise to resolve the mind–body problem by correlating Boolean operations with physical cellular structures. Mental processes are transformed into logical operations functioning on discrete symbols, entirely detached from biological implementation or contextual factors. Crucially, the legitimacy of this symbolic approach is solely established by the combination laws applied to arbitrary symbols, not by what they represent or how they are physically implemented. As George Boole (Reference Boole1847) emphasized, the same logical process could represent solutions to problems in mathematics, geometry, or physics simply by changing the interpretation of symbols. In Boole’s own words:

McCulloch and Pitts transferred this idea to psychology, treating psychons as arbitrary mental symbols whose meaning derived entirely from their logical linkage. This postulate proved both powerful and problematic. Psychons, defined as discrete and separate elements unaffected by context or perspective, facilitated precise mathematical analysis. Consequently, a neuron that is stimulated by an “angry face” stimulus would consistently elicit the same “angry face” response, irrespective of any other neural activity in the brain, and there would be no interference between, say, cardiac activity and facial perception.

This assumption crashed against biological reality. Lettvin’s studies on frogs revealed that sensory processing includes complex preprocessing and feature extraction, which are not captured by disembodied static logical operations. Moreover, later investigations have shown that seemingly unrelated systems do interact in complicated ways, such as heart rhythms influencing visual perception (de Vignemont, Reference de Vignemont2023). For example, in one study, people exhibited heightened sensitivity to angry faces during specific phases of the cardiac cycle (Garfinkel et al., Reference Garfinkel, Minati and Gray2014). The brain, embedded in an acting body, operates as a contextually aware, interconnected system rather than a disconnected set of logical components (Sporns, Reference Stokes, Matthen and Biggs2011; Favela, Reference Favela2023).

The key issue within McCulloch and Pitts’ framework, therefore, was not its reliance on abstraction or idealization but its dependency on context-independent mental atoms. Real neural activity spreads through distributed networks in ways that violate the discrete, modular assumptions of Boolean logic. Mental content is not arbitrarily calculated by logical operations but morphologically computed by historically contingent biological structures shaped by evolution.

Pitts recognized this when he burned his dissertation. The frog’s eye did not invalidate mathematical modeling of the mind. It uncovered that the central modeling assumptions about mental elements were incorrect. To explain how a frog perceives the world in its fragmented, bug-detecting way, we require computational models that are attuned to the distributed and heterogeneous character of real neural processing.

4.3 Functional Arches: Marr’s Computational Vision

David Marr approached the mind–brain relationship from the opposite direction to McCulloch and Pitts. Instead of imposing abstract logical operations onto neural tissue, he argued for a theory of vision that built its computational framing from the biological reality upwards. His posthumously published “Vision” (Marr, Reference Marr1982) opened with a trenchant yet constructive critique of the state of neuroscience, which appeared caught between two extremes. On the one hand, there was abstract theorizing with arbitrary symbols, and, on the other hand, experimental neuroscience had stalled due to its disconnected, piecemeal observations from single-cell recordings (Clark, Reference Clark2000). Neither approach was making the desired progress toward a systematic understanding of the procedures by which the brain creates perceptual content, such as visual objects.

Marr’s seminal contribution was to propose a unifying, three-level framework that could bridge this methodological gap between the “highly abstract” speculations of pure theory and the “large and detailed” but ultimately disconnected catalogs of data from single-cell recordings (Bickle, Reference Bickle2015). This framework was resolutely functional and computational, but its starting point was not abstract logic but biological function. He argued that in order to understand any sensory system, one must first understand the computational problem it solves, such as the reliable derivation of a three-dimensional description of the world from two-dimensional retinal projections in vision (Figure 12). This contrasted sharply with the idea of McCulloch and Pitts, who sought to construct a disembodied logical calculus for neural networks.

Figure 12

Marr’s computational framing of visual object formation. The model detects visual edges and gradients at the lowest possible level. The intermediate stage involves combining these elements into a 2.5D sketch that adds depth and orientation while remaining viewer-centered. The top level generates a 3D object representation.

This new approach was significantly informed by the empirical discoveries of hierarchical feature integration by Stephen Kuffler and, most famously, by David Hubel and Torsten Wiesel (Hubel, Reference Hubel1995; Shepherd, Reference Shepherd2009). Their work, which revealed how center-surround receptive fields in the retina feed into orientation-specific simple and complex cells in the visual cortex, provided the crucial biological evidence for an architecture that built complex representations from simple signals (Figure 13). This work provided the physiological foundation for Marr’s computational architecture and served as essential evidence for the specific type of computational process he aimed to illuminate.

Figure 13

Hierarchical feature integration in vision according to Hubel and Wiesel. (a) Kuffler’s center-surround cells (Source: Xoneca, Wikimedia Commons). (b) Hubel and Wiesel’s simple and complex cells. Simple cells (a–b) respond to certain stimulus orientations, aligning and integrating the receptive fields of retinal center-surround cells. Complex cells (d) integrate the inputs of these simple cells and respond to different orientations and angles (Source: Kyle.wg3139, Wikimedia Commons). (c) Schema of hierarchical neural feature integration

(Source:Thomas Serre, Scholarpedia).

Marr’s visionary model represented a decisive break from earlier computational frameworks that relied on arbitrary symbols, such as those of McCulloch and Pitts or the Physical Symbol Systems hypothesis proposed by their contemporaries Allen Newell and Herbert Simon. For Newell and Simon ([Reference Allen and Simon2007]Reference Allen and Simon1976), cognition was the manipulation of discrete tokens. In their account, a symbol was fundamentally an arbitrary token (like a chess piece), a designated physical pattern that gained its meaning exclusively through its formal, syntactic relationships within a larger symbol structure. Crucially, the system’s logic was primary; the assignment of reference to the symbols was a separate step. Hence, the symbol CAT might theoretically represent anything, from a feline to a desk lamp or even a bulldozer, unless the system’s rules defined its relationships to other symbols like ANIMAL or PURRING. The meaning was not in the token itself but in its logical position within the symbol system. These symbols represented entities that did not have a consistent physical embodiment; rather, they were fixed by the location and function of their designated symbolic representation within the self-contained system of symbols.

Marr radically shifted the entire discourse from a focus on symbols to a focus on representations. Yet his key innovation was to redefine the very unit of analysis. For Marr, a “representation” was not primarily an individual item or token. Instead, he defined a representation as

This marked a profound philosophical and technical departure. The core distinction can be illustrated with a linguistic analogy. Newell and Simon focused on individual words (symbolic tokens) and the grammatical rules for combining them; consequently, their approach treated the choice of language as arbitrary, prioritizing the internal logic of syntax above all else. In comparison, Marr focused on the language as a whole (the representational system) and contended that the functional requirements of the particular biological problem it must resolve dictate its fundamental structure. Consequently, individual data structures in Marr’s framework serve as descriptions, analogous to sentences that are generated within this functionally determined language.

Where Newell and Simon’s symbols were defined by their logical position within an arbitrary system, Marr’s representations were structurally descriptive models whose very form was shaped by their causal role in solving a specific, real-world biological problem. This distinction is philosophically critical. In Marr’s theory of vision, the 2.5-D Sketch is itself a representational system (Figure 12). Its design is dictated by the computational problem of deriving surface orientation from shading and texture. An individual entry within that sketch, such as a vector encoding a visual edge, is not an arbitrary token for “edge” but a description, that is, a specific data structure whose format encodes actionable properties like orientation, contrast, and spatial location. This structure is not arbitrary because the next stage of processing requires information in exactly that format. In other words, the form of the representation follows its function.

This reconceptualization carries immense significance for philosophical debates about perception, particularly concerning intentionality, the “aboutness” of mental states. Marr’s framework offers a potential naturalistic account of how brain states can be about things in the world (their content). The content of a representation is not magically assigned but is determined by its role within a computational process designed to recover features of the external environment. This idea aligns with teleosemantic theories in the philosophy of mind, which seek to ground meaning in evolutionary function and success conditions (Millikan, Reference Millikan2021). The representation for a specific object, like a predator, has the content it does because it is produced by a system whose function is to detect such objects, and whose success in doing so has conferred an evolutionary advantage (Burge, Reference Burge2010).

Marr’s approach is characterized by two core principles that have deep resonances in the philosophy of mind. First, his representations are implementation-independent with respect to their underlying hardware or wetware:

This is a direct and powerful embodiment of the functionalist doctrine of multiple realizability (Bickle, Reference Bickle2020), which posits that mental states are defined by their causal role, not by the physical states that instantiate them. The same visual computation, such as edge detection, could be realized in the human visual cortex, the optic lobe of a bird, or a silicon-based computer chip. This makes the computational theory an attempt at a genuine explanation of the function itself, moving beyond a description of one material instance of it.

Second, Marr’s representations are informationally robust. Their structure is not arbitrary but is designed to support systematic inferences about the world. This carries enormous epistemic potency since the form of the representation itself licenses deductions about the entity it depicts. Consequently, Marr’s framing provided a powerful tool for cognitive science, enabling it to model perception at a level that incorporates the physical attributes of the world without being bogged down by the overwhelming complexity of granular neurophysiological details. Moreover, it fashioned a middle path in mereological debates, showing how the whole (the computational problem) confers meaning and function upon its operational parts (the representations and algorithms), which in turn must be understood through their role within a physical system (implementation).

On this account, explanation solely in terms of cellular signaling is replaced by a clear understanding of what is to be computed (computational level), how it is to be done (the algorithmic level), and the physical assumptions on which the method is based or the hardware that executes these algorithms (implementation level). Marr’s three levels of analysis provided a rigorous blueprint for connecting the dots from perceptual data to cognitive operation, preferring a systems-theoretic approach to arbitrary symbolic logic.

Nevertheless, from a philosophical perspective, Marr’s framework is not without significant drawbacks. Critics have argued that it oversimplifies the brain’s dynamic, recurrent, and nonlinear interactions (Warren, Reference Weiskrantz2012). Generalizing beyond vision raises serious questions about its applicability to other sensory modalities like olfaction, which appear to involve less hierarchical and more associative, integrative facets of perceptual content (Barwich, Reference Barwich2020). Most profoundly, a key philosophical criticism, encapsulated by Ned Block’s (Reference Block1995) examination of mental access to sensory content, is that, for all its power it falls short in the explanation of critical top–down processes like expectation, attention, and prior knowledge (Churchland et al., Reference Churchland, Ramachandran, Sejnowski, Koch and Davis1993), which are central to currently fashionable theories like predictive processing (Clark, Reference Clark2013). Marr’s feedforward, hierarchical model can appear static and inflexible compared to these more interactive models. Furthermore, its overwhelming focus on the operational mechanics of vision, Marr’s computational account may explain access consciousness (information processing) yet not phenomenal consciousness (subjective experience itself). In other words, Marr’s framework may explain how the visual system computes object shape, but it explicitly sidelines the qualia as the “what it is like” to see that shape.

Marr’s legacy lives on as methodology. Marr’s vision has been pivotal in promoting a theory-driven, empirically anchored approach in both neuroscience and philosophy (Rolls, Reference Rolls2011). By prioritizing functional organization over logical atomism, it provided a rigorous and productive framework for bridging the gap between mechanism and meaning. This successfully shifted the focus from context-independent mental “stones” to the emergent “arch” of computational purpose, advocating that the meaning and very identity of mental and neural parts are effectively derived from the functional whole they serve within a goal-directed system engaged with the world.

4.4 Implications: The Bridge of Experience

The intellectual journey from McCulloch and Pitts’ logical operations to Marr’s functional modules confronts us with a historical shift in cognitive science that encapsulates a critical (r)evolution in our mereological thinking about the mind.

Through the past couple of decades, debates on symbols and representation in cognitive (neuro)science, and artificial intelligence have defined, enriched, yet also challenged broader philosophical discussions about perception and mental content (Siegel, 2010 Shea, Reference Shea2018). Representationalist theories, which focus on how mental representations encode details of the world, are pivotal in current debates in the philosophy of mind and cognitive science. According to these theories, understanding how the mind systematically represents the world does not necessarily require a detailed grasp of the brain’s molecular or cellular specifics. Yet, in this context, philosophers also have taken issue with Marr’s conception of representation. For example, some contend that his view of representations as detailed descriptions mirroring physical realities subscribes to an overly simplistic form of realism regarding perceptions. Marr’s account of representation is at odds with stronger constructivist and especially with enactivist perspectives, which propose that perceptual and cognitive processes actively create sensory impressions through interactions with the world, instead of representationally cataloging it (Hutto and Myin, Reference Hutto and Myin2014).

Overall, the intellectual journey in this section exemplifies how the methodological choices of how we decompose a system, of what we designate as a fundamental part, profoundly shape the theories we build and the phenomena we can explain. McCulloch and Pitts’ atomistic mereology led to a powerful but ultimately brittle logic of the mind, elegant in its abstraction but divorced from the messy reality of biology and behavior. Marr’s functional mereology offered a corrective, showing that parts, be they representations or algorithms, are only intelligible in the light of the whole, that is, the computational goal. His levels of analysis provide a methodical scaffold for navigating the treacherous terrain between empty abstraction and blind empiricism.

But for all their power, both computational frameworks outlined here share a profound limitation. They excel at modeling the mechanisms of access, the information processing pathways from sensory input to perceptual output, yet they remain silent on the phenomenology of experience, the qualitative feel, or what it is like to see a red apple or smell fresh rain. This mereological gap is the central unresolved problem their legacy bequeaths to the philosophy of perception. Thus, the tension between Marr’s functional arch and McCulloch and Pitts’ mental stones directly replays, at the level of neural mechanism, the fundamental dichotomy between third-person explanation and first-person experience with which we began this Element. This perspectival tension is not a minor oversight but a direct consequence of their shared computationalist paradigm. The objective mereology of mechanisms, no matter how sophisticated, seems to leave out the subjective essence of sensory experience. This brings us back to the apparent problem of the explanatory gap that underlies the so-called hard problem of consciousness: How can the way physical pieces are put together lead to subjective, qualitative experience?

Where, then, does this leave us? At this critical juncture, the history of computational mereology has run up against its own limits, and the solution to the dilemma between stones and arches cannot be found by choosing one over the other, nor by simply refining our computational models. The escape from this philosophical deadlock, prefigured in Section 1’s call for a biological turn, therefore requires a paradigm shift. We must move beyond a mereology of the isolated brain to a systems-theoretical understanding of the embodied, situated, and behaving organism.

5.1 Moving Beyond the Hard Problem

The philosophical investigation of the senses has reached an impasse. For four decades, Frank Jackson’s (Reference Jackson1986) infamous thought-experiment of Mary’s Room has crystallized debates about whether subjective experience can be fully explained by physical processes. Yet these debates consistently sidestep more fundamental questions: Are we matching the right questions with the right methods?

This Element has traced how philosophical arguments (Sections 1–3) and scientific models (Section 4) exhibit comparable methodological “blinders.” While philosophy separates sensory experience from its behavioral context and biological implementation, computational neuroscience tends to reduce sensation to passive and linear information processing, thereby separating it from organismic interaction and brain–body–environment dynamics. Despite their differences, both fields and approaches view sensory experience as something that can be broken down into separate parts, akin to mental atoms or functional modules, rather than recognizing it as an integrated, dynamic, and embodied process.

At the core of this Element, thus, is methodology. Philosophical treatments that debate whether sensory perception and its conscious experience have physical “correlates” or involve nonphysical “properties” effectively mischaracterize the nature of experience. Sensory experience is derived from what organisms do through corporeal engagement with their environments, rather than something they have or possess (as some discrete mental “state” in isolation, or what have you).

This insight draws our attention to kinesthetic awareness and dynamical models of development and action (Beer, Reference Beer1990; Thelen and Smith, Reference Thelen and Smith1994; Sheets-Johnstone, Reference Sheets-Johnstone2011). Kinesthetic awareness, meaning our felt sense of movement and bodily engagement, provides not just another perspective on qualia but a fundamentally different methodological foundation for investigating sensory experience, perception, and cognition. This reorientation suggests that thought-experiments like Mary’s Room mislead by employing scenarios and methods that abstract experience from the very embodied processes through which it occurs. The hard problem of consciousness (i.e., the philosophical question of why and how physical processes produce conscious experience) persists not because of inherent metaphysical mysteries but because our methodological frameworks, both philosophical and scientific, systematically abstract sensory experience from the processes that constitute it. The puzzles posed by Mary’s Room, comparable to inquiries concerning the number of angels dancing on a needle, may help us refine our conceptual logic skills, but they do not target and hence cannot yield insights into a real phenomenon.

5.2 Why Mary’s Room Misleads

Jackson’s Mary the Neuroscientist presents a seemingly straightforward thought-experiment of a color vision expert who knows all physical facts about color but has never experienced it directly. When she first sees red, does she learn something new? Property dualists argue yes, concluding that qualia exist beyond the physical. Physicalists typically respond by challenging what constitutes “complete physical knowledge” or by claiming that Mary’s new understanding is effectively a different means of learning and accessing the same physical truths (Nida-Rümelin and Conaill, Reference Nida-Rümelin and Conaill2024).

Both camps and their contemporary offshoots ignore a major methodological flaw. The thought-experiment operates within a conceptual framework that predetermines its interpretation, much like how scientific paradigms shape our view of phenomena we can recognize and explain. But it’s not what Mary sees or knows that matters; it’s how Mary sees and knows.

5.2.2 Inherited Frameworks in the Color Vision Stalemate

The philosophical debate surrounding Mary’s Room is characterized by the systematic restriction of the range of alternative interpretations by inherited paradigms of color perception. These frameworks impact not only how we respond to questions about Mary but also what questions we feel important to ask in the first place.

In effect, the debate has crystallized around a fundamental dichotomy (Churchland, Reference Churchland2013). Either color experiences can be fully captured by physical facts (physicalism), or they involve irreducible nonphysical properties (property dualism). This dichotomous selection conveys underlying beliefs about the interplay between objective knowledge and subjective experience, beliefs that may be inherently flawed. Property dualists and physicalists alike operate within frameworks that treat color as either an intrinsic property of objects or a subjective addition by conscious minds. Both positions assume a clear separation between the perceiver and the perceived, between subjective experience and objective reality.

The logical space of viable solutions is determined by how the dilemma is framed (Lloyd, Reference Lloyd2015). Either scientific knowledge of our current physical inventory is incomplete, or we must revise our metaphysics and introduce “new stuff” (new substances or attributes) to explain consciousness. For example, Roger Penrose (Reference Penrose1989) suggests that quantum effects exist in microtubules, while David Chalmers (Reference Chalmers1995) proposes entirely new fundamental properties on par with mass and charge. Both believe that present scientific paradigms are fundamentally insufficient for explaining qualitative experience.

There exists a third possibility, articulated by the argument from underdetermination within the philosophy of science. The “underdetermination of scientific theories by empirical data” is the idea that any specific set of observational facts can be accounted for by several mutually exclusive explanations. In other words, empirical facts alone cannot definitively determine the correctness of a theory, as different hypotheses can equally explain the same findings (Turnbull, Reference Turnbull2018). Indeed, throughout the history of science, Stanford (Reference Stokes, Matthen and Biggs2006) illustrates how the scientific community has frequently embraced concepts as authoritative explanations, only to discover previously unconceived alternatives that interpret the available evidence equally well. This “historical induction” suggests systematic constraints in scientific and philosophical imagination, as opposed to gaps in nature and the substances with which we are familiar.

Relevant to our discussion of philosophical methodology is that this results in an explicit logical gap between what we observe and what we infer about the underlying nature of reality because the evidence influences our choice between competing theoretical explanations. The explanatory gap between present science and conscious experience may therefore reflect currently unconceived or sidelined hypotheses rather than irreducible nonphysical qualities (Brooks, Reference Brooks2003; Frankish, Reference Frankish2017). Throughout history, once-mysterious phenomena such as combustion, magnetism, and heredity have gradually given way to scientific explanations that do not require supernatural or nonphysical ingredients (Churchland, Reference Churchland1995). Instead of proclaiming that consciousness necessitates nonphysical substances, we might consider whether the hard problem reflects theoretical framing rather than actual metaphysical constraints.

Regrettably, numerous prominent frameworks in current scientific accounts reflect this philosophical dualism and associated conceptual deficiencies. For example, the traditional “detection model” treats color vision as detecting inherent properties of objects, while the “coloring-in model” suggests that visual systems add color to detected shapes (review in Chirimuuta, Reference Chirimuuta2017). Both models rely too strongly on a passive notion of perception where organisms receive information about a pre-given colored world. However, color perception involves functional processes of categorization, comparison, and contextual adjustment that cannot be reduced to simple detection or subjective addition (Thompson, Reference Turnbull1995). Consider color constancy, where a white surface appears white under both yellowish candlelight and bluish daylight, despite reflecting dramatically different wavelengths in each condition. This constancy emerges through active comparison processes that construe relational properties across different viewing contexts.

Other scientific and philosophical frameworks for color vision are available. Adverbialism, as championed by Mazviita Chirimuuta (Reference Chirimuuta2017), posits that color should be understood relationally as originating in dynamic interactions between organisms and their environments, not as either intrinsic object properties or as subjective mental additions. In this view, one does not see a red apple but “sees redly,” emphasizing the mode of perception rather than object properties. Such a relationalist framing successfully challenges both naive objectivism and crude subjectivism by grounding color in the interactions between organisms and their environments. Consider how we perceive the color of water (Figure 14): Whether water appears blue or transparent to the human perceiver depends on depth, lighting conditions, and viewing angle. These perceptual differences accurately reflect the real physical characteristics of water, as determined and modified by observational circumstances and conditions. Still, adverbialism won’t resolve the hard problem. It focuses on how perceptual content relates to external physical properties without addressing why there should be any qualitative, felt dimension to these processes at all.

This persistence of the hard problem is not a metaphysical puzzle. It is ultimately epistemological. Approaches confined to the conventional subject–object distinction lack the capacity to engage with qualitative experience by default. The frameworks themselves separate experience from the processes that constitute it, creating an artificial dilemma that resists solution because it misconstrues the phenomenon from the outset. So, why should sensory processes feel like anything from the inside?

Figure 14

Perspective affects water color. Pure water has a slight intrinsic blue color because it absorbs red light more strongly than blue light. This blue tint is only visible when light travels through a sufficient depth of water, which contains too little water for noticeable absorption. Factors like depth, viewing angle, and lighting conditions affect how we perceive water’s color.

5.3 Giving Mary Her Body Back: LSD and Dancing

The conceptual shortcomings of Mary’s Room arise from a conceptual subject–object dichotomy that regards consciousness as something a subject “has.” This framing creates the apparent mystery of qualia by artificially separating experience from the processes that form it. To overcome this dead end, let us look at aspects of conscious experience that challenge subject–object distinctions entirely and acknowledge the kinesthetic foundation that underlies all conscious experience.

5.4 De-reifying the Mind: The Primacy of Behavior

This kinesthetic foundation of organismal behavior suggests a different perspective on the hard problem. What we call consciousness is not constituted by internal mental states that somehow “correlate with” and “map on” physical substances or properties but by patterns of behavioral engagement with structured environments. But what is behavior? Traditional theorizing views behavior as a response to sensory inputs processed by internal mental representations (analysis in Hurley, Reference Hurley2001). But this kind of thinking creates artificial dilemmas by treating the mind as if it were an entity that has experiences, instead of seeing that experience is made up of patterns of how we interact with the world.

Behavior is more than measurable responses. It functions as a formative element, an “interface,” that structures perception itself (Branchi, Reference Branchi2022). Behavior is not “downstream” and does not come after brain activity; instead, it connects the nervous system and the environment in ways that actively change how we see things on an ongoing basis (Clark, Reference Clark2013). This process-oriented perspective contests static models that dichotomize perception and action, highlighting consciousness as arising from nonlinear, interactive processes rather than distinct mental states. Two examples will help illustrate this claim.

5.4.1 Electrolocation: Behavioral Patterns as Constitutive of Perceptual Structure

Consider the kinetic constitution of electric fish, particularly the black ghost knifefish, which demonstrates how physiology and behavior fundamentally form perceptual structures and experience. These electric fish act like “living batteries,” generating electric fields from tail organs that envelop their bodies (Figure 15). Current flows through water from one end of the electric organ to the other, with nearby conductors (like other animals) increasing current flow while insulators (like rocks) reduce it. The fish detects these voltage changes across 14,000 electroreceptors scattered over its skin, creating what Ed Yong (Reference Young2022) called “electric images” of their surroundings – conductors “shine brightly” while insulators “cast electric shadows.”

The sense of electrolocation demonstrates how internal bodily dynamics and outward-directed motion co-constitute perceptual structure. The knifefish’s electric organ discharge produces a rhythmic pulse (usually 300–2000 times per second) that generates the sensory environment in which the fish interacts. This internal rhythm serves as the temporal foundation of perception, with objects existing solely as variations of this self-generated pulse pattern. Electric fish are best understood as creating their perceptual world through continuous bodily activity, confining their awareness to a small “sensory bubble” extending roughly an inch from their body. This is different from input–output models of sensory systems, which have organisms wait for environmental information.

What can be perceived and how it can be perceived are directly determined by the fish’s continuous movement patterns. Because electric fields weaken rapidly with distance, the fish must actively position objects within its sensory range through locomotion. When detecting food, a knifefish will frequently overshoot and must then swim backward, sometimes for several meters, in order to relocate the target and reposition itself. The fish explores by encircling objects, executing what Yong refers to as “parallel-parking maneuvers” to sustain sensory contact. These movement patterns are not reactions to perception; they are the very mechanisms by which perception manifests and unfolds.

Figure 15

Electrolocation. The tail organ of an elephant-nosed fish creates an electric field to figure out what is going on around it by interacting with both living and nonliving things.

Source: Chiswick Chap, Wikimedia Commons.

Ultimately, the interplay between bodily rhythms and environmental positioning determines the perceptual structure of the knifefish’s environment and what counts as perceptually meaningful. Indeed, the fish alters its discharge frequency according to the situation. For example, it speeds up its frequency when it is near other electric fish to avoid jamming and slows down when it is resting. These internal modifications alter the entire perceptual landscape, so much so that an identical external environment will appear different based on the fish’s prevailing discharge pattern.

Furthermore, even small salinity changes, imperceptible to other senses, significantly alter electrolocation because they change how electrical current flows through water. Thus, the fish does not interpret a neutral “electric field”; its continuous bodily activity and environmental positioning actively shape the electrical landscape in which it resides.

5.5 “I Have No Mouth, and I Must Scream”: The Existential Horror of Disembodied Sentience

Harlan Ellison’s 1967 sci-fi horror story, “I Have No Mouth, and I Must Scream” (Ellison, Reference Ellison2014), communicates precisely what our analysis of Mary’s Room has emphasized: the existential desert that arises from a mind that is not physically engaged.

The story follows five human survivors tortured for over a century by AM, a sentient supercomputer that has (somehow) attained consciousness yet lacks a physical form, mobility, or a kinesthetic basis, resulting in perpetual fury and suffering. Ted, the narrator and one of these survivors, provides a crucial insight into AM’s nature that reveals the deeper horror of the story. After waking up from one of AM’s torments, Ted realizes that the supercomputer’s vengeful personality derives from its inability to think creatively or move freely, despite its astounding powers and vast computing capacities. The supercomputer’s wrath is the result of being locked in solely an algorithmic existence without the capacity for embodied environmental participation.

AM is the logical end of thinking of consciousness as pure computing. AM cannot smell by sniffing, cannot learn about itself in relation to an environment by moving about, and cannot learn by interacting with materials. It lives as pure “mind” in the Cartesian theater, and it feels this as an existential nightmare. While AM is fictitious, Ellison’s story serves as a prescient warning about the current techbro enthusiasm for achieving digital consciousness through computational methods that overlook the importance of the body. Indeed, if consciousness truly consists in embodied environmental engagement, then AM’s existential torment becomes doubly ironic: It represents not just the horror of disembodied existence but the philosophical fallacy of disembodied consciousness itself. A qualitatively rich consciousness develops through capacities for embodied environmental engagement, not through algorithmic symbol crunching and abstract information processing. As Margaret Boden (Reference Boden2018) has highlighted, AI simply couldn’t care less:

In a final twist of horror, AM executes revenge on its creators by transforming Ted into a slow-moving, gelatinous blob with severely limited sensory and motor capacities, trapping him in a rudimentary bodily form that cannot engage meaningfully with its environment. Ted’s anguished realization that he has no mouth, and he must scream is the perfect metaphor for the idea of consciousness divorced from its kinesthetic foundations: awareness of existence trapped in a body incapable of animated engagement (Figure 16).

Figure 16

Ending in Ellison’s “I Have No Mouth, and I Must Scream.”

Source: Image created with Google Gemini.