1. Introduction
Modern engineering design and development effectively mandates both physical and digital activities throughout its process (Reference Ulrich and EppingerUlrich & Eppinger, 2016). In the creation of objects and machines, stakeholders must engage with myriad ideas, tools, and technologies across both physical and digital representations while moving towards successful solutions. The form of representations can vary hugely, ranging from (I.e.) low fidelity physical mock-ups, to full production-ready prototypes, to knowledge databases, to digital simulation and analytic models. A divide often exists between physical and digital media (Reference Snider, Kent, Goudswaard and HicksSnider et al., 2022). While each play complementary roles during design, the need to transfer between domains, tools, and forms of representation introduces cost via the need for inter-domain knowledge transfer, increasing iteration time, complexity and time-cost of replication (i.e. digitising a physical prototype for subsequent analysis). Recently, a new technological paradigm is allowing blurring of the boundaries between physical and digital. Termed Spatial Computing (SC), these systems propose the integration of physical and digital worlds, such that dynamic physical reality is used to generate digital space (Reference CaoCao, 2024), enabling users to access, interact with, and augment digital content within the physical environment (Reference Yenduri, M, Maddikunta, Gadekallu, Jhaveri, Bandi, Chen, Wang, Shirawalmath, Ravishankar and WangYenduri et al., 2024). Importantly, the digital systems involved also maintain spatial awareness (Reference BellBell, 2023; Reference GreenwoldGreenwold, 2003), allowing them to react to, analyse, and augment physical elements. Such systems create a streamlined capability, allowing digital creation (i.e. modelling, visualisation) in context of physical objects / surroundings, and allowing digital (i.e. simulations) to react to real physical state. Proven benefits include reduced cycle time by reducing transfer between domains (Reference Kent, Snider and HicksKent, Snider, & Hicks, 2021), increased spatial awareness and knowledge development during training (Reference Bisson, Mahdjoub, Zare, Goutaudier, Ravier and SagotBisson et al., 2023) or design reviews (Reference Horvat, Martinec, Uremović and ŠkecHorvat et al., 2024), increased understanding during digital activity (Reference Kim and HyunKim & Hyun, 2022), higher technical accessibility across stakeholders (Reference Mariani, Kooijman, Shah and StoimenovaMariani et al., 2021), and increased immersion, awareness, and empathy (Reference Trump and ShealyTrump & Shealy, 2023).
Figure 1. Spatial systems: (left to right) physical / digital prototyping (Reference Cox, Gopsill, Snider and HicksCox et al., 2024); spatial design reviews (Reference Horvat, Martinec, Uremović and ŠkecHorvat et al., 2024), spatial workshop training (Reference Bisson, Mahdjoub, Zare, Goutaudier, Ravier and SagotBisson et al., 2023), OpenCap1
While recent extant research exists, within the design domain the application of SC requires further research. At time of writing, no papers in the Design Society repository return when searching for the term “spatial computing”. This paper aims to map existing understanding of SC to contribute an operational framework for SC in the design sector, identify underpinning technologies identify value propositions across design activities, and present initial challenges and opportunities for the field.
2. Spatial Engineering systems
Spatial Computing (SC) can be defined as any human-machine system in which the computer Footnote takes in and gives out data relative to real objects and spaces (Reference BellBell, 2023; Reference GreenwoldGreenwold, 2003). Others expand this to include the ‘combination of physical and virtual worlds, allowing users to perceive and interact with digital content in their physical environment’ (Reference Yenduri, M, Maddikunta, Gadekallu, Jhaveri, Bandi, Chen, Wang, Shirawalmath, Ravishankar and WangYenduri et al., 2024), the construction of digital space using physical data (Reference CaoCao, 2024), and the ability of devices to be aware of and digitally represent their surroundings (Reference Delmerico, Poranne, Bogo, Oleynikova, Vollenweider, Coros, Nieto and PollefeysDelmerico et al., 2022). It promises efficient, human-centric activity, with in-context information reacting to and augmenting the physical world. Timeliness is shown by prominence in industry, with Deloitte, Accenture, PwC, and Gartner all listing SC as a key trend for 2024/25Footnote 2.
Figure 2. (left) Prominence of research on “Spatial Computing” since 2000. (Right) Gartner Tech Trends for 2024, showing SC lower left as an ‘Innovation Trigger’
While traces of SC research may be found from 2003 (see Reference GreenwoldGreenwold, 2003), it has only drawn focus recently. Taking “spatial computing” as search criteria, Fig.2 shows 675 papers retrieved across IEEE and SpringerLink databases, with clear acceleration since 2023. Many older works are not relevant; pre-2020 many concern geo-spatial / geographic analysis (i.e. Shekar et al. (2015)), and pre-2010 many consider distributed computing (i.e. Reference Zambonelli, Mamei and SmirnovZambonelli et al. (2005)). It is hence evident that little academic research currently focuses on the major technical trend that SC is expected to become.
2.1. Operational elements of a Spatial Engineering system
To contextualise SC to the design domain, we refer to systems targeted for engineering as Spatial Engineering (SE) systems. Research in the XR domain has highlighted the core elements required for a human to interact with a mixed physical / digital system. Drawing initially from Milgram (1994) both the role of digital and physical domains must be considered, where each may represent objects (i.e. products, machines) and environments. Users must also be considered part of the system (Reference Kent, Snider and HicksKent, Snider, & Hicks, 2021; Reference Snider, Kent, Goudswaard and HicksSnider et al., 2022), both as input and receiver of information from it. Finally, the way in which the physical and digital elements interact with each other must also be considered, where options range from no interaction at all, to realistic interaction (i.e. digital ball bouncing on physical table), to augmented interactions (i.e. digital analyses overlaid on physical objects) (Reference Snider, Kukreja, Cox, Gopsill and KentSnider et al., 2024). Bringing these together gives a framework as shown in Fig.3 and Table 1.
Figure 3. Framework for spatial systems in design
Tabel 1. Description of elements within the Spatial Framework
The core elements of a Spatial Engineering system are the user, their environment, and the objects with which they interact. These are interconnected including across the physical / digital boundary, each with the ability to understand, interact with, and potentially manipulate all other elements (i.e. user picks up digital objects, digital objects react to physical environment, digital environment responds to user movement). This is enabled by a data system that manages capture, storage, and manipulation of digital elements, including alignment to the physical. Technologies must to enable elements and facilitate interaction between them, termed functional capabilities in Table 1. These include synchronisation; the ability to transition between physical and digital forms, interaction; the ability to interact between elements including the user, and compute; the ability to store, manipulate, and analyse system data.
2.2. Recent works in design
While not explicitly Spatial Engineering (SE) systems, design researchers have investigated technology and capabilities that are recognisably relevant to the concept. In particular, many have explored XR technologies for designers, as a 3D design tool (Reference Feeman, Wright and SalmonFeeman et al., 2018; Reference Nandy, Smith, Jennings, Kuniavsky, Hartmann and Goucher-LambertNandy et al., 2023), for sketching (Reference Seybold and MantwillSeybold & Mantwill, 2021), for product visualisation (Reference Goethem, Verlinden, Watts and VerwulgenGoethem et al., 2021; Reference Harlan, Goetz and WartzackHarlan et al., 2023), for design review (Reference Horvat, Martinec, Uremović and ŠkecHorvat et al., 2024; Reference Romero, Pinquié and NoëlRomero et al., 2021), for training (Reference Brunzini, Papetti, Germani and AdrarioBrunzini et al., 2021), and for increasing empathy and understanding (Reference Scurati, Dozio, Ferrise and BertoniScurati et al., 2023). The application spaces explored by these works are exceptionally broad and show potential for the design sector but also only scratch the surface (Reference Snider, Kukreja, Cox, Gopsill and KentSnider et al., 2024), with many employing only partial capabilities in their current toolsets.
Other works consider underpinning technologies and functions, including hand tracking (Reference Gopsill, Kukreja, Cox and SniderGopsill et al., 2024), gesture control (Reference Harlan, Goetz and WartzackHarlan et al., 2023; Reference Jain and JalluJain & Jallu, 2023), object tracking and digitisation (Reference Barhoush, Nanjappan, Thiel, Georgiev, Swapp and LoudonBarhoush et al., 2021; Reference Dammann, Steger and Paetzold-ByhainDammann et al., 2023), and digital human modelling (Reference Latif, Gong, Nanjappan and GeorgievLatif et al., 2023; Reference Ormerod, Dybvik, Fraser and SniderOrmerod et al., 2024). Much work is currently the focus of the human-computer interaction field, but as it reaches technological maturity moves increasingly into the realms of possibility for design researchers to study. One area of opportunity is the digitisation and representation of environments. It is well recognised that the environment plays a major role when designing (Reference Nanjappan, Uunila, Vaulanen, Välimaa and GeorgievNanjappan et al., 2023; Reference Scurati, Dozio, Ferrise and BertoniScurati et al., 2023), and using spatial technology scope exists to replicate, transfer, and manipulate this environment to (i.e.) create more realistic user experiences, or increase empathy of designers (Reference Trump and ShealyTrump & Shealy, 2023). Recent maturity of approaches such as Gaussian Splatting (Reference Kerbl, Kopanas, Leimkuhler and DrettakisKerbl et al., 2023) show photorealistic digitisation of any space in minutes. While increasingly numerous, relevant works in the design field often presented isolated cases and hence only a partial picture. While value has been asserted, there is a need for research that generalises and aggregates opportunities for the wider community.
3. Technical elements of Spatial Engineering systems
A key reason for the ‘Why Now?’ of Spatial Engineering lies in recent maturation of a range of technologies that blur the boundaries between the physical and digital (Reference Yenduri, M, Maddikunta, Gadekallu, Jhaveri, Bandi, Chen, Wang, Shirawalmath, Ravishankar and WangYenduri et al., 2024), principally Extended Reality technologies such as virtual and augmented reality (collectively termed XR) (Reference CaoCao, 2024) but also including improved communications (i.e. bluetooth, 5G), interaction technologies, Digital Twins, sensing and mapping, and system architectures (Reference Cronin and ScobleCronin & Scoble, 2020). This maturation is visible in XR, where rapidly growing capabilities are now approaching mainstream usage with approachable costs. Headsets have transitioned from low resolution virtual-only systems tethered to a PC, to high-fidelity, untethered, systems that combine physical and digital and use spatial interaction by default. Within the next 5 years headsets will similar in form to sunglasses, with a race now existing to reach the market first and analysts predicting importance akin to the invention of the mobile phoneFootnote 3. Taking functional capabilities from Table 1 and extracting from literature (Reference Cronin and ScobleCronin & Scoble, 2020; Reference Yenduri, M, Maddikunta, Gadekallu, Jhaveri, Bandi, Chen, Wang, Shirawalmath, Ravishankar and WangYenduri et al., 2024), several technologies are required for implementation of an SE system, see Fig.5.
Figure 4. Evolution of extended reality technology. left to right: NASA (∼1990), HTC Vive (2015), Microsoft Hololens (2016), Meta Quest 3 (2023), Meta Project Orion (late 2020s)
Figure 5. Technical components of Spatial Engineering systems
Representation
considers how spatial data, (i.e. models, information, analysis) are presented to the user. Within SE this forms a relatively well-studied area, with researchers considering differing forms of physical and virtual representation ranging from traditional monitors (Reference Horvat, Martinec, Uremović and ŠkecHorvat et al., 2024), to XR (Kent, Snider, Gopsill, et al., 2021), to methods that integrate domains (Reference Cox, Gopsill, Snider and HicksCox et al., 2024). Importantly, as SE systems incorporate both physical and digital elements they need not only implement XR visualisation methods, and instead should focus on appropriate balance of the affordances of differing physical and digital media against activity needs to best realise value (Reference Snider, Goudswaard, Ranscombe, Hao, Gopsill and HicksSnider et al., 2023).
Digitisation
considers how physical elements including the user are tracked and digitised within the system. This is a whole-system endeavour that, depending on application, may include object capture (i.e. geometry, position, dynamics), environment capture, digital human modelling, and bespoke sensing (i.e. IoT). Synchronicity and required precision are critical, with many technologies struggling to maintain useful levels of precision the real-time performance (Reference Kent, Snider and HicksKent, Snider, & Hicks, 2021). Technical research challenges are frequent in literature, with many studying how accuracy, robustness, and capture speed may be improved. For design, these must be considered in context of the requirements of the activity - prototyping for example often draws value from lower fidelity modelling, while high value manufacture and assembly requires precision of the order of microns. Equally, with a potentially large number of elements to digitise in any given activity or space, there is a need to consider priority and value of inclusion of elements to avoid excessive cost and technical overhead.
Interaction
considers how system elements interact, including user interaction. In addition to traditional interfaces, this may include tracked spatial interactions with physical or digital objects, machines, and environments (i.e. with AR menus or gesture control (Reference Jain and JalluJain & Jallu, 2023) or as-real physical interaction (Reference Cox, Gopsill, Snider and HicksCox et al., 2024)) and/or digital information overlays in the physical space. This spatial interaction mimics the tangible physicality of our interaction with the real world, and improves understanding, accessibility to technical information, and simplifies control. While much current research is considering technical capability of spatial interaction methods, questions also exist in the form of interaction best suited to different activities (Reference Harlan, Schleich and WartzackHarlan et al., 2021) and across stakeholder groups.
Computation
once physical components have been digitised, the flexibility and algorithmic power of digital tools are unlocked. This creates a wealth of opportunity for rapid, interactive analysis that grant users deeper understanding that would otherwise require a costly analysis loop and advanced technical expertise (Kent, Snider, Gopsill, et al., 2021). Researchers highlight the opportunity for AI techniques (Reference CaoCao, 2024), with ML approaches already common in some sectors (i.e. for digital human modelling), although more traditional analyses may also be leveraged (i.e. see (Reference Kent, Snider and HicksKent et al., 2019; Reference Shekhar, Feiner and ArefShekhar et al., 2015)). One key challenge lies in computational overhead given the complexity of spatial data, requirements of engineering analytic toolsets (i.e. CFD, FEA), and need for near real time response. Another lies in how analysis control and output can be best aligned with spatial interactions and representations, where human-controlled spatial interactions are typically imprecise, and options exist in how results are presented in context of both the physical and digital world around them.
Data Management
building on common understanding of the value of inter-connected digital systems brought forth by Digital Twinning, there is a need for a common platform to manage the array of data that SE requires. This is a challenging endeavour, requiring aggregation and management of multiple inputs and data types as well as the analyses, representation, and control methods that the system employs. Many implementations recognisable as SE use bespoke software built on (i.e.) game engine technology such as Unity and Unreal, although solutions such as Nvidia Omniverse and PTC Vuforia (itself manufacturing-focused) show broader promise. Focusing on the design context, specific challenges exist in incompatibility with data formats that are common in the engineering domain. Engineering is replete with highly structured formats (i.e. parametric CAD files) creating interoperability challenges when digitising and communicating between different system elements, and the surface geometry and point cloud data that many digitisation technologies currently employ. Further, as a data-heavy domain engineering creates challenge in the technical architectures required to enable streamlined communication and system management between a potentially high number of elements.
4. Spatial computing in engineering design
The preceding sections have framed the operational structure of SE, its underpinning technologies, and presented relevant works from the design field. This section situates SE in design, moving from ‘What’ into ‘Why’ and ‘When’. With close alignment between SE and XR, core references in this section were found by extracting all papers within the Design Society repository that mention ‘Augmented Reality’ or ‘Virtual Reality’ since 2021 31 papers remained following pruning, which rejected all papers that did not explicitly claim a value contribution or present an XR system applied to a design-relevant activity.
4.1. Proposed value of Spatial Engineering systems
The paper corpus was analysed to categorise papers that either explicitly considered value, or presented a working spatial system and proposed the value that it provided, see Table 2. Most highlight the improved understanding that use of these systems enables. Here, researchers highlighted improvement through a better appreciation for detail or system structure, wider simulation capability, higher degrees of immersion leading to empathy, and a lower bar to accessibility for a range of stakeholders. Of secondary focus was the reduced cycle time that these tools enable, typically due to the ability to spatial understanding and interact with as-physical systems while they remain in the digital world, and the quicker generation of understanding enabled by embedded information. Several other categories were also given in the work, ranging from social (communication and collaboration) to design-focused (creation and interaction), to efficiency (reduced cost and real-time information).
Tabel 2. Value propositions identified by extant works. References given in footnote4
4.2. Spatial Engineering across design activities
Opportunities for SE exist throughout the development process. Table 3 presents activities Footnote in which reviewed papers claim value, as well as aligned value propositions from Table 2.
Tabel 3. Activities investigated for SE systems in design. Letters indicate value, see Table 2
The majority of examples focus on either design activities (i.e. modelling [5], configuration [14][30], CAD [3][29]) or design reviews, either as formal tools to support reviews (i.e. [21]) or as visualisation tools to better understand systems (i.e. [17][19]). The improved spatial awareness gives clear benefit, although there remain open questions regarding good practice implementation, scenarios of highest value, and avoidance of negative effects, such as increased fixation. Beyond these, literature shows examples of SE approaches applied to many activities across the design and development process, from early ideation phases through to production. Across these the form of implementation changes greatly, from 3D sketching systems (i.e. [1]) to synchronised physical / digital objects (i.e. [12]) to digital human models (i.e. [11]) to information visualisation systems (i.e. [10]). While this provides little guidance on the nature of implementation for a specific case or broadest set of capabilities for the a wide range of cases, it does show that the promise of SE systems appears possible to realise - that it may be applied to the majority of the process, and generate better understanding with improved efficiency in many.
5. Challenges for Spatial Engineering systems
While examples of SE show value they are specific in their capabilities, broad in their implementations, and typically developed as part of research studies. There are few generic toolsets and no ‘killer apps’ to drive industry uptake. Current implementations do not have the range of functionality that designers need. With much research deriving from technology rather than application domains (Reference Xu, Papangelis, Tigwell, Lalone, Zhou, Saker, Chamberlain, Dunham, Luna and SchwartzXu et al., 2024), SE currently lacks refinement, generalisability, and knowledge of scope for engineering design. There is hence a need to expand, aggregate, and frame the use of spatial toolchains, such that as technology develops it may be efficiently leveraged. This section explores challenges and opportunities for the research domain, to drive the development and use of SE in engineering design.
How should SE systems be tailored to support engineering design?
Design is reliant on both physical and digital media with frequent transitions between the two. As a process, design can be understood as an iterative learning cycle, where partial solutions are created and explored to refine knowledge of the problem, then spurring another, more refined set of ideas (Reference Smulders, Reyman and DorstSmulders et al., 2009). In this context, the promise of SE systems is substantial. Integrating physical and digital allows streamlined exploration of ideas and designs earlier in design with reduced cycle time, allowing the benefits of each domain to be simultaneously leveraged. Inclusion of user, object, and environment creates an immersive and explorative world, where designs can be understood in-situ, rapidly iterated, physically experienced, digitally analysed, and tested across stakeholders earlier in the process. Allowing human-centric interaction with digital and physical representations creates an accessible, understandable way of working that supports decision making and reduces design cycle time.
While relevant systems exist and have shown value, there is little guidance on how SE should be tailored to any specific scenario. Each element (user, environment, object) can be represented in many ways, with higher time, cost, and skill required for more sophisticated forms. During their processes, designers explore numerous partial representations in a wide range of ways, and spatial tools must able to support this breadth. Currently there are no standardised SE tools to streamline the creation and testing process and limited compatibility with engineering platforms, requiring bespoke creation of SE systems. With this degree of variety, uncertainty, and potential for cost, it is critical that focused effort considers how SE tools should develop in order to best align with the design process and its requirements, both to establish core functionalities and good practice for implementation.
What are the technical challenges for SE tools?
Section 3 presented five technical capabilities for SE: digitisation, representation, interaction, computation, and data management. While spatial representation, review, and visualisation have been explored by several, digitisation of designs is less well considered. Particularly where a designer is working physically this is a significant technical challenge; rapidly and accurately capturing geometry, motion, and function is difficult and often costly. Similarly, while object positional tracking methods exist, cheap options lack precision, and unintrusive systems (i.e. OptiTrack) are prohibitively costly. Similar challenges exist in digitisation of other system elements, with user-tracking becoming more common but often at lower fidelities, and environment digitisation often requiring expensive equipment and significant processing. As technology develops there is a need to map requirements for design, impact on implementation, and how sophistication of system elements can be balanced between cost and capability. Open questions also exist in how best to interact with SE systems. While gesture and hand-tracked control is often more natural it lacks the depth of control and precision of traditional interfaces. SE interfaces must react and operate across domains; digital elements must respond to physical inputs, and physical elements need to respond to digital inputs. This creates technical and UI research opportunities, exploring how interfacing can align with intuition and accessibility, maintain control and precision, and operate across physical/digital domains.
Challenges in data management align with those seen by other systems. Spatial tools have high computational overhead, with challenges in interoperability, bandwidth, and processing. Challenges also exist in the translation of current spatial tools to the design sector. Coming primarily from the fields of entertainment and human-computer interaction, spatial technologies are not natively compatible with engineering technologies; i.e. surface models used in spatial tools are not compatible with the parametric models required for engineering CAD. There is then a need for either translation or redevelopment to integrate the data and computational requirements that design imposes.
A substantial opportunity exists in application of analyses in spatial tools. As they digitise by default and are replete with data, they are a prime candidate for advanced analytic techniques. Opportunities exist in the creation of intuitive analytics that respond to physical, digital, and user interaction, supplementing learning by exploring performance spatially. While some in literature do include live analysis this is an understudied area, with high potential for emergent value.
What is the balance between value and cost, and where do the strongest opportunities lie?
With current state of technology and high computational overhead it is important to consider the technical cost associated with SE development. With few commercially available tools outside of sketching systems (i.e. GravitySketch), time and effort currently need to be given to develop bespoke tools. While barriers to entry are decreasing and commercial platforms targeted to engineering are emerging (i.e. PTC Vuforia), there remains a cost to early adoption that must be taken into account. For researchers opportunity exists in considering how SE may be implemented to balance this cost and the value that the system provides. Rather than assuming that SE should maximise novelty, there should be consideration of how each element is employed in its simplest form to support the activity and achieve value; i.e. using traditional interfaces, reducing fidelity, or minimising physical tracking. As technology improves so may sophistication increase, but simplest useful systems must be considered first.
6. Conclusions and outlook
The blurring of physical and digital that Spatial Computing offers holds clear benefits in the engineering design domain, where use of and transition between many physical and digital representations is part of day-to-day operation. While few have directly studied it as a named concept, there has been rapid recent growth in research in related fields, particularly through XR design systems. This paper has shown the value of SC across near-all stages of the development process and particularly in human-centric and designer-led activities, where they are shown to improve understanding and reduce cycle time. From recently emerged extant research this paper has extracted and contributed an operational framework for Spatial Engineering systems that leverage the benefits of SC in the design context, and five technical capabilities that such systems require to operate. It has also highlighted a range of operational and technical challenges for the community, centred primarily on establishing good practice, system interface and technology, and balancing creation cost with the value that such systems provide.
Open access
