2.1. Early childhood second language acquisition

Second language acquisition (SLA) in early childhood is grounded in the critical period hypothesis, which posits that language learning capacity is maximized during the first decade of life due to increased neural plasticity [Reference Lenneberg1]. Empirical studies confirm that preschool students acquire syntactic structures, phonological distinctions, and vocabulary with greater ease and long-term retention when exposed early to L2 environments [Reference Hartshorne, Tenenbaum and Pinker2, Reference Unsworth3]. In particular, ESL instruction in non-English-speaking countries is often introduced in early education settings to leverage this developmental advantage. However, TLLL in preschools frequently fails to sustain attention or offer differentiated input tailored to individual learners’ cognitive states [Reference Krashen4, Reference Hsin, Li and Tsai5].

Sociocultural learning theory emphasizes that linguistic development occurs most effectively in socially mediated contexts where children co-construct meaning through guided interaction [Reference Vygotsky6]. In early childhood education, this suggests the need for responsive and interactive instruction that supports scaffolding and immediate feedback. Despite this, ESL instruction often lacks multi-modal input, embodied experiences, and adaptivity – factors that are essential for young learners who rely heavily on sensorimotor processing.

2.2. Robot-assisted language learning

RALL has emerged as a promising approach for addressing the cognitive and attentional limitations of conventional ESL teaching. Social robots leverage physical embodiment, gesture, and speech to engage children in verbal and non-verbal interaction. Studies have shown that young learners exhibit increased engagement, verbal output, and motivation when interacting with socially assistive robots compared to screen-based applications [Reference Belpaeme, Kennedy, Ramachandran, Scassellati and Tanaka7, Reference Kory and Breazeal8]. RALL systems also offer consistency and repeatability in content delivery, an advantage over human instruction in environments where teacher resources are limited.

Multiple studies have demonstrated the potential of robots to reinforce vocabulary acquisition, assist pronunciation, and support turn-taking behavior [Reference Van den Berghe, Verhagen, Oudgenoeg-Paz, Van Der Ven and Leseman9, Reference Xueqing and Li10]. However, most existing RALL systems are characterized by limited adaptability. Interaction flows are typically pre-scripted, with little response to real-time emotional states or levels of comprehension of learners [Reference Ho and Lim12]. Moreover, physical embodiment is often underutilized: many platforms rely solely on verbal instruction or animated facial expressions without integrating gesture, movement, or object manipulation [Reference Tanaka, Takahashi and Isshiki13]. In some systems, robots act merely as animated storytellers or question prompts, rather than as embodied, task-oriented tutors.

Few studies have implemented RALL in authentic preschool environments with participants who have no prior exposure to English. Most are carried out under laboratory conditions, with short intervention periods and limited ecological validity [Reference Iio, Shinozaki and Tanaka14]. Furthermore, system evaluation is often limited to short-term vocabulary gains, rather than comprehensive measures of engagement, interaction quality, or long-term retention.

2.3. Multi-modal interaction in language learning

Multi-modal interaction – defined as the integration of verbal, visual, and kinesthetic stimuli – has long been identified as a critical factor in enhancing early language learning outcomes. Based on dual coding theory [Reference Paivio15] and supported by the principles of embodied cognition [Reference Brown and Ellis16], multi-modal instruction reinforces semantic encoding by associating spoken words with concrete sensory input. In the context of preschool ESL learners, combining auditory instruction with visual imagery, touchable objects, and physical actions improves attention, comprehension, and recall [Reference Brown and Ellis16, Reference Blazejowska, Gruba, Indurkhya and Gunia17].

Some RALL studies have attempted to introduce multi-modal elements through gesture recognition, visual displays, or tablet-based interfaces [Reference Ho and Lim12, Reference Tanaka, Takahashi and Isshiki13]. However, few systems implement true sensory integration, where verbal, visual, and kinesthetic feedback are synchronized in real time and tailored to the learner’s affective state. Coordination across modalities is essential in preschool contexts, where embodied interaction and concrete associations support early semantic development. The present study addresses this gap by integrating physical pick-and-place tasks, robot-led drawing, and dynamic, emotion-aware feedback into a unified multi-modal framework for ESL instruction.

2.4. Emotion-adaptive systems in educational robotics

Affective responsiveness, the ability of an educational system to interpret and respond to the emotional states of learners, has become an increasingly important focus in CRI. Preschool learners are emotionally reactive and require frequent validation and encouragement to maintain learning focus. Real-time affective cues such as facial expressions, hesitation, or gaze aversion offer valuable indicators of readiness or disengagement of the learner [Reference Breazeal, Gordon and Morency18]. Educational robots that detect and adapt to these cues have been shown to improve learning outcomes, user satisfaction, and session duration [Reference Finders, Wilson and Duncan19, Reference Ackerman and Friedman-Krauss20].

Despite its relevance, few RALL systems incorporate real-time emotion recognition or affective adaptation. Interaction scripts are often fixed and feedback is triggered by static event thresholds rather than context-sensitive cues [Reference Breazeal, Gordon and Morency18, Reference Park and Kim21]. Integration of emotion recognition in early ESL learning remains underexplored in RALL research, especially in developmental settings like preschools.

2.5. Research gap and system positioning

Although prior research affirms the value of RALL in improving early language learning, several persistent limitations remain. Most existing systems lack coordinated multi-modal integration and operate without affective adaptivity. Their implementations are often short-term, decontextualized, and do not reflect the complexity of real-world preschool settings. Few studies incorporate physical manipulation of objects or robot-led motor tasks that link vocabulary to embodied experience. Furthermore, systems are rarely evaluated using both quantitative and behavioral metrics under ecologically valid classroom conditions.

The present study addresses these limitations by developing a multi-modal, embodied, emotion-adaptive framework for ESL instruction in preschoolers. The proposed system integrates physical manipulation, collaborative drawing, and adaptive verbal cues in a unified robot-tablet platform. It is evaluated through an experimental design within the subject in an authentic Korean preschool setting, offering new empirical insights into the design and deployment of scalable, developmentally appropriate RALL systems.

3.1. Hardware configuration

The physical components of the system were assembled to balance expressive interaction with usability and safety for children aged 4–8. The robotic arm used is the OpenManipulator-X (ROBOTIS) [25], a 4-DOF platform mounted on a standard classroom table. Motor ID14, which is part of the OpenManipulator-X arm, was equipped with a custom pen holder to enable the robot to draw predefined shapes (e.g., heart, house, sun, circle, square, and others) on paper and whiteboard surfaces using calculated Cartesian paths translated into joint-space trajectories. A U2D2 communication controller enabled real-time motor commands.

The robotic arm was controlled using a desktop computer running Ubuntu 20.04 and ROS Noetic. Communication between the robot and the computer was facilitated via the U2D2 interface, which allowed for real-time execution of motion commands. The robot’s tasks – including both drawing and pick-and-place actions – were implemented by running custom-developed Python and C++ scripts based on the official OpenManipulator ROS packages.

For facial and emotional detection and expression, the system utilized a display with an integrated camera on the top that visualized animated facial expressions in response to the child’s affective state. A Logitech C920 USB camera, positioned at the child’s eye level in front of the table, captured facial data in real time for emotion classification. The tablet interface, implemented on an iPad (3rd generation, 2021, 10.2-inch), served as the primary platform for educational games and digital feedback, enabling a two-channel interaction environment: physical (robot) and digital (tablet). Two external speakers were connected to the main computing unit to deliver clear auditory output. Together, these components constituted a multi-modal configuration designed to facilitate socially engaging and developmentally appropriate CRI.

3.2. Software architecture and module integration

The system architecture was designed with modularity and real-time responsiveness as core principles. ROS was used to control robot actions and synchronize service calls between nodes responsible for drawing, pick-and-place, and timed motion sequences. The SetDrawingTrajectory and goal_task_client() services facilitated parameterized execution of drawing tasks, with inverse kinematics ensuring consistent pen-surface contact (Figure 2) [25]. Moreover, the pick-and-place task was implemented through the detection of AR markers using an Intel RealSense camera, enabling the robot to accurately identify initial cube positions and relocate them to predefined target locations (Figure 2).

A Streamlit-based EduApp (Figure 3), launched on the iPad, offered four structured learning tasks: vocabulary learning, math problems, color matching, and a word scramble game. The vocabulary content used in the application was derived from the Cambridge English: Pre A1 Starters Word List Picture Book to ensure age-appropriate and pedagogically aligned material [26]. The application was developed independently from the ROS node, but manually synchronized during sessions to maintain a coherent instructional flow.

OpenAI-generated Text-to-Speech (TTS) files [27] were generated for all vocabulary items, instructions, and motivational phrases. These were manually triggered by the operator in coordination with task progression during the interaction between child and robot for drawing and drawing tasks, allowing adaptive pacing and tone modulation in real time.

Figure 2.

Robot-executed drawing and vocabulary-based pick-and-place tasks using OpenManipulator-X to reinforce embodied ESL learning.

Figure 3.

Tablet-based EduApp interface with vocabulary, math, color-matching, and scramble tasks supporting robot-led instruction.

3.3. Multi-modal interaction

Each learning session was divided into two sequential phases: robot-led physical interaction followed by tablet-based digital reinforcement. During the first phase, the robot executed pick-and-place tasks using AR-labeled vocabulary cubes or drew shapes symbolizing target words (e.g., “sun”, “cloud”) (Figure 2). Children were prompted to repeat the words or describe the object being drawn, anchoring vocabulary in sensorimotor experience.

In the second phase, children interacted with the tablet application, which offered tasks aligned with the robot-led activities. These included visual-matching tasks, math exercises, color-word associations, and scrambled word challenges (Figure 3) [Reference Cox and Haebig29–Reference Engwall and Lopes32]. Each task featured animated feedback, embedded videos, and narration through TTS audio. The sequence was manually timed by the instructor to ensure smooth handoff between the physical and digital domains.

3.4. Emotional and facial expression and recognition

The system included a lightweight affective feedback loop based on facial expression recognition [Reference Shahab, Taheri, Mokhtari, AsemanRafat, Kermanshah, Shariati and Meghdari34]. Using a webcam mounted at eye level, the system captured the real-time emotional state of children and classified expressions into five categories: neutral, happy, sad, angry, or surprised (Figure 4). The classification was implemented in Python using the DeepFace library [Reference Serengil28], which internally supports models for real-time facial expression analysis. These detected emotions were displayed on a separate screen as an animated face that mimicked the child’s emotional state, providing intuitive and immediate visual feedback.

Although the robot did not autonomously change behavior based on emotion input, the mirroring effect encouraged self-awareness and allowed the instructor to adapt the pace or offer verbal reinforcement accordingly. This mechanism was particularly effective in maintaining engagement during longer sessions and provided a foundation for future work on fully autonomous emotional adaptation. The system emphasized emotional transparency and safety over complexity, aligning with the developmental needs of early childhood.

3.5. Design priorities and constrains

The framework was built around three guiding principles: developmental appropriateness, modular synchrony, and scalability. Tasks were short, goal-oriented, and visually reinforced. Robot movement was fine-tuned through pilot testing with adults and adjusted based on children’s observed engagement levels. The use of manual synchronization (rather than full automation) was a deliberate choice to respect natural instructional pacing and maintain operational control in a dynamic classroom environment.

Although the robot and tablet were not fully integrated via shared control architecture, their coordinated deployment created a unified learning experience. The design demonstrated that meaningful robot-assisted ESL learning is achievable without high computational complexity – provided that interaction is structured, feedback is responsive, and engagement is multi-sensory.

4.1. Participants

A total of 22 Korean preschool children, aged 4–8 years, were initially recruited for the study. Fourteen children participated in Round 1 (9 sessions), and an additional eight were enrolled in Round 2 (4 sessions). However, one child from Round 2 was excluded from the final analysis due to insufficient participation (attendance below 80% of the sessions), resulting in a final sample of 21 children. None of the children had received prior formal instruction in English.

To provide a clearer developmental profile, the final sample comprised 3 children aged 4 years, 8 children aged 5 years, 4 children aged 6 years, 5 children aged 7 years, and 1 child aged 8 years, spanning the transition from preschool to early primary education. Of the 21 children, 16 were boys and 5 were girls. The participants were recruited from a daycare center affiliated with the Korea Institute of Science and Technology (KIST), and all experimental procedures were approved by the Institutional Review Board (IRB) of the institute (approval number KIST-202412-HR-001). Written informed consent was obtained from the children’s guardians, and verbal assent was obtained from each child prior to participation. To ensure instructional focus and minimize group-induced distractions, children were divided into small sub-groups during sessions. Tasks were conducted in a dedicated CRI space, with alternating robot-led and tablet-based activities designed to maintain engagement and reduce fatigue.

In the South Korean educational system, children between the ages of 4 and 7 typically attend kindergarten, while formal elementary school begins at age 8. Therefore, the selected age range of 4–8 years represents the upper spectrum of early childhood education within this national context. This period is particularly formative for language acquisition, cognitive flexibility, and socio-emotional development. The RALL framework was intentionally designed to support learners across this transitional developmental span through differentiated, scaffolded interactions and age-appropriate multi-modal engagement strategies. By targeting this age group, the study aims to evaluate the framework’s adaptability and instructional effectiveness during a critical window for early second language learning.

To ensure instructional focus and avoid group-induced distractions, children were divided into small subgroups during sessions. Tasks were conducted in a dedicated CRI space, with alternating robot-led and tablet-based activities designed to maintain engagement and reduce fatigue.

4.2. Experimental design

The study adopted a within-subject crossover design to enable direct comparison between the two instructional modalities – TLLL and RALL – under equivalent task conditions (Figure 5). Each child participated in both instructional modes over the course of multiple sessions, allowing matched data collection across conditions.

The experiment consisted of two rounds:

• • Round 1 involved 14 participants and consisted of 9 sessions: 4 TLLL sessions, 4 RALL sessions, and 1 debrief and survey session (Figure 6).

• • Round 2 involved 7 participants and included 4 sessions: 2 TLLL and 2 RALL. To control for order effects, the instructional sequence was reversed in Round 2, with RALL delivered before TLLL (Figure 7).

Each session lasted approximately 60–65 min, encompassing robot-assisted or teacher-led instruction, interactive tablet engagement, and observational or survey-based data collection. The instructional content and complexity of the task were kept constant under all conditions to ensure a fair comparison.

Figure 6.

Round 1 experimental sessions comparing TLLL (left) and RALL (right) modalities in real classroom setting.

Figure 7.

Round 2 instructional sequence: reversed order of RALL (left) and TLLL (right) conditions for crossover validation.

All instructional content was based on beginner-level ESL vocabulary selected from the Cambridge curriculum [26], adapted for preschool comprehension and engagement. Tasks were thematically aligned across both instructional modalities. The RALL condition, however, uniquely incorporated robotic embodiment through pick-and-place manipulation and drawing trajectories, in addition to synchronized verbal and digital feedback.

The implemented tasks included:

• • Vocabulary Learning: In both conditions, English vocabulary items were introduced per session. In TLLL, the teacher explained each word verbally and used flashcards or printed visuals. In RALL, the robot vocalized the target words using pre-recorded TTS audio, while the tablet displayed animated images of the objects. The children were encouraged to repeat each word aloud in both conditions.

• • Pick-and-place: To reinforce vocabulary through embodied action, the robot manipulated labeled cubes or toy objects associated with the vocabulary items (e.g., “sun”, “apple”, “cloud”). The robot identified the correct item using predefined AR marker IDs and performed a pick-and-place movement to place it in a matching bin or zone. This task involved spatial reasoning, object-word association, and joint attention.

• • Drawing Task: Using a pen attached to the gripper of the OpenManipulator-X, the robot performed shape-drawing tasks on A4 paper and later switched to the whiteboard fixed in the workspace. Each shape corresponded to a target vocabulary word (e.g., heart, house, smiley, sun). Drawings were executed using pre-programmed joint trajectories derived from 2D coordinates and implemented via the SetDrawingTrajectory ROS service. The children were prompted to identify the drawn object and verbally describe it.

• • Math and Color-Matching Tasks: Children engaged in simple arithmetic and color-categorization games. These were implemented as drag-and-drop activities in the tablet interface during RALL, and as verbal or card-based games during TLLL. The robot narrated instructions in RALL sessions using synchronized audio and visual cues.

• • Word Scramble Game: To assess vocabulary retention and spelling recognition, the children completed a scramble game in which they reconstructed target words from jumbled letters. The RALL version used the tablet interface with input and submit functionality, while the TLLL version involved teacher-facilitated oral guidance.

Each session in RALL began with robot-led physical interaction (drawing or pick-and-place), followed by digital tablet-based engagement. In contrast, TLLL sessions followed a linear format with verbal instruction, guided repetition, and teacher-directed activities. The tasks were intentionally short and interactive, with break intervals to maintain attention.

4.3. Data collection and evaluation measures

To evaluate the effectiveness of the proposed framework, a mixed-methods data collection strategy was used, which incorporated both quantitative and qualitative measures. The study focused on capturing instructional outcomes across the two conditions – RALL and TLLL – by assessing learner performance, engagement, emotional response, and subjective feedback.

Following the completion of each instructional condition, children participated in structured, one-on-one oral surveys administered in English by a trained Korean research assistant (Figure 8). These surveys were designed to be developmentally appropriate for young learners and employed a simplified 5-point Likert-type scale, ranging from 1 (“not at all”) to 5 (“very much”). The assistant provided consistent verbal scaffolding, supportive hand gestures, and visual cues to help children understand each item and respond meaningfully. This support enabled reliable data collection across the full 4–8 age range. To ensure accessibility, responses were collected with the aid of consistent verbal prompts and hand gestures. The survey assessed self-reported understanding, ease of learning, confidence, and enjoyment across four task categories: vocabulary learning, mathematics, color matching, and scrambled word reconstruction. In the RALL condition, additional items probed children’s impressions of robot behavior, ease of tablet interaction, and the perceived helpfulness of the integrated system.

Figure 8.

Structured post-session user experience survey conducted with children following TLLL and RALL sessions.

All instructional tasks – including vocabulary learning, mathematics, color matching, and word reconstruction – were conducted primarily in English, aligning with the ESL learning objectives of the study. For tasks with less inherent linguistic content (e.g., color matching and mathematics), instructions and prompts were delivered in simple English, accompanied by visual aids and occasional Korean clarification to support comprehension. This hybrid linguistic strategy was intended to balance meaningful English exposure with cognitive accessibility. Children were encouraged to respond in English when possible, although Korean responses were accepted without penalty. This flexible approach fostered a supportive, low-pressure environment conducive to language development and task engagement.

The mathematics task was structured as an interactive problem-solving activity in which children were presented with simple visual number problems and prompted to input their answers using the tablet interface. Instructions and questions were delivered in simple English (e.g., “What is 3 + 2?”), accompanied by intuitive visual cues to support comprehension. Upon submission, children were shown a short educational animation in English that explained the solution through gamified storytelling. This format helped reinforce early arithmetic reasoning while simultaneously providing consistent exposure to instructional English in an engaging, age-appropriate manner.

The color-matching task focused on supporting receptive and productive vocabulary skills through guided multi-modal interaction. Each session began with a short English-language animated video introducing basic color names and associated objects (e.g., “red apple”, “blue sky”). Following the video, children were shown an image and asked in English, “What color is it?” They were then required to type the correct English color name (e.g., “yellow”, “green”) and submit their answer. Immediate feedback indicated whether their response was correct or incorrect, reinforcing both spelling and semantic understanding. While the task was primarily conducted in English, minimal Korean support was offered when necessary to ensure comprehension. This sequence enabled children to first internalize the vocabulary and then apply it in a low-pressure, interactive setting, maintaining alignment with the study’s ESL learning objectives.

In parallel, behavioral observations were recorded during each session using structured annotation protocols. Observational data focused on attentional markers (e.g., gaze direction, verbal responsiveness), behavioral cues indicating cognitive effort or confusion, and spontaneous emotional expressions. The observer also noted levels of physical engagement, including participation in robot-led drawing and pick-and-place tasks.

In the RALL sessions, real-time affective data were captured using a webcam positioned at eye level with the participant. The system classified facial expressions into five categories: neutral, happy, sad, angry, and surprised – using a lightweight convolutional neural network (CNN)-based model trained on child-appropriate datasets. The implementation was supported by OpenCV and TensorFlow libraries, enabling efficient image capture and classification on the local system. Although the robot did not autonomously adapt to emotional input, mirroring of facial expressions on an adjacent display served to increase emotional awareness and indirectly supported session pacing. These emotional logs were used as supplemental data to interpret overall engagement and response.

Quantitative analysis was conducted using paired-sample t-tests to examine differences between instructional conditions for each task and survey item. Repeated measures ANOVA was used to assess instructional effects between rounds and to assess whether instructional order influenced the outcomes. This method is particularly suitable for within-subject designs, as it accounts for the correlation between repeated observations from the same individuals. By partitioning the variance into within- and between-subject components, ANOVA provides a robust framework for examining changes over time or across multiple interventions, while controlling for individual variability. Where appropriate, the effect sizes were calculated using Cohen’s d and f to provide estimates of practical significance.

To complement the numerical results, open-ended responses from the surveys were thematically analyzed. Children frequently highlighted the robot voice, patient pacing, and visual gestures as helpful in remembering words and completing tasks. Comments such as “the robot waits until I’m ready” or “it helps me understand” reflected a high degree of perceived comfort with the system. These qualitative findings reinforced quantitative trends and were consistent in both rounds of instruction, suggesting that the positive reception of the robot was not simply a novelty effect but indicative of the alignment of the framework with the cognitive and emotional needs of preschool learners.

4.4. Ethical and developmental considerations

The experiment was carried out according to the ethical guidelines for research with minors. IRB (KIST-202412-HR-001) approval was secured, and informed consent was obtained from all legal guardians. Sessions were designed to be playful, low-stress, and developmentally appropriate. All interaction cues were delivered at a pace suitable for preschoolers, and robot movements were optimized for safety and clarity. Facial data used for emotion recognition was processed in real time and not stored beyond session classification logs, and all data were anonymized prior to analysis.