Development of adaptive human–computer interaction games to evaluate attention

Abstract In this paper, several physical activity-based human–computer interaction (HCI) games which are developed and implemented for the improvement of attention, emotion, and sensory–motor coordination will be presented. The interface and the difficulty levels of these games are specially designed for the use of people with different age groups and disabilities. The games involve physical activities for the fulfillment of some basic HCI tasks which require hand and arm motion for control, such as fruit picking and air hockey, with adaptive difficulty levels based on varying parameters of the games and human performance. In the fruit picking game, several fruit images are moving from top to the bottom of the screen. Objective is to collect apples while avoiding the pears. The player’s hand will control the basket that collects the fruits. In the air hockey game, the player will try to score goals against computer-controlled opponent. The player’s hand will control the paddle to hit the puck to score or to defend his/her goal area. The player’s hand is recognized by Kinect RGB-D sensors in both games. Aim of the adaptive difficulty-based system is keeping the players engaged in the games. The games are tested with a group of deaf children (3.5–5 years) as a part of an ongoing project,1 to decrease the stress of the children and increase their positive emotions, attention, and sensory–motor coordination before the audiology tests. The game performances and the evaluation of the therapists show that the games have a positive impact on the children. The games are also tested with a group of adults as a control group, where a mobile EEG device is employed to detect the attention levels. For this purpose, the adults also attended a third game featuring a maze and controlled with Myo sensors.


Introduction
In the recent years, game playing is successfully used for increasing the learning capacity or developing cognitive skills [1,2,3,4,5,6,19]. Games that are not designed for special purposes can also assist in the development of such skills as presented in ref. [7]. Some games encourage the players to develop strategies in the game and concentrate on the game in order to improve their performance [36]. Game playing is useful for this purpose, since it assists in developing attention. Attention indicates the intensity of mental focus. The attention level increases when a user focuses on a single thought or an external object and decreases when distracted. Game playing is also important in the development of planning and sensory-motor coordination skills, and people tend to spend more time or get motivated for playing games than spending time with other educational or therapeutic activities that are designed and implemented for the same purpose [9]. But a main drawback of this approach is the requirement of sitting idle without any physical activity for a long time for some of the games. While these games might still help in developing cognitive skills, in the long term they can become physically unhealthy. Also, physical activities themselves help developing cognitive skills since they encourage sensory-motor coordination [35]. The positive effects of an after school physical activity program on working memory in preadolescent children are reported in ref [16]. Although there is a large number of studies on general purpose exergames as well as movement-based education games, a comprehensive set of guidelines to guide the game design process is lacking. There is a detailed review on physical activity-based studies and suggested design phases, design requirements, and corresponding design decisions presented in ref. [22] for adolescents. Combining physical activities and game playing is beneficial to improve the learning experience while avoiding remaining idle for extended amounts of time. Jones et al. discuss the need to support the classroom activities by promoting more physical activity to reduce sedentary and idle time in children in their detailed review [14].
In this work, two Kinect-based games and a Myo Armband-based game are designed and implemented in order to show that physical activity-based games support the development of cognitive skills by encouraging the participant's increased attention in order to be successful in the game. The games are summarized as follows:

Fruit picking game
In this game, players will try to pick fruits in the screen by their movement. This game will motivate users to be engaged in physical activities while implementing strategies to pick most of the fruits to maximize their game score. Since fruits will fall with different velocities, players will try to prioritize fruits to pick, and this will encourage them to be engaged in cognitive activities as well. Due to these reasons, this game is suitable for encouraging both physical and cognitive activities as well as this research.

Air hockey game
Players will try to score against computer player using the motion of their hands. This game is a heavily based on physical activity, but it still requires cognitive activities as well. Player might want to predict where the puck is going to be, where and how fast they should hit the puck in order to score a goal. In order to implement these strategies, players will be engaged in heavy cognitive activities as well.

Myo Armband-based game
This game aims to improve the cognitive skills. In the game, Myo Gesture Control Armband ( Fig. 1) is employed for the game control. The Myo Armband is worn to wrist/lower arm and uses muscle data to detect the hand/finger/wrist motion. We employed Up, Down, Left, and Right arm moves in the game for possible control actions (Fig. 2). These moves of the players are detected by an artificial neural networks-based approach.
The games are tested with a group of deaf children (3.5-5 years) as a part of an ongoing project "Roborehab: An Assistive Audiology Rehabilitation Robot," in collaboration with the audiology research group of a Medical School, where the main goal is to decrease the stress of the children and improve their positive emotions, attention, and sensory-motor coordination during the audiology tests. For this purpose, the project involves human-computer interaction (HCI) and human-robot interaction (HRI) games, different sensors, tablets, and a socially assistive robot (Pepper humanoid robot) [40]. This paper presents the first step of the project. In the current study, we also aimed to design and develop games which are simple yet enjoyable enough to be played by the children with hearing impairments and enable the researchers in this project to use the different sensors to collect the required data to fulfill the aim of the project. Within this study, the games are also tested with a group of adults as a control group, where a mobile electroencephalogram (EEG) device is employed to detect the attention levels, and Myo sensor as a controller. The adult control group is used to test and verify the game design and sensory setup before the tests with children. Detailed discussion and statistical analysis on the results are presented. The children are tested using the EEG device in the second step of the study; therefore, the device is not integrated into their game sessions presented in this paper.

Related work
In the literature, there are several physical activity-based HCI studies. A research for studying the effects of Kinect-based games on students with special educational needs is presented in ref. [19]. The survey presents several Kinect-based games that have been designed for learning purposes and categorizes them according to their goals and ability to adapt and analyze. Some of these studies have shown that the physical activity-based games help to improve cognitive skills, help students to improve their concentration, and to get better scores on academic tests. And finally, it has shown that these type of games have positive impact on children's cognitive and motor skills by employing experiments that consist of pretest and posttest questionnaires, and interviews of the game, and analyses on the game results.
Role of attention in implicit learning is investigated in ref. [36]. This paper presents several studies investigating the relation between attention and the memory. In one case, participants cannot recognize the word that is presented to one ear, while a shadowed message is given to the other ear. Another study includes participants that are assigned to multiple tasks at the same time thus leading to an divided attention. These experiments' results suggest that divided attention disrupts the learning process. Refs. [8,32,33] discuss further aspects of attention and learning.
In addition to these studies, game-based learning is investigated in refs. [7,9,12,17,18,20,21,23,25,28,39,41,42]. In ref. [7], an RPG (role-playing game) involving geological contents is implemented. In ref. [9], a literature review is provided. It summarizes the effects of computer games on children's motivation and learning. It also investigates the research on mathematical games that aim to support the attitude of children of different ages toward the subject of mathematics. Importance of the attention on learning is investigated in ref. [36]. Effects of Kinect-based games on learning are investigated in several studies such as ref. [31]. In ref. [13], a gesture-based game on Kinect has been implemented. Children's social preferences for playing the game is examined. In ref. [38], a Kinect-based game is developed to improve spatial and learning skills. The role of Kinect-based games on the improvement of the performance of students with cognitive disabilities and its application on learning different fields, such as mathematics and physics, are examined in ref. [34]. Cognitive benefits of computer games on elderly are presented in refs. [37,43]. The relation between motor control and the learning process is studied in ref. [35].
Physical activity-based games are examined under the name of exergames in some studies such as refs. [26,29] . In ref. [30], it mentions that several studies show the impact of exergames on quality of life. This study focuses on obtainability of personal data using Kinect-based exergames, compared to other gaming devices like Nintendo Wii and PlayStation Move. In ref. [27] exergames are used as a tool to provoke different emotions and evaluate individual differences of an entertaining and balanced workout program. Facial expressions and physiological data are collected throughout the gameplays in experiments to evaluate physical and cognitive stress and to observe emotional reactions. Experimental setup consists of physiological sensors (sensors for respiration, body temperature change, and galvanic skin conductance), Kinect v2 device (for collecting facial expressions) and a virtual bicycle game. Different emotions are triggered using different games scenes like jumpscare scene and challenge scene. In one of the other studies [11], it mentions the lack of physical activity of children, its consequences, and a solution to this based on exergames. Game enjoyment, mood, and attitude toward exergaming are also examined in this study. Ref. [24] examines the effects of exergaming. This study mentions about importance of physical activity to public health and importance of exercises. Study consist of reviews of multiple other studies related to effectiveness of exergames on promoting exercises among people. Ref. [15] investigates the nonphysical effects of exergames by reviewing the other studies which investigate psychological effects of exergames. Ref. [44] investigates reliability and effectiveness of motion sensors which are used with exergaming experiments.

Proposed HCI games
In this research, the aim is to show the positive impact of the physical activity-based games on the learning experience and cognitive skills.

Fruit picking game
In this game, the player will see a number of fruits falling from top to the bottom of the screen. Objective is to collect apples while avoiding the pears. The player will do that using one of their hands (default is right hand, but the player can switch between hands at any time of the game session). The player's hand will control the basket that collects the fruits. Each collected apple will increase the game score by 1, while each collected pear will decreased the score by 1.
There are three stages of the game where difficulty level of the game increases by increasing the average falling speed of fruits. Every fruit's falling speed will be randomized based on a predefined range for each stage. Maximum of four fruits will be presented on the game screen at every time. Creation of the fruits will be randomized, with probability of 0.5. Each stage's maximum score is 10 (ten apples will be created at each stage). Physics needed for simulating the fall of the fruits is implemented at this stage.
Adaptive difficulty of this game will make the fruits fall faster if the player is performing well and collecting apples, otherwise it will make fruits fall slower. At each collection of a fruit, adaptive difficulty algorithm will run to find adaptive difficulty coefficient that will be applied to falling speed of the fruits on top of stage's own randomized falling speed value. Adaptive difficulty coefficient can be 0.25, 0.5, 0.75, 1, 1.5 and 2 depending on the performance. If the current performance is high, adaptive difficulty algorithm will pick a high valued coefficient. The game starts with the coefficient as 1, if the participant fails, smaller coefficients starting from 0.75 are applied. With each failure, the coefficient gets smaller. The smallest value is 0.25 and after that, the coefficient does not change. On the other hand, if the participant is successful, starting from the coefficient 1.5 the coefficient increases. The highest value is 2, and if the participant keeps a successful performance, the coefficients stays at this value. According to the success/failure of the participant the coefficient increases or decreases by one level, which changes the falling speed of the fruit, accordingly.
Size of the fruits and the background image of the game can be changed at any time using the menu bar at the top. Score, hit and miss values will be shown on the upper right of the screen. Hit is the number of apples collected, miss is the number of apples that player was unable to collect, and score is the overall game score calculated by subtracting the number of pears collected from hit value (as in Eqs. (1), (2), (3)). A sample image from the game is shown in Fig. 3.

Air hockey game
In this game, the participant plays against computer-controlled opponent. Both the human player and the computer-controlled player cannot reach beyond their half of the playfield. Both the players will try to hit the puck to score or to defend their goals. The game does not have a maximum score and time limitation.
Playfield is in the center of the screen. A button that resets the scores is placed at the top. Scores of both human-and computer-controlled player is at the bottom. Legends for the game are placed at right side of the screen as seen in (Fig. 4). "CPU" represents the paddle of the computer-controlled player, "YOU" represents the paddle of human player, and "Ball" represents the puck.
Rule-based control module of the computer-controlled player and physics needed to simulate collisions are designed and implemented based on several rules, such as hitting the puck with paddles(players) and reflecting the puck against borders of the field. The algorithm of the rule-based control module can be found in Algorithm 1. An adaptive difficulty system is implemented for this game, by adjusting the speed of the computercontrolled player based on human player's performance in terms of goals scored. The speed is adjusted with the same coefficients as described in the fruit picking game. The game starts with the coefficient 1, and the better the performance, the higher the coefficient is, which directly increases the speed of the computer-controlled player.
Human player's paddle is controlled by player's hand motion, which is detected by the Kinect device. Rule-based system (RBS) first determines if it is in the attacking or defending mode based on the direction of the puck. If puck is going toward RBS's goal, RBS will try to defend, otherwise it will attack. If puck is not in the RBS's half area, RBS's paddle will go to attack position if it is in attacking mode, otherwise it will go to defence position if it is in defending mode. Attack position is the center of RBS's half area, while defence position is the front of RBS's goal. In both cases of attacking and defending modes, RBS's paddle will try to get behind of the puck without touching it, if the paddle is front of the puck. If RBS's paddle is already behind of the puck, RBS's paddle will try to score a goal by shooting straight to the human player's goal if the RBS is in the attacking mode, or RBS's paddle will try to clear the puck out of RBS's goal by going directly to the puck if RBS is in the defending mode.

Myo Armband-based maze game
This game uses electromyography (EMG) data which is retrieved by the Myo Armband to recognize arm movements for the game control. Myo Armband has a data collection rate of 200 sample per seconds. Each data consist of eight different EMG values. Data are collected in 5-minute sessions for each move, which is used to train and test the neural network. Neural network has been tuned for best parameters which are 2 hidden layers, and 400 units in the each layer with learning rate of 0.02 and momentum of 0.9. Input layer consists of 40 units (each 5 consecutive data items have been combined together to   Table I.   This game is designed to control an object in a maze (Fig. 7). This game aims to improve muscle activities and useful for people who cannot perform the physical activities required to play Kinect-based games. Players will try to find an optimized way of reaching the objective while they are using their muscle movements to control the object. This fact suggests that this game will help to improve cognitive skill while being physically active during the gameplay.

Experimental Setup
During the first set of experiments, a brain-computer interface (BCI) device called Mindwave EEG headset is used for collecting EEG data of the participants, both while playing the games and while waiting idle. In these experiments, only the attention data are collected. The attention value is in 0 and 100 range. Attention represents the concentration of the participant who wears the Mindwave headset. This data will allow us to detect if the participant is paying attention and focusing on the game.
For the first set of experiments, experimental setup (Fig. 8) consists of a laptop, a Kinect RGB-D device (Fig. 9) and the Mindwave set ( Fig. 10), for the first two games (fruit picking and air hockey), and the Kinect is replaced by a Myo sensor for the maze game.
For the second set of experiments, The Mindwave device is not used, only a laptop and a Kinect RGB-D device are employed. In the Kinect games, the participants are asked to stand almost 1.5 mt away from the laptop and Kinect, facing toward them. The Myo-based maze game can be played in both standing and sitting position.

Kinect RGB-D device
Kinect device is used for detecting body movements of the participants by tracking the spatial information of body parts such as hands, wrists, shoulders, and elbow. We have used the locations of hands and wrists of the participants for the game control.

Mindwave EEG headset
Mindwave is the device for collecting EEG signals. Its data collection rate is 512 sample per second. The signals are preprocessed by the software of the headset. The collection rate of the preprocessed data is 1 sample per second. Preprocessed data consist of Delta, Theta, Low Alpha, High Alpha, Low Beta, High Beta, Low Gamma, High Gamma, Meditation, and Attention information. In our experiments, we have only used attention information. Mindwave device is connected to the computer via a bluetooth connection.

Software
Games are implemented in .NET environment. Games are implemented as WPF (Windows Presentation Foundation) applications. Integration of Kinect RGB-D device is done via Kinect's SDK (Software Development Kit). Integration of Mindwave EEG headset is done via a socket connection to device's software. All of the projects are implemented with C# language.

Experiments
Each of the three proposed games have different properties which require different physical activities and different levels of attention and coordination.
In the first set of experiments, both the game performances and the attention values from each participant are collected throughout the game sessions using the EEG headset. The baseline attention values are collected while the participants are sitting idle. Attention values while playing these games are expected to be significantly higher than the baseline condition. This may support the claim that physical activity-based games also increase brain activity.
In the second set of experiments, only the game performances (score values) are collected. The EEG headset is not employed in these games. Every participant take part in either first or second set of experiments but not both. Both set of experiments consisted of several games, for example, the first set of experiments consists of air hockey, fruit picking, and Myo maze game, which are played by each participant in random order.
Ten adults have participated in the first set of experiments and 6 adults and 12 children have participated in the second set of experiments. Myo game and EEG headset are used only for the first set of experiments. The experiment setups are first tested by adults as a pilot study, then with children. This enables the researchers to evaluate the test conditions, games, and sensors before the tests with children. Also the analysis of the games, their effects, and sensory setup with different age groups helps the researchers to detect the differences and define the design principles according to the needs of these groups. Twelve children (3.5-5 years old) from 8 families took part in the study. These children regularly visit the Audiology Department of Cerrahpasa Medical School for their audiometry tests and therapies. They were recruited by the audiologists and pedagogue taking part in this collaborative project. Children within the given age limits, who are deaf and wearing cochlear implants, but do not have any other medical conditions were recruited to take part in the study.
There may be some possible limitations in this study due to the sensors used and the special requirements of the participants. The games in the study are specially designed and developed to be used to collect data from these sensors. Some of the sensors were wearable sensors (Myo and EEG), whereas the others (Kinect) should be used from a predefined distance. The wearable sensors were hard to use with children, and their sizes were not fit for children and might require the participants to stay still during the game (EEG) which might not suitable for the children of this age group. Also children had implants which makes the usage of wearable sensors such as EEG and need careful planning. The study is also limited in terms of the participant numbers, since it is hard to draw firm conclusions with a small number of participants. On the other hand, it is a hard task to recruit a large amount of children with similar cognitive and hearing conditions and medical records. The difficulty levels and the game interface including the graphics of the games are also important. Long verbal or written guidelines or feedback mechanisms were not used, since the children do not know how to read and lose their attention very easily. We finally come up with the designs, adaptive difficulty levels, and game setups following the suggestions of collaborating experts.

First set of experiments-experiments for measuring concentration
In the first set of experiments, 10 adult participants' attention values are collected both while they are playing the games and while they are sitting idle. Results that show mean, standard deviation, and statistical significance values of attention values are shown in Tables II, III, and IV for fruit picking game, air hockey game, and Myo-based game, respectively. ANOVA tests are conducted with confidence level of 95%.
In more than 70% of the participants, there is significant difference between attention values collected while participants are sitting idle and while they are playing the games. Here, sitting idle can be interpreted a baseline condition, and when the participant is dealing with a cognitive task such as the proposed games, higher attention levels are expected. Also there are differences between games themselves. Myo-based game has the highest average of attention, and attention values are significantly higher in 80% of participants as opposed to 70% in other two Kinect-based games. Myo-based game  was harder to play than the Kinect-based games. This might be the reason for the differences between games. Another observation from these results is the attention values of the seventh participant. There is not any significant difference between attention values of while playing games and sitting idle. The reason for this would be that this particular participant found the games easy, and these games did not affect the attention value of this participant.

Relation between attention and difficulty levels
Ten participants' attention data have been collected throughout their gameplay. Their performance in the game is also recorded while they are playing the games. The relation between players' performance in terms of their current adaptive difficulty level and their attention level is investigated. Visual illustration of each participants' attention level and their adaptive difficulty level is given in Appendix. Timestamp values of attention values and adaptive difficulty values (timestamps where adaptive difficulty of the game has been changed) are used to create these figures. Note that since the ranges of attention values and the adaptive difficulty values of two games are different, attention values are scaled down to 0-5 range for fruit picking game, and to 0-3 range for air hockey game. Original range of the attention value was between 0 and 100. As observed in the figures, adaptive difficulty system has an effect on attention. In some of the participants' gameplays, game adapts itself to the decreasing attention of the participant by increasing the  difficulty and this action increases the attention value of the participant. In other gameplays, adaptive difficulty system manages to help the person to keep a high, steady amount of attention throughout the gameplay.

Second set of experiments -game scores
In order to evaluate the performance of the participants, game scores are recorded. These experiments include gameplays of both 6 adult and 12 child participants. Results of both adults and children are evaluated separately. Histograms of game scores for fruit game are in Fig. 11(a) and (b) for adults and children, and histograms for air hockey are in Fig. 12(a) and for adults and (b) for children, respectively. Histograms show the game score ranges as bins on the x-axis and number of experiments that fall into the game score value of each bin on the y-axis. Game score values for air hockey game are calculated as in Eq. (4).
While figures for both the fruit game and the air hockey game show that adults performed better, some of the children performed almost as good as adults, although their age group is very small and they did not have experience with such sensors and game setup before. This fact shows that games are suitable for all ages. All children who are asked to attend the game voluntarily played, and their motivation and reaction to the game is declared as positive by the therapist accompanying the test.
One of the reasons of the fact that performance gap between adults and children is small is due to the adaptive difficulty system. If the player's performance is low, game will help them to play better by decreasing the difficulty. If the player's performance is high, game will increase the difficulty to increase the challenge. In both cases, adaptive difficulty system makes the games more enjoyable. Some selected   Fig. 13 for fruit game and Fig. 14 for the air hockey game, respectively. These figures show that although some of the players performed poorly at the start and with the help of adaptive difficulty, they achieved good scores (e.g., user 2 in the Figs. 13 and 14). Whereas, the games started as more challenging for a skilled player and this player's performance started to drop by the end of the game session (e.g., user 1 in the Figs. 13 and 14).

Conclusions
In this study, adaptive human-computer interaction games with Kinect, EEG headset, and Myo sensors are designed, developed, and tested with volunteering participants of different age and disability levels (adults with no disabilities and children with hearing disabilities). Experiments in this study show that physical activity-based games are increasing the attention of individuals which will help them increase the efficiency of their learning process. The three games have adaptive difficulty levels and require different physical and cognitive requirements. We did not use commercially available games but designed and developed these games due to the requirements and restrictions of our project. The games have simple rules to be played easily with different age groups and disabilities, such as a fruit picking game, an air hockey game, and a maze game. The fruit picking and air hockey games are controlled by Kinect sensor, and maze game is based on Myo sensor.
The first set of tests is performed with 10 adults for measuring and comparing concentration. Both Kinect and Myo sensors and EEG headset are employed in these experiments. Results of these tests show that attention value is significantly increased for more than 70% of the participants during the game, compared to the idle times. The second set of experiments was conducted first by 6 adults, then with 12 deaf children of 3.5-5 years in the clinical setting of a collaborating audiology department. 2 This set of experiments only consisted of Kinect-based fruit picking and air hockey games. EEG headset was not employed in these games, since the children used small-sized EEG caps afterward in another experimental setup.
Both the adults and the children showed good performance in the games. A pedagogue and three audiologists attended this part of the study and reported the positive emotion and attention of the children participating in the games. When the children showed high performance and motivation in the games, in both games, the difficulty level adaptively increased. The audiologists and the pedagogue reported that, the children, who took part in the games, more willingly continue with the following therapy and test sessions, and communicated and collaborated with the researchers more. These positive claims are verified with the game scores, and the observations and subjective evaluations of the audiologists and the pedagogue attending the experiments. We plan to use these games as an alternative way to stimulate the emotions of children instead of the videos and pictures as used in adults.
Performance reports of both games suggest that games are suitable for all ages. This fact makes this study more valuable, since it can be used for all people to increase their capacity to learn, help them become physically healthy, and enjoy their time. Games with fixed parameters are found to be less engaging over time, especially with children under special treatment or education and reduces the player performance. Also these games target children with disabilities in the long run, and adaptivity of difficulty makes the games appropriate for a larger target group. If the player's performance is relatively low, game will help them play better by decreasing the difficulty. If the player's performance is relatively high, game will increase the difficulty to increase the challenge. In both cases, adaptive difficulty system makes the games more enjoyable.
This pilot study is a part of an ongoing work on the assistive robotic platforms for the rehabilitation of the deaf children [10]. We plan to utilize these games with the assistance of the robot, robot being the part of the game and give feedback about the game. These interaction games will enable the therapists to build a bridge between the robot and the deaf children and familiarize them with the robot. The adaptive difficulty levels in the games keep them engaged with the game, refine the sensory-motor coordination and attention of the children, and help to decrease the reaction time of the children to the stimuli in the therapy. The games will be played in the assistance of the robot (the display screen on the robot will be employed, and the robot will give feedback about the success of the game), and the effect of the robot's presence will be analyzed. This work is the first step of this work. Also the attention and the facial expression of the participants collected during the game are used in the emotion recognition of the children. Especially, this is important where it is hard to stimulate the children's reactions when the difficulty level of the tasks increase.