Objective Evaluation of the Effects of Human-Robot Communication Using Gaze and Cerebral Blood Flow Volume 56- Issue 1

Hirokazu Ito^1* and Nao Ihara²

• ¹Department of Nursing, Institute of Biomedical Sciences, Tokushima University Graduate School, Japan
• ²Graduate School of Technology, Industrial and Social Sciences, Tokushima University, Japan

Received: March 25, 2024; Published: April 08, 2024

*Corresponding author: Hirokazu Ito, Department of Nursing, Institute of Biomedical Sciences, Tokushima University Graduate School, Tokushima, Japan

DOI: 10.26717/BJSTR.2024.56.008787

Abstract PDF

Aim: With the rapidly declining birth rate and population aging in Japan, the government supports the use of robots to address the problems of insufficient human resources and economic funding to provide adequate care to older adults. This study aimed to analyze the changes in eye gaze and cerebral blood flow during conversations between humans and robots to objectively evaluate its impact on humans.

Methods: The research was conducted from April to December 2022. The study participants were three nursing students in their 20s to 40s at University A. PALRO (Fujisoft) was used as the communication robot (CR). The participants wore a HOT-2000 (NeU) brain activity meter and had a 15-minute conversation with the CR after resting for 15 min in a room separate from the experimental room. The gaze data were divided into six areas, and descriptive statistics were obtained.

Results: The cerebral blood flow (right and left) was categorized into that obtained at rest time, during the conversation, and during characteristic scenes (scenes from which participant smiles and laughter were extracted). Welch's one-way ANOVA was performed. The participants most commonly gazed at the area around the face, even if the other party was the robot. Blood flow was significantly higher during scenes involving laughter than during other scenes. The heart rate also increased, suggesting an effect on the human autonomic nervous system.

Conclusion: Although the participants who experienced this experiment were not older adults, many communication robots have been installed in facilities for the elderly in Japan. This result considers that conversations with a robot are effective in preventing lonely older adults from decreasing their activities and the resultant decline in activity of daily living (ADL) and function.

Keywords: Gaze; Blood Flow; Communication; Nursing; Robot

The birth rate is rapidly declining while the population is aging in Japan. The aging rate, which is the percentage of the total population older than 65 years, was 28.4% in 2019 and is expected to rise to 38.4% by 2065 (Cabinet Office and Government of Japan, [1]). This rapid population aging has caused a shortage of human resources and economic funding to provide adequate care for older adults (Muramatsu & Akiyama, [2]). To address this issue, the Ministry of Health, Labor, and Welfare in Japan has launched the "Promotion of the Development and Dissemination of Nursing Care Robots" project to support the introduction of robots and their practical use (Ministry of Health, Labor, and Welfare, [3]). The Ministry of Economy, Trade, and Industry (METI) and the Japan Agency for Medical Research and Development (AMED) launched the Robot Care Equipment Development Project to support the development of robots to solve problems in nursing care facilities (Ministry of Economy, Trade, and Industry & Japan Agency for Medical Research and Development Announcements, [4]). In medical and welfare fields, advanced technology is required to detect information, make judgments, and take action.

A robot that assists in nursing care is defined as an intelligent mechanical system that possesses the three elemental technologies of "sensing” (sensor system), "judging” (intelligence/control system), and "operating” (drive system). Consequently, the introduction of robots in the medical and nursing care fields is expected to be an effective countermeasure in an aging society with a declining birth rate. Humans engage in extremely complex communication, and a combination of verbal and nonverbal cues are used to effectively send and receive information to others. According to Birdwhistell [5], only 30–35% of meaning is conveyed through words, and the rest is conveyed through nonverbal communication. Mehrabian [6] also stated that only 7% of the attitude and personality tendencies of individuals are estimated using words; 38% and 55% are based on peripheral language and facial expressions, respectively. In addition, communication in medical care settings requires more advanced techniques than ordinary communication due to a combination of factors, such as the illness and age of a person.

If robots in medical and nursing care settings cannot make appropriate nonverbal expressions or understand the nonverbal responses of users, their conversations will feel unnatural, and good relationships will not be established with them (Mathur & Reichling, [7]). However, only a few studies have objectively evaluated the reactions of participants to using communication robots. An appropriate evaluation of eye gaze and nonverbal responses of humans will enable robots to be equipped with functions to compensate for unnaturalness in the future. Therefore, this study aimed to analyze the changes in gaze and cerebral blood flow during conversations between humans and robots to objectively evaluate their impact on humans.

Study Design

The study was conducted from April to December 2022, and the participants were three nursing students in their 20s and 40s at a university. PALRO (Fujisoft) was used as the communication robot (CR). Before the experiment, adjustments were made to set the eye level to that of the CR, and the distance between the CR and the participants was 50 cm. The participants wore a HOT-2000 (NeU Corporation) brain activity meter and had a 15-minute conversation with the CR after resting for 15 min in a room separate from the experimental room. The participants were asked to speak naturally using the same dialogue program built into the robot when it was sold. During the conversation, gaze information was collected using a gaze analyzer (Tobii Technologies), and changes in heart rate (HR) and cerebral blood flow (HbT: total hemoglobin [Hb] change (left/right)) were measured using the HOT-2000. Video recordings were used for subjective evaluations (Figure 1).

Figure 1

Ethical Considerations

This study was conducted with the approval of the ethics committee of the institution of the researchers (No. 4221). No harm to the participants was observed. The participants were notified that their privacy would be protected.

Previous Studies on Eye Gaze

We used a Tobii Eye Tracker 4C from Tobii Technologies (150 eye movements/s) to perform gaze analysis. The Tobii Eye Tracker 4C is a noncontact device that detects the gaze of a person using infrared reflections, and is currently used in home video game consoles. It has been used in a nursing study that reported differences in how nurses gaze at electronic medical records (Takami, et al. [8]) and intravenous injections (Sugimoto, et al. [9]) depending on their clinical experience.

Previous Studies on Cerebral Blood Flow

The HOT-2000 meter was developed by the NeU Corporation. It is used to measure cerebral blood flow, and it uses light at a wavelength that is easily absorbed by hemoglobin (approximately 800 nm) based on the principle of optical topography. When brain regions on the light path are activated, blood flow and light absorption increase, allowing for the non-invasive evaluation of brain activity. The system can be used only by attaching a headset; therefore, data collection can be performed with minimal burden on the participant. It has been used in medical studies and has shown the relationships between brain oxygenation levels during sleep deprivation and poor cognitive performance of nurses during night shifts (Durán-Gómez, et al. [10]). It has also been used to study the effects of sex on brain activity during mental workload (Keshmiri, et al. [11]), limitations of near-infrared spectroscopy (NIRS) data analysis (Keshmiri, et al. [12]), and the relationship between cerebral blood flow and mental stress using NIRS (Komuro, et al. [13]). Figure 2 shows a conversation between a human and the CR.

Figure 2

Statistical Analysis

The gaze data were divided into six (2 × 3) locations, with (X, Y) = (0, 0) as the coordinates of the center of the participant’s gaze range (the central part of the CR face), and descriptive statistics (frequency and percentage) were obtained. Cerebral blood flow was divided into that before the conversation, during the conversation, and characteristic scenes (scenes in which participant smiles and laughter were extracted). Welch's one-way ANOVA was analyzed, followed by the Games-Howell method for post-hoc testing. Statistical significance was set at p-values of <0.05. All statistical analyses were performed using SPSS for Windows (version 20.0; SPSS Inc., Chicago, IL, USA).

The results of the gaze analysis showed that Area 2 (n=2016, 44.15%) was the most frequent for Participant 1, followed by Area 3 (n=1708, 37.41%). For Participant 2, Area 2 (n=14606, 96.44%) was the most frequent, followed by Area 3 (n=362, 2.39%). For Participant 3, Area 2 (n=21750, 94.07%) was the most frequent, followed by Area 1 (n=680, 2.94%) (Figure 3). The changes in cerebral blood flow and heart rate are shown below. The changes in the right cerebral blood flow in Participant 1 were highest for the characterized scenes (0.91±0.14), followed by during (0.87±0.18) and before (0.50±0.15) conversation, with significant differences for all scenes (p<0.05). The changes in the left cerebral blood flow were significantly different (p<0.05). The highest value was obtained before the conversation (-1.47±0.21), followed by that for the characteristic scene (-1.57±0.12), and that observed during the conversation (-1.58±0.13). The heart rate changes during conversation (83.18±5.67) were significantly higher than before conversation (80.47±14.88) (p<0.05), and those occurring during the characteristic scenes (83.11±5.98) were significantly higher than those before the conversation (80.47±14.88) (p<0.05).

Figure 3

The changes in the right cerebral blood flow in Participant 2 significantly differed (p<0.05) for the characterized scene (0.62±0.07), during the conversation (0.23±2.37), and before the conversation (0.15±0.14). The changes in the left cerebral blood flow were also significantly different (p<0.05) for the characteristic scenes (0.53±0.24) and during (0.10±2.21) and before (0.01±0.10) the conversation. The heart rate changes were higher during the conversation (81.47±5.76) than for the characteristic scenes (78.41±2.78) and before the conversation (73.91±3.37) (p<0.05). The changes in the right cerebral blood flow in Participant 3 differed significantly (p<0.05) before (-0.33±0.09) and during (-0.46±0.12) the conversation and for the characterized scenes (-0.52±0.13). The changes in the left cerebral blood flow were higher during the conversation (0.09±0.10) than for the characteristic scenes (0.07±0.10) and before the conversation (-0.38±0.18) (p<0.05). The changes in the heart rate were higher for the characteristic scenes (74.97±3.24) and before (73.66±2.60) and during (73.09±3.48) the conversation (p<0.05) (Table 1).

Table 1: Welch’s Analysis of Variance Results for robot and human conversation.

Note: We1ch’s Analysis of variance, abbreviations: SD, standard deviation, RtHb: Right to Homogrobin, HR: Heart Rate, *p<0.05, **p<0.01, ***p<0.001. Post hoc test Games-Howe.

The gaze analysis showed that the participants most frequently gazed at the periphery of the face of the robot. This means that basic communication skills, such as making eye contact with another human (or patient) and gazing around the face (University of Minnesota, [14]), were used during conversation with a robot. However, PALRO, a CR, is designed to show various responses with 60 LED lights placed around its face, instead of nonverbally demonstrating pleasure, anger, sadness, or pleasure like a human face. Therefore, it is possible that the participant was trying to obtain response information by gazing at the area around the face of the PALRO (FUJISOFT website, [15]). The cerebral blood flow values were the highest for participants 1 and 2 during the characterized scenes. Laughing during conversation is considered an emotional change and may affect cerebral blood flow. Sugawara et al. [16] stated that laughing causes changes in blood pressure and heart rate, resulting in temporary but beneficial changes in the vascular system. The ability of a robot to stimulate emotional changes in humans is considered effective in maintaining cognitive function, and robot use in facilities for older adults is expected to have benefits.

Studies have shown that robotic interventions have improved the state of asthenia in patients with dementia (Valentí Soler, et al. [17]). However, the left cerebral blood flow changes varied depending on the participant. The involvement of the right brain has also been studied. Brain activity changes varied when the participant was engaged in conversation, as the left brain generally concerns language functions during conversation (Taylor & Regard, [18]). It is possible that the value changed significantly for the scenes during which the participant actively spoke to the robot, and it is necessary to scrutinize the content of the conversation and the number of statements made. Heart rate is known to fluctuate with psychological factors, and transient changes occur with excitement and pain, among others (Levy, et al. [19]). The heart rates of participants 1 and 2 were significantly higher during conversations. Communication with a robot is considered to affect human autonomic nervous system activity, leading to changes in heart rate. Based on the above, conversations with robots can be objectively evaluated for their effect on human cerebral blood flow, heart rate, and the autonomic nervous system activity that controls them. Starting a conversation with a robot may effectively prevent lonely older adults from decreasing their activities and the decline of ADLs and function. We will continue to conduct experiments under the same conditions on the older adults and evaluate the results.

While changes in eye-gaze frequency and cerebral blood flow were measured, it is difficult to directly compare a robot and a human in an experiment. It is also difficult to maintain similar conversational content across all participants.

In this study, we focused on human-robot conversational situations and objectively evaluated the impact of communication with robots on humans. The results showed that even if the other party was a robot, humans frequently gaze at the area around the face. The changes in cerebral blood flow showed differences; blood flow increased significantly for scenes involving laughter relative to the others. While addressing problems such as human resource shortages, robots may also have a positive effect on humans by stimulating conversations and increasing human activity such as recreation, conversation, physical exercise etc..

We would like to express our sincere gratitude to the participants of this study. This study was supported by JSPS KAKENHI (grant number: 19K1973500).

1. (2019) Cabinet Office, Government of Japan. Measures to address Japan’s aging society.
2. Muramatsu N, Akiyama H (2011) Japan: super-aging society preparing for the future. Gerontologist 51(4): 425-432.
3. (2018) Ministry of Health, Labour and Welfare. Promoting the development and popularization of care robots.
4. Ministry of Economy, Trade and Industry, Japan Agency for Medical Research and Development Announcements (2021) Robotic Care Devices Portal. With a nursing robot to a comfortable future.
5. Birdwhistell RL (1970) Kinesics and context: essays on body motion communication. University of Pennsylvania Press.
6. Mehrabian A (1971) Silent messages. Wadsworth Publishing.
7. Mathur MB, Reichling DB (2016) Navigating a social world with robot partners: a quantitative cartography of the Uncanny Valley. Cognition 146: 22-32.
8. Takami M, Nishiumi E, Nakanishi E, Ishigaki K (2021) Analysis of gaze information patterns of nurses gathering patient information using electronic health records. Studies in Health Technology and Informatics 284: 531-533.
9. Sugimoto M, Tomita A, Oyamada M, Sato M (2022) Eye-tracking-based analysis of situational awareness of nurses. Healthcare 10(11): 2131.
10. Durán Gómez N, Guerrero Martín J, Pérez Civantos D, López Jurado CF, Montanero Fernández J, et al. (2021) Night shift and decreased brain activity of ICU nurses: a near-infrared spectroscopy study. International Journal of Environmental Research and Public Health 18(22): 11930.
11. Keshmiri S, Sumioka H, Yamazaki R, Ishiguro H (2017) A non-parametric approach to the overall estimate of cognitive load using NIRS time series. Frontiers in Human Neuroscience 11: 15.
12. Keshmiri S, Sumioka H, Yamazaki R, Ishiguro H (2018) Differential entropy preserves variational information of near-infrared spectroscopy time series associated with working memory. Frontiers in Neuroinformatics 12: 33.
13. Komuro Y, Sato Y, Lin L, Tang Z, Hu L, et al. (2018) Reliability of wearable two channel CW-NIRS in measurements of brain function. Oxygen Transport to Tissue XL 1072: 301-305.
14. (2016) University of Minnesota. Communication in the real world: an introduction to communication studies. University of Minnesota libraries publishing.
15. (2023) FUJISOFT. What is PALRO?.
16. Sugawara J, Tarumi T, Tanaka H (2010) Effect of mirthful laughter on vascular function. American Journal of Cardiology 106(6): 856-859.
17. Valentí Soler M, Agüera Ortiz L, Olazarán Rodríguez J, Mendoza Rebolledo C, Pérez Muñoz A, et al. (2015) Social robots in advanced dementia. Frontiers in Aging Neuroscience 7: 133.
18. Taylor KI, Regard M (2003) Language in the right cerebral hemisphere: contributions from reading studies. News in Physiological Sciences 18(6): 257-261.
19. Levy RL, White PD, WD (1945) Transient tachycardia: prognostic significance alone and in association with transient hypertension. JAMA 129(9): 585-588.