Utilization of AI Voice Interaction for Educational Experiment SystemBased on Biological Data Analysis

Takaya Masaki, Tomohiro Nakahara, Hatsuho Nankai, Ryoya Hayashi, Akihito Fujimoto, Tetsuya Sato*

Department of Electrical Engineering, Kobe City College of TechnologyJapan

tesato@kobe-kosen.ac.jp

Abstract: This paper details the development of voice interaction on our proven KCCT-VB(KobeCity College of Technology Vocabulary Builder). Though our students have to touch a button onthe screen to choose an answer after listening to the question on our conventional KCCT-VB whichis an Android based English vocabulary building application, the new voice interaction allows themto answer by pronouncing the answer word. As many educators have stressed the importance ofreading aloud, our new system which allows our students to answer by their voice, could affectpositively regarding their concentration/relaxation on the course of the activities. As which canmeasure the students' real-time biological data through the brainwave sensor and the pulseoximeter, the newly developed system is proven to be used to unveil the effect of voice interactionon the vocabulary building activities. The examinations of the attention and the meditationevaluation value based on the brainwave, the pulse rate of the heartbeat, the blood oxygenlevel(SpO2), and the sphygmographic-wave, suggested a positive effect regarding the students'relaxation on the course of the activities.Keywords: Voice Interaction, Reading Aloud, English Vocabulary, Learning Behavior, BiologicalData Measurement, Brainwave, Pulse Wave, Pulse Rate, Blood Oxygen Level

Introduction

As global cooperation and collaborations have become a significant responsibility for all engineers,EFL(English as a Foreign Language) is a necessary skill for all of the engineer candidates. In our college which hasbeen established and dedicated for young engineer education after completing 9th grade through 14th grade with a5-year intensive course, successful students earn an associate degree in engineering. In our department of electricalengineering, about a half graduates become engineers in major manufacturers including Panasonic, Canon,Mitsubishi Electric and other electronics companies. Another half students continue their study for additional twoyears in our college as advanced course students in order to earn a full bachelor degree which allows them to enterany graduate school, while some of the graduates enter other universities as junior students. For those youths whodesignated their career in engineering in their younger days, EFL learning is a rather difficult subject, as theirmother language has also not yet matured. As their performances in math, science, and engineering subjects aretalented, most of them tend to prefer technical activities rather than training their English communication skills.Considering those circumstances, in order to fulfill the emerging necessity to train our students as engineers ready tocollaborate globally, we have elaborated to develop a novel approach utilizing our students' familiarities to ICTtechnologies for our students' English vocabulary building, as the importance of vocabulary building is receivingmore attention these days.

Meanwhile young engineer’s QRP(Quick Response Performance) is believed to play an important role todevelop their communication capability, based on the last author’s* experience as a leading engineer in a globalelectronics company. We have assumed that the training the QRP in EFL learning should enhance the developmentof their skills. Based on this assumption, we have developed a novel English vocabulary building application, namedKCCT Vocabulary Builder(KCCT-VB) using smartphones linked with a cloud data server, here KCCT stands forKobe City College of Technology. While our intention partially includes the utilization of our students' familiaritywith ICT technologies including smartphones, the developed system has allowed us to utilize two major advantages.The first one is the intuitive UI(User Interface) based on the built-in touch panel display, which allows us to delivera question with multiple choices as possible answers. As students do not have to operate any keys or pointingdevice, enabling them to choose the answer just by touching the particular choice, the developed application hasallowed us to measure accurate response time for each answer choice. Those response time data are accumulatedautomatically in a cloud data server, and we have reported the findings which revealed our students' Englishvocabulary building process in comparison with students from other countries at eLearn2011.1)

The second advantage is the real-time measurements of the students' biological data through the sensorslinked with the smartphones. We have utilized the brainwave sensor(Mindwave Mobile), made and supplied byNeuroSky Inc., as a real-time biological information in order to measure the student's psychological response data interms of the attention(a degree of concentration) and the meditation(a degree of relaxation), and reported thefindings on our students' attention characteristics at eLearn2013.2) The attention and meditation data are calculatedbased on the brainwave processing that is the proprietary technology developed by NeuroSky Inc. Though thesystem uses only a single dry electrode without any conductive gel at the forehead of the examinee(student), manyresearch articles have already reported that its usefulness is comparable with the conventional methods usingconductive gel,3) that the system can discriminate between reading easy and hard sentences,4) that it can assess acognitive load,5) and that it can indicate an individual's emotional change in terms of the attention and the meditationlevels.6)

Furthermore, in order to train the students' listening capability, we have incorporated a listening mode ontop of the reading mode in our KCCT-VB. The platform was carefully redesigned for a little bit larger populardevice(Google's NEXUS 7-inch tablet), and reported in our former papers.7-8) Moreover, as a trial to enhance ourstudents' engagements in the vocabulary building application, we have incorporated and looked into the battle modeas a gamification approach. The developed application can accumulate not only our students' learning performances,but also some learning behavioral data with some biological data, which shows more concentration on the battlemode compared to the single mode.9) Besides, as a further biological data measurements, we have incorporated apulse oximeter(SR-700bs, Konica Minolta, Inc.) linked with our Android device, and reported about some possibleanalyses and utilization of additional biological data based on the heart pulse measurement.10-11)

Meanwhile, as many educators have stressed the importance of reading aloud, we have a long history ofutilization of reading aloud. Currently, most of the English teacher even in elementary school encourages students toread aloud, and which is believed to be useful to keep the students’ positive attitude. At the same time, manyresearch proved the fact that even elderly seniors who have senile dementia can halt the advance of the symptomsand improve their ability to recognize objects after they continue to read aloud. In this paper, in order to allow ourstudents to answer the question by pronouncing the answer word instead of touching the button silently, a utilizationof emerging AI(Artificial Intelligence) voice interaction technology was detailed. We have also looked into thepossible effectiveness by examining the students’ biological data on the course of the activities.

Development of the system

On top of the advancement of the voice recognition and the natural language understanding, the latestAI(Artificial Intelligence) voice interaction technology based on the predetermined possible voice utterancesamples, allows us to utilizethe voice interaction in ourpractical education. Amongthose emerging technology,we have utilized the voiceinteraction service availablefrom Amazon, which allowsus to register 4000 basicEnglish words, and recognizesour students' voice as one ofthose words so that ourstudents can answer bypronouncing the answer wordinstead of touching the buttonwhich represents the correctword. The explanatory systems c h e m a t i c s o f t h econventional KCCT-VB andthe newly developed KCCT-VB with voice interaction aredepicted in Fig.1 and Fig.2respectively. Fig.1. The explanatory system diagram of the conventional KCCT-VB

Fig.2. The explanatory system diagram of the newly developed KCCT-VB with voice interaction

The significant difference between the conventional KCCT-VB and the new KCCT-VB with voiceinteraction is the main program which controls the question delivery and the student's answer management. For theconventional KCCT-VB, as shown in Fig.1, the main program is written in JAVA, and controls everything on thestudent's Android terminal. Only students' practice data and biological data are sent to the cloud server for thefeedback to the students and the analyses by researchers.10-11)

On the other hand, for the new KCCT-VB with voice interaction as shown in Fig.2, the main program iswritten in Python on the AWS(Amazon Web Services) server, and controls the question delivery and the answerrecognition through ASK(Alexa Skill Kit) server and Alexa Echo Spot terminal. At the same time, a student'sbiological data is measured by the KCCT-VB in Alexa mode, which is sent to the cloud server for the analyses byresearchers. The KCCT-VB in Alexa mode is designed to measure the student's biological data in step with the newKCCT-VB with voice interaction working on the AWS server, and allows us to consolidate the biological data withthe student's practice data(information about the delivered questions and student's answers) on our cloud server. Asthe Echo Spot terminal has the display, the main program can deliver any text message onto the screen. Since ourobjective is a utilization of the voice interaction, in order to minimize the difference between the conventionalKCCT-VB and the KCCT-VB with voice interaction, four possible answer choices are displayed on the screen, sothat our students can choose an answer from four choices, which allows us to compare our students' learningbehaviors with biological data depending on the answer methods(answer by touching the button silently, or answerby pronouncing the answer word).

Another newly developed feature is some feedback to each student. Firstly we implemented a user nameregistration to identify each student, and display the registered user name at the upper left portion as shown in Fig.3.Furthermore, we have newly designed to display the feedback information to each student at the lower part of thescreen, where the information about each student's last record, each student's highest record, and the highest recordsin all KCCT-VB users. Additional latest three records can be also displayed for their information if necessary. Webelieve all of those features could affect positively keeping their motivation in a long term vocabulary buildingactivities, but we will report on it in other paper separately, as the main focus of this paper is the utilization of thevoice interaction.

Fig.3. The screen capture of the KCCT-VB with new feedback features

Experiments

Although both of the conventional KCCT-VB and the new KCCT-VB with voice interaction implemented4000 basic English words(classified into 4 levels from 1st 1000 words, 2nd 1000 words, to 4th 1000 wordsrespectively) and its definition, and which can deliver any words depending on the level, the part of the speech, orrandomly, we designated 30 words starting from level1 to level4 in order to compare both answering methodconsistently. The experiments details are summarized in Table.1.

Table.1. Conditions for the experiments to compare both answering methods

conventional KCCT-VB new KCCT-VB with voice interaction

a definition announcementas a question delivery

announced by Google's text to speechon NEXUS 7inch tablet

announced by Alexa's text to speechon Amazon Echo Spot

answer options(four choices) displayed on NEXUS 7inch tablet displayed on Amazon Echo Spot

student's answer method touching a button on the screen pronouncing the answer word

examinees(students being tested)

6 students were tested both in the conventional KCCT-VB and in the new KCCT-VB with voice interaction. All of them are 5th-year student in our college.(equivalent to a sophomore in a university)

Results and Discussions

Although no significant difference was observed in the attention and the meditation evaluation values basedon the brainwave analysis from the NeroSky brainwave sensor, remarkable differences were found in the biologicaldata from the pulse oximeter(SR-700bs, Konica Minolta, Inc.). As similar differences were observed in all 6students, a typical example is shown in Fig.4. Since it took some preparatory time for the KCCT-VB with voiceinteraction including time for the guiding announcement in these experiments, the upper two data weredemonstrated according to the time, while the lower two data were demonstrated according to the questionnumber(1st word to 30th word), so that we can more easily compare the situation working on the same word.

(a) conventional KCCT-VB (b) new KCCT-VB with voice interactionFig.4. Comparison of the heartbeat pulse rate(beat per minute) and the blood oxygen level(SpO2(%))

As a result, considerable fluctuation of the heartbeat pulse rate was observed in the new KCCT-VB withvoice interaction compared to the conventional KCCT-VB where students touch a button silently. At the same time,remarkable fluctuations and a decrease were observed in the blood oxygen level(SpO2).

In general, the fluctuations in the heartbeat pulse rate is believed to have a deep connection to the autonomicnerve system, and considered as somewhat relaxed or unstressed mental condition. Meanwhile, the decrease of theblood oxygen level suggests consumption of oxygen in the student's brain. Since the oxygen is consumed mostsignificantly in the brain, and believed to reflect its activities. Therefore these results suggest that the utilization ofthe voice interaction somehow affect our students learning behaviors. As the average time to answer one word byvoice is around two times compared to answer by button, the results mentioned above could come from it. Butusually the heartbeat pulse rate and the blood oxygen level doesn't fluctuate so much even students took longer timeto answer by button, we believe these results should mainly come from the voice utterance.

On top of the utilization of voice interaction for the answering, another new feature utilizing the voiceinteraction technology is a brush-up mode for further training of incorrect words as shown in Fig.2. Since the newKCCT-VB with voice interaction can record each student's incorrect words, each student is asked to read aloud theincorrect words to remember them. We will also explain about this new feature together with the feedback feature indetail at our poster presentation.

Summary

Since many educators have stressed the importance of reading aloud, in order to utilize the latest voiceinteraction technology, the new KCCT-VB with voice interaction was designed and developed. Furthermore,practical experiments were carried out for 6 students at our college. The experiments were carefully designed tocompare the difference in the answering method, by touching a button silently or by pronouncing an answer word.Based on the biological data measured on the course of the experiments, the results suggested that the utilization ofthe voice interaction somehow affect our students learning behaviors.

For further research, as we reported our new evaluation measure based on our in-depth analysis of thefluctuation of the heartbeat intervals in another paper,12) we will utilize the particular method to look into theeffectiveness of the voice interaction. Furthermore, as we implemented the new feedback feature in the conventionalKCCT-VB and the brush-up mode in the new KCCT-VB with voice interaction, we will carry out the experiments tolook into the effect of the particular new features based on our biological data measurements and the latest analysesreported in another paper.12)

Acknowledgment

This work was supported by JSPS KAKENHI Grant Number JP18K02926.

References

1) Yukihito Nishida, Tetsuya Sato: A Study on ESL Learning in Early Engineering Education using Smart Phones, Proc. of E-Learn2011-World Conference on E-Learning in Corporate, Government, Healthcare & Higher Education-organized by AACE(the Association for the Advancement of Computing in Education), pp.337-342, 2011.2) Satoshi Itakura, Toshiya Kira, Shigeki Suehiro, Tetsuya Sato: A Study on ESL Learning in Early Engineering Education using Smart Phones linked with Brainwave sensor, Proc. of E-Learn2013, pp.798-803, 2013.3) An Luo, Thomas J.Sullivan: A user-friendly SSVEP-based brain-computer interface using time-domain classifier,Journal of neural Engineering, Vol.7, pp.1-10, 2010. 4) Jack Mostow, Kai-min Chang, Jessica Nelson: Toward Exploiting EEG Input in a Reading Tutor, Proc. of the 15th International Conference on Artificial Intelligence in education(AIED)2011, Lecture Notes in Computer Science(LNCS), Vol.6738, pp.230-237, 2011.5) Eija Haapalainen, SeungJun Kim, Jodi F. Forlizzi, Anind K. Dey: Psycho-Physiological Measures for Assessing Cognitive Load, Proc. of the 12th ACM international conference on Ubiquitous Computing (Ubicomp'10), pp.301-310, 2010.6) Katie Crowley, Aidan Sliney, Ian Pitt, Dave Murphy: Evaluating a Brain-Computer Interface to Categorize Human Emotional Response, Proc. of 2010 IEEE 10th International Conference on Advanced Learning Technologies (ICALT), pp.276-278, 2010.7) Yuki Kurata, Naoki Funahara, Junichi Hayashi, Eiichiro Ohashi, Nana Otsuka, Tetsuya Sato: Development of an Experimental Platform for English Vocabulary Learning in Early Engineering Education using Smartphones, Proc. of E-Learn2015, pp.844-849, 2015.8) Naoki Funahara, Junichi Hayashi, Eiichiro Ohashi, Nana Otsuka, Tetsuya Sato: A Study on English Vocabulary Learning Behaviors Based on Biological Data Analysis, Proc. of E-Learn2015, pp.1721-1726, 2015.9) Yuki Kurata, Daichi Minayoshi, Shingo Morioka, Yoshiki Yasufuku, Tetsuya Sato: Development of a Gamificational Experimental Platform Utilizing a Fighting Game Style Learning and an Avatar Encouragement for English Vocabulary Building Application on Smartphones, Proc. of E-Learn2016, pp.1016-1021, 2016.10) Ryosuke Shimmura, Ryohei Konishi, Ken Nakajima, Ryoya Hayashi, Tetsuya Sato: Development of a Versatile Experimental Platform for English Vocabulary Learning Based on Extended Biometric Information Analyses, Proc. of E-Learn2017 organized by AACE, pp.771-776, 2017.11) Ryohei Konishi, Ryosuke Shimmura, Ken Nakajima, Ryoya Hayashi, Tetsuya Sato: A Study on InconscientLearning Behaviors and Psychological Evaluations Using Biological Data for an Image-Based Vocabulary BuildingApplication, Proc. of E-Learn2017 organized by AACE, pp.718-723, 2017.12) Ryoya Hayashi, Tomohiro Nakahara, Hatsuho Nankai, Akihito Fujimoto, Takaya Masaki, Tetsuya Sato: AStudy of Pulse Wave Analysis Based on Biological Data Measurements in Educational Experiments, Proc. of E-Learn2018 organized by AACE, to be published, 2018.