A Cross-Platform SmartphoneBrain Scanner

Jakob Eg LarsenTech. University of DenmarkDTU InformaticsRichard Petersens Plads, 3212800 Kgs. Lyngby, Denmarkjel@imm.dtu.dk

Michael Kai PetersenTech. University of DenmarkDTU InformaticsRichard Petersens Plads, 3212800 Kgs. Lyngby, Denmarkmkp@imm.dtu.dk

Arkadiusz StopczynskiTech. University of DenmarkDTU InformaticsRichard Petersens Plads, 3212800 Kgs. Lyngby, Denmarkarks@imm.dtu.dk

Lars Kai HansenTech. University of DenmarkDTU InformaticsRichard Petersens Plads, 3212800 Kgs. Lyngby, Denmarklkh@imm.dtu.dk

Carsten StahlhutTech. University of DenmarkDTU InformaticsRichard Petersens Plads, 3212800 Kgs. Lyngby, Denmarkcs@imm.dtu.dk

Copyright is held by the author/owner(s).CHI’12, May 5–10, 2012, Austin, Texas, USA.

AbstractWe describe a smartphone brain scanner with a low-costwireless 14-channel Emotiv EEG neuroheadset interfacingwith multiple mobile devices. This personal informaticssystem enables minimally invasive and continuouscapturing of brain imaging data in natural settings. Thesystem applies an inverse Bayesian framework to spatiallyvisualize the activation of neural sources real-time in a 3Dbrain model or to visualize the power of brainwaves withinspecific frequencies. We describe the architecture of thesystem and discuss initial experiments.

Author KeywordsEEG, brain scanner, smartphone, personal informatics,BMI

ACM Classification KeywordsH.5.2 [Information Interfaces and Presentation]:Miscellaneous

General TermsDesign, Measurement

IntroductionIn recent years personal informatics tools and applicationshave gained increased attention. More recently the trendtowards self-monitoring appears to be moving from an

early adopter phenomenon to gain more wide-spreadappeal. A lot of attention has been on simple devices tocapture data about physical activity or sleep patterns.However, better understanding of the intricate relationsbetween our brains and behaviors could be a key to futureimprovements in well-being and productivity, but in thepersonal informatics domain less attention has been oncontinuous monitoring of brain signals.

The main reason has been that traditional functional brainimaging techniques including fMRI and PET typically relyon complex, heavy hardware that offer limited comfortand mobility for the user. Thus the understanding of brainactivity in real-life settings has been limited, as it hasmainly been possible to measure brain activity inlaboratory settings, which may induce largely unknownbiases. In particularly for studies of emotion and socialcognition [2] this may be problematic. The ability tomonitor brain activity continuously under naturalisticconditions to study how we perceive our surroundings inmobile real-life settings have a set of obvious applicationsand advantages [2].

In this paper we describe a minimally invasive brainmonitoring system offering real-time brain state decodingand 3D cortical activity visualization on multiple low-costmobile device form factors (from smartphones to tabletdevices). The system can be used to continuously recordbrain states, e.g. emotional responses, and the 3Dvisualization or brainwaves within specific frequencies canbe used for simple bio-feedback.

Applications of continuous bio-feedback is interesting inthe context of personal informatics. It has been shownthat such bio-feedback may lead to improvements inbehavior, reaction times, emotional responses, andmusical performance. Within the clinical domain it has

been shown to have a positive effect on attention deficit,hyperactivity disorder, and epilepsy [4]. For suchapplications a low-cost and easy-to-use brain monitoringsystem enabling mobility can be beneficial.

The adoption of Brain Machine Interfaces (BMIs) has sofar been limited by the cost of the EEG equipment,cumbersome setup, and low portability. Also advancedanalysis algorithms have traditionally not been applied inreal-time, limiting the potential of BMIs to decoding ofsimple states. The presented system addresses thoseissues by focusing on a low-cost consumer grade solution(an off-the-shelf neuroheadset combined with asmartphone or tablet device) with complex real-timeanalysis on the portable device, not limiting the use-casesto the classical stationary setup. The implementedmodules constitute a flexible framework enabling variousBrain Machine Interfaces to be built utilizing affective andcognitive states of various complexity as well as enablingmobility during EEG monitoring.

Smartphone Brain ScannerThe Smartphone Brain Scanner provide multiple functionsincluding stimulus delivery, data acquisition, logging, brainstate decoding and 3D brain activity visualization. In thecurrent system a wireless Emotiv EPOC 14 channelneuroheadset is used to acquire the raw EEG data with asampling rate of 128 Hz. All subsequent processing isperformed directly on the mobile device by custom-madesoftware. In the current configuration the software is ableto perform real-time brain state decoding as well asprovide bio-feedback to the user through a 3D renderingof the active cortical EEG sources.

The 3D brain model contains 1028 vertices and 2048triangles and the user can interact with the 3D model on

the device using touch gestures. The source activity isreconstructed using a low resolution brain electromagnetictomography (LORETA) approach, performed in 16samples window, resulting in the snapshots update rate of8 Hz. Based on the source activations, a spectrogram canbe calculated using Fast Fourier Transform performed onall vertices on the 16 samples window resulting in aspectrogram from 0 to 64 Hz with a 1 Hz resolution. Thebrain activity is rendered using ranges of RGB colorvalues, providing a performance of approximately 30 fpsand fluent touch-based interaction with the 3D model.

Figure 1: The Smartphone Brain Scanner running on a Linux-based EXOPC tablet (topleft), the Emotiv neuroheadset, Android-based Acer tablet (top right), Android-basedHTC smartphone (bottom left), Maemo-based N900 smartphone (middle), and aMeeGo-based N950 smartphone (bottom right).

In a configuration using the Nokia N900 smartphone thesystem has a delay of less than 150 ms. between thesignal emerging in the brain and being visualized on thesmartphone. The delay depends on the amount ofprocessing enabled and the mode of operation. Thesystem implementation is based on the Qt framework andbenefits from the cross-platform support of multiple

hardware platforms (smartphones, tablet devices,netbooks, and PCs) that are based on the Linux operatingsystem. Thus the system runs on multiple platforms,including Maemo/MeeGo based smartphones,Android-based smartphones and tablet devices, as shownin Figure 1.

DiscussionThe main challenge for the mobile real-time brain scanneris the power consumption on the mobile device(e.g. smartphone) and the neuroheadset, which both puta constraint on the duration the system can be used. Wehave carried out experiments with continuous locallogging of EEG data, which allowed 7.5 hours of usage.However, with online data analysis only 3.5 hours of usageis possible.

To validate the brain state decoder we examined its abilityto distinguish between left and right imagined fingertapping on a single subject. We applied the CommonSpatial Patterns (CSP) algorithm combined with lineardiscriminant analysis (LDA) as it has shown useful fordiscriminating between different motor tasks [3]. Inputdata was bandpass filtered 8-32 Hz and focused on theinterval 0.75-2.00s after the stimuli, which resulted in anaccuracy of 64%. Additionally, we examined if thesmartphone brain scanner could distinguish pleasant andunpleasant images based on the EEG recordingsrepresenting the subjects affective response. The imageswere selected from the international affective picturesystem (IAPS) based on their classes and in accordance to[1]. LDA was used to separate the two classes given inputfeatures focusing on the power in the frequency interval4-48Hz and time interval 100-500ms after stimuli. Six outof eight subjects could be classified with an accuracyabove a random baseline model.

To analyze whether the generic design of theneuroheadset is capable of capturing common patternsacross the 8 subjects, we reduced the dimensionality usingprincipal component analysis (PCA) and applied aK-means algorithm (N=10) to cluster 8 x 14 ICAindependent components based on scalp maps and powerspectrum. The results show that 32, 19 and 34 out of 112ICA components were shared among 8, 7 and 6 subjectsrespectively – all within 3 standard deviations of thecentroids. This indicates an ability to consistently capturecommon patterns of brain activity across subjects, evenwhen taking into account the less accurate positioningand limited number of electrodes on the neuroheadsetcompared to traditional EEG caps. While the clusteredICA components do not represent absolute scalp mappolarities as such, they indicate common sources ofsynchronous brain activity, consistent with activities incentral, temporal and parietal cortex previously observedto differentiate responses when viewing affective picturescompared to neutral content. The described classificationswere carried out offline, although a classification directlyon the device is currently in progress.

ConclusionsWe have demonstrated that complex reconstruction ispossible on multiple mobile device platforms, including 3Dreconstruction and spectrograms, and have carried outinitial experiments. The wireless system enable completeuser mobility and continuous logging of brain activitieseither for real-time neurofeedback purposes or for lateranalysis, which are both considered highly relevant forpersonal informatics applications.

The cross-platform support offers different kinds of brainmachine interaction and applications which facilitatepersonalized interfaces reflecting emotional state or

intelligent interaction on different device form factors.The ability to runtime visualize the changes in power fromthe raw EEG data in a 3D model of activations, mayenable localized control of specific brainwave frequencypatterns used for training neurofeedback control in brainmachine interfaces.

Unlike traditional lab-based EEG monitoring, a majoradvantage of the wireless setup is that it enables bothmobility and for EEG monitoring in natural settings, andthereby enabling continuous monitoring of brain signalsfor personal informatics applications.

AcknowledgmentsThis work is supported in part by Danish LundbeckFoundation through Center for Integrated Molecular BrainImaging (CIMBI).

References[1] Lang, P. J., Bradley, M. M., and Cuthbert, B. N.

International affective picture system (iaps): Affectiveratings of pictures and instruction manual. Tech. Rep.A-8, University of Florida, Gainesville, FL, 2008.

[2] Makeig, S., Gramann, K., Jung, T.-P., Sejnowski,T. J., and Poizner, H. Linking brain, mind andbehavior. Int J Psychophysiol 73, 2 (2009), 95–100.

[3] Muller-Gerking, J., Pfurtscheller, G., and Flyvbjerg, H.Designing optimal spatial filters for single-trial EEGclassification in a movement task. Clinicalneurophysiology 110, 5 (1999), 787–798.

[4] Weiskopf, N., Scharnowski, F., Veit, R., Goebel, R.,Birbaumer, N., and Mathiak, K. Self-regulation oflocal brain activity using real-time functional magneticresonance imaging (fMRI). J of Psysiology 98, 4-6(2004), 357.