Designing Games for All: Exploring Output and Immersion.

Investigating the impact of modality perception on immersion in digital games

Five Primary Sensory Modalities • Vision • Auditory • Somatosensory (Touch and Proprioception) • Olfactory (Sense of Smell) • Gustatory (Sense of Taste)

The Senses

• Human Interaction and Communication is inherently multimodal – An everyday conversation includes more than just

auditory cues… – A touch can convey a whole idea…

• Human Computer Interaction remains limited – Multimodal Input is common.

• Voice, Touch, Gaze, Motion, Gesture, BMI, etc. – Output tends to be limited

• Visual, Auditory and occasionally haptic (vibration)

Multimodal Interaction

• Subjective by nature • Difficult to achieve and easy to destroy • Engagement is a vital ingredient

– Mental engagement is intensely subjective but promotes immersion very effectively.

– Physical engagement is easier to achieve and directly promotes immersion.

Immersion

• Immersion is an acknowledged ingredient of human crafted experiences – Books, Films, Theatre, Games…

• The majority of these constructs are limited by their medium with regards to their form and function

• Even when technology comes into play, the extents are often limited – 3D Films, Force Feedback

The importance of immersion

Investigates the role of two sensory modalities, specifically visual and auditory perception, in achieving effective immersion and presence. • Establishing an experimental design to provide balanced and

comparable sets of modal cues. Do so with natural interfaces for low physical engagement barriers.

• Performing a user study that would evaluate the modalities both in isolation and in combination in order to study the impact of each modality.

The Project

What did we do? • We made participants dodge cars. • Sometimes while blindfolded.

The Experiment

• Simulated a collision avoidance scenario – Virtual Cars on a highway – Mental Engagement

• Multimodal Input – Motion sensing technology (Kinect) – High Physical Engagement

• Multimodal Output – Visual and Auditory Feedback

The Setup

• The Participants where asked to dodge the incoming cars for as long as possible.

• Nine measured sessions in total – 3 With both visual and auditory feedback – 3 With only visual feedback – 3 With only auditory feedback

The Specifics

Primary desired data element: • Qualitative Immersion Measures

– Method: IMEX questionnaire Secondary data element • Quantitative performance data

– Participant survival time per session

Data Collection and Evaluation

The analysis showed that the modal output had a significant impact on immersion.

Quantitative Results - Immersion

0

10

20

30

40

50

60

70

80

90

General Immersion

Action Visceral Mental Visceral

Mea

n I

MX

sco

re o

n 0

-10

0 s

cale

Both Modalities Audio Only Visual Only

• Performance was measured by the survival time of each participant in each play through – Insignificant difference between Audio-Visual and Visual-

only sessions – Significant difference between Audio-Visual and Audio-

only sessions – as was expected • With regards to preference

– 75% preferred the Audio-Visual version – 25% preferred the Audio-only version – Noce preferred the visual only version

Quantitative Results - Performance and Preference

Overall, feedback from the users shows promising results: • Positive reactions to the experience • High levels of Immersion • Interesting responses regarding value of audio

– A/V is optimal – Visual only is “muted” and “disconnected” – Audio only is “intense” and “thrilling”

Qualitative Results & User Feedback

Three themes surfaced: • Rapid Immersion

– Physical and Mental engagement equals rapid immersion – “Damn this is intense!”

• The importance of sound – Removing a “secondary” modality has an impact – “Something was missing, I was waiting for the crunch.”

• Engaging audio experience – Removing a “primary” modality has a severe impact – “That was absolutely terrifying! I was all there!”

Qualitative Results & User Feedback

• The Starting point – The objective: Design and implement a highly immersive

digital experience – The hypothesis: Unimodal and multimodal experiences

create different levels of immersive response.

• The findings – Achieving Immersion: Relatable scenarios and natural

input greatly promote immersion – Unimodal Output: Removing primary senses leads to

intense experiences

Conclusions

• Enhanced Entertainment Experiences • Games Designed for All

– Vision Impairment • AudioGames.net

– Dexterity/Mobility Impairment • OneSwitch.org.uk

• Games Designed for All vs. Accessible Games

Potential Application

http://www.gameaccessibilityguidelines.com

• Contextual modality suppression – Scenario based sensory deprivation

• Further immersion enablers – Oculus rift

• Introduce additional output modalities – Haptic feedback

• Investigate immersive scenarios

The next step

Thanks for listening!