AR-VR Workshop

DESIGNING COMPELLING AR AND VR EXPERIENCES

Mark Billinghurst [email protected]

Zi Siang See [email protected]

vsmm2016.org

October 17th 2016

About Us • Mark

• PhD University of Washington • Founder, HIT Lab NZ • Professor, University South Australia

• Zi Siang • Faculty member, Sunway • Creative Director, Reina Imaging • Academic, University Tunku Abdul Rahman

Overview • 9:30 Introduction (Mark + Zi Siang) • 9:35 Introduction to Virtual Reality (Mark) • 10:00 Developing VR with ENTiTi (Zi Siang) • 10:30 Introduction to Augmented Reality (Mark) • 11:00 Developing AR with ENTiTi (Zi Siang) • 11:30 Building Outdoor AR with Wikitude (Zi Siang) • 12:00 Research Directions/Questions (Mark) • 12:30 Finish

What You Will Learn • Definitions of AR/VR • History of AR/VR • Example applications • How to make AR/VR experiences • Hands-on with authoring tools • Best interaction methods • Research directions in AR/VR

Authoring Tools Used

1)  Augmented Reality (AR) 2)  Virtual Reality (VR) 3)  Outdoor AR experience

- ENTiTi Creator - ENTiTi Creator

- Wikitude World

ENTiTi Creator (Desktop)

• AR/VR application building for non-programmers • Available from http://www.wakingapp.com

Install for PC or Mac

ENTiTi Mobile Application

• Download and Install the ENTiTi app • Search for ENTiTi on Android or iOS stores

Wikitude Mobile Application

• Download and Install the Wikitude Mobile app • Search for Wikitude on Android or iOS stores

Logistics •  Install software on own machines/phone

• ENTiTi desktop/mobile applications • Wikitude mobile application

• Share VR viewers

• Using WIFI (in Sunway University) • Download workshop content

•  http://www.su2crcdm.org/vsmm2016/workshop/arvr

Introduction to Virtual Reality

Ivan Sutherland (1963)

• Sketchpad – first interactive graphics program

The Ultimate Display “The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal”.

Ivan Sutherland, 1965

An Invisible Interface

Virtual Reality

Computer generated multi-sensory simulation of an artificial environment that is interactive and immersive.

What is Virtual Reality? Virtual reality is..

a computer technology that replicates an environment, real or imagined, and simulates a user's physical presence and environment to allow for user interaction. (Wikipedia)

• Defining Characteristics • Environment simulation • Presence •  Interaction

First VR Experience

• “This is so real..” • https://www.youtube.com/watch?v=pAC5SeNH8jw

Key Technologies • Autonomy

• Head tracking, body input •  Intelligent systems

•  Interaction • User input devices, HCI

• Presence • Graphics/audio/multisensory output • Multisensory displays

•  Visual, audio, haptic, olfactory, etc

Types of VR

19

https://immersivelifeblog.files.wordpress.com/2015/04/vr_history.jpg

Ivan Sutherland HMD (1968)

https://www.youtube.com/watch?v=NtwZXGprxag

Early Experimenters (1950’s – 80’s)

Helig 1956

Sutherland 1965

Furness 1970’s

The First Wave (1980’s – 90’s)

NASA 1989 VPL 1990’s

Virtuality 1990’s

Desktop VR - 1995 • Expensive - $150,000+ •  2 million polys/sec • VGA HMD – 30 Hz • Magnetic tracking

Rise of Commercial VR Companies • W Industries/Virtuality (1985 - 97)

•  Location based entertainment •  Virtuality VR Arcades

• Division (1989 – 1998) •  Turn key VR systems •  Visual programming tools

• Virtual i-O (1993 -1997) •  Inexpensive gamer HMDs

• Sense8 (1990 - 1998) •  WorldToolKit, WorldUp •  VR authoring tools

• April 2007 Computer World • VR Voted 7th on of 21 biggest flops

• MS Bob #1

Second Wave (2010 - ) • Palmer Luckey

• HMD hacker • Mixed Reality Lab (MxR)

• Oculus Rift (2011 - ) •  2012 - $2.4 million kickstarter •  2014 - $2B acquisition FaceBook •  $350 USD, 110o FOV

Desktop VR 2016 • Graphics Desktop

• $1,500 USD • >4 Billion poly/sec

• $600 HMD • 1080x1200, 90Hz

• Optical tracking • Room scale

Market Size

Computer Based vs. Mobile VR

Oculus Rift

Sony Morpheus

HTC/Valve Vive

2016 - Rise of Consumer HMDs

HTC Vive

• Room scale tracking • Gesture input devices

Example Vive App – Tilt Brush

https://www.youtube.com/watch?v=ijukZmYFX-0

Mobile VR: Google Cardboard

• Released 2014 (Google 20% project) • >5 million shipped/given away • Easy to use developer tools

+ =

Multiple Mobile VR Viewers Available

•  In 2016 – 46m possible desktop VR users vs. 400 m mobile VR users •  https://thoughts.ishuman.co/vr-will-be-mobile-11529fabf87c#.vfcjzy1vf

•  zxcvz

Mobile VR Applications

Types of VR Experiences •  Immersive Spaces

•  360 Panorama’s/Movies • High visual quality •  Limited interactivity

•  Changing viewpoint orientation

•  Immersive Experiences •  3D graphics

•  Lower visual quality • High interactivity

•  Movement in space •  Interact with objects

Immersive Panorama

• High quality 360 image or video surrounding user • User can turn head to see different views •  Fixed position

Example: Cardboard Camera

• Capture 360 panoramas • Stitch together images on phone • View in VR on Cardboard

Example Applications • VRSE – Storytelling for VR

•  http://vrse.com/ • High quality 360 VR content

• New York Times VR Experience • NYTVR application • Documentary experiences

• Vrideo •  http://vrideo.com/ • Streamed immersive movies

Capturing Panoramas • Stitching photos together

•  Image Composite Editor (Microsoft) • AutoPano (Kolor)

• Using 360 camera • Ricoh Theta-S •  Fly360

Google Cardboard App • 7 default experiences

• Earth: Fly on Google Earth

•  Tour Guide: Visit sites with guides

• YouTube: Watch popular videos

• Exhibit: Examine cultural artifacts

• Photo Sphere: Immersive photos

• Street View: Drive along a street

• Windy Day: Interactive short story

100’s of Google Play Cardboard apps

Sample Applications

Example: Vanguard V application

https://www.youtube.com/watch?v=YOiQ01Mxuo4

Building VR Experiences

What You Need • Cardboard Viewer/VR Viewer

•  https://www.google.com/get/cardboard/

• Smart phone • Android/iOS

• Authoring Tools/SDK • Google VR SDK • Unity/Unreal game engine • Non programming tools

• Content •  3D models, video, images, sounds

Software Tools • Low level SDKs

• Need programming ability •  Java, C#, C++, etc

• Example: Google VR SDK (iOS, Android) •  https://developers.google.com/vr/

• Game Engines • Powerful, need scripting ability

• Unity - https://unity3d.com/

• Unreal - https://www.unrealengine.com/vr • Combine with VR plugins (HMDs, input devices)

•  Google VR Unity plugin

Unity Interface

Tools for Non-Programmers •  Focus on Design, ease of use

•  Visual Programming, content arrangement

• Examples •  Insta-VR – 360 panoramas

•  http://www.instavr.co/ •  Vizor – VR on the Web

•  http://vizor.io/ •  A-frame – HTML based

•  https://aframe.io/ •  ENTiTi – Both AR and VR authoring

•  http://www.wakingapp.com/ •  Eon Creator – Drag and drop tool for AR/VR

•  http://www.eonreality.com/eon-creator/

Designing Mobile VR Applications • Things to consider

• Ease of use •  Type of experience

•  Immersive images vs. 3d interaction •  Length of experience

•  2D versus 3D information presentation • Constraints

•  Limited graphics power •  Limited user input/interaction

•  Head pointing, button •  Limited feedback (audio, video, no haptic)

Physiological Considerations • Factors to Consider

• Head tracking • User control of movement • Use constant velocity • Grounding with fixed objects • Brightness changes

Universal VR Interaction Tasks • Object Interaction

• Selection: Picking object(s) from a set • Manipulation: Modifying object properties

• Navigation •  Travel: motor component of viewpoint motion • Wayfinding: cognitive component; decision-making

• System control •  Issuing a command to change system state or mode

Interactive Patterns – Setup/Control • Setup factors to consider:

• Entering and exiting • Headset adaptation • Full Screen mode • API calls •  Indicating VR apps

Entering VR

• Provide user setup instruction

Example: GearVR Interface

•  2D Interface in 3D Environment • Head pointing and click to select

Interactive Patterns - Display Reticle

• Easier for users to target objects with a display reticle • Can display reticle only when near target object • Highlight objects (e.g. with light source) that user can target

Example: Gaze Selection

https://www.youtube.com/watch?v=zwtJLE69uR4

Example: Gaze Menu Selection

https://www.youtube.com/watch?v=T0PNfc_Yibk

Interactive Patterns - Controls

• Fuze buttons •  Time based head pointing with no click input

•  Visual countdown, button placement

• Gaze and click •  Target size and selection

Example

• Show pointing reticle • Countdown timer with activated

Example: Fuze Button Selection

https://www.youtube.com/watch?v=lJmBEWkWSBY

Example: Bubble Menus

https://www.youtube.com/watch?v=Eq09WERtA3M

Example: User Interface Toggling

https://www.youtube.com/watch?v=QSYLOc5nf10

Navigation: Gaze Directed Walking

• Move in direction that you are looking • Very intuitive, natural navigation • Can be used on simple HMDs (Google Cardboard • But: Can’t look in different direction while moving

Example

https://www.youtube.com/watch?v=ZZC0ef604WU

Guided Navigation Technique

• Water skiing metaphor for VR movement

Example: VR Roller Coaster

VR Coaster Demo

https://www.youtube.com/watch?v=JZNiHI6aN2Y

Interactive Patterns - Feedback

• Use audio and haptic feedback • Reduce visual overload • Audio alerts • 3D spatial sound • Phone vibrations

Google Design Guidelines

• Google’s Guidelines for good VR experiences: • Physiological Considerations •  Interactive Patterns

•  Setup •  Controls •  Feedback •  Display Reticle

•  From http://www.google.com/design/spec-vr/designing-for-google-cardboard/a-new-dimension.html

Cardboard Design Lab Application

• Use Cardboard Design Lab app to explore design ideas

More Reading

• UX of VR website: http://www.uxofvr.com/

VR Authoring

Trends or Fab? The 80s 2016 up+

Conventional Mobile AR & VR

Conventional Mobile AR

Conventional Mobile VR360

AR & VR on Head Mount Devices

AR & VR on Head Mount Devices Mobile device

(as computing module)

•  Carl Zeiss just announced VR One, a virtual reality headset for use with a smartphone. It is a viewer designed to work with phones between 4.7 and 5.2 inches (Zeiss, 2014).

Google Cardboard

•  Carl Zeiss just announced VR One, a virtual reality headset for use with a smartphone. It is a viewer designed to work with phones between 4.7 and 5.2 inches (Zeiss, 2014).

Programmable NFC Tag

Low cost resin /plastic lens

Mobile device as computing module

Endless configurable size and shapes (and materials)

Magnetometer

Google Cardboard (v1)

more reading at http://www.gizmag.com/google-cardboard-2-review-initial/37777/

Presses the screen

Figure: google cardboard v2 supports larger phone.

Google Cardboard (v2)

Figure: Ikea AR (using metaio)

Figure: European researchers used virtual content to recreate Mosul Museum destroyed in civil war.

VR AR

Sourcedandreinterpretedfromh1p://www.augment.com/blog/which-headset-is-right-for-you/

Challenges • Consideration

• Mostly everyone has a mobile device • Only few individuals have expensive HMD • Everyone can access to low-cost HMD

• Design and Development • Content creation, authoring • Usable user interface (minimal?) • Display platforms or systems • Ease-of-use • Useful Experience

Local or Cloud-based App? Cloud-based

•  Unlimited numbers of recognition •  Unlimited content from server •  Requires network •  Stability relies on network speed •  OS update safe for content •  Shows new content automatically •  Users does not own content

Local-based •  Limited numbers of recognition •  Limited content in-App •  Works offline •  Stable •  OS update affects App & content •  Requires users to update App •  Users can own content

Authoring • Virtual Reality

• Virtual Reality (VR) which can be referred to as immersive multimedia or computer-simulated life, replicates an environment that simulates physical presence in places in the real world or imagined worlds and lets the user interact in that world.

• Augmented Reality • Augmented Reality (AR) mixes a live real-world view

with virtual interactive content on a mobile or wearable device. One of the key enablers for this is tracking technology, such as computer vision techniques for tracking off pre-defined markers or markerless images.

VR using ENTiTi

ENTiTi /Waking App: VR

Experience

ENTiTi Creator – http://www.wakingapp.com

ENTiTi Creator

ENTiTi Creator

Virtual Reality

Project name

Choose “VR Images Presentation”

ENTiTi Creator

ENTiTi Creator

In library, import all T-shirt images (any square images)

ENTiTi Creator

Library is now updated with new assets

ENTiTi Creator

Double click on items to insert T-shirt images

ENTiTi Creator

insert T-shirt images

ENTiTi Creator

Save, Publish

Mobile App Preview: ENTiTi

Search in ENTiTi: name of your project

Select “Virtual Reality”

Mobile App Preview

Mobile App Preview

Experience •  Hands-free navigation •  Insert in google cardboard

VR using ENTiTi (moving in VR)

ENTiTi /Waking App: VR Search: VSMM2016

Experience

ENTiTi Creator

Username: [email protected] Password: XXXXX

ENTiTi Creator

Open “Hello VSMM”

ENTiTi Creator

Project > Copy Project > New Name (Put your new name)

ENTiTi Creator

Now a new copy of the project is created under your name

ENTiTi Creator

Several assets were pre-uploaded in this project simulation

ENTiTi Creator

footprint

Object

0.3

Point 1

ENTiTi Creator

Save and publish

ENTiTi /Waking App: VR Search: Your project name 1)  Gaze at the footprint 2)  Observe the footprint 3)  Is it moving?

ENTiTi Creator

Delete this string

ENTiTi Creator

footprint

ENTiTi Creator

Save and publish

ENTiTi /Waking App: VR Search: Your project name 1)  Gaze at the footprint 2)  Observe movement

Experience Gaze at the footprint

Experience Moving towards Point 1

Visual Programming • Benefits

• Minimum coding required • Prototyping basic VR scenario

• Recommendation • Storyline of scenario is essential • Content quality requires lengthy development • Choose suitable platforms for specific need

Introduction to Augmented Reality

1977 – Star Wars – Augmented Reality

Augmented Reality Definition

• Defining Characteristics [Azuma 97] • Combines Real and Virtual Images

• Both can be seen at the same time • Interactive in real-time

• The virtual content can be interacted with • Registered in 3D

• Virtual objects appear fixed in space

Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

2008 - CNN

https://www.youtube.com/watch?v=lbcarXDDqMk

•  Put AR pictures here

Augmented Reality Examples

AR vs VR

Milgram’s Reality-Virtuality continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment

"...anywhere between the extrema of the virtuality continuum."

P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

AR History

Pepper’s Ghost (1862)

• Dates back to Giambattista della Porta (1584)

Sutherland HMD (1968) • 1968: Sutherland / Sproull’s first HMD system •  see-through stereo display • head tracking

US SuperCockpit Program (1970’s-80’s)

Superimpose flight information over real world

Industrial and Academic Research (1990’s- )

• Early 1990’s: Boeing coined the term “AR.” • Mid 1990’s AR research in tracking and display (UNC)

Early Commercialization (2000 – 2010)

• 2000: Augmented sports broadcasts • 2007: PlayStation Eye of Judgement

Consumer Adoption (2009 - )

• Web pages with AR experiences integrated into them • Smart phones with built-in sensors suitable for mobile AR

Mobile Augmented Reality

CPU: 300 Mhz HDD; 9GB RAM: 512 mb Camera: VGA 30fps Graphics: 500K poly/sec

1998: SGI O2 2008: Nokia N95

CPU: 332 Mhz HDD; 8GB RAM: 128 mb Camera: VGA 30 fps Graphics: 2m poly/sec

2005 - Mobile Phone AR

• Mobile Phones • camera • processor • display

• AR on Mobile Phones • Simple graphics • Optimized computer vision • Collaborative Interaction

AR Advertising (HIT Lab NZ 2007)

• Txt message to download AR application (200K) • See virtual content popping out of real paper advert • Tested May 2007 by Saatchi and Saatchi

2008: Location Aware Phones

Nokia Navigator Motorola Droid

2009 - Outdoor Information Overlay • Mobile phone based • Tag real world locations

• GPS + Compass input • Overlay graphics on live video

• Applications • Travel guide, Advertising, etc

• Wikitude, Layar, etc.. • iOS/Android, Public API released

Layar Demo (2009)

•  https://www.youtube.com/watch?v=b64_16K2e08

Augmented Reality Business Today • Rapidly Growing

• > $80 Billion USD by 2020

• Wide range of HW/SW available • HMD, mobile phones, PCs • Many easy to use developer tools

• Many application areas • Marketing, gaming, education • Mobile AR

Pokemon GO

Killer Combo: brand + social + mobile + geo-location + AR

Pokemon GO Effect

•  Fastest App to reach $500 million in Revenue •  Only 63 days after launch, > $1 Billion in 6 months •  Over 500 million downloads, > 25 million DAU •  Nintendo stock price up by 50% (gain of $9 Billion USD)

AR Technology

Key Enabling Technologies

1.  Combines Real and Virtual Images Display Technology

2.  Registered in 3D Tracking Technologies

3.  Interactive in real-time Interaction Technologies

AR Display Technologies

• Handheld Displays • Mobile phone, tablets

• Head mounted displays • Optical/video see-through

• Fixed Displays • Desktop, large screen

• Projected Displays • Projected images on real world

Hololens (2016)

•  Integrated system – Windows • Stereo see-through display • Depth sensing tracking • Voice and gesture interaction

View Through Hololens

https://www.youtube.com/watch?v=RddvMLwT__g

AR Interaction • Natural user interaction

• Gesture, body input

• Handheld •  Touch based interaction • Device motion

• Physical object •  Familiar tool, object

AR Tracking • Goal

•  Find users viewpoint

• Outdoor Tracking • GPS, compass

•  Indoor • Computer vision

•  Tracking known features

Example: Vuforia Tracking

https://www.youtube.com/watch?v=MHUncdOytuM

Tracking Targets

Image

Object

Environment

• Weak AR •  Imprecise tracking •  No knowledge of environment •  Limited interactivity •  Handheld AR

• Strong AR •  Very accurate tracking •  Seamless integration into real world •  Natural interaction •  Head mounted AR

Strong vs. Weak AR

Architecture

AR Applications

• Web based AR •  Flash, HTML 5 based AR •  Marketing, education

• Outdoor Mobile AR •  GPS, compass tracking •  Viewing Points of Interest in real world

• Handheld AR •  Vision based tracking •  Marketing, gaming

•  Location Based Experiences •  HMD, fixed screens •  Museums, point of sale, advertising

Typical AR Experiences

Medical Applications

• Using AR to see imagery superimposed inside body • Enables doctor to see information at body site

Example: Ankle Joint

Gaming: Rock-em Sock-em

• Shared AR Demo • Markerless tracking

Rock’em Sock’em Demo

https://www.youtube.com/watch?v=hXtq1qBMLIw

Pepsi AR Experience (2014)

• Video see-through AR in bus shelter • Bus shelter appears under attack

Pepsi Demo

https://www.youtube.com/watch?v=Go9rf9GmYpM

CityViewAR Application (Android)

• Visualize Christchurch before the earthquakes • Search for CityViewAR on Android play store

CityViewAR

https://www.youtube.com/watch?v=fdgrXxJx4SE

Education: Quiver (iOS/Android)

•  Interactive Colouring Books •  Children colour their own AR scenes

• Wide range of educational pages available •  Animals, cells, volcanos, etc

•  http://www.quivervision.com/

Quiver Demo

https://www.youtube.com/watch?v=xirCqQFr6K8

AR Interface Design

• Interface Components • Physical components • Display elements

• Visual/audio

• Interaction metaphors

Physical Elements

Display Elements Interaction

Metaphor Input Output

AR Design Principles

AR Design Space

Reality Virtual Reality

Augmented Reality

Physical Design Virtual Design

Tangible User Interfaces (Ishii 97) • Create digital shadows for physical objects

• Foreground • graspable UI

• Background • ambient interfaces

i/O Brush (Ryokai, Marti, Ishii)

Tangible AR Interaction Metaphor

• AR overcomes limitation of TUIs • enhance display possibilities • merge task/display space • provide public and private views

• TUI + AR = Tangible AR • Apply TUI methods to AR interface design

Tangible AR Design Principles

• Tangible AR Interfaces use TUI principles • Physical controllers for moving virtual content • Support for spatial 3D interaction techniques • Time and space multiplexed interaction • Support for multi-handed interaction • Match object affordances to task requirements • Support parallel activity with multiple objects • Allow collaboration between multiple users

VOMAR Interface

Handheld HCI

• Consider your user • Follow good HCI principles • Adapt HCI guidelines for handhelds • Design to device constraints • Rapid prototyping • User evaluation

Consider Your User • Consider context of user

•  Physical, social, emotional, cognitive, etc

• Mobile Phone AR User

•  Probably Mobile • One hand interaction •  Short application use • Need to be able to multitask • Use in outdoor or indoor environment • Want to enhance interaction with real world

Applying Principles to Mobile AR • Clean •  Large Video View •  Large Icons •  Text Overlay •  Feedback

Design to Device Constraints • Understand the platform and design for limitations

• Hardware, software platforms • Eg Handheld AR game with visual tracking

• Use large screen icons • Consider screen reflectivity • Support one-hand interaction • Consider the natural viewing angle • Do not tire users out physically • Do not encourage fast actions • Keep at least one tracking surface in view

Art of Defense Game

HandHeld AR Wearable AR

Output: Display

Input

Input & Output

HMD vs Handheld AR Interface

Handheld Interface Metaphors

• Tangible AR Lens Viewing •  Look through screen into AR scene •  Interact with screen to interact with AR

content •  Eg Invisible Train

• Tangible AR Lens Manipulation •  Select AR object and attach to device •  Use the motion of the device as input

•  Eg AR Lego

AR using ENTiTi

ENTiTi /Waking App: AR

AR with image-based tracking

ENTiTi Creator

Augmented Reality

Project name

Choose “Video Business Card”

ENTiTi Creator

New image target to be scanned

ENTiTi Creator

Select namecard sample image

ENTiTi Creator

1)  Assign your image 2)  Click “Preview”

ENTiTi Creator


We need to upload what to be viewed or experienced

2) Upload 1) Library

ENTiTi Creator


Choose an mp4 video

ENTiTi Creator


Insert video

ENTiTi Creator


“OK” for including new video

Sample is provided “entiti_ar_showreel_small.mp4”

ENTiTi Creator

Click on “LOGIC” to provide info /setting

Video is now uploaded!

ENTiTi Creator

Parameters can be provided •  Facebook •  Linkedin

ENTiTi Creator

Click on “LOGIC” to provide info /setting

Mobile App Preview: ENTiTi Overlay it to your namecard

Search in ENTiTi: name of your project

Experience AR content (video, fb, linkedin)

Location-based AR

Wikitude App

AR Point of Interest


Video: Microsoft future


Video: Layar, Impactful Augmented Reality in Your Everyday Life

Challenges • Design and Development

• Content creation, authoring • Display platforms or systems •  Tracking approaches

Sensors

Wikitude World: location-based AR

Wikitude World: location-based AR • KML

• Keyhole Markup Language is a standardized format used in Google Earth. It can provide basic information for POIs and easily uploaded (or using hyperlink) into Wikitude.

• KML files can be created with the Google Earth user interface (in Google Map)

• Alternatively, it can be created using XML or simple text editor to work on raw KML scripts from scratch.

https://developers.google.com/kml/documentation/?hl=en

Wikitude World: location-based AR • ARML

• Augmented Reality Markup Language is an open exchange format based on KML but extends the format for useful data around AR data sets.

• ARML 2.0 is used in the live versions of the 3 leading Augmented Reality Browsers (Junaio, Layar and Wikitude), where it is used to make the AR Browsers interoperable.

• ARML 1.0 file format is currently used in the Wikitude World Browser.

http://openarml.org/wikitude4.html

Steps for Using Wikitude 1.  Register for Wikitude account

•  http://www.wikitude.com 2.  Register for web hosting account

•  http://www.000webhost.com • Own domain and server hosting

3.  Content authoring •  https://www.google.com/mymaps/ •  http://studio.wikitude.com • Scripting and customization

Publish in Wikitude • KML

•  [put stuff here] • ARML

•  [put stuff here]

Free Web Hosting • KML



Part 1: KML

https://www.google.com/mymaps/ • KML



KML (Google Map) • KML



KML: XML Scripting • KML



Steps for Setting up Wikitude 1.  Register for Wikitude account


•  http://www.000webhost.com • Or use own domain and server hosting

Wikitude (Registration)

KML • Benefits

• Generated from Google Earth • Allows basic editing

• Current limitation •  Limited scripting option for Wikitude World •  Lack of options for POI details in Wikitude World • Range is confined to ~20km radius

Publish in Wikitude: KML • KML



Publish in Wikitude: KML • KML



provide URL, host the *.kml file from your own server

Wikitude Mobile App

Part 2: ARML

Managing files in domain + server •  Download ARML Script (for workshop)

•  http://www.laboratoryworkshop.net23.net

•  Examples of URL of our uploaded files •  http://www.markbillinghurst.com/testing.png •  http://www.laboratoryworkshop.net23.net/hitlab_512x512.png •  http://www.zisiangsee.com/hitlab_512x512.png •  http://www.elnadiana.net23.net/upsiAR.png •  http://www.laboratoryworkshop.net23.net/wikitude-

arml_workshop_utm.xml

Publish in Wikitude • KML



ARML: XML Scripting • KML




wikitude-arml_workshop.xml <?xml version="1.0" encoding="UTF-8"?> <kml xmlns="http://www.opengis.net/kml/2.2" xmlns:ar="http://www.openarml.org/arml/1.0" xmlns:wikitude="http://www.openarml.org/wikitude/1.0"> <Document> <ar:provider id=“vsmm-workshop-arml"> <ar:name>AR Workshop (ARML)</ar:name> <ar:description>Creating POI location-based AR</ar:description> <wikitude:providerUrl>http://www.hitlabnz.org</wikitude:providerUrl> <wikitude:logo>http://www.zisiangsee.com/wikitude/hitlabnz/hitlab_512x512.png</wikitude:logo> </ar:provider> <Placemark id="123"> <ar:provider>vsmm-workshop-arml</ar:provider> <name>LEGOLAND</name> <description>A fantastic place to visit in Iskandar Johor. Open 10:00AM-7:00PM.</description> <wikitude:info> <wikitude:thumbnail> http://www.zisiangsee.com/wikitude/hitlabnz/hitlab_512x512.png </wikitude:thumbnail> <wikitude:phone>+6075978888</wikitude:phone> <wikitude:url>http://www.legoland.com.my</wikitude:url> <wikitude:email>[email protected]</wikitude:email> <wikitude:address>7, Jalan Legoland, Bandar Medini,, 79250 Nusajaya, Johor, Malaysia</wikitude:address> </wikitude:info> <Point> <coordinates>103.63179030000003,1.426637,0.0</coordinates> </Point> </Placemark> </Document> </kml>


Keep consistent


Allows replication (use unique numeric id) For example <Placemark id=“124”> or <Placemark id=“125”>


Longlitude, Latitude, Altitude (altitude is optional)

ARML: Tags


Publish in Wikitude: ARML • KML



Publish in Wikitude: ARML • KML



provide URL, host the *.xml file from your own server

Wikitude Mobile App

ARML • Benefits

• Extended scripting option for Wikitude World • Additional POI details in Wikitude World

• Current limitation • Propagation: Wikitude World may take time to be

visible. • Range is confined to ~20km radius

•  Future possibilities • AR link may be interoperated with other AR browser.

Authoring Options : Unity + Vuforia

Other Authoring Options: A-FRAME

Other Authoring Options: Vizor

Other Authoring Options: Insta-VR

Other Authoring Options: EON Reality

AR Use Cases

Discussion • Potentials

• Other experience we can include for mixed reality?

Figure: Scentee Figure: Senseg

Discussion

• Considerations • Variety of options for AR/VR authoring. • Design industries - easy, useful, empathic content. • Mixed-reality experience - taste, scent, haptic, touch,

audio, motion.

Research Directions

Key Enabling Technologies

1.  Combines Real and Virtual Images Display Technology

2.  Registered in 3D Tracking Technologies

3.  Interactive in real-time Interaction Technologies

DISPLAY

• Past • Bulky Head mounted displays

• Current • Handheld, lightweight head mounted

• Future • Projected AR • Wide FOV see through • Retinal displays • Contact lens

Evolution in Displays

Wide FOV See-Through (3+ years) • Waveguide techniques

• Wider FOV •  Thin see through • Socially acceptable

• Pinlight Displays

•  LCD panel + point light sources •  110 degree FOV • UNC/Nvidia

Lumus DK40

Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH 2014 Emerging Technologies (p. 20). ACM.

Pinlight Display Demo

https://www.youtube.com/watch?v=tJULL1Oou9k

Light Field Displays

https://www.youtube.com/watch?v=J28AvVBZWbg

Retinal Displays (5+ years) • Photons scanned into eye

•  Infinite depth of field • Bright outdoor performance • Overcome visual defects •  True 3D stereo with depth modulation

• Microvision (1993-) • Head mounted monochrome

• MagicLeap (2013-) • Projecting light field into eye

Contact Lens (10 – 15 + years) • Contact Lens only

• Unobtrusive • Significant technical challenges

•  Power, data, resolution • Babak Parviz (2008)

• Contact Lens + Micro-display • Wide FOV •  socially acceptable •  Innovega (innovega-inc.com)

http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/

TRACKING

Evolution of Tracking • Past

•  Location based, marker based, • magnetic/mechanical

• Present •  Image based, hybrid tracking

• Future • Ubiquitous • Model based • Environmental

Model Based Tracking (1-3 yrs) • Track from known 3D model

• Use depth + colour information • Match input to model template • Use CAD model of targets

• Recent innovations •  Learn models online •  Tracking from cluttered scene •  Track from deformable objects

Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., & Navab, N. (2013). Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In Computer Vision–ACCV 2012 (pp. 548-562). Springer Berlin Heidelberg.

Deformable Object Tracking

https://www.youtube.com/watch?v=KThSoK0VTDU

Environmental Tracking (3+ yrs)

• Environment capture • Use depth sensors to capture scene & track from model

•  InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/) • Real time scene capture on mobiles, dense or sparse capture • Dynamic memory swapping allows large environment capture • Cross platform, open source library available

InfinitAM Demo

https://www.youtube.com/watch?v=47zTHHxJjQU

Fusion4D (2016)

• Shahram Izhadi (Microsoft + perceptiveIO) • Real capture and dynamic reconstruction • RGBD sensors + incremental reconstruction

Fusion4D Demo

•  https://www.youtube.com/watch?v=rnz0Kt36mOQ

Wide Area Outdoor Tracking (5+ yrs)

• Process • Combine panorama’s into point cloud model (offline) •  Initialize camera tracking from point cloud • Update pose by aligning camera image to point cloud • Accurate to 25 cm, 0.5 degree over very wide area

Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.

Wide Area Outdoor Tracking

https://www.youtube.com/watch?v=8ZNN0NeXV6s

Outdoor Localization using Maps

• Use 2D building footprints and approximate height • Process

•  Sensor input for initial position orientation •  Estimate camera orientation from straight line segments •  Estimate camera translation from façade segmentation •  Use pose estimate to initialise SLAM tracking

• Results – 90% < 4m position error, < 3o angular error Arth, C., Pirchheim, C., Ventura, J., Schmalstieg, D., & Lepetit, V. (2015). Instant outdoor localization and SLAM initialization from 2.5 D maps. IEEE transactions on visualization and computer graphics, 21(11), 1309-1318.

Demo: Outdoor Tracking

•  https://www.youtube.com/watch?v=PzV8VKC5buQ

INTERACTION

Evolution of Interaction • Past

•  Limited interaction • Viewpoint manipulation

• Present • Screen based, simple gesture •  tangible interaction

• Future • Natural gesture, Multimodal •  Intelligent Interfaces • Physiological/Sensor based

Natural Gesture (2-5 years) • Freehand gesture input

• Depth sensors for gesture capture • Move beyond simple pointing • Rich two handed gestures

• Eg Microsoft Research Hand Tracker •  3D hand tracking, 30 fps, single sensor

• Commercial Systems • Meta, MS Hololens, Occulus, Intel, etc

Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S. (2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).

Hand Tracking Demo

https://www.youtube.com/watch?v=QTz1zQAnMcU

Multimodal Input (5+ years) • Combine gesture and speech input

• Gesture good for qualitative input • Speech good for quantitative input • Support combined commands

•  “Put that there” + pointing

• Eg HIT Lab NZ multimodal input •  3D hand tracking, speech • Multimodal fusion module • Complete tasks faster with MMI, less errors

Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.

HIT Lab NZ Multimodal Input

https://www.youtube.com/watch?v=DSsrzMxGwcA

Intelligent Interfaces (10+ years) • Move to Implicit Input vs. Explicit

• Recognize user behaviour • Provide adaptive feedback • Support scaffolded learning • Move beyond check-lists of actions

• Eg AR + Intelligent Tutoring • Constraint based ITS + AR • PC Assembly (Westerfield (2015)

•  30% faster, 25% better retention

Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent Augmented Reality Training for Motherboard Assembly. International Journal of Artificial Intelligence in Education, 25(1), 157-172.

ENHANCED EXPERIENCES

Gilmore + Pine: Experience Economy

experiences

services

products

components

Valu

e

Function

Emotion

Crossing Boundaries

Jun Rekimoto, Sony CSL

Invisible Interfaces


Milgram’s Reality-Virtuality continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)


Virtual Environment

The MagicBook

Reality Virtuality Augmented Reality (AR)


Example: Visualizing Sensor Networks

•  Rauhala et. al. 2007 (Linkoping) • Network of Humidity Sensors

•  ZigBee wireless communication

• Use Mobile AR to Visualize Humidity

UbiVR – CAMAR

CAMAR Companion

CAMAR Viewer

CAMAR Controller

GIST - Korea

ubiHome @ GIST

©ubiHome

What/When/How

Where/When

Media services

Who/What/ When/How

ubiKey

Couch Sensor

PDA

Tag-it

Door Sensor

ubiTrack

When/How

When/How

Who/What/When/How

Light service MR window

Example: Social Panoramas

• Google Glass • Capture live image panorama (compass + camera)

• Remote device (tablet) •  Immersive viewing, live annotation

Reichherzer, C., Nassani, A., & Billinghurst, M. (2014). Social panoramas using wearable computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 303-304). IEEE.

Social Panorama Demo

https://www.youtube.com/watch?v=vdC0-UV3hmY

Empathy Glasses (CHI 2016)

• Combine together eye-tracking, display, face expression •  Impicit cues – eye gaze, face expression

+ +

Pupil Labs Epson BT-200 AffectiveWear

Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.

AffectiveWear – Emotion Glasses

• Photo sensors to recognize expression • User calibration • Machine learning • Recognizing 8 face expressions

Remote Collboration

• Eye gaze pointer and remote pointing • Face expression display •  Implicit cues for remote collaboration

Example: Holoportation

• Augmented Reality + 3D capture + high bandwidth •  http://research.microsoft.com/en-us/projects/holoportation/

Holoportation Video

https://www.youtube.com/watch?v=7d59O6cfaM0

Example: Social Acceptance

• People don’t want to look silly • Only 12% of 4,600 adults would be willing to wear AR glasses •  20% of mobile AR browser users experience social issues

• Acceptance more due to Social than Technical issues • Needs further study (ethnographic, field tests, longitudinal)

TAT Augmented ID

TAT AugmentedID

Scaling Up

• Seeing actions of millions of users in the world • Augmentation on city/country level

AR + Smart Sensors + Social Networks

• Track population at city scale (mobile networks) • Match population data to external sensor data • Mine data for applications

Example: MIT SENSEable City Lab

https://www.youtube.com/watch?v=eXOCbrQYqbY http://senseable.mit.edu/wikicity/rome/

Example: CSIRO WeFeel Tool

• Emotionally mining global Twitter feeds

• http://wefeel.csiro.au

Thank you

www.empathiccomputing.org

@marknb00

[email protected]

Resources

Books • Unity Virtual Reality Projects

•  Jonathan Linowes

• Holistic Game Development with Unity •  Penny de Byl

Useful Resources • Google Cardboard main page

•  https://www.google.com/get/cardboard/

• Developer Website •  https://www.google.com/get/cardboard/developers/

• Building a VR app for Cardboard •  http://www.sitepoint.com/building-a-google-cardboard-vr-app-in-unity/

• Creating VR game for Cardboard •  http://danielborowski.com/posts/create-a-virtual-reality-game-for-

google-cardboard/

• Moving in VR space •  http://www.instructables.com/id/Prototyping-Interactive-Environments-

in-Virtual-Re/

Resources • Excellent book

•  3D User Interfaces: Theory and Practice •  Doug Bowman, Ernst Kruijff, Joseph, LaViola, Ivan Poupyrev

• Great Website •  http://www.uxofvr.com/

•  International 3DUI group •  Mailing list, annotated bibliography •  www.3dui.org

•  3DI research at Virginia Tech. •  research.cs.vt.edu/3di/

Additional Slides

Optional: Wikitude Studio: Setting up

Wikitude Studio

Steps for Setting up Wikitude 1.  Register for Wikitude account


•  http://www.000webhost.com • Or use own domain and server hosting

Wikitude Studio

Optional: Wikitude Studio: AR with 3D

Wikitude Studio

Wikitude Studio

Drag any image here that you want to track for AR

Naming

Wikitude Studio

Wikitude Studio

*.wt3 3D file (encoded by wikitude encoder)

Wikitude Studio

Drag and align the object to centre (to preview in mobile wikitude App)

Wikitude Studio

export

Wikitude Studio

Fill up all the required details, and then Publish now!

Mobile App Preview: Wikitude

Search in wikitude: name of your project

AR Use Cases

Optional: Wikitude Studio: AR Button & VR360

Customize Additional Features

AR Application

Additional VR360 on HMD


AR Application

Button

1

2 Additional VR360 on HMD


AR Application

Button

How it works? •  Additional HTML5, WebVR •  Allows customization •  Mobile devices for HMD (eg. Google Cardboard)

1

2 Additional VR360 on HMD

Customize Additional Feature • AR Application /AR Browser

•  2D image-based AR tracking • HTML5 in-app web browser for additional feature •  Include a series of VR360 content

• Requirements • AR authoring software /platform • Online hosting for additional feature /content

Wikitude Studio

Wikitude Studio

button

URL

Managing files in domain + server •  Download files needed for this workshop

•  http://www.laboratoryworkshop.net23.net

•  Examples of URL of our uploaded files •  http://www.markbillinghurst.com/virtualreality360tour •  http://www.laboratoryworkshop.net23.net/virtualreality360tour •  http://www.zisiangsee.com/virtualreality360tour •  http://www.elnadiana.net23.net/virtualreality360tour •  http://www.human.net23.net/virtualreality360tour

Uploading online content

Click on “upload”

Uploading online content • KML



File Manager


upload

Choose the zipped file provided


Open this folder “virtualreality360tour”


view “index.html”


Copy this URL (to paste it in wikitude)

Wikitude Studio

URL updated

Wikitude Studio

export

Wikitude Studio

Fill up all the required details, and then Publish now!

Mobile App Preview: Wikitude

Search in wikitude: name of your project

Mobile App Preview

Navigation •  Next /previous button •  Gallery selection •  Gyroscope activation •  Stereo view •  Full screen

Mobile App Preview

Stereo view •  Head mount device (HMD) e.g. google cardboard

Mobile App Preview

Mobile App Preview

Experience •  Hands-free next/forward access •  Insert in google cardboard

What Other Experience?

HMD 360 Experience

HMD 360 Experience • Design and Development

• Content creation • Display platforms or systems •  Tracking approaches

What Other Experience?

Currency

Wikitude Studio (cloud-based) •  Benefits

•  Quick authoring process •  Instant AR/VR experience generated •  Generated content can be used in SDK

•  Current limitation •  Limited control for some multimedia elements •  Fully rely on network performance •  SDK may be costly for further implimentation

•  Suitability •  Exhibition •  Prototyping •  Info /edutainment multimedia projects •  Industrial applications

AR-VR Workshop

Technology

Transcript of AR-VR Workshop