(모바일) 증강현실은Hype(거품)인가?

Woontack Woo (禹 雲澤), Ph.D.

http://twitter.com/wwoo_ct

GIST CTI/U-VR Lab

Gwangju, Korea

MOBAS 2012@ 숙명여대 (2012.2.2)

Outline

VR vs. AR : Is AR Hype?

Where are we now?

What’s Next?

Why Ubiquitous VR, DigiLog & CAMAR?

Summary and Q&A

Virtual Reality vs. Augmented Reality

Virtual Reality (out there):

Puts people in a computer-generated world

Augmented Reality (here):

Puts virtual content in a real physical world

Ubiquitous VR : VR over ubiComp Env (here & right now):

Forces the VR/MR/AR to live out here in the world with people

Johnny Mnemonic Minority Report

Augmented Reality vs. Virtual Reality

AR VR

Supplements reality, rather than completely replacing it

Completely immerse a user inside a synthetic environment

User maintains a sense of presence in real world (Here and Right Now!)

While immersed, the user cannot see the real world around him (Out there!)

• Apply to all senses, not just sight: sound, haptics, etc. • Extend and enhance user’s ability (performance/perception of the

world)

What’s AR?

What’s Augmented Reality?

It is really an old idea, e.g. “Terminator vision in 1990”

Milgram's Reality-Virtuality Continuum [94]

Azuma's definition on AR [97]

combines real and virtual

is interactive in real time

is registered in real 3D world

AR is Hype?

The web is dead. Long Live the Internet!

Aug. 2010 Web

Google Trend (VR vs. AR)

AR is Hype?

A: Virtual Reality Embraced by Businesses B: Another use for your phone: 'augmented reality C: Qualcomm Opens Austria Research Center to Focus on Augmented Reality D: Qualcomm Launches Augmented Reality Application Developer Challenge E: Review: mTrip iPhone app uses augmented reality F: Toyota demos augmented-reality-enhanced car windows

AR is Hype?

Half century after its birth, the VR is in decline.

Two decades after its birth, the AR is in saturation.

AR over web for interoperability?

Smart AR Apps for just-in-time services?

AR authoring and sharing for sustainable eco-system?

So, will AR be dead?

@ Wired September 2010

Hype Cycle of AR

CAMAR 2.0

• Mobile IT Infra

• CV HW/SW Tech

• AI (context) Tech

What is CAMAR? U-VR?

I might be able to make money

I hate AR

Nope, no money here

I am wondering how I lived without AR

Mobile 3D AR

3D AR

Mobile 2D AR

LBS-based AR

Marker-based AR

Hype Cycle of AR 2011

2010

2008

2009

Augmented Reality

• MIT’s annual review; “10 Emerging Tech.s 2007”

• Gartner: top 10 disruptive tech 2008-12

• Juniper: mAR 1.4B downloads/y, revenue $1.5B/y

by 2015 (11M in 2010)

Outline

VR vs. AR : Is AR Hype?

Where are we now?

What’s Next?

Why Ubiquitous VR, DigiLog & CAMAR?

Summary and Q&A

Augmented Reality

What is involved in the AR process?

Get video from camera

Estimate position and orientation of the camera

Render the augmented

scene

Process the interaction

Update the status

AR Trend

3D vs. Informative AR

Items 3D AR Informative AR

Main focus Realistic 3D augmentation Information augmentation

Input Camera Sensors (GPS, Compass, etc.)

CoI Object Location (POI)

Core Tech Object recognition/tracking 3D registration Realistic 3D rendering

POI detection Overlay Information filtering

Interaction 3D virtual objects vs. physical space & user

2D text vs. user

Authoring Offline In-situ

Killer apps Games, training, education Navigation, adv, sns

Platform Standalone Networked

PC-based Sony, Total Immersion WearableAR with HMD

Phone-based Vuforia(Qualcomm), Junaio/iLiving(metaio),

Layar, wikitude, sekaiCam, Argon

Tracking @ U-VR Lab 2007-8

BilliARd (2005)

2D Picture Tracking (K.Kim) 3D Vase Tracking (Y. Park) 3D Vase Tracking (Y. Park)

Tracking (W. Baek) Tracking MO (W.Back) Multiple object Tracking

(Y.Park)

Tracking @ U-VR Lab 2008-11

ISMAR2008 ISMAR 2009 VC2010

3D objects Tracking,

ISMAR2008 (K.Kim)

Motion Blur in 3D tracking, ISMAR2009 (Y. Park)

Scalable Tracking, VC 2010 (KKim)

ISMAR2010 ISMAR 2011 ICAT2011

Modeling & Tracking,

ISMAR2010 (KKim)

Depth-assisted Tracking,

ISMAR2011 (Y.Park) Depth-assisted Detection,

ICAT2011 (W.Lee)

Enhance Experience, Engage, Educate & Entertain

Hongkil Dong Technologies in Chosun

Storytelling application Integrated with virtools*

Storytelling application Integrated with virtools*

Digilog Apps 2010

DigiLog Miniature

Hong-Kildong Asadal-Asaneo

Storytelling application Desktop AR Integrated with virtools*

Storytelling application Desktop AR + mobile interaction Integrated with virtools*

Digilog Apps 2011

Why Mobile AR?

Why phone-based AR ? 968M(40M) m-phones worldwide (Korea) in 09

64M(15M) s-phone worldwide(Korea) in 11

Low cost, ubiquity, robust, self-contained

From terminal to gateway: With advances in tracking and increased computing power, mobile AR systems are developing

Leading graphics board vendors release new chipset that will enable HW-accelerated 3D CG on mobile phones

AR Apps in iPhone AppStore 54 in July, 2010 and 605 total. (260K Apps in Oct., 2010)

http://www.capv.com/home/Multiclient/MobileImaging.html

Mobile UI/UX

Camera for Physical Interaction

Connecting People, Places, Things, and Ideas

The environment as part of the interface

Camera phones as “bridging” devices

Integration with the user’s activities

Click real world !!!

Mobile AR : Informative

Informative AR Applications

Wikitude Layar TwitAround

Junaio Aurasma vuforia

Informative mobile AR

How does it work?

Illustrations courtesy of GeoVector (left)

and Layar & Junaio by Metaio (right)

QualComm SDK:

http://developer.qualcomm.com/dev/augmented-reality

https://ar.qualcomm.com/qdevnet/

Mobile AR SDK

KHARMA by GATECH

KML/HTML Augmented Reality Mobile Architecture

An Open Platform for Delivering Mobile Augmented Reality Experiences

https://research.cc.gatech.edu/polaris/

CAMAR 2.0 by GIST

Context-aware mobile AR

CAMAR + LBS + SNS

Mobile AR SDK

LBS

SN

S

CONTEXT

CONTENT

Embedded Sensors

Outline

VR vs. AR : Is AR Hype?

Where are we now?

What’s Next?

Why Ubiquitous VR, DigiLog & CAMAR?

Summary and Q&A

What is still holding us back?

Processing power <10fps ~30fps

Registration accuracy 1cm 1deg

Screen size Limited FOV Optical-see-through

Interaction Limited to touching screen

NUI with 5 senses

GPS Limited speed and accuracy

AGPS and wifi with Vision-assisted

Network BW/latency 400ms pings are not so responsive

<50ms for cloud

Authoring Just emerging In-situ mash-up

Battery power Devour batteries Full day usage

Platform Dual core smart phone Quad core smart phone

Apps Navigation M-commerce

What to Expect in 2012?

Matthias Galica (CEO of ShareSquare)

Quick Response (QR) Codes : simple yet effective…

Augmented Reality (AR)

Near Field Communication (NFC)

The Field (Everybody Else)

The effects of AR

Increased transparency

Reduced privacy

Timely access to information

Juniper Research report on AR

predicted that annual revenues from mobile AR apps will reach

$732M by 2014,

$1.5B by 2015,

up from less than $1M in 2009.

What to Expect in 2012?

Where is Digital World headed?

Wearable, Cloud computing

Internet of things with sensor proliferation

Digital immersion with Rich UI/UX over ubiComp

SNS over networked urbanism

SNS needs AR!

More than AR needs SNS, SNS needs AR to improve the UI/UX!

Augmented Content is a King, then Context is a queen consort controlling the King!

So, ubiComp and AR are tightly coupled ‘Outernet’!

Mobile : smart phone for mobility of people

Social : layers of AR for cultural/collaborative experience

Cloud : bringing down all the resources of the internet to reality

eco-system: publishing sharable cloud content

How to make Mobile AR App’s useful?

Beyond the hype:

Moving from simple gimmicky experiments to real world applications

Current AR experience

shows generally low social maturity

Is stalling, waiting for breakthrough either technological or conceptual

Future AR needs to

Enhance social interactions, instead of replacing them with AR gimmicks

Be optional for social interaction

Impractical AR Useful AR

•3D models placed in a webcam with little or no interactivity

•Connectivity is always ON •The cloud is the computer •Engaging, persistent experience for the user

•Layered animation with little or no feedback

•MAR that uses solely GPS, compass, and accelerometer input •[LBS + SNS + MAR] drawing from

networked large DBs with customization features •MAR where geo-tagging doesn't

serve an everyday purpose

What’s Next?

Social AR?

Issues of Social AR

Physical self along with a digital profile

Unauthorized Augmented Advertising

Privacy: Augmented Behavioral Targeting

Safety: Physical danger

Spam

Social AR?

Cases against AR

Outline

VR vs. AR : Is AR Hype?

Where are we now?

What’s Next?

Why Ubiquitous VR, DigiLog & CAMAR?

Summary and Q&A

Smart Holistic Space

Convergence

Between Real & Virtual

Space for experience and expression

Personal space (3rd skin): an individual has a complete control of data/info flow, i.e., incoming, filtering, selecting, outgoing (using a user profile)

Social space: a group has some control, ownership and regulatory power, but as a result of negotiations and transactions

General space: a temporary quality and free access, but regulated by a public social contract

Real space

Seams btw two worlds How to

combine

Seamlessly?

Virtual space

Dual space {R, R’}

V

R R’

RE

RE RE

RE’

VE’

RE’

RE

VE’

What’s VR, AR and UVR?

What’s Ubiquitous VR?

Woo’s Definition [11] : U-VR is

3D Link btw dual (real & virtual) spaces with additional info

CoI augmentation, not just sight: sound, haptics, smell, taste, etc.

Bidirectional UI for H2H/H2S/S2H/S2S communication in dual spaces

Real space

Social Networks

Seamless Augmentation

btw dual spaces

How to

Link Seamlessly?

CoI

LINK

U-Content

Virtual space

AR for DigiLog Experience

AR platform for DigiLog Experience will be

User participation

Context-awareness

Realistic 3D augmentation

• Multicore/GPGPU

• Object recognition/tracking

• Light source estimation

• Physics engine

• AI engine

• CoI-aware

• Environmental context:

• User context: knowledge,

experience, preference

• Just-in-time visualization

• Interaction

• Mash-up authoring

• SNS

• Extension of human

perception

How to avoiding the fate of VR ?

"to look for DigiLog, the intersections

between the digital world (3D, SNS)

and analog world (CoI)”

Technical Challenges

CoI Localization: Context of Interest (CoI): Space vs. Object

Accurate CoI Recognition and Tracking

3D Interaction

Ubiquitous Augmentation

LBS/SNS-based Authoring and Mash-up

Smart UI for Intuitive Visualization

AR-Infography + Organic UI

Networking and public DB management

U-VR ecosystem with SNS, LBS, CaS

Interoperable representation

HW wish list

Better camera/GPS/compass, CPU/GPU, I/O, battery

CAMAR 2.0: Context-aware mobile AR

Direct response

Reflective response Planned

response

Mashup

Sharing

CAMAR 2009-10

Mobile AR: WHERE to augment?

Concept Context-aware Annotation (H. Kim)

Plan Recognition (Y. Jang) Multi-page Recognition (J.Park) LBS + mobile AR (W. Lee)

[Paper] Y. Jang and W. Woo, “Stroke-based semi-automatic region of interest detection for in-situ painting recognition", 14th International Conference on Human-Computer Interaction (HCII 2011), Jul. 9-14,

Orlando, USA, accepted.

[Patent] W. Woo, Y. Jang, “현장에서 그림 인식을 위해 선긋기 상호작용을 통한 반자동식 관심영역 검출 알고리즘 ,” 2010. (출원 중)

Adaptive Content Recommendation

Recommend user-preferred content

Retrieve content efficiently using hierarchal context model

Context @ U-VR Lab 2010-11

J. Han, H. Schmidtke, X. Xie, and W. Woo, “Adaptive Content Recommendation using Hierarchical Context Model with Granularity for Mobile Consumer,” in Pers. Ubiqu. Comp

ut., pp.000-000, 2012. (Submitted)

ARWand: Phone-based 3D Object Manipulation in AR

Exploits a 2D touch screen, a 3DOF accelerometer, and compass sensors information to manipulate 3D objects in 3D space

Design transfer functions to map the control space of mobile phones to an AR display space

Interaction @ U-VR Lab 2011

Taejin Ha, Woontack Woo, "ARWand: Phone-based 3D Object Manipulation in Augmented Reality Environment," ISUVR, pp. 44-47, 2011.

ARWand: Phone-based 3D Object Manipulation in AR

Interaction @ U-VR Lab 2011

Taejin Ha, Woontack Woo, "ARWand: Phone-based 3D Object Manipulation in Augmented Reality Environment," ISUVR, pp. 44-47, 2011.

Experiment and application

◦ Low control-to-display gain: a sophisticated translation could be possible but this requires a significant amount of clutching

◦ High gain could reduce the frequent clutching, but accurate manipulation could be difficult

◦ Therefore, we need to consider an optimal control function that satisfies both fast and accurate manipulation

In Situ Video Tagging on Mobile Phones

In situ Planar Target Learning on Mobile Phones

Sensor-based Automatic Fronto-parallel View Generation

Fast Vanishing Point Computation

CAMAR @ U-VR Lab 2010

W. Lee, Y. Park, V. Lepetit, W. Woo, "In-Situ Video Tagging on Mobile Phones," Circuit Systems and Video Technology, IEEE Trans. on, Vol. 21, No. 10, pp. 1487-1496, 2011.

W. Lee, Y. Park, V. Lepetit, W. Woo, "Point-and-Shoot for Ubiquitous Tagging on Mobile Phones," ISMAR10, pp. 57-64, 2010.

Input Image

Fronto-parallel View Generation

Target Learning on the mobile GPU

Real-time Detection

Horizontaltarget ?

Vanishing Points Estimation

In Situ Video Tagging on Mobile Phones

CAMAR @ U-VR Lab 2010

데모비디오

In situ Augmentation of Real World Objects Vertical Target Learning & Detection

- In situ augmentation of real world objects

without pre-trained database

- Fast target learning in a few seconds

- Real-time detection from novel viewpoints

- Learning a vertical target from an arbitrary

viewpoint

- Vanishing point-based fronto-parallel view

generation

- Real-time detection from unseen viewpoints

Available at : http://youtu.be/vaaFhvfwet8

W. Lee, Y. Park, V. Lepetit, W. Woo, "In-Situ Video Tagging on Mobile Phones," Circuit Systems and Video Technology, IEEE Trans. on, Vol. 21, No. 10, pp. 1487-1496, 2011.

W. Lee, Y. Park, V. Lepetit, W. Woo, "Point-and-Shoot for Ubiquitous Tagging on Mobile Phones," ISMAR10, pp. 57-64, 2010.

Interactive Annotation on Mobile Phones for Real and Virtual Space Registration

Allows to quickly capture the dimensions of a room

Operates at interactive frame-rates on mobile device and provides simple touch-interaction

Serves as anchors for linking virtual information to the real space represented by the room

CAMAR @ U-VR Lab 2011

H. Kim, G. Reitmayr and W.Woo, “Interactive Annotation on Mobile Phones for Real and Virtual Space Registration,” in Proc. ISMAR 2011, pp.265-266, Oct. 2011.

Interactive Annotation on Mobile Phones for Real and Virtual Space Registration

CAMAR @ U-VR Lab 2011

Youtube share link http://www.youtube.com/watch?v=I00I-phmPbI

데모비디오 Demo #1 Demo #2

In office room and seminar room, - Capture the dimensions of a room,

approximated as a room - Annotate a virtual content on rectangular

areas on the room’s surface

In ART gallery, - Load an AR zone-based room model - Annotate a virtual content on rectangular areas

on the room’s surface

In-situ AR Mashup for AR Content Authoring

Easily create AR contents from Web contents

Context-based content recommendation

User-similarity, item similarity, social relationship

Configure AR content sharing setting

To Whom, When, in What conditions

CAMAR @ U-VR Lab 2011

H.Yoon and W.Woo, “CAMAR Mashup: Empowering End-user Participation in U-VR Environment,” in Proc. ISUVR 2009, pp.33-36, July. 2009. (Best Paper Award)

H.Yoon and W.Woo, “Concept and Applications of In-situ AR Mashup Content,” in Proc. SCI 2011, pp. 25-30, Sept. 2011.

In-situ AR Mashup for AR Content Authoring

CAMAR @ U-VR Lab 2011

H.Yoon and W.Woo, “CAMAR Mashup: Empowering End-user Participation in U-VR Environment,” in Proc. ISUVR 2009, pp.33-36, July. 2009. (Best Paper Award)

H.Yoon and W.Woo, “Concept and Applications of In-situ AR Mashup Content,” in Proc. SCI 2011, pp. 25-30, Sept. 2011.

In-situ Content Mashup

• Extract query keywords based on context of object • Content recommendation based on personal context and social context • Access related Flickr, Twitter, Picasa contents in-situ

Some Issues: Standard

Interoperability (Standard)

W3C : HTML5 (ETRI): http://www.w3.org/2010/06/w3car/report.html

web3D : X3D (Fraunhofer)

OGC : ARML (GATECH)

ISO/IEC JTC1: SC24 WG9 [X3D(KU), XML(GIST)] vs. SC29 [X3D(ETRI)]

Real Content, Virtual Content, ***Link***

Virtual

Contents

#1

Virtual

Content

#2

Virtual

Contents

#3

Virtual

Contents

#4

Link Link Link Link

MR Metadata Format

id ID

who owner, author

when creation time

where location

what tag, description

how movable, access

Real

Contents

id ID

who owner, author

when creation time

where location

what tag, type

how movable, access

id ID

who author

when creation time

where real-virtual relation

what ID of real and virtual

contents

how access

AR/MR Link Lookup Table

POI: Metadata

AR Content: Metadata

2

1

3

Standard by GIST

Server/DB Mobile Device Real World

Image Acquisition (Grab)

-FP Recognition

-AR Content Management

Feature Points or

Content ID

R-ID, R-Pose

TTAK.KO-10.0434

Data Representation for

AR/MR (=Link)

TTAK.KO-10.0428

QR Code Data Representation

for (Mobile) AR

LID WHO WHEN WHER

E

WHAT HOW

R-ID V-ID

GUID1 … … V-

Pose Image ID1

VID1 …

GUID2 …

GUID3 … … V-

Pose Image

ID1 VID4 …

Link DB

AR

Viewer Object of Interest (OOI)

Context of Interest

(COI)

Point of Interest (POI)

What’s Next?

Next?

People don’t necessarily want to walk around holding phone in front of their faces

CAMAR 2.0 vs. Wearable CAMAR 2.0

<Open + Participate + Share>

LBS + In-situ Mashup + SNS + CAMAR for sustainable AR eco-system

Outline

VR vs. AR : Is AR Hype?

Where are we now?

What’s Next?

Why Ubiquitous VR, DigiLog & CAMAR?

Summary and Q&A

Q & A

“The future is already here. It is just not uniformly distributed” by William Gibson (SF writer)

More Information

Woontack Woo, Ph.D.

Twitter: @wwoo_ct

FB: @wtwoo

Mail: wwoo@gist.ac.kr

Web: http://uvr.gist.ac.kr

1st UVR Workshop @ KAIST GSCT, Mar. 16-17, 2012

7th ISUVR 2012 @ KAIST GSCT, Aug. 22 - 25, 2012