1 © 2015 Gachon University. All rights reserved.

360 Degrees Video Coding 기술동향

May. 15. 2016

Yeongil Ryu and Eun-Seok Ryu

(wlrmwlrm99@gc.gachon.ac.kr, esryu@gachon.ac.kr)

Multimedia Communications and Systems Lab(MCSL)

http://mcsl.gachon.ac.kr

Department of Computer Engineering

Gachon University

2 © 2015 Gachon University. All rights reserved.

Emerging 360 Degrees Videos

• Digital Camera 기반의 360 Degrees Video Systems는 최근 크게 주목

• 최근 360 Degree Video Viewer로서 다양한 HMD들이 출시

다양한 HMD의등장(Gear VR, Oculus Rift, Cardboard VR Kit, PlayStation VR)

360 Degree Video의예시

3 © 2015 Gachon University. All rights reserved.

HMD Revenues by Device Type

• World Market, Forecast: 2014 to 2020 (Source: ABI Research)

Device Type Revenues 2014 2015 2016 2017 2018 2019 2020

VR Standalone (US$ Thousands) 935 3,980 19,504 94,592 269,586 501,431 902,575

VR Mobile (US$ Thousands) 55,250 164,093 321,621 561,551 853,557 1,190,712 1,607,462

VR Tether (US$ Thousands) 55,250 111,605 876,546 1,938,042 3,038,851 4,691,985 6,906,602

Enterprise Smart

Glasses

(US$ Thousands) 576,790 1,051,088 2,046,876 3,849,328 6,715,196 9,522,589 11,693,589

Consumer Smart

Glasses

(US$ Thousands) 148,317 27,280 526,339 989,827 1,726,765 2,448,666 3,006,923

Total (US$ Thousands) 836,543 1,601,046 3,790,886 7,433,340 12,603,955 18,355,383 24,117,151

4 © 2015 Gachon University. All rights reserved.

Samsung GearVR versus Oculus Rift

Oculus Rift Gear VR

Optical Lens 110 degrees or greater 96° field of view

Display2160 x 1200 pixel, 90Hz built-in OLED

displays2560 x 1440 pixel Super AMOLED

Refresh Rate 90 Hz 60 Hz

Required Hardware

PC with Windows 7 SP1 or newer, 8GB of

RAM or more, Intel i5-4590 equivalent or

greater, Nvidia GeForce GTX 970/AMD 290

equivalent or greater

Galaxy Note 5, Galaxy S6, S6 Edge, and

S6 Edge+

SensorGyrocope, Accelerometer, Magnetometer,

Constellation array

Accelerator, Gyrometer, Geomagnetic,

Proximity

Focal Adjustment None Focus adjustment wheel

Interpupillary Distance

Coverage

Default distance set at 64 mm, adjustable range

TBA54~70 mm

Physical UI Xbox controller, Oculus Touch controllers Touchpad, back button, volume key

ConnectionHDMI 1.3 video output to headset•2x USB 3.0

ports for peripherals

MicroUSB connection to Galaxy Note 5,

S6, S6 Edge, S6 Edge+

5 © 2015 Gachon University. All rights reserved.

VR 제조사별기술적특징비교 (source: ABI Research)

DEVICE HMD TypeDisplay Type

DisplayResolution

RefreshRate

Field of View

AudioPosition Tracking

Head Tracking InputRelease Date

Price

Oculus Rift CV1

VR tethered

- 2160X1200 90 Hz -Built-in

Headphones

IR Camera 360°

GyroscopeAccelerometer Magnetometer

OculusTouch

Standard controller

Q1 2016US$400(est.)

HTC ViveVR

tethered- 2160X1200 90 Hz 110°

Headphone jack

Lighthouse basestation

360°

GyroscopeAccelerometer Magnetometer

Lighthouse controllers Standard controller

Nov 2015

US$500(est.)

Sony’s Project

Morpheus

VR tethered

OLED 1920X1080 120 Hz 100°Headphone

jackIR Camera

360°Gyroscope

Accelerometer

Standard controllerPS Move

1Q 2016US$400(est.)

FOVEVR

tetheredWQHD 2560X1440 90 Hz 100°

Headphone jack

Lighthouse basestation

360°

GyroscopeAccelerometer Magnetometer

Eye tracking Standard controller Lighthouse controllers

May 2016

US$399

OSVRVR

tetheredQLED 1920X1080 60 Hz 100° None

IR Camera 210°

GyroscopeAccelerometer

Compass

No standard input

Available now

US$200to

US$300

Google Cardboard

VR mobileDependenton phone

Dependenton phone

Dependenton phone

Dependenton phone

Headphone jack

NoneDependent on

phoneSingle button

Available now

US$5 to US$20

Samsung Gear VR

VR mobileSuper

AMOLED2560X1440 60 Hz 96°

Headphone jack

NoneGyroscope

Accelerometer Magnetometer

Standard controller Touchpad

Available now

US$199(plus

phone)

Mattel View-Master

VR mobile Dependent on phone

Dependenton phone

Dependenton phone

Dependenton phone

Headphone jack None

Dependent on phone

Single button

Available now

US$30

ZEISS VR ONE

VR mobileDependent on phone

Dependenton phone

Dependenton phone

Dependenton phone

Headphone jack None

Dependent on phone None

Available now US$99

StarVRVR

tetheredLCD 5120X1440 - 210°

Headphone jack

IR camera Fiducial markers

360°

GyroscopeAccelerometer Magnetometer

No standard input

SurviosVR

standalone- - - - - - - - - -

6 © 2015 Gachon University. All rights reserved.

PC기반 VR 대비 Mobile VR이갖는약점

NOWMouse/Keyboard,Controller, Touch

SOONVoice, Gesture,

Eye-Tracking

LATERBrain-Machin

Interface (BMI)

*일부Source: ABI Research

• 모바일 기기의 비디오 처리 성능의 한계로 사용자시선의 빠른 대응이 어려움

• 현재 사용자 Tracking 기능이 제한적임

• 사용자 Input 기기가 제한적임

VR360 ISSUES• Bigger File Size• Higher Spatial Resolution• Higher Complexity• Tracking User’s Movements Issue• Stitching Artifacts

Mobile VR ISSUES• Less capable tracking• Simpler content• Lack of input options

7 © 2015 Gachon University. All rights reserved.

Conventional 360 Degree Video Processing

Image stitching,Equirectangular

mapping

Video rendering on a sphere

Video decoding

Video encoding

Stitched video is coded as regular 2D video by using H.264/AVC and H.265/HEVC

Input from seven cameras

Corresponding stitched image

Stitched image texture mapped onto a sphere

8 © 2015 Gachon University. All rights reserved.

Related Work: Region Adaptive Smoothing Method

• Samsung Research America, 2015 360 Degree Video의 높은 Resolution으로 인한 Bitrate 증가를 최소화하기 위한 기법

촬영된 영상의 Top, Bottom Region은 인코딩하기 위하여 Equirectangular Mapping된 Stitched Image에서는 큰

부분을 차지하지만, 실제로 HMD 상에서 Sphere Mapping되어 사용자에게 보여질 때에는 다른 Region과 같은

크기로 Mapping되어 렌더링 됨

이러한 특성을 활용하여 Equirectangular Mapping된 Stitched Image의 Top, Bottom Region을 Low-pass Filter의

일종인 가우시안 필터로 Smoothing하여 최소한의 화질 저화로 20% Bitrate 감소 효과를 내는 기법

Equirectangular Mapping된 Stitched Image HMD에 Rendering하기위하여Sphere Mapping된 Stitched Image

Proposed 360 Degree Video Processing Chain

source: Budagavi, Madhukar, et al. "360 degrees video coding using region adaptive smoothing." Image Processing (ICIP), 2015 IEEE International Conference on. IEEE, 2015.

9 © 2015 Gachon University. All rights reserved.

Related Work: Tile Based Panoramic Streaming

Using SHVC• Fraunhofer Heinrich Hertz Institute, 2015

Scalable Extension of HEVC (SHVC)을 기반으로, 사용자가 바라보는 지점인 Region-of-Interest

(ROI) 의 변화에 따른 화면 전환 지연 및 Bitrate를 최소화 하기 위한 기법

360 Degree Video를 여러 개의 Tile로 분할하여, ROI에 포함된 Tile만 Streaming

네트워크 환경의 변화나 Zoom-in 이벤트 발생 시 적응적이고, 끊김 없는 Video Streaming을 위하여

SHVC의 Multi Layer를 활용

사용자의 ROI가우측으로변화하는경우의 Streaming 되는 Tile의변화

source: Sanchez de la Fuente, Yago, Robert Skupin, and Thomas Schierl. "Compressed Domain Video Processing for Tile Based Panoramic Streaming using SHVC." Proceedings of the 3rd International Workshop on Immersive Media Experiences. ACM, 2015.

10 © 2015 Gachon University. All rights reserved.

Related Work: Pyramid Mapping Method

• Facebook, 2016

Equirectangular Mapping Method의 Top, Bottom 의 낭비되는 공간을 절약하기 위하여 이미지를

피라미드 형으로 Mapping

Equirectangular Mapping Method에서 발생하는 Bended Motion Vector를 제거할 수 있음

사용자가 바라보는 지점의 반대편으로 갈수록 점진적인 Down-Sampling을 하여 적응적 Video

Streaming 지원

1. A viewer inside the 360 video sphere

2. Put the sphere inside a pyramid

3. Unfold the sides of the pyramid

4. Fit the frame into a rectangle

A 360 Video in Equirectagular Layout

A 360 Video in Pyramid Layout

source: https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/

11 © 2015 Gachon University. All rights reserved.

Thank You