Holographic Whisper - CHI2017 oral presentation by Yoichi Ochiai

Holographic Whisper: Rendering audible sound spots in three-dimensional space by focusing ultrasonic wavesYoichi Ochiai*13, Takayuki Hoshi*23, Ippei Suzuki*1

*1 University of Tsukuba, Digital Nature Group *2 University of Tokyo, *3 Pixie Dust Technologies, inc.*joint first authorship, Contact: [email protected]

Assistant Professor at University of Tsukuba, Head of Digital Nature Group CEO of Pixie Dust Technologies, inc

Motivation and Introduction“Render the Aerial Sound Sources separated from Machines”

Copyright Yoichi Ochiai, University of Tsukuba, Digital Nature Group, Pixie Dust Technologies, inc.


We envision the post-pixel world

Release data from material existence and regenerate Material Properties by using out of range frequencies of our sensory organs.

P i x e l s P i x i e s2D to 2.5D

Physical Device

3 DC o m p u t a t i o n a l W a v e D e s i g n

M a g i c a l E x p e r i e n c e w i t h N o C o n t a c t D e v i c e

T o w a r d s


Pixel Image

HapticHaptic

Sound Manipulation

2.5D Voxel

Human

How can we update the pixel world (2.5D) towards pixie world (3D)?

Scenery of 2000s to 2020

2.5D device

Physical Laptops

Traditional Scenery


Post-Pixel Alternatives By Computational Wave Synthesis

Manipulation of Object

Aerial HapticsAerial Image and Haptics

Voxel Graphics

holographic laser plasma

Holographic laser plasma and ultrasonic

holographic ultrasonic levitation


We have studied to envision the alternatives by computational wave synthesis.


Post-Pixel Alternatives By Computational Wave Synthesis

Manipulation of Object

Aerial HapticsAerial Image and Haptics

Voxel Graphics

holographic laser plasma

Holographic laser plasma and ultrasonic



Next! Holographic AudioWe have studied to envision the alternatives by computational wave synthesis.

U n i v e r s i t y o f Ts u k u b a , Yo i c h i O c h i a i L a b r a t o r y


Towards three dimensional audio communication

19c-20c 1990-Broad Ultrasonic Beam

Mass-Communication Towards Individual Communication

ABC.com, Ponpei,et,al.

2017-Holographic Points and Steerable BeamsTowards Personalized Communication with Ubiquitous Environment

Our Study


Overview

Designing narrow audio distribution for individual communication

Our Study

Conventional


Contribution

The method enables production of aerial sound point sources with an ultrasonic phased array for three-dimensional (3D) audio interaction which includes:

Principles

Theory

Simulated Result

Implementation

User Study

Evaluation

Application

Discussion

Related Works“How to design the Acoustic Environment”



The Audio Communication

Audible Communication has been separated since Edisons Era.

Individual Broad


Efforts towards Holographic Individual Audio

[3]BOSE. F1 flexible array loudspeaker system. from http://www.bose.com/prc.jsp?url=/promotions/entry_pages/f1/index_en.jsp[4]JBL. IntelliDisc-DS90. from http://www.jblpro.com/www/products/installed-sound/intellidisc-ds90[5]M. Yoneyama, J. Fujimoto, Y. Kawamo, and S. Sasabe. 1983. The audio spotlight: An application of nonlinear interaction of sound waves to a new type of loudspeaker design. The Journal of the Acoustical Society of America 73, 5, 1532-1536.[6]K. Aoki, T. Kamakura, and Y. Kumamoto. 1991. Parametric loudspeaker - Characteristics of acoustic field and suitable modulation of carrier ultrasound, Electronics and Communications in Japan 74, 9, 76-82.[7]F.J. Pompei. 1999. The use of airborne ultrasonics for generating audible sound beams, Journal of the Audio Engineering Society 47, 9, 726-731.[9]K. Shinagawa, Y. Amemiya, H. Takemura, S. Kagami, and H. Mizoguchi. 2007. Three dimensional simulation and measurement of sound pressure distribution generated by 120 ch plane loudspeaker array. In Proceedings of IEEE International Conference on Systems, Man and Cybernetics, 278-283.[10]T. Matsui, D. Ikefuji, M. Nakayama, and T. Nishiura. 2014. Multiple audio spots design based on separating emission of carrier and sideband waves. In Proceedings of International Congress and Exposition on Noise Control Engineering, 6313-6317.[12]C. Shi, Y. Kajikawa, and W.-S. Gan. 2014. An overview of directivity control methods of the parametric array loudspeaker. APSIPA Transactions on Signal and Information Processing 3, e20.



Efforts towards Holographic Individual Audio

[3]BOSE. F1 flexible array loudspeaker system. from http://www.bose.com/prc.jsp?url=/promotions/entry_pages/f1/index_en.jsp[4]JBL. IntelliDisc-DS90. from http://www.jblpro.com/www/products/installed-sound/intellidisc-ds90[5]M. Yoneyama, J. Fujimoto, Y. Kawamo, and S. Sasabe. 1983. The audio spotlight: An application of nonlinear interaction of sound waves to a new type of loudspeaker design. The Journal of the Acoustical Society of America 73, 5, 1532-1536.[6]K. Aoki, T. Kamakura, and Y. Kumamoto. 1991. Parametric loudspeaker - Characteristics of acoustic field and suitable modulation of carrier ultrasound, Electronics and Communications in Japan 74, 9, 76-82.[7]F.J. Pompei. 1999. The use of airborne ultrasonics for generating audible sound beams, Journal of the Audio Engineering Society 47, 9, 726-731.[9]K. Shinagawa, Y. Amemiya, H. Takemura, S. Kagami, and H. Mizoguchi. 2007. Three dimensional simulation and measurement of sound pressure distribution generated by 120 ch plane loudspeaker array. In Proceedings of IEEE International Conference on Systems, Man and Cybernetics, 278-283.[10]T. Matsui, D. Ikefuji, M. Nakayama, and T. Nishiura. 2014. Multiple audio spots design based on separating emission of carrier and sideband waves. In Proceedings of International Congress and Exposition on Noise Control Engineering, 6313-6317.[12]C. Shi, Y. Kajikawa, and W.-S. Gan. 2014. An overview of directivity control methods of the parametric array loudspeaker. APSIPA Transactions on Signal and Information Processing 3, e20.


Pickup


How the ultrasonic directional audio works?

Why??? How did it achieved?


Beam Forming[5]M. Yoneyama, J. Fujimoto, Y. Kawamo, and S. Sasabe. 1983. The audio spotlight: An application of nonlinear interaction of sound waves to a new type of loudspeaker design. The Journal of the Acoustical Society of America 73, 5, 1532-1536.[6]K. Aoki, T. Kamakura, and Y. Kumamoto. 1991. Parametric loudspeaker - Characteristics of acoustic field and suitable modulation of carrier ultrasound, Electronics and Communications in Japan 74, 9, 76-82.[7]F.J. Pompei. 1999. The use of airborne ultrasonics for generating audible sound beams, Journal of the Audio Engineering Society 47, 9, 726-731.


Beam Forming

WaveFront = Plane Wave = Sound Beam

[5]M. Yoneyama, J. Fujimoto, Y. Kawamo, and S. Sasabe. 1983. The audio spotlight: An application of nonlinear interaction of sound waves to a new type of loudspeaker design. The Journal of the Acoustical Society of America 73, 5, 1532-1536.[6]K. Aoki, T. Kamakura, and Y. Kumamoto. 1991. Parametric loudspeaker - Characteristics of acoustic field and suitable modulation of carrier ultrasound, Electronics and Communications in Japan 74, 9, 76-82.[7]F.J. Pompei. 1999. The use of airborne ultrasonics for generating audible sound beams, Journal of the Audio Engineering Society 47, 9, 726-731.


Beam Forming

WaveFront = Plane Wave = Sound Beam

ex. LASER

Coherent Plane Wave

[5]M. Yoneyama, J. Fujimoto, Y. Kawamo, and S. Sasabe. 1983. The audio spotlight: An application of nonlinear interaction of sound waves to a new type of loudspeaker design. The Journal of the Acoustical Society of America 73, 5, 1532-1536.[6]K. Aoki, T. Kamakura, and Y. Kumamoto. 1991. Parametric loudspeaker - Characteristics of acoustic field and suitable modulation of carrier ultrasound, Electronics and Communications in Japan 74, 9, 76-82.[7]F.J. Pompei. 1999. The use of airborne ultrasonics for generating audible sound beams, Journal of the Audio Engineering Society 47, 9, 726-731.


Ultrasonic Self-demodulation generates audible sound

temporal change of ultrasonic (primary) waves

the radiation of audible sound (secondary) waves

where c [m/s] is the speed of sound in air, β is the nonlinear parameter, and ρ [kg/m3] is the mass density of air

Audible

Ultrasonic

→In the beam you can hear the audible sound


Beam Forming to the point audio

[10]T. Matsui, D. Ikefuji, M. Nakayama, and T. Nishiura. 2014. Multiple audio spots design based on separating emission of carrier and sideband waves. In Proceedings of International Congress and Exposition on Noise Control Engineering, 6313-6317.


Beam Forming to the point audio

Audible Area

[10]T. Matsui, D. Ikefuji, M. Nakayama, and T. Nishiura. 2014. Multiple audio spots design based on separating emission of carrier and sideband waves. In Proceedings of International Congress and Exposition on Noise Control Engineering, 6313-6317.


Steering Beam

[12]C. Shi, Y. Kajikawa, and W.-S. Gan. 2014. An overview of directivity control methods of the parametric array loudspeaker. APSIPA Transactions on Signal and Information Processing 3, e20.

θ [rad]: When the spacing between the transducers is d [m] in the case of a 1D transducer array, the phase delay of the i-th transducer, ∆φi [rad] where k [rad/m] is the wavenumber of the ultrasound.


Steering Beam

WaveFront = Plane wave = Beam

[12]C. Shi, Y. Kajikawa, and W.-S. Gan. 2014. An overview of directivity control methods of the parametric array loudspeaker. APSIPA Transactions on Signal and Information Processing 3, e20.

θ [rad]: When the spacing between the transducers is d [m] in the case of a 1D transducer array, the phase delay of the i-th transducer, ∆φi [rad] where k [rad/m] is the wavenumber of the ultrasound.


Sound Focus

WaveFront = Spherical Wave


Sound Focus



Sound Focus

Focal Point



Sound Focus

Focal Point

Objective Lens

Plasmaex. LASERCoherent Spherical Wave

Focal Point

Hoshi et al, 2008

PTH:Audible-sound radiation threshold, self-demodulation effect becomes obvious when Pu is effectively high and its square value is not negligible.


Make it narrow area: Threshold of self-demodulation

temporal change of ultrasonic (primary) waves

the radiation of audible sound (secondary) waves

where c [m/s] is the speed of sound in air, β is the nonlinear parameter, and ρ [kg/m3] is the mass density of air

Audible Area Pu > PTH

*notice: recording mic is more sensitive than human auditory & reflection of mic itself recorded


Recorded Sound

Implementation“Designing Holographic Loudspeaker”



Electrical Circuit Design


Phase Modulation

Evaluation“How to design the Acoustic Environment”



Audio Distribution

targetnoise by interference

Ex, UltrasonicSuper-directional Speaker


Audio Distribution

targetnoise by interference

Ex, UltrasonicSuper-directional Speaker

Minimum Focal Area: 20mm x 20mm

Focal Area: 1600mm x 1600mm


Simulated Results of the Shape of Sound Source

The size of speaker decides fundamental limitation of beam forming.


User Study1

Direction and Location for x-y plane is understandable for users.

x y

Z


User Study2

User can distinguish these three points (Z axis)

x y

Z


User Study3

Precise audible height perception is not easy

for users (for Z axis)

x y

Z


Results and User Study2&3

Make Larger Difference (Z). or Change the x-y Direction

x y

Z


Sound Quality

*notice: recording mic is more sensitive than human auditory & reflection of mic itself recorded


Sound Quality

conventional conventional This studyoriginalwave shape

FFT spectrums

Application“I can hear Holographic Whisper”



Application for Interaction

Interactive Sound Distribution

Features for Novel Applications

Narrow and Broad

Trace and Focus

Haptic and Push


Place the sound spot, more precise and interactive


Application for Interaction (Focusing inner music instruments)


Application for Interaction (Whispered Voxels)


Place the sound spot for pick up target


Super Narrow Distribution for spotting captions


Super Narrow Distribution (active design of sound field)


Super Narrow and Wide Distribution Switchable


Super Narrow and Wide Distribution Switchable

training and instruction


3D manipulation of Object with Audible Annotations

Audible Information can be recognized separately.

Audible Information can not be delivered separately.


This technique will expand audio recognitions


Possible Scenario Super Narrow Distribution

Make Sound Difference

navigation play

shopping


Super Narrow Distribution (combine with large monitor)

User A

User B

User C

Individual Audio Interaction for voice control


Super Narrow Distribution (Multi-language conference)

Discussion and Conclusion“Summery of Holographic Whisper”



Discussions

Ultrasound exposure

Multiple Sound Points by CGH

It cannot say no effect. Lower than conventional ones (because this methods generate radiation point Lower intensity than haptic device

Available by generating multiple focus by multiple modulation. but it l imited spatiotemporal resolution.

Lower intensity(eg.haptic)

Direct Exposure is not necessary

AvailableAdvantage

Advantage


Discussions

Amplitude of Audio Signals

Grating Lobes

Audible

Ultrasonic

Amplitude of Audio signal decides the characteristics of sound focus, it is typical trade off of this methods.

it generates grating lobes around the focus (40 degree). This is typical l imitation of this methods.

It affects both amplitude and audio distribution

It generated around the focal point

limitation and tradeoff

limitation and tradeoff

Discussions

The size of speaker decides fundamental limitation of beam forming.Copyright Yoichi Ochiai, University of Tsukuba, Digital Nature Group, Pixie Dust Technologies, inc.

Discussions


Overview

Designing narrow audio distribution for individual communication

Our Study

Conventional


Conclusion

Production of aerial sound point sources with an ultrasonic phased array for three-dimensional (3D) audio interaction includes:

Principles

Theory

Simulated Result

Implementation

User Study

Evaluation

Application

Discussion



Release data from material existence and regenerate Material Properties by using our of range of our sensory organs.

P i x e l s P i x i e s2D to 2.5D 3 D

Copyright Yoichi Ochiai, University of Tsukuba, Digital Nature Group, Pixie Dust Technologies, inc.Copyright Yoichi Ochiai, University of Tsukuba, Digital Nature Group, Pixie Dust Technologies, inc.

Team

Dr. Yoichi OchiaiAssistant Prof, Advisor to President Head of Digital Nature Group, University of TsukubaCEO and coFounder of Pixie Dust Technologies, inc.

Dr. Takayuki HoshiAssistant Prof, University of Tokyo

co Founder of Pixie Dust Technologies, inc.

Ippei SuzukiUndergrad Student, Digital Nature Group, University of Tsukuba


Holographic Whisper: Rendering audible sound spots in three-dimensional space by focusing ultrasonic wavesYoichi Ochiai*13, Takayuki Hoshi*23, Ippei Suzuki*1

*1 University of Tsukuba, Digital Nature Group *2 University of Tokyo, *3 Pixie Dust Technologies, inc.*joint first authorship, Contact: [email protected]

Assistant Professor at University of Tsukuba, Head of Digital Nature Group CEO of Pixie Dust Technologies, inc


FAQ

Oh what amazing! Can I buy?Yes, you can buy soon ( this year) We provide this technique for B to B for now.

Is it safe???

Our auditory is sensible and it quite lower intensity than Ultrasonic Haptic Applications

call or mail

Oh, complex!! Just Use Headphones! Easy!

Nice idea, it can suppress others voice to concentrate.

W h a t i s t h e n e x t C h a l l e n g e t o w a r d s P o s t M u l t i m e d i a ? ( P o s t P i x e l s i n t e r a c t i o n )

V i s u a l : H D ( 1 0 8 0 p ) i n S p a c e

6 0 H z i n T i m e

A u d i o : 2 2 . 1 k H z i n T i m e



Go out of the range



6 0 H z i n T i m e

A u d i o : 2 2 . 1 k H z i n T i m e

M u l t i m e d i a s y s t e m s a r e r e s t r i c t e d f r o m l i m i t a t i o n a n d i m i t a t i o n o f t h e s p e c o f h u m a n s e n s o r y o r g a n s



Go out of the range



6 0 H z i n T i m e

A u d i o : 2 2 . 1 k H z i n T i m e

M u l t i m e d i a s y s t e m s a r e r e s t r i c t e d f r o m l i m i t a t i o n a n d i m i t a t i o n o f t h e s p e c o f h u m a n s e n s o r y o r g a n s

1 . H i g h I n t e n s i t y 2 . H i g h R e s o l u t i o n 3 . T r u e H o l o g r a p h y



Go out of the range

“Focused Ultrasound and Lasers by Field Generators” U n i v e r s i t y o f Ts u k u b a , Yo i c h i O c h i a i L a b r a t o r y

Spatial light modulator

(SLM)

CGH

Laser

Lens

Focal Points

Acoustic

Optic

Modulation on Reflection

Time delay generation

Acoustic Pressure and Standing Waves

Laser Induced Plasma



Release data from material existence and regenerate Material Properties by using out of range frequencies of our sensory organs.

P i x e l s P i x i e s2D to 2.5D

Physical Device

3 DC o m p u t a t i o n a l W a v e D e s i g n

M a g i c a l E x p e r i e n c e w i t h N o C o n t a c t D e v i c e

T o w a r d s

Holographic Whisper - CHI2017 oral presentation by Yoichi Ochiai

Engineering

Transcript of Holographic Whisper - CHI2017 oral presentation by Yoichi Ochiai