Realistic Object Appearance using Bidirectional Texture Functions

Institute of Computer Science II

Computer Graphics

Realistic Object Appearance using Bidirectional Texture Functions Christopher Schwartz, Reinhard Klein

3D COFORM STAR – VAST 2011 Prato, Italy – Christopher Schwartz 1 20/10/2011

INTRODUCTION


Object Appearance

• Common presentation: Geometry (+ Texture)

• … but is this sufficient?


Geometry only With texture Correct appearance

Importance of Surface Appearance

• Same shape, different materials

• Different „look-and-feel“ • Important hints about the object • Misleading vs. better understanding


Object Appearance

• Impression of reflection of incident light

• Influenced by features on different scales

• Macroscopic

• Mesoscopic

• Microscopic

• Viewpoint and Illumination dependent


Form of Representation

Macroscopic scale

• 3D shape

• Explicit representation (e.g. polygon mesh)

Mesoscopic scale

• Individually resolved by human perception

• Statistical representation not accurate

• Explicit representation too costly

Microscopic scale

• Alignment of microscopic structures

• Statistical representation (e.g. BRDF)


Bidirectional Reflectance Distribution Function

• Opaque, uniform Materials • No texture!

• Ratio of incident irradiance to outgoing radiance • Defined over local

hemisphere

• Depends on • Solid angle of light ωi

• Solid angle of view ωo


Bidirectional Reflectance Distribution Function

• Example BRDF • sampled at discrete angles


ωi

ωo

BRDF Tabulated Example: BRDF on Sphere Discrete sample positions on Hemisphere

Specular reflection

Model-driven vs. Data-driven

• Matusik et al. 2003 [1]: Measured BRDF ground-truth data

• 100 real world materials

• 1° resolution for view- and light directions

• > 1,000,000 samples

• Ngan et al. 2005 [2]: Experimental analysis of BRDF models




Red phenolic

Measured distribution from [1]

Fitted model (Cook-Torrance) from [2]

Data-driven Reflectance

• Example Surface with Meso structure

• Results are „ABRDFs“ (apparent BRDFs [3])


ωi

ωo

ωi

ωo

Specular reflection

Retro-reflection

Hard to fit with analytical model

influence from neighborhood

Model-driven Reflectance

• Loss of Mesoscale depth impression…


Fitted analytical SVBRDF Photograph McAllister 2002 [10]

Data-driven Reflectance


Texture Mesoscale approximated by Bump-mapping

Data-Driven (BTF)

Images taken from Müller et al. 2005 [7]


Macroscopic scale

• 3D shape


Mesoscopic scale




Microscopic scale





Macroscopic scale

• 3D shape


Mesoscopic scale




Microscopic scale




Data-driven: Image based


Macroscopic scale

• 3D shape


Mesoscopic scale




Microscopic scale




Bidirectional Texture Function

ACQUISITION


First use of BTF: CUReT Database

• 1996 – 1999 by Dana et al. [4] • 61 materials

• 205 different view- and light directions

• 24-bit RGB images

Manual placement of camera

BTF only partially measured


University of Bonn BTF Database

• 2001 – 2003 Sarlette et al. [5] • 6561 view and light directions

• 36-bit RGB images

• Fully automated

• 12 hrs per acquisition


University of Bonn Multiview Dome


• 2004 – now by Sarlette et al.

• 22,801 view and light directions

• HDR images

• Fully automated

• No moving parts

• 2hrs per acquisition

First Capture of Complete Objects with BTF

• Furukawa et al. 2002 [6] • Laser scanned geometry

• Separate BTF capture

Very sparse sampling

Alignment errors


Photograph Rendering

First Capture of Complete Objects with BTF

• Furukawa et al. 2002 [6] • Laser scanned geometry

• Separate BTF capture

Very sparse sampling

Alignment errors



Integrated Acquisition of Objects with BTF

• Müller et al. 2005 [7] • Use University of Bonn Dome

• Dense sampling (22,801), 2hrs/acquisition

• Measurements integrated in one setup • No registration necessary

• Geometry via Shape-from-Silhouette


Integrated Acquisition of Objects with BTF

• Müller et al. 2005 Drawbacks:

No radiometric calibration

• Misleading colors

• Only LDR

Shape-from-Silhouette

• Not automatable

• Coarse geometry – Misleading Shape

– Blur due to misalignment


Example from Havemann et al. 2008 [12]

Integrated HQ Acquisition

• Holroyd et al. 2010 [17] • Integrated setup

• Geometry with Structured Light

• 42 view and light directions


Sparse sampling Model-driven

• Fit Cook-torrance BRDFs



Integrated HQ Acquisition with BTF

• Schwartz et al. 2011 [8] • Use University of Bonn Dome

• Extended with projectors for Structured Light

• integrated measurement

• Rapid: 3.7 hrs per acquisition

• Proper calibration and HDR

• Geometry: Weinmann et al. 2011 [11]


Visual Hull [7], [12] Laser Scan Proposed Method [11]

Quality


Faithfulness


Photographic picture

(tonemapped HDR)

BTF + Geometry Schwartz et al. 2011 [8]

(tonemapped HDR)

Polynomial Texture Maps

• Malzbender et al. 2001 [9]: PTM

• Image-based, Sampling of Light (ωi)

Incomplete appearance information

• View-dependent part of reflectance missing

For 3D Objects: only one fixed viewpoint


PTM

Texture

Faithfulness


Photographic picture

(tonemapped HDR)

BTF + Geometry Schwartz et al. 2011 [8]

(tonemapped HDR)

Polynomial Texture Map Malzbender et al. 2001 [9]

(Single view and LDR!)

Multiview PTMs

• Gunawardane et al. 2009 [18]: • PTMs from multiple viewpoints

No macro scale

interpolation of views via optical flow

Limited amount of views

• Combining multiple objects is hard • incorrect silhouttes, occlusion,

shadows, etc.

• Full light transport (i.e. path-tracing) not possible

33 20/10/2011 3D COFORM STAR – VAST 2011 Prato, Italy – Christopher

Schwartz

Full Light Transport with BTFs


COMPRESSION, RENDERING AND TRANSMISSION


Datasizes

• Schwartz et al. 2011 [8]:

• Uncompressed BTF: ≈ 500 GB per object

• Not feasible even for offline rendering…


BTF Compression

• Fitting analytical models

• Wu et al. 2011 [15]: SPMM

• Model-driven…

• lost meso-structure


BTF

SPMM

BTF Compression

• High degree of redundancy in the BTF

• Perform statistical data analysis

• Find low dimensional basis

• learn how to best describe the data


Compression using Statistical Analysis

• Organization of the discrete BTF

• As matrix

• As Tensor


Angles ωi, ωo

Pix

els

(x,y

,co

lor)

Pixels (x,y)

ωo

Note: Color (or wavelength) can be additional dimension

Full Matrix Factorization

• E.g. Liu et al. 2004 [14]: FMF

• Representation is compact and realtime renderable [5]


…

...

M

… V

Angular components

“Eigen-ABRDFs“

U …

Spatial components

“Eigen-Textures“ SVD

TM USV

Importance

Decorrelated Full Matrix Factorization

• Gero Müller 2009 [13]: DFMF

• Insight from image compression (e.g. JPEG)

• Human perception is more sensitive to variation in intensity than color

• Also: chromacity in images exhibits less variation

• Decorrelate the BTF data into luminance and chrominance – BTFRGB BTFY BTFU BTFV

• Use fewer components for chrominance channels U, V


FMF Rendering

• Uncompressed: ≈ 500GB

• Compressed: ≈ 640MB

• 32 components

• Even fits on the GPU

• Random access to BTF

• Angular combination a = (ωi, ωo)

• Pixel p = (x,y)


com

po

nen

ts

Angular component #1

Spatial component #1

Angular component #2

Spatial component #2

…

…

pixel angles

BTF(a,p) = < , >

Interactive Inspection via GPU Rendering


Streaming of BTF over the Internet

• Schwartz et al. 2011 [16]: • Spatial components

≈ natural images

• Angular components ≈ low frequency

• Apply additional wavelet compression


0.4bpp Wavelet

16bpp Reference

Streaming of BTF over the Internet


0.87 MB First renderable

version

1 MB 7 MB 46.4 MB Fully

transmitted

534 GB Reference

Questions?

Learn more about BTFs for Cultural Heritag on Wednesday [8] and Thursday [16]


References [1] “A Data-Driven Reflectance Model”, Matusik W., Pfister H., Brand M. and McMillan L., ACM TOG 22, 3(2003), 759-769. [2] “Experimental Analysis of BRDF models”, Ngan A., Durand F. and Matusik W., Proceedings of EGSR, 2005, 117-226. [3] „Image-based Rendering with Controllable Illumination“, Wong T., Heng P., Or S., and Ng W., Proceedings of EGWR, 1997, 13-22 [4] „Reflectance and Texture of Real-World Surfaces “, Dana K.J., van Ginneken B., Nayar S.K. and Koenderink J.J., Proceedings of CVPR, 1997, 151-157 [5] „Efficient and Realistic Visualization of Cloth“, Sattler M., Sarlette R. and Klein R., Proceedings of EGSR, 2003 [6]„Appearance based object modeling using texture database: acquisition, compression and rendering“, Furukawa R., Kawasaki H. Ikeuchi K. and Sakauchi M., Proceedings of EGRW 2002, 257-266 [7] „Rapid Synchronous Acquisition of Geometry and BTF for Cultural Heritage Artefacts“, Müller G., Bendels G.H. and Klein R., Proceedings of VAST, 2005 [8] „Integrated High-Quality Acquisition of Geometry and Appearance for Cultural Heritage”, Schwartz C., Weinmann W., Roland R. and Klein R., Proceedings of VAST, 2011 [9] „Polynomial Texture Maps“, Malzbender T., Gelb D. and Wolters H., Proceedings of SIGGRAPH, 2001 [10] „A Generalized Surface Appearance Representation For Computer Graphics“, McAllister D.K., PhD. Thesis, University of North Carolina at Chapel Hill, 2002 [11] „A Multi-Camera, Multi-Projector Super-Resolution Framework for Structured Light“, Weinmann M., Schwartz C., Ruiters R. and Klein R., Proceedings of 3DIMPVT, 2011, 397-404 [12] „The Presentation of Cultural Heritage Models in Epoch“, Havemann S., Settgast V., Fellner D., Willems G., Van Gool, L., Müller G., Schneider M. and Klein R., EPOCH Conference on Open Digital CH Systems, 2008 [13] „Data-Driven Methods for Compression and Editing of Spatially Varying Appearance“, Müller G., PhD. Thesis, University of Bonn, 2009 [14] „Synthesis and Rendering of Bidirectional Texture Functions on Arbitrary Surfaces“, Liu X., Hu Y., Zhang J. Tong X., Guo B. and Shum H.-Y., IEEE Transactions on Visualization and Computer Graphics 10 (3), 2004, 278-289 [15] „A Sparse Parametric Mixture Model for BTF Compression, Editing and Rendering“, Wu H., Dorsey J. and Rushmeier H., Computer Graphics Forum 30 (2), 2011, 465-473 [16] „WebGL-based Streaming and Presentation Framework for Bidirectional Texture Function“, Schwartz C., Ruiters R., Weinmann M. and Klein R., Proceedings of VAST, 2011 [17] „A coaxial optica scanner for synchronous acquisition of 3D geometry and surface reflectance“, Holroyd M., Lawrence J. and Zickler T., ACM Trans. Graph. 29 (4), 2010 [18] „Optimized Image Sampling for View and Light Interpolation“, Gunawardane P., Wang O., Scher S., Rickards I., Davis J. And Malzbender T., Proceedings of VAST, 2009 [19] „Principles and practices of robust, photography-based digital imaging techniques for museums.“, Mudge M., Schroer C., Ear G., Martinez K., Pagi H., Toler-Franklin C., Rusinkiewicz S., Palma G., Wochowiak M., Ashley M., Matthews N., Noble T. and Dellepiane M., Proceedings of VAST, 2010


Realistic Object Appearance using Bidirectional Texture Functions

Technology

Transcript of Realistic Object Appearance using Bidirectional Texture Functions