Interactive Matting
description
Transcript of Interactive Matting
![Page 1: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/1.jpg)
Interactive MattingChristoph Rhemann
Supervised by:Margrit Gelautz and Carsten Rother
![Page 2: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/2.jpg)
Matting and compositing
![Page 3: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/3.jpg)
Matting and compositing
![Page 4: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/4.jpg)
Outline
Talk Outline:
• Introduction & previous approaches
• Our matting model
• Evaluation strategy
![Page 5: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/5.jpg)
= +
Cr,g,b = α Fr,g,b + (1 - α) Br,g,b
● ●
● ●
Inverse process of compositing:
Determine: F, B, αGiven:C
Matting is ill posed
![Page 6: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/6.jpg)
= +
Underconstrained problem:7 Unknowns in only 3 Equations
● ●
Cr = α Fr + (1 - α) Br
Cg = α Fg + (1 - α) Bg
Cb = α Fb + (1 - α) Bb
Cr,g,b = α Fr,g,b + (1 - α) Br,g,b● ●
Matting is ill posed
![Page 7: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/7.jpg)
Trimap
Scribbles
Background
Background
Unknown
Foreground
Unknown
Foreground
User interaction
![Page 8: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/8.jpg)
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
![Page 9: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/9.jpg)
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
Closed Form Matting [Levin et al. 06]
R
B
G
![Page 10: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/10.jpg)
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
R
B
G
Closed Form Matting [Levin et al. 06]Assumption: F and B colors in a local window lie on color line
![Page 11: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/11.jpg)
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
R
B
G
Closed Form Matting [Levin et al. 06]Assumption: F and B colors in a local window lie on color line Analytically eliminate F,B. Alpha can be solved in closed form
![Page 12: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/12.jpg)
Result of [Levin et al 06]True Solution Input image + Trimap
Result of Closed Form Matting [Levin et al. 06]:• Result imperfect: Hairs cut off• Problem: Cost function has large solution space
Previous approaches
![Page 13: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/13.jpg)
What are the reasons for pixels to be transparent?
Segmentation – based matting
Defocus Blur
![Page 14: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/14.jpg)
Lens Camera sensor
Point spread
function
Point Spread Function
Focal plane
Lens’ aperture
Lens and defocus
Slides by Anat Levin
![Page 15: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/15.jpg)
LensObject Camera sensor
Point spread
function
Lens’ aperture
Focal planeSlides by Anat Levin
Lens and defocus
Point Spread Function
![Page 16: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/16.jpg)
What are the reasons for pixels to be transparent?
Segmentation – based matting
Defocus Blur Motion Blur
PSF forMotion Blur
![Page 17: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/17.jpg)
What are the reasons for pixels to be transparent?
Segmentation – based matting
Defocus Blur Motion Blur
Discretization
![Page 18: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/18.jpg)
What are the reasons for pixels to be transparent?
Observation: Apart from translucency mixed pixels are caused by camera’s Point Spread Function (PSF)
Segmentation – based matting
Defocus Blur Motion Blur
Discretization Translucency
![Page 19: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/19.jpg)
Basic idea:Model alpha as convolution of a binary segmentation with PSF
Approach taken [Rhemann et al. 08]:Use this model as prior in framework of [Levin et al. 06]
Model for alpha
Binary segmentation PSF Observed alphaInput image + Trimap
![Page 20: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/20.jpg)
Matting process
Initial alpha using [Wang et al. ´07]
(Result is imperfect)
Initialize PSF/deblur alpha
Deblured (sparse) alpha
Binarized (sparse) alpha using gradient
preserving MRF prior
Iterate a few times
Input image
![Page 21: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/21.jpg)
Matting process
Binarized (sparse) alpha using gradient
preserving MRF prior
Segmentation prior
Final alpha
Ground truth
![Page 22: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/22.jpg)
Result for [Levin et al. ’06]
Input image
Input image + trimap
Comparison
![Page 23: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/23.jpg)
Result of [Wang et al. ’07]
Input image
Input image + trimap
Comparison
![Page 24: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/24.jpg)
Input image
Input image + trimap
Result of [Rhemann et al. ’08]
Comparison
![Page 25: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/25.jpg)
Input image + trimap [Levin et al. ’06]
[Wang et al. ’07] [Rhemann et al. ’08] Ground truth alpha
[Levin et al. ’07]
Comparison – Close up
![Page 26: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/26.jpg)
Evaluation of matting algorithms
How to compare performance of algorithms?
Showing some qualitative results
OR
Quantitative evaluation using reference solutions
![Page 27: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/27.jpg)
Evaluation of matting algorithms
• Key Factors for a good quantitative evaluation
• Ground truth dataset
• Online evaluation
• Perceptual error functions
![Page 28: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/28.jpg)
• 35 natural images• High resolution• High quality
Triangulation Matting [Smith, Blinn 96]- Photograph object against 2 different backgrounds True solution to matting problem
Input image Ground truth Zoom in
Ground truth dataset
![Page 29: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/29.jpg)
Data and evaluation scripts online
Advantages:• Investigate results• Upload novel results
www.alphamatting.com
Online evaluation
![Page 30: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/30.jpg)
Motivation:Simple metrics not always correlated with visual quality
Input image Zoom in Result 1SAD: 1215
Result 2SAD: 806
Perceptually motivated error functions
![Page 31: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/31.jpg)
Develop error measures for two properties:• Connectivity of foreground object• Gradient of the alpha matte
Perceptually motivated error functions
Input image Zoom in Result 1SAD: 312
Result 2SAD: 83
![Page 32: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/32.jpg)
User Study:• Goal: Infer visual quality of image compositions• Task: Rank to according to how realistic they appear
Perceptually motivated error functions
Gradient artifacts Connectivity artifacts
![Page 33: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/33.jpg)
Correlation of error measures to average user ranking
Gradient data Connectivity data0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8Grad.
Grad.
Conn.
Conn.
GradientConnectivitySADMSE
Correlation
Perceptually motivated error functions
![Page 34: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/34.jpg)
• Model for alpha overcomes ambiguities
• Model-based algorithm: Performs better than competitors
• Perceptual motivated evaluation
• Message to you: Evaluation of your algorithm is important• Use ground truth data to make quantitative comparisons• Use a large dataset• Use a training / test split
Conclusions
![Page 35: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/35.jpg)
![Page 36: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/36.jpg)
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
R
B
G
Model of F
Model of B
Observed color
Data driven approaches (e.g. [Wang et al. 07])• Model color distribution of F and B (from the user defined trimap)• Observed color more likely under F or B model?• Use likelihood in framework of [Levin et al 06]
![Page 37: Interactive Matting](https://reader036.fdocument.pub/reader036/viewer/2022062222/568164da550346895dd72c34/html5/thumbnails/37.jpg)
Result of data driven approaches [Wang et al. 07]:• Hair is better captured• Many artifacts in the background
Previous approaches
Result of [Levin et al 06]True Solution Input image + Trimap Result of [Wang et al 07]