Background Blurring
description
Transcript of Background Blurring
Background [email protected]
2
Outline Introduction
Documentation◦Object Segmentation◦Depth Map Generation◦ Image Defocus
3
Method
Lazy Snapping
Perspective Box
Pop-up Card
Camera Settings
Defocus BlurDefocus Blur
Privacy PreservingPrivacy Preserving
Photo BrowserPhoto Browser
Object Segmentation
Depth Map Generation
Applications
Image Defocus
Lazy Snapping
Perspective Box
Pop-up Card
Camera Settings
MethodSegment◦Lazy snapping◦Mean shift
5
Resize to 20%
Do hard graph cut by lazy snapping
User Stroke
Mean shift
Lazy Snapping. ACM Trans. On Graphics 2004.Mean Shift: A Robust Approach Toward Feature Space Analysis. PAMI 2002.A Bayesian Approach to Digital Matting. CVPR 2001.
MethodDepth map
◦ Perspective Box Vanish point Rear wall
Depth Map
Perspective Box
Normal vector
Pop-up card
6
Depth Map
Normal vector
Pop-up card
Tour Into the Picture: Using a spidery mesh user interface to make animation from a single image. SIGGRAPH 1997
MethodDepth map
◦ Pop-up card
Depth Map
Perspective Box
Normal vector
Pop-up card
7
Depth Map
Normal vector
Pop-up card
MethodDepth map
◦ Normal vector
Depth Map
Perspective Box
Normal vector
Pop-up card
8
Depth Map
Normal vector
MethodDefocus blur◦bokeh
𝐼𝑑𝑒𝑓𝑜𝑐𝑢𝑠 ሺ𝑖,𝑗ሻ= σ𝐸𝑛ሺ𝑖,𝑗ሻσ𝑤𝑒𝑖𝑔ℎ𝑡𝑛(𝑖,𝑗) + 𝛼𝑛(𝑖,𝑗)
𝑤𝑒𝑖𝑔ℎ𝑡𝑛ሺ𝑖,𝑗ሻ= 1 𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛,𝑠,𝑏)Τ
𝛼𝑛(𝑖,𝑗) =൜𝑤𝑒𝑖𝑔ℎ𝑡𝑛(𝑖,𝑗), 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ< 𝑇ℎ𝑏𝑜𝑘𝑒ℎ0, 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ≥ 𝑇ℎ𝑏𝑜𝑘𝑒ℎ
9
𝑤𝑒𝑖𝑔ℎ𝑡𝑛ሺ𝑖,𝑗ሻ= 1𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛ሺ𝑥𝑛,𝑦𝑛,𝑠,𝑏ሻ× 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛)𝑙𝑢𝑚𝑖𝑛𝑜𝑢𝑠(𝑥𝑛,𝑦𝑛)
……
……E1(i,j)
I(x1,y1) I(x2,y2) I(x3,y3)
I(xn,yn) I(xn+1,yn+1)
E2(i,j) E3(i,j)
En(i,j) En+1(i,j)
En+2(i,j)
I(xn+2,yn+2)
𝐸𝑛ሺ𝑖,𝑗ሻ= 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ 𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛,𝑠,𝑏)Τ 𝑤𝑒𝑖𝑔ℎ𝑡𝑛ሺ𝑖,𝑗ሻ= 1 𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛,𝑠,𝑏)Τ x
Method
10
Defocus blur……
……E1(i,j)
I(x1,y1) I(x2,y2) I(x3,y3)
I(xn,yn) I(xn+1,yn+1)
E2(i,j) E3(i,j)
En(i,j) En+1(i,j)
En+2(i,j)
I(xn+2,yn+2)
Method
◦Near-by object case
11
Focus on the flower
Focus on the flower
◦Near-by object case
Method Blur circle diameter
Objects in front of the focus
Objects behind
the focus
Defocus Blur
Texture synthesis inpainting
Blur shape as alpha map
Defocus Blur
Interpolation by alpha map
Shallow focus image 12
Experimental Result 1 Defocus blur method proposed in the system
comparing with other blur filter results
13Deep focus Shallow focus
Experimental Result 1 Defocus blur method proposed in the system comparing
with other blur filter results
Gaussian blur Defocus blur Defocus blur + Bokeh
14
Experimental Result 1
15
Deep focus
Shallow focus
Defocus blur
Experimental Result 1
16
Gaussian blur Defocus blur + Bokeh
Original photograph
• Defocus blur method proposed in the system comparing with other blur filter results
Experimental Result 2
◦Near-by object case
17
Focus on the flower
Focus on the flower
◦Near-by object case
Without inpainting
Interpolation result
Experimental Result 2
18
Applications - Privacy Preserving
19
Applications - Image Transition at Photo Browser
20
21
Applications - Image Transition at Photo Browser
Movie with defocus blur
effect 22
Applications - Image Transition at Photo Browser
Documentation
23
./ 系統主要分成三部份,前兩塊產生出 depth 和segment 的資訊之後,可以在第三部份根據不同參數設定產生出不同的效果◦system
1_segmentation Tool1: lazy-snapping with simple UI Tool2: 光影魔術手
2_depthmap depth of background scene (3d-box) depth of each segment
3_refocusblur camera parameter refocusing (inpainting, defocus blur)
◦sample // sample input, parameter and result
24
Segmentation◦Lazy snapping
Tool1: preprocessing(resize for mean shift ) + lazy snapping
Lzsegment.m Input : parameter file name先做 preprocessing ,這裡可以用 meanshift 或是直接
downsample 成 64 色階 ,並且在 meanshift 的地方 resize為五分之一加快速度,
讀取 stroke_1.bmp, stroke_2.bmp… 等前後景筆劃圖,前景是紅色,後景是藍色。然後執行 lazy snapping 處理
◎ 每個 segment 都存一個灰階的 bmp 檔,檔名分別是seg_1.bmp, seg_2.bmp… 沒有先後順序。將每個 segment扣除剩下的 background 的部份,即是 seg_segN.bmp
◎ 需要 libgil2 的 library
Tool2: 光影魔術手 每個 segment 都存一個 png 檔,檔名分別是 obj_1.png,
obj_2.png… 將這些 png 檔的 alpha 值轉成 bmp, 並且扣除只剩下 background 的部份 , 即是 seg_segN.bmp 25
Depth map◦depth.m (function depth(parafilename))
Input: parameter file name dep_1.dat, dep_2.dat, …dep_(segN-1).dat★ 每個 dep_n.dat 的內容 :
nx ny
p1x p1y
p2x p2y
… ★ n 表示 normal vector, 預設是 (0,0), vector 方向代表物體在
平面上的法向量方向,長度表示深度變化量 ★ p1,p2… 是 user 劃下的筆劃座標,即此 segment 的深度參考
點,預設是 segment 最下方的點。如果沒有 p1,p2… 系統會自動取 segment 最下方一列的點作為參考點。
Output: dep_1.bmp, dep_2.bmp … dep_segN.bmp
26
Refocus Blur◦ refocus.m ( function [finalresult]=defocus(parafilename) )
Get parameter from file, and load the segment/depth bmp files
Get the focus position from focusposition.dat Compute diffusion blur circle diameter for each
pixel Devide image into two parts, the part in front of the
focus, and run defocus_blur.m Use alpha.m to get alpha map, and find the pixels to
run inpaint.m Use the result by inpaint.m to call defocus_blur.m Combine the front part and back part by alpha map return the result image
27
others◦ Inpainting function needs the
estexemplarhelper.c run mex estexemplarhelper.c before the
function call at first timeThe inpainting code (inpaint.m) is from
http://www.cc.gatech.edu/~sooraj/inpainting/
◦Meanshift code is downloaded from◦http://www.caip.rutgers.edu/riul/
◦
28
29
中間檔案 /參數 Generate from for
focuspoint.dat UI畫定 refocus.m 的聚焦點資訊
dep_1.dat, dep_2.dat …
UI畫定 depth.m每個segment的深度參考點和 normal值,如果沒有給予參考點則視為最下方一排
stroke_1.bmp, stroke_2.bmp…
UI畫定 Lazy snapping 的前後景分割依據
seg_1.bmp, seg_2.bmp…
lzsegment.m 產生,或是用 tool2取得obj.png之後,用
seg_png2bmp.m產生
depth.m和 refocus.m都需要用到的 segment map
dep_1.bmp, dep_2.bmp…
depth.m產生 refocus.m需要用到的深度資訊
v1,v2,a1,a2,b1,b2.. Vanish box和消失點座標 ,可用 UI介面畫
定
depth.m產生背景深度資訊所需要
其他 camera 參數 使用者自行調整
Input parameter file 1.dirname diratory of the files, include '/' on the tail, ex: 'sample1/' 2.imagename file name of input image, ex: test.jpg 3.segN number of segment image, include the background.
ex: 2 (foreground*1+background) 4.vx position of vanish point, must be in the rear wall, 5.vy but can be outside of the image 6.ax 7.ay left_top position of the rear wall 8.bx 9.by right_bottom position of the rear wall 10.dc_f lens focal length,(mm) ex: 50 mm, 100 mm ...etc. 11.dpmax the depth of pixels with color level 255 in depth map, (mm) 12.dnear the depth of pixels with color level 0 in depth map, (mm) 13.dc_N the aperture number 14.thr=250 threshold of bokeh color level 15.light_e=0.5 little constant number for light_weight function 16.lens shape default setting 0 is flat disk
30
Camera parameter dc_f 相機的焦距,單位是 mm ,數值越大景深越淺,
建議在 50~100 之間 Dpmax rear wall 距離成像平面的距離,建議為 5~50 公尺
(5000~50000)
Dnear 場景裡距離成像平面最近的物體距離,建議為 1 公尺,至少要在兩倍焦距以上
dc_N 光圈值,數值越大,光圈孔隙越小,景深越深,為達到淺景深效果建議值在 1.4~2.8 之間,越小效果越
強 thr=250 將 pixel 產生散景效果的臨界值,建議不要低於 240 light_e=0.5 weight function 的明度微調參數,為一個大於 0 的值,
數值越大明度影響越低,建議為 0.5~1 之間 lens shape 光圈孔隙的形狀,預設為 0
對應到 refocus/lens/ 資料夾底下的 0.bmp, 1.bmp
31
Sample◦ ./
parameter01-.dat
◦ ./sample01 original image seg_1.bmp seg_2.bmp … dep_1.dat dep_2.dat …
可由 UI 產生 focusposition.dat
可由 UI 產生,給定要聚焦的點,可以為單一或是多個點,將會取其對應到的深度平均值,格式為
x1 y1x2 y2…
dep_1.bmp dep_2.bmp …
32
執行:◦用 system 的前兩步驟產生出前頁所述的參數和
bmp files ,並放在所屬資料夾下◦ result = refocus(parameter file name);
33
34
END
THANK YOU