以碎形正交基底和時間情境圖為基礎進行之視訊檢索 Video retrieval based on...

$: 以碎形正交基底和時間情境圖為基礎進行之視訊檢索 Video retrieval based on fractal orthogonal bases and temporal graph$
以碎形正交基底和時間情境圖為基礎進行之視訊檢索

Video retrieval based on fractal

orthogonal bases and temporal graph

研究生：張敏倫指導教授：蔣依吾博士國立中山大學資訊工程學系

視訊檢索關鍵字 - 檔名

人工定義 - 見解不同定義有所差別關鍵字意義不同

Content based video retrieval(CBVR) 傳統方法–影片摘要

缺乏視訊結構上觀點本論文

視訊結構 - 時間情境圖視訊內容 - 碎形正交基底

Li Gao and Zhu Li, “An Efficient Video Indexing and Retrieval Algorithm Using the Luminance Field Trajectory Modeling,” IEEE Trans. on circuits and systems for video technology, vol. 19, no. 10, p.1566~1570, Oct. 2009.

2009

C.-W. Su, H.-Y. Mark Liao, K.-C. Fan, C.-W. Lin and H.-R. Tyan, ”Motion Flow-Based Video Retrieval,” IEEE Trans. Multimedia, Vol. 9, No. 6, pp. 1193-1201, 2007.

2007

W. Hu, D. Xie, Z. Fu, W. Zeng, and S. Maybank, “Semantic-Based Surveillance Video Retrieval,” IEEE Trans. Image Processing, Vol. 16, No. 4, April 2007

2007

X. Zhu, A. K. Elmagarmid, X. Xue, L. Wu, and Ann Christine Catlin, “InsightVideo: Toward Hierarchical Video Content Organization for Efficient Browsing, Summarization and Retrieval,” IEEE Trans. Multimedia , vol. 7, no. 4, pp. 648~665, Aug. 2005.

2005

分鏡關鍵影像擷取

群集場景

相似性量測測試序列

K-d Tree

N(Q) is the total number of frames in the leaf nodes traversed by QN is the total number of frames in the database

Open Video


群集場景


Motion-tolerant dissolve detection algorithm[1] and Shot change detection based on the Reynolds transport theorem [2]

以 trajectory 來取代Key-frame


群集場景


1. Query-Keyword2. Multiple object

queries3. Query by sketch

Surveillance video


群集場景


1. Frame Level Similarity2. Shot Level Similarity

1. Shot-Length Matching

2. Average Color Histogram Matching

3. Camera Motion Matching

4. Key-frame Matching3. Group Level Similarity4. Scene Level Similarity5. Video Level Similarity

Open video

依據比率失真度準則進行 shot segment並擷取關鍵影像

影片序列輸入

Shot 內變異數達預設比率

N

NCut分群

建立時間情境圖

有向圖轉字串

相似性量測 :1. 視訊結構2. 視訊結構和視訊內容

FOB

frame

String

Video

Y

HSV粗略分割

字串剖析

依據比率失真度準則進行 shot segment並擷取關鍵影像

影片序列輸入


N

NCut分群


有向圖轉字串

FOB

frame

String

Video

Y

HSV粗略分割

字串剖析


HSV粗略分割 -降低複雜度 HSV 初始分割

影像之 HSV histogram 交集定義影像之間相似度令為第 x 張影像之彩色直方圖，影像 x 與

影像 y 之色彩相似度定義為xH

binsh

yx hHhHyxSim ))(),(min(),(

擷取關鍵影像根據失真變異度

每個分鏡選取一張關鍵影像若分鏡差異失真度大於 threshold 則分割

0 2 3 4 5 6 8 9 10 11 12 131 7 14

0 2 3 4 5 6 8 9 10 11 12 131 7 14

motion compensated

....0f 1f 2nf 1nf...

mf

'0f

'1f

'2f

... '1nf

'2nf

'3nf

current frame

),( ji

mfS ),( yx

'1

'1

'0

' ,..., nS fffV

),( ji

2f

...0f 1f 2nf 1nf...mf2f mfreference frame

reconstructed frame

3nf

3nf

1

0

'' ),(1

) , (n

jjjS ffDis

nVVDis

Rate- distortion criterion

)( mfDistortion),( 21 nffDistortion

1f 3nfmfmf

...0f 1f 2nf 1nf...

mf2f 3nf

'

0f'

1f'

2f ... '1nf

'2nf

'3nf

'mf...

Rate- distortion criterion

1f 2nfmfmf

...0f 1f 2nf 1nf...

mf2f 3nf

'

0f'

1f'

2f... '

1nf'

2nf'

3nf'

mf...

依據比率失真度準則進行shot segment並擷取關鍵

影像

影片序列輸入


N

NCut分群


有向圖轉字串

FOB

frame

String

Video

Y

HSV粗略分割

字串剖析


Clustering Using Normalized Cut 每個 segment 代表節點，邊 (edge) 上之權重值為

解公式 (1) 藉由合適特徵值所對應之特徵向量

),(exp)j , ( jisimT

ffiw

ji

j nddddiagDjiw

yDzwherezzDWDD

)...,(),,(d where

)1(,)(

21i

2

1

2

1

2

1

),(

),(

),(

),(),(

VBassoc

BAcut

VAassoc

BAcutBANcut

Temporal graph generation

0 1 2 3 2 2 84 5 6 7 91


影像

影片序列輸入


N

NCut分群


有向圖轉字串

相似性量測FOB

frame

String

Video

Y

HSV粗略分割

字串剖析

有向圖轉字串 - Algorithm

[ 0 1 3 4 8 ]

[ 0 1 2 6 7 8 9 ][ 0 1 2 6 7 8 9 [ 2 5 2] ][ 0 1 2 6 7 8 9 [ 2 5 2 [ 2 3 4 2] ] ]

[ 0 1 3 4 8 [ 1 2 1] ][ 0 1 3 4 8 [ 1 2 1] [ 3 5 6 7 4 ] ] [ 0 1 3 4 8 [ 1 2 1] [ 3 5 6 7 4 [ ( 53 45) ] ] ]


影像

影片序列輸入


N

NCut分群


有向圖轉字串

FOB

frame

String

Video

Y

HSV粗略分割

字串剖析


相似性量測 - 視訊結構利用 edit-distance 對兩 string 的主結構字串做

相似性量測

InsertjdistC

DeleteidistC

matchcomputeC

C

ji

ji

ji

ji

),,(

),,(

,

,1

1,

1,1

,

single node 有結構 node 無結構 node

single node 0 (n-1)/n (n-1)/n

有結構 node (n-1)/n 1

無結構 node (n-1)/n 1 1

'' , jiC

Edit Distance To find a smallest edit process between two

strings.

S1: TAGTCAC G

S2: AG ACTGTC

Operation: DMMDDMMIMII

Insertbdistc

Deleteadistc

baMatchc

c

jji

iji

jiji

ji

),(

),(

)(0

min

1,

,1

1,1

,

.1),(),( Suppose ji bdistadist lengthlength

jilengthlength

DQ

CDQDQd

,),(

0 1 2 12 [2 11 2 [ 2 3 2 [ 2 8 9 10 2 [ 2 4 5 6 7 2 ] ] ] ]

lengthlength

jilengthlength

DQ

CDQDQd

,),(

影片名稱時間種類影像數關鍵影像數出處

New Indians, Segment 02 00:03:28 Documentary 6231 10University of Maryland, College of Information

StudiesNASA 25th Anniversary Show, Segment 01

00:03:30 Documentary 6414 13University of Maryland, College of Information

StudiesNASA 25th Anniversary

Show, Segment 0300:02:22 Documentary 4718 22

University of Maryland, College of Information Studies

The Colorado, segment 01 of 10

00:03:13 Documentary 6164 7Carnegie Mellon University,

Informedia ProjectA New Horizon, segment

05 of 1300:01:59 Documentary 4046 17

Carnegie Mellon University, Informedia Project

Bor 00:01:06 Public Service 1989 7 JHUCCPDrift Ice as a Geologic Agent, segment 7 of 11

00:01:05 Documentary 1920 11 , Informedia Project

Drift Ice as a Geologic Agent, segment 10 of 11

00:03:46 Documentary 1380 10 , Informedia Project

Family Planning TV Spots Around the World:

Kyrgyzstan00:01:14 Public Service 2442 15 JHUCCP

How Water Won the West, segment 03 of 8

00:01:25 Documentary 2805 19Carnegie Mellon University,

Informedia Project


StudiesNASAKSNN - How Do Space Shuttles Blast Off

00:00:30 Educational 902 6 NASA

New Indians, Segment 111

00:01:02 Documentary 1860 14 of , Studies

Senses And Sensitivity, Introduction to Lecture 1

presenter00:02:20 Lecture 4620 25

University of North Carolina at Chapel Hill, Interaction Design Lab

The Voyage of the Lee 00:01:15 Documentary 2272 12 Geological Survey


Studies

NASA Connect - ISS - Space Station Parts

00:04:25 Educational 8745 31 NASA

視訊結構

碎形正交基底 (FOB) 相似影像有相似碎形函數；相似歸結圖有相似碎形函數；兩碎形函數不相似，其歸結圖亦不相似；兩影像不相似，其碎形函數亦不相似；

相似性量測 - 視訊結構和視訊內容

利用 edit-distance 對兩 string 主結構計算互相之 distance

InsertjdistC

DeleteidistC

matchcomputeC

C

ji

ji

ji

ji

),,(

),,(

,

,1

1,

1,1

,

single node 有結構 node 無結構 node

single node 0+ (n-1)/n+1/n* (n-1)/n+1/n*

有結構

node

(n-1)/n+1/n* 1

無結構

node

(n-1)/n+1/n* 1 1

),( jifob

),( jifob

),( jifob),( jifob ),( jifob

'' , jiC

視訊結構和視訊內容

30

0 1 2 12 [2 11 2 [ 2 3 2 [ 2 8 9 10 2 [ 2 4 5 6 7 2 ] ] ] ]

0 1 [ 1 2 3 1 [ 3 4 3 ] ]

視訊結構

視訊結構和視訊內容

34

結論與未來展望結論

分群準則 - 影響時間情境圖時間複雜度 - 視訊結構和視訊內容

未來展望依個人喜好建立個人化之影片類別降低計算時間

報告完畢

謝謝

以碎形正交基底和時間情境圖為基礎進行之視訊檢索 Video retrieval based on...

Documents

Transcript of 以碎形正交基底和時間情境圖為基礎進行之視訊檢索 Video retrieval based on...