What, Where & How Many? Combining Object Detectors and CRFs
description
Transcript of What, Where & How Many? Combining Object Detectors and CRFs
2023年4月21日 星期五
1
What, Where & How Many?Combining Object Detectors and CRFs
L’ubor Ladický, Paul Sturgess, Karteek Alahari, Chris Russell, and Philip H.S. Torr
Lecturer : Zhiguo Ma
Outline
• Authors
• Abstract
• Background
• Hierarchical CRF
• Object detector potential in CRF
• Experiments & Conclusion
2023年4月21日 星期五
2
作者介绍
L’ubor Ladický8 papers on CVPR,ICCV,BMVC,ECCV ,etc.
Best paper of BMVC 2010 & ECCV 2010
Website : http://sots.brookes.ac.uk/lubor/
No information for Paul Sturgess & Chris Russell
2023年4月21日 星期五
3
Karteek Alahari
2023年4月21日 星期五
4
10+ papers on ACCV, ICPR, CVPR, BMVC,PAMI, ECCV, etc.Website: http://www.di.ens.fr/~alahari/
Philip H.S. Torr
PhD at the Robotics Research Group of the University of Oxford.
Oxford as a research fellow, and is currently a Visiting Fellow in Engineering Science at the University of Oxford
Research scientist for Microsoft Research
Many papers on Journal & conference in fields of CV,ML, PR.
2023年4月21日 星期五
5
Abstract
Computer vision algorithms for individual tasks such as object recognition, detection and segmentation have shown impressive results in the recent past. The next challenge is to integrate all these algorithms and address the problem of scene understanding. This paper is a step towards this goal. We present a probabilistic framework for reasoning about regions, objects, and their attributes such as object class, location, and spatial extent. Our model is a Conditional Random Field defined on pixels, segments and objects. We define a global energy function for the model, which combines results from sliding window detectors, and low-level pixel-based unary and pair wise relations. One of our primary contributions is to show that this energy function can be solved efficiently. Experimental results show that our model achieves significant improvement over the baseline methods on CamVid and PASCAL VOC datasets.
2023年4月21日 星期五
6
摘要针对单独任务(如物体识别、检测和分割)的计算机视觉算法,在近几年取得很大的进步。下一个挑战是整合这些算法,解决场景理解的问题,本篇文章是向此目标前进的一步。我们提出了一种概率性框架用于推断区域、物体及其属性(如物体类别,位置及空间范围等)。我们的模型是一个定义在像素、区域、物体上的条件随机场。模型定义了一个全局能量函数,整合来自滑动窗口物体检测器、底层像素级的一元和二元信息。我们的一个主要贡献是展示这个能量函数可以被有效地求解。在 CamVid 及 PASCAL VOC 数据集上的结果显示,我们的模型比基准算法获得了很大的性能提升。
2023年4月21日 星期五
7
Background
2023年4月21日 星期五
8
( a )原始图像 ( b )物体类别分割 ( c )物体检测 ( d )检测与分割结合(本文)
物体类别分割会丢失一些物体,且不提供物体数目信息;物体检测能检测到此类物体,但不提供前、背景分割结果。整合分割与检测,可以解决上述问题。
Related Work
Stuff and ThingsStuff: homogeneous or reoccurring pattern of fine-scale properties, but no specific spatial extent or shape
Things: have distinct size and shape.
Object class segmentationSuccessful on stuff, but fails on things
Foreground (thing) object detectionGood at things, but fail on stuff, which is amorphous
2023年4月21日 星期五
9
CRF
Label set object class( such as car, airplane, bicycle, etc.)
Random variables Image pixel
Clique c set of pixels conditionally dependent on each other
Labeling x any possible assignment of labels to pixels2023年4月21日 星期五
10
1 2{ , ,..., }nL l l l
1 2{ , ,..., }nX X X X
Posterior distribution & energy of CRF
2023年4月21日 星期五
11
Labeling
Data
Normalized factor
Clique Set Potential function
*
Engery Function: ( ) log Pr( | ) log ( )
MAP estimation of X: argmax Pr( | ) arg min ( )
c c
x L x L
E x X D Z xc C
X x D E x
1Pr( | ) exp( ( ))c c
c C
X D xZ
Potentials in energy function
Unary potentialLocal feature responses , the likelihood of a pixel taking a certain label
Pairwise potentialEncourage neighboring pixels take the same label
Higher order potential between segmentsModel relationship between segments, object, etc.
Color potential for instance of objectsForeground and background estimation
2023年4月21日 星期五
12
Object detector potential in CRF
2023年4月21日 星期五
13
The set of pixels in a detection d1 is denoted by Xd1 , yd1 represent the validity of detector
Energy function with detector potential
2023年4月21日 星期五
14
( ) ( ) ( , , )pix d d dd D
E X E x x H l
Pixel-based energy
Set of detections Detection score
Pixels in detection Detected Label
{0,1}
d
( , , ) min ( , , , )
y indicats whether detector hypothesis is validdd d d y d d d dx H l y x H l
2023年4月21日 星期五
15
{0,1}( , , ) min ( ( , ) ( , ) )
( , ) max(0,H ) define the strength of hypothesis
( , )( , ) = define the cost of label inconsistency
is detector threshold
N is t
dd d d d d d d d d
y
d d d d d t
d d dd d
d d
t
d
x H l f x H y g N H y
f x H w x H
f x H Ng N H
p x
H
d
henumber of inconsistent pixels
p is acceptable percentage of inconsistent pixels
Inference for detector potentials
Rewrite detector potential:
2023年4月21日 星期五
16
N
( , , ) min(0, ( , ) ( ))
( , ) min( ( , ), ( )).
Form of robust P potential can be soloved by graph
( ) min( , ( (
cut
( ), (
)
,
)
) ,
)
d
d
d d d d d d i di x
d d d d d i d
h max
i x
max l
l l il
i x
x H l f x H k x l
f x H f
x
x H k x
min k
l
f f l d
x l
'
' '
0
( ( , ) ( , ) ).d
d
d d d d d d dy
y
y arg min f x H y g N H y
Experimental Results
DatasetCamVid
10 minutes of high quality 30HZ
960 X 720 resolution
Three of four sequences shot in daylight, one shot in dusk
32 classes totally, 11 classes used in this papers
PASCAL VOC 200914743 images, 20 foreground class and 1 background class
749 training, 750 validation and 750 test images.
2023年4月21日 星期五
17
Details of CRF framework
Two level hierarchy CRF based on pixels and segmentsPixel-based potentials
Use TextonBoost to estimate the probability of a certain label by boosting weak classifiers based on a set of shape filter responses.
Segment-based potentialsSegments or super-segments based on Mean shift Joint Boosting algorithm
2023年4月21日 星期五
18
Detection-based potentials
DetectorsHistogram-based detector
Multiple features( bag of word, self-similarity, SIFT and oriented edges descriptors)Cascaded classifier composed of SVMs
Parts-based detectorHOG descriptorsDeformable parts and global templateLatent SVM
Output of detectorsBounding boxes with response scores
Foreground and background color modelGMM
2023年4月21日 星期五
19
Results on CamVid dataset
2023年4月21日 星期五
20
Result on PASCAL VOC dataset
2023年4月21日 星期五
21
2023年4月21日 星期五
22
Result on PASCAL VOC dataset
Summary
Integration of detectors with CRF.
Can handle occluded objects and false detections
Efficient and tractable with graph cut.
2023年4月21日 星期五
23
2023年4月21日 星期五
24
Thank you!
Any Question?