Talal_Imran
-
Upload
tariq-mahmood -
Category
Documents
-
view
218 -
download
0
Transcript of Talal_Imran
-
8/2/2019 Talal_Imran
1/7
Image Classification and Retrieval using Correlation
Imran Ahmad
School of Computer ScienceUniversity of Windsor
Windsor, ON N9B 3P4 - Canada
Muhammad Talal Ibrahim
Dept. of Computer ScienceCOMSATS Institute of Information Technology
Islamabad - Pakistan
Abstract
Image retrieval methods aim to retrieve relevant images
from an image database that are similar to the query image.
The ability to effectively retrieve non-alphanumeric data is
a complex issue. The problem becomes even more difficult
due to the high dimension of the variable space associatedwith the images. Image classification is a very active and
promising research domain in the area of image manage-
ment and retrieval. In this paper, we propose a new image
classification and retrieval scheme that automatically se-
lects the discriminating features. Our method consists of
two phases: (i) classification of images on the basis of max-
imum cross correlation and (ii) retrieval of images from the
database against a given query image. The proposed re-
trieval algorithm recursively searches similar images on the
basis of their correlation against a given query image from
a set of registered images in the database. The algorithm
is very efficient, provided that the mean images of all of the
classes are computed and available in advance. The pro-
posed method classifies the images on the basis of maximumcorrelation so that the images with more similarities and,
hence, exhibiting maximum correlation with each other are
grouped in the same class and, are retrieved accordingly.
1. Introduction
With advances in computing and digital imaging tech-
nologies, the number of images is increasing rapidly, thus,
making it necessary to provide techniques for efficient man-
agement and retrieval of stored images. Image classifica-
tion and retrieval is also a major issue in the areas of pat-
tern recognition, robotics and artificial intelligence. Exam-ples of systems requiring classification include but are not
Author would liketo acknowledge partialsupport provided by the Nat-
ural Sciences and Engineering Research Council (NSERC) of Canada and
Higher Education Commission (HEC) of Pakistan for completion of part
of this work.
limited to visual tracking [7], image registration [6], and
content-based image retrieval [2].
Essentially, there are two main image retrieval tech-
niques: (i) text-based retrieval and (ii) content-based re-
trieval. All of the earlier image retrieval techniques were
text based and involved annotations of salient image fea-
tures or association of keywords with the image contents.Retrieval was done by issuing a text-based query and only
on the basis of successful match of the keywords or the
annotations. Even though text-based approaches are able
to capture the high level abstractions and concepts asso-
ciated with the image contents, it is believed that due to
manual procedures for annotations of image contents, such
approaches are fraught with difficulties. Manual proce-
dures are not only time consuming but are highly subjec-
tive. Moreover, some of the visual aspects of images are
inherently difficult to describe while others could equally
be described in many different ways [4]. Given the huge
amount of image data that exists now or will be collected
in future, manual approaches are clearly inadequate. In
order to eliminate such problems and to make image re-
trieval more efficient, there has been a great emphases on
retrieval techniques that are based on automatic extraction
of visual features and mathematical attributes from the im-
age contents. Such type of retrieval is generally known as
the Content-based Image Retrieval (CBIR) and has been
an active area of research for many years. Even though
CBIR approaches (see [12] for a partial list of such tech-
niques) can capture directly computable low level features
from the images, efficient and precise image retrieval still
remains to be an open problem. Despite development of
many commercial and/or academic/research CBIR systems
over the years, researchers are still actively working to find
a more comprehensive solution with an objective to provideimproved precision and recall.
Auto correlation and cross correlation are standard statis-
tical techniques and are used in the areas of image process-
ing and pattern recognition to estimate the degree to which
two given patterns are correlated. These techniques have
Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV06)
0-7695-2542-3/06 $20.00 2006IEEE
-
8/2/2019 Talal_Imran
2/7
been successfully applied to estimate the motion of moving
objects [10], relevance feedback in CBIR systems [8] and
image registration, etc. [10].
In this paper, we propose an image classification and re-
trieval scheme that is based on the concept of maximum
cross correlation between images with promising results. In
the proposed approach, we first classify images on the ba-
sis of maximum cross correlation with each other and storethem accordingly whereas for the retrieval of an image, this
classification is used.
The rest of the paper is organized as follows: in Section
2, a brief review of few important existing techniques and
those related to our approach is presented. In Section 3,
the proposed system is discussed whereas some of the ex-
perimental results are presented and discussed in Section 4.
Finally, Section 5 provides some concluding remarks.
2. Related Work
Since 1990s, there has been a considerable progress in
the area of content-based image retrieval. In order to re-trieve and browse image data on the basis of their contents
and pictorial queries, many content-based image retrieval
systems have been proposed [1, 3, 4, 5, 15, 11, 13]. A sur-
vey of some of the important techniques is given in [12].
Such systems are essentially based on low level image fea-
tures that are directly computed from the image contents.
Some of the most commonly used features are the color, the
shape, the texture and the spatial locations and distributions
of the image objects [1, 4, 13].
Image retrieval is merely not restricted to search and rep-
resentation only. New issues are continuously introduced by
the subjectivity of the human perception and need to be ad-
dressed to provide satisfactory retrieval performance. Sys-
tems with relevance feedback are an attempt to address such
issues and to improve the quality of the retrieval. Such sys-
tems allow interaction with the user, who in turn, is respon-
sible for determining the quality of match and retrieval [9].
A new perspective in image retrieval method involving
a combination of factor analysis with relevance feedback
method is introduced in [9] whereas a statistical correlation
model for the retrieval of relevant images is presented in [8].
In this model, an estimate of the correlation between two
images based on the number of search sessions in which im-
ages have been marked relevant is calculated. Since in the
process of relevance feedback, main emphasis is to improve
the retrieval accuracy, several passes are made through the
database during the retrieval process and a correlation is dy-namically calculated by interacting with the user. However,
establishing a correlation dynamically is not only a time
consuming process but it also makes it difficult to incor-
porate positive and negative examples in query and/or the
similarity refinement process.
3. Proposed System
In Statistics terminology, correlation is a measure of the
relation between two or more variables. As mentioned ear-
lier, cross correlation and normalized correlation are stan-
dard statistical methods that have been successfully used
in various image related applications. Given the two same
size vectors or matrices, two-dimensional cross correlationbetween them can be calculated using the 2-D Correlation.
Assuming two matrices A and B of the same dimension
(m,n), the 2-D discrete normalized correlation can be com-puted as:
r =
m
n
(Amn A)(Bmn B)(m
n
(Amn A)2)(m
n
(Bmn B)2)
where A is the mean of the values of matrix A and B is
the mean of the values of matrix B and are given as:
A =1
mn
m1i=0
n1j=0
A(i, j)
and
B =1
mn
m1i=0
n1j=0
B(i, j)
The calculated correlation coefficients can range from -1
to +1 such that a value of -1 represents no correlation be-
tween the matching entities whereas a value of +1 repre-
sents a perfect positive correlation or a perfect match. Any
value in between is an indication of the degree of correla-
tion, and depending on application, can be used to make a
proper selection/retrieval decision.
Our proposed system has two phases: (i) classification or
clustering and (ii) retrieval. In the classification phase, im-
ages are classified on the basis of their maximum cross cor-
relation with each other according to the expression given
above and a collection of mean images corresponding to
each class is maintained. During the retrieval process, cross
correlation between the query image and the mean images is
determined and the class with maximum correlation satisfy-
ing the threshold criteria provided by the user is returned to
the user. A block diagram of the proposed system is given
in Figure 1. It is important to note that before the classifi-
cation process begins, all of the images are normalized to
make them all of same size.
Let the image database contains n discrete images andthat represents a user-defined threshold - minimum ac-
ceptable correlation between the query and the database im-
ages. The proposed recursive image classification algorithm
is then carried out by using one of the following three clas-
sification schemes:
Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV06)
0-7695-2542-3/06 $20.00 2006IEEE
-
8/2/2019 Talal_Imran
3/7
New Image
CorrelationDetermination
Calculated correlationinformation of classified
images
ImageDatabase
Image Classification
Query Image
CorrelationDetermination
Image Retrieval
Retrieve class withmaximum correlation
ImageDisplay
Figure 1. System Architecture.
Linear classification: In this approach, images are
classified by making several passes over the database.
In the classification process, the first image from the
database is taken and its cross correlation with rest of
the images in the database is calculated. As a result, an
image with maximum correlation with the first one is
picked up and the two jointly form a new class. Now
we have two classes: one containing only two images
as discussed above and another one containing remain-
ing (n 2) images. We then take the first image fromthe class containing (n 2) images and calculate itscorrelation with rest of the images in its own class and
the class containing only two images. If its correla-
tion is maximum with the first class then it is moved
to the first class otherwise we pick the image from the
second class with which it has maximum cross correla-
tion and make a separate new class which will contain
only these two images. The same process is repeated
over and over until all of the images in the initial larger
class are classified. This scheme is very time consum-
ing with all worst, average and best case classification
time complexities ofn2 and results in maximum num-
ber of classes among the three classification methods
used in our experiments.
Selective classification: In this classification scheme,
we initially start with only two classes such that eachcontains only a single randomly picked image from the
database. The rest of the images are then classified on
the basis of their maximum correlation with the two
initial classes. Same process is repeated recursively
within each of the newly formed class and stops only
when it fails to satisfy the given threshold criteria,
defined at the beginning of the classification process. It
is important to note that the threshold can be the value
of the correlation coefficient such that 0 1 orsome number of images in a class. Result of the clas-
sification process is a binary tree in which leaf nodes
represent classes in which the images are maximally
cross correlated with other images in the same classand represent similar images.
Figure 2. Classes obtained using selectiveclassification approach.
Auto classification: In this scheme, we start by se-
lecting an image from the database and calculating its
correlation with rest of the images in the database and
choose the one that has the minimum correlation with
the first image. In this way, two initial classes are
formed that are totally independent of each other andhave minimum correlation between them. Now one by
one, we pick each of the remaining (n2) images andmake them part of one of the two newly formed classes
on the basis of their cross correlation. Same process is
repeated within each class as long as each class satis-
Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV06)
0-7695-2542-3/06 $20.00 2006IEEE
-
8/2/2019 Talal_Imran
4/7
fies the criteria provided by the threshold parameter .
For retrieval against a given query image, the query im-
age is compared with the mean of each of the classes
and a correlation between the two is calculated. Im-
ages in a class having maximum cross correlation with
the query image are retrieved as the possible similar
images to the query image with a similarity measure
or the quality of match given by the difference in themagnitude of the cross relation of the query image and
that of the images in the retrieved class.
Figure 3. Classes obtained using auto classi-
fication approach.
The classification process in our approach is an expen-
sive process. Even though all of the three classificationschemes mentioned above have same time complexity n2,
there is a difference in the number of classes formed and,
hence, in the retrieval performance. The auto classification
method results in the least number of classes, and hence, the
number of leaf nodes.
Result of any of these classification schemes is an un-
ordered binary tree with following properties:
The root of the tree represents the mean of all of the
images in the database.
Each of its two children represent classified images
such that the images in the same child are maximally
correlated. Subsequent iterations are represented by
the addition of more nodes to the tree.
The internal nodes of the tree represent mean image of
the images in their children.
The leaf nodes contain maximally cross correlated
classified images whereas the number of leaf nodes
represent the number of distinct classes.
Arguably, the classification process in our scheme is the
most time consuming process since the cross correlation of
all of the images in the database against those which have
already been classified needs to be determined. However,
it is important to note that the images are classified only
once and the classification process could be done off-line.
Further, to avoid repetitive computations, at each level of
processing, we store the mean of the classes as well. Dur-
ing the retrieval phase, only these mean images are used to
make a proper selection against the given query image as
shown in Figure 1.
As mentioned earlier, the classification process recur-
sively groups images in two classes such that the images in
the same class have maximum correlation with each other
whereas the correlation is minimum between the classes.
The recursive process stops only when a threshold conditionspecified at the beginning is satisfied. It is important to note
that the threshold can be defined either in terms of number
of images in a class or a specific value of the correlation co-
efficient. The corresponding image retrieval process works
as follows:
1. Get the query image and recursively compare it with
the mean images stored in the nodes of the tree.
2. Retrieve the class having maximum correlation with
the query image.
The retrieval in this system is very efficient since thequery image is compared only with the mean images of the
classes only rather than all of the images in the database.
If the resulting binary tree is kept as an ordered tree, the
worst case performance of the retrieval process is only
O(log2(n)).
Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV06)
0-7695-2542-3/06 $20.00 2006IEEE
-
8/2/2019 Talal_Imran
5/7
Figure 4. Sample images from the database.
4. Experimental Results
We have tested our approach on an image database con-
taining 5000 images. Our database is essentially a sub-
set of the A Very Large Image Database by the Labora-
tory for Engineering Man/Machine Systems (LEMS) at the
Brown University, Rhode Island, USA [14]. Images in this
database are binary images from a variety of different cat-
egories such as animals, people, arts, etc. An example of
images in this data collection and used in our experiments
is shown in Figure 4.
For the purpose of classification, we trained our system
and obtained statistics using all of the 5000 images in our
data collection that are size normalized. Results of classi-
fication were collected by employing the above mentionedclassification methods.
Figure 5. Mean Image of the class using auto
classification approach.
The proposed system works more efficiently if the resul-
tant binary tree is a skewed tree. In this case, retrieval will
require at most 2h iterations to retrieve relevant subset ofimages where h is the height of the tree.
Figure 2 is an example of selective classification pro-
Figure 6. A sample sub-class obtained usingauto classification approach
cess involving only 100 random images from the database
whereas the result of classification of same 100 images us-
ing auto correlation approach is shown in Figure 3. It is
important to note that with only 100 images in the two clas-
sifications schemes, the number of leaf nodes and the mean
images are roughly the same. However, it is not the case
when all of the images are classified.
Figure 5 is an example of one of the mean images repre-
senting a class in the auto classification process. The actual
images corresponding to this mean image are shown in Fig-ure 6. As can be seen, this class contains two major groups
of the images as determined by the threshold of the sys-
tem. If the value of is readjusted, the two groups will be
classified further as shown in Figure 7.
The number of leaf nodes as a function of number of
images in the database for the three mentioned classification
schemes is given in Figure 8. As one might expect, the
number of classes and the leave nodes are more in the linear
classification scheme than those in the selective and the auto
classification schemes and, hence, it may take more to both
classify images as well retrieve relevant images from the
database.
Figure 9 shows the precision-recall graph for our re-trieval results. The query image used in some of these cases
is part of the database and in some other cases, it is a totally
different image. As one can observe from the graph, the av-
erage prcission achieved for these two cases is about 90%.
It should be observed that, on the average, about 90% of the
Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV06)
0-7695-2542-3/06 $20.00 2006IEEE
-
8/2/2019 Talal_Imran
6/7
(a)
(b)
Figure 7. Resultant sub-classes (a) and (b) af-
ter adjusting for the class in Figure 6.
relevant images can be retrieved by the system.
5. CONCLUSIONS
To estimate the degree of correlation between two given
patterns, auto correlation and cross correlation are the com-
mon statistical techniques used in the areas of image pro-
cessing and pattern recognition. In this paper, we have pre-
sented a new image classification and retrieval approach
that is based on the concept of correlation. In this ap-
proach, images are classified through an off-line process on
the basis of their cross correlation with other images in the
database. Images with maximum cross relation are recur-
sively grouped in the same class. The resultant hierarchy
is maintained as a binary tree in which each root node rep-
resents the mean of the images in its subtrees such that the
leaf nodes contain maximally correlated images, thus, mak-
ing the retrieval process very efficient.
We are well aware of the fact that the classification pro-
cess in our scheme is very expensive and has time complex-
ity of(n2
) where n is the number of images in the database.However, once classified, retrieval becomes very efficient
and requires only O(log2(n)) comparisons.At this point, we have not tested our system for insertion
of new images. Since insertion will involve recomputation
of the mean images and reclassification, it is expected to be
0
50
100
150
200
250
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
No. of images
No.ofleafnodes
..
Auto Classification Selective Classification Linear Classification
Figure 8. Graph showing the Comparison of
Selective classification and Auto classifica-
tion.
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0.00 0.20 0.40 0.60 0.80 1.00
Recall
Precision
.
Figure 9. Precision-recall graph for the aver-
age case
as expensive as the original classification process. Due to
this limitation, the proposed system may only be suitable for
those limited environments which deal with batch updates
such as the library archives.
References
[1] I. Ahmad and W. I. Grosky. Indexing and retrieval of images
by spatial constraint. Journal of Visual Communication and
Image Representation, 14(3):291320, Sept. 2003.
[2] L. G. Brown. A Survey of Image Regisration Techniques.
ACM Computing Survey, 28(4):325372, 1992.
[3] R. Brunelli and O. Mich. Image Retrieval by Example. IEEE
Transactions on Multimedia, 3(2):164171, 2000.
[4] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang,
B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic,
D. Steele, and P. Yanker. Query by Image and Video Con-
tent: The QBIC System. IEEE Computer, 28(9):2332,
Sept. 1995.
Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV06)
0-7695-2542-3/06 $20.00 2006IEEE
-
8/2/2019 Talal_Imran
7/7
[5] T. Gevers and A. Smeulders. Pictoseek: Combining color
and shape invariant features for image retrieval. IEEE Trans-
actions on Image Processing, 9(1):102119, 2000.
[6] S. Kruger and A. Calway. Image Registration using Mul-
tiresolution Frequency Domain Correlation. In Proceedings
of the British Machine Vision Conference, pages 316325,
1998.
[7] K.-C. Lee, J. Ho, M.-H. Yang, and D. Kriegman. Vi-
sual Tracking and Recognition using Probabilistic Appear-
ance Manifolds. Computer Vision and Image Understanding
(CVIU), 99(3):303331, 2005.
[8] Z. C. M. Li, W. Liu, and H. J. Zhang. A Statistical Cor-
relation Analysis in Image Retrieval. Pattern Recognition,
35:26872693, 2002.
[9] A. Machado, C. N. J. Marinho, M. Fernando, and M. Cam-
pos. An image retrieval method based on factor analysis. In
Proc.of XVI BrazilianSymp. on Computer Graphics and Im-
age Processing (SIBGRAPI03), pages 191196, Oct. 2003.
[10] S.-H. Park and H. J. Sung. Correlation-based image regis-
tration for applications using pressure-sensitive paint. AIAA
Journal, 43(1), Jan. 2005.
[11] A. Pentland, R. W. Picard, and S. Sclaroff. Photobook: Tools
for Content-Based Manipulation of Image Databases. In
Storage and Retrieval for Image and Video Databases II,
volume 2185, pages 3447. SPIE, 1994.
[12] A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and
R. Jain. Content-based image retrieval at the end of the early
years. IEEE Transaction on Pattern Analysis and Machine
Intelligence (PAMI) , 22(12):13491380, Dec. 2000.
[13] J. R. Smith and S.-F. Chang. VisualSEEk: A Fully Auto-
mated Content-Based Image Query System. In Proceedings
of ACM Conference on Multimedia, pages 211218, 1996.
[14] URL. http://www.lems.brown.edu/dmc, 2006.
[15] B. M. Wei-Ying Ma. NeTra: A Toolbox for Navigating
Large Image Database. Multimedia Systems, 7(3):184198,
1999.
Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV06)
0-7695-2542-3/06 $20.00 2006IEEE