Scale-Invariant Feature Transform (SIFT)Jinxiang Chai
ReviewImage Processing - Median filtering
- Bilateral filtering
- Edge detection
- Corner detection
Review: Corner Detection1. Compute image gradients
2. Construct the matrix from it and its neighborhood values
3. Determine the 2 eigenvalues (i.j)= [1, 2].
4. If both 1 and 2 are big, we have a corner
The Orientation FieldCorners are detected where both 1 and 2 are big
Good Image FeaturesWhat are we looking for?Strong featuresInvariant to changes (affine and perspective/occlusion)Solve the problem of correspondenceLocate an object in multiple images (i.e. in video)Track the path of the object, infer 3D structures, object and camera movement,
Scale Invariant Feature Transform (SIFT)Choosing features that are invariant to image scaling and rotationAlso, partially invariant to changes in illumination and 3D camera viewpoint
InvarianceIlluminationScaleRotationAffine
Required ReadingsObject recognition from local scale-invariant features [pdf link], ICCV 09
David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004), pp. 91-110
Motivation for SIFTEarlier MethodsHarris corner detectorSensitive to changes in image scaleFinds locations in image with large gradients in two directionsNo method was fully affine invariantAlthough the SIFT approach is not fully invariant it allows for considerable affine changeSIFT also allows for changes in 3D viewpoint
SIFT Algorithm Overview
Scale-space extrema detectionKeypoint localizationOrientation AssignmentGeneration of keypoint descriptors.
Scale SpaceDifferent scales are appropriate for describing different objects in the image, and we may not know the correct scale/size ahead of time.
Scale space (Cont.)Looking for features (locations) that are stable (invariant) across all possible scale changesuse a continuous function of scale (scale space)
Which scale-space kernel will we use?The Gaussian Function
Scale-Space of Image
variable-scale Gaussianinput image
Scale-Space of Image
variable-scale Gaussianinput imageTo detect stable keypoint locations, find the scale-space extrema in difference-of-Gaussian function
Scale-Space of Image
variable-scale Gaussianinput imageTo detect stable keypoint locations, find the scale-space extrema in difference-of-Gaussian function
Scale-Space of Image
variable-scale Gaussianinput imageTo detect stable keypoint locations, find the scale-space extrema in difference-of-Gaussian function
Look familiar?
Scale-Space of Image
variable-scale Gaussianinput imageTo detect stable keypoint locations, find the scale-space extrema in difference-of-Gaussian function
Look familiar? -bandpass filter!
Difference of GaussianA = Convolve image with vertical and horizontal 1D Gaussians, =sqrt(2)B = Convolve A with vertical and horizontal 1D Gaussians, =sqrt(2)DOG (Difference of Gaussian) = A BSo how to deal with different scales?
Difference of GaussianA = Convolve image with vertical and horizontal 1D Gaussians, =sqrt(2)B = Convolve A with vertical and horizontal 1D Gaussians, =sqrt(2)DOG (Difference of Gaussian) = A BDownsample B with bilinear interpolation with pixel spacing of 1.5 (linear combination of 4 adjacent pixels)
Difference of Gaussian PyramidInput ImageBlurBlurBlurDownsampleDownsampleB2B3A2A3A3-B3A2-B2A1-B1DOG2DOG1DOG3Blur
Other issuesInitial smoothing ignores highest spatial frequencies of images
Other issuesInitial smoothing ignores highest spatial frequencies of images - expand the input image by a factor of 2, using bilinear interpolation, prior to building the pyramid
Other issuesInitial smoothing ignores highest spatial frequencies of images - expand the input image by a factor of 2, using bilinear interpolation, prior to building the pyramid
How to do downsampling with bilinear interpolations?
Bilinear FilterWeighted sum of four neighboring pixels xyuv
Bilinear FilterSampling at S(x,y):(i+1,j)(i,j)(i,j+1)(i+1,j+1)S(x,y) = a*b*S(i,j) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1)
uvyx
Bilinear FilterSampling at S(x,y):(i+1,j)(i,j)(i,j+1)(i+1,j+1)S(x,y) = a*b*S(i,j) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1)
Si = S(i,j) + a*(S(i,j+1)-S(i))Sj = S(i+1,j) + a*(S(i+1,j+1)-S(i+1,j))S(x,y) = Si+b*(Sj-Si)To optimize the above, do the followinguvyx
Bilinear Filter(i+1,j)(i,j)(i,j+1)(i+1,j+1)yx
Pyramid ExampleA1B1DOG1DOG3DOG3A2A3B3B2
Feature DetectionFind maxima and minima of scale spaceFor each point on a DOG level: Compare to 8 neighbors at same levelIf max/min, identify corresponding point at pyramid level belowDetermine if the corresponding point is max/min of its 8 neighborsIf so, repeat at pyramid level aboveRepeat for each DOG levelThose that remain are key points
Identifying Max/MinDOG L-1DOG LDOG L+1
Refining Key List: IlluminationFor all levels, use the A smoothed image to computeGradient Magnitude
Threshold gradient magnitudes: Remove all key points with MIJ less than 0.1 times the max gradient valueMotivation: Low contrast is generally less reliable than high for feature points
Results: Eliminating FeaturesRemoving features in low-contrast regions
Results: Eliminating FeaturesRemoving features in low-contrast regions
Assigning Canonical OrientationFor each remaining key point:Choose surrounding N x N window at DOG level it was detectedDOG image
Assigning Canonical OrientationFor all levels, use the A smoothed image to computeGradient Orientation
+Gaussian Smoothed ImageGradient OrientationGradient Magnitude
Assigning Canonical OrientationGradient magnitude weighted by 2D Gaussian with of 3 times that of the current smoothing scaleGradient Magnitude2D GaussianWeighted Magnitude*=
Assigning Canonical OrientationAccumulate in histogram based on orientationHistogram has 36 bins with 10 incrementsWeighted MagnitudeGradient OrientationGradient OrientationSum of Weighted Magnitudes
Assigning Canonical OrientationIdentify peak and assign orientation and sum of magnitude to key pointWeighted MagnitudeGradient OrientationGradient OrientationSum of Weighted MagnitudesPeak*
Eliminating edgesDifference-of-Gaussian function will be strong along edgesSo how can we get rid of these edges?
Eliminating edgesDifference-of-Gaussian function will be strong along edgesSimilar to Harris corner detector
We are not concerned about actual values of eigenvalue, just the ratio of the two
Eliminating edgesDifference-of-Gaussian function will be strong along edgesSo how can we get rid of these edges?
Local Image DescriptionSIFT keys each assigned:LocationScale (analogous to level it was detected)Orientation (assigned in previous canonical orientation steps)Now: Describe local image region invariant to the above transformations
SIFT: Local Image DescriptionNeeds to be invariant to changes in location, scale and rotation
SIFT Key Example
Local Image DescriptionFor each key point: Identify 8x8 neighborhood (from DOG level it was detected)Align orientation to x-axis
Local Image DescriptionCalculate gradient magnitude and orientation mapWeight by Gaussian
Local Image DescriptionCalculate histogram of each 4x4 region. 8 bins for gradient orientation. Tally weighted gradient magnitude.
Local Image DescriptionThis histogram array is the image descriptor. (Example here is vector, length 8*4=32. Best suggestion: 128 vector for 16x16 neighborhood)
Applications: Image MatchingFind all key points identified in source and target imageEach key point will have 2d location, scale and orientation, as well as invariant descriptor vectorFor each key point in source image, search corresponding SIFT features in target image. Find the transformation between two images using epipolar geometry constraints or affine transformation.
Image matching via SIFT featruesFeature detection
Image matching via SIFT featrues Image matching via nearest neighbor search - if the ratio of closest distance to 2ndclosest distance greater than 0.8 then reject as a false match. Remove outliers using epipolar line constraints.
Image matching via SIFT featrues
SummarySIFT features are reasonably invariant to rotation, scaling, and illumination changes.
We can use them for image matching and object recognition among other things.
Efficient on-line matching and recognition can be performed in real time
Top Related