Andrew Rosenberg- Lecture 21: Spectral Clustering
Transcript of Andrew Rosenberg- Lecture 21: Spectral Clustering
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
1/43
Lecture21:SpectralClustering
April22,2010
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
2/43
LastTime
GMMModelAdapta=onMAP(MaximumAPosteriori)MLLR(MaximumLikelihoodLinearRegression)
UMB-MAPforspeakerrecogni=on
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
3/43
Today
GraphBasedClusteringMinimumCut
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
4/43
Par==onalClustering
Howdowepar==onaspacetomakethebestclusters?
Proximitytoaclustercentroid.
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
5/43
DifficultClusterings
Butsomeclusteringsdontlendthemselvestoacentroidbaseddefini=onofacluster.
Spectralclusteringallowsustoaddressthesesortsofclusters.
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
6/43
DifficultClusterings
Thesekindsofclustersaredefinedbypointsthatarecloseanymemberinthecluster,
ratherthantheaveragememberofthecluster.
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
7/43
GraphRepresenta=on
Wecanrepresenttherela=onshipsbetweendatapointsinagraph.
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
8/43
GraphRepresenta=on
Wecanrepresenttherela=onshipsbetweendatapointsinagraph.
Weighttheedgesbythesimilaritybetweenpoints
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
9/43
Represen=ngdatainagraph
Whatisthebestwaytocalculatesimilaritybetweentwodatapoints?
Distancebased:d(xi, xj) = expxi xj
2
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
10/43
Graphs
NodesandEdges Edgescanbedirectedorundirected Edgescanhaveweightsassociatedwiththem
Heretheweightscorrespondtopairwiseaffinitywij = d(xi, xj) = exp
xi xj
2
wijxi
xj
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
11/43
Graphs
Degree
Volumeofaset
D(xi) =
jV
wij
V ol(C) =
iC
D(xi)
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
12/43
GraphCuts
Thecutbetweentwosubgraphsiscalculatedasfollows
Cut(C1, C2) =
iC1
jC2
wij
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
13/43
GraphExamples-Distance
Height Weight
20 5
8 6
9 4
4 4
4 51
5D
E
C
5.1
2.2
B
4.5
4.1
A
12
11
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
14/43
GraphExamples-Similarity
Height Weight
20 5
8 6
9 4
4 4
4 51
.2D
E
C
.19
.45
B
.22
.24
A
.08
.09
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
15/43
Intui=on
Theminimumcutofagraphiden=fiesanop=malpar==oningofthedata.
SpectralClusteringRecursivelypar==onthedataset
Iden=fytheminimumcut Removeedges Repeatun=lkclustersareiden=fied
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
16/43
GraphCuts
Minimum(bipar==onal)cutminCut(C1, C2) =
iC1
jC2
wij
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
17/43
GraphCuts
Minimum(bipar==onal)cutminCut(C1, C2) =
iC1
jC2
wij
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
18/43
GraphCuts
Minimal(bipar==onal)normalizedcut.
Unnormalizedcutsarearactedtooutliers.
minCut(C1, C2)
V ol(C1)+
Cut(C1, C2)
V ol(C2)= min
1
V ol(C1)+
1
V ol(C2)Cut(C1, C2)
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
19/43
Graphdefini=ons
-neighborhoodgraphIden=fyathresholdvalue,,andincludeedgesif
theaffinitybetweentwopointsisgreaterthan.
k-nearestneighborsInsertedgesbetweenanodeanditsk-nearest
neighbors.
Eachnodewillbeconnectedto(atleast)knodes. Fullyconnected
Insertanedgebetweeneverypairofnodes.
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
20/43
Intui=on
Theminimumcutofagraphiden=fiesanop=malpar==oningofthedata.
SpectralClusteringRecursivelypar==onthedataset
Iden=fytheminimumcut Removeedges Repeatun=lkclustersareiden=fied
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
21/43
SpectralClusteringExample
MinimumCutHeight Weight
20 5
8 6
9 4
4 4
4 51
.2D
E
C
.19
.45
B
.22
.24
A
.08
.09
Cut(BCDE,A) = 0.17
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
22/43
SpectralClusteringExample
NormalizedMinimumCutHeight Weight
20 5
8 69 4
4 4
4 51
.2D
E
C
.19
.45
B
.22
.24
A
.08
.09
NormCut(BCDE,A) = 1.067
NormCut(C1, C2) =Cut(C1, C2)
V ol(C1)+
Cut(C1, C2)
V ol(C2)
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
23/43
SpectralClusteringExample
NormalizedMinimumCutHeight Weight
20 5
8 69 4
4 4
4 51
.2D
E
C
.19
.45
B
.22
.24
A
.08
.09
NormCut(BCDE,A) = 1.067
NormCut(ABC,DE) = 1.038
NormCut(C1, C2) =Cut(C1, C2)
V ol(C1)+
Cut(C1, C2)
V ol(C2)
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
24/43
Problem
Iden=fyingaminimumcutisNP-hard. Thereareefficientapproxima=onsusinglinear
algebra. BasedontheLaplacianMatrix,orgraphLaplacian
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
25/43
SpectralClustering
ConstructanaffinitymatrixA B C D
A .4 .2 .2 0
B .2 .5 .3 0
C .2 .3 .6 .1
D 0 0 .1 .1
.1
.3
.2.2
A
B C
DW =
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
26/43
SpectralClustering
ConstructthegraphLaplacian
Iden=fyeigenvectorsoftheaffinitymatrix
D = diag(D0, D1, . . . , Dn)
L = DW
Lv = v
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
27/43
SpectralClustering
K-Meansoneigenvectortransforma=onofthedata.
Projectbacktotheini=aldatarepresenta=on.
k-eigenvectors
n-datapoints
EachRowrepresentsa
datapointintheeigenvector
space
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
28/43
Overview:whatarewedoing?
Definetheaffinitymatrix Iden=fyeigenvaluesandeigenvectors. K-meansoftransformeddata Projectbacktooriginalspace
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
29/43
Whydoesthiswork?
IdealCase
Whatareweop=mizing?Whydotheeigenvectorsofthelaplacianincludeclusteriden=fica=oninforma=on
1 0
1 0
1 0
0 1
0 1
0 1
1 1 1 0 0 0
1 1 1 0 0 0
1 1 1 0 0 0
0 0 0 1 1 1
0 0 0 1 1 1
0 0 0 1 1 1
Lv = v
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
30/43
Whydoesthiswork?
Howdoesthiseigenvectordecomposi=onaddressthis?
ifweletfbeeigenvectorsofL,thentheeigenvaluesaretheclusterobjec=vefunc=ons
f(xj) = fj clusterassignmentL = D W
fT
Lf = fT
DffT
Wf
=i
dif2
i
ij
fifjwij
=1
2
i
j
wij
f2i 2ij
fifjwij +ji
wij
f2j
=1
2
ij
wij(fi fj)2
Clusterobjec=vefunc=onnormalizedcut!
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
31/43
NormalizedGraphCutsview
Minimal(bipar==onal)normalizedcut.
Eigenvaluesofthelaplacianareapproximatesolu=onstomincutproblem.
minCut(C1, C2)
V ol(C1)
+Cut(C1, C2)
V ol(C2)
= min1
V ol(C1)
+1
V ol(C2)Cut(C1, C2)
NCut(A,B) =yT(D W)y
yTDy
minyyT(D W)y subject to yTDy = 1
(DW)y = Dy
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
32/43
TheLaplacianMatrix
LD-W Posi=vesemi-definite Thelowesteigenvalueis0,eigenvectoris Thesecondlowestcontainsthesolu=on
Thecorrespondingeigenvectorcontainstheclusterindicatorforeachdatapoint
xTLx 0
1
2 =Cut(A,B)
|A|+Cut(A,B)
|B|
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
33/43
Usingeigenvectorstopar==on
Eacheigenvectorpar==onsthedatasetintotwoclusters.
Theentryinthesecondeigenvectordeterminesthefirstcut.
Subsequenteigenvectorscanbeusedtofurtherpar==onintomoresets.
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
34/43
Example
Denseclusterswithsomesparseconnec=ons
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
35/43
3classExample
Affinitymatrix eigenvectors rownormaliza=on output
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
36/43
Example[Ngetal.2001]
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
37/43
k-meansvs.SpectralClustering
K-means SpectralClustering
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
38/43
Randomwalkviewofclustering
Inarandomwalk,youstartatanode,andmovetoanothernodewithsomeprobability.
Theintui=onisthatiftwonodesareinthesamecluster,youarandomlywalkislikelyto
reachboth
points.
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
39/43
Randomwalkviewofclustering
Transi=onmatrix: Thetransi=onprobabilityisrelatedtotheweightofgiventransi=onandtheinverse
degreeofthe
currentnode.
Lrw= ID
1W = D
1L
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
40/43
Usingminimumcutforsemi
supervisedclassifica=on? Constructagraphrepresenta=onofunseendata. Insertimaginarynodessandtconnectedtolabeledpoints
withinfinitesimilarity.
Treatthemincutasamaximumflowproblemfromstot
s
t
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
41/43
KernelMethod
Theweightbetweentwonodesisdefinedasafunc=onoftwodatapoints.
Wheneverwehavethis,wecanuseanyvalidKernel.
wij = d(xi, xj) = expxi xj
2
wij = K(xi, xj)
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
42/43
Today
Graphrepresenta=onsofdatasetsforclustering
SpectralClustering
-
8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering
43/43
NextTime
Evalua=on.Classifica=onClustering