Manifold learning
Transcript of Manifold learning
05/03/2023 1
A Brief Introduction to Manifold Learning
Wei Yang
Some slides are from Geometric Methods and Manifold Learning in Machine Learning (Mikhail Belkin and Partha Niyoqi). Summer School (MLSS), Chicago 2009
05/03/2023 2
What is a manifold?
https://en.wikipedia.org/wiki/Manifold
05/03/2023 3
Manifolds in visual perception
Consider a simple example of image variability, the set of all facial images generated by varying the orientation of a face
• 1D manifold:– a single degree of freedom: the angle of rotation
• The dimensionality of would increase if we allow– image scaling
– illumination changing
– …
The Manifold Ways of Perception•H. Sebastian Seung and •Daniel D. LeeScience 22 December 2000
05/03/2023 4
Why Manifolds?
• Euclidean distance in the high dimensional input space may not accurately reflect the intrinsic similarity– Euclidean distance
– Geodesic distance
Linear Manifold VS. Nonlinear Manifold
05/03/2023 5
Differential Geometry
• Embedded manifolds
• Tangent space
• Geodesic
• Laplace-Beltrami operator
05/03/2023 6
Embedded manifolds
• Locally (not globally) looks like Euclidean space
𝑆 2⊂ℝ3
05/03/2023 7
Example: circle
• Charts: continuous, invertible
• Atlas: charts covered the whole circle
• Transition map 𝑥
𝑦
¿𝜙 h𝑟𝑖𝑔 𝑡 (𝑎 ,√1−𝑎2)¿√1−𝑎2
𝑎
𝜙𝑡𝑜𝑝❑ − 1 (𝑎 )
𝜙 h𝑟𝑖𝑔 𝑡 (𝜙𝑡𝑜𝑝❑ −1 (𝑎) )
http://en.wikipedia.org/wiki/Manifold
05/03/2023 8
Tangent space
• -dimensional affine subspace of
𝑇 𝑝ℳ 𝑘⊂ℝ𝑁
𝑝
05/03/2023 9
Tangent vectors and curves
• Tangent vectors <———> curves.
Geometric Methods and Manifold Learning – p. 1
𝑣
𝜙 (𝑡 )
𝑝
05/03/2023 10
Tangent vectors as derivatives
• Tangent vectors <———> Directional derivatives
Geometric Methods and Manifold Learning – p. 1
𝑣
𝜙 (𝑡 )
𝑝
05/03/2023 11
Riemannian geometry
• Norms and angles in tangent space
Geometric Methods and Manifold Learning – p. 1
𝑣
𝑤𝑝
05/03/2023 12
Length of curves and geodesics
• Can measure length using norm in tangent space.
• Geodesic — shortest curve between two points.
Geometric Methods and Manifold Learning – p. 1
𝑝
05/03/2023 13
Gradient
• Tangent vectors <———> Directional derivatives
• Gradient points in the direction of maximum change.
Geometric Methods and Manifold Learning – p. 1
¿𝛻 𝑓 ,𝑣>≡ 𝑑𝑓𝑑𝑣
𝑣
𝜙 (𝑡 )
𝑝
05/03/2023 14
Exponential map
• Geodesic:
Geometric Methods and Manifold Learning – p. 1
𝑞
𝑝
𝑟
𝑣𝑤
𝜙 (𝑡 )
05/03/2023 15
Laplace-Beltrami operator
• Orthonormal coordinate system.
Geometric Methods and Manifold Learning – p. 1
𝑝 𝑥2𝑥1
exp𝑝 :𝑇 𝑝ℳ 𝑘→ℳ 𝑘
05/03/2023 16
Linear Manifold Learning
• Principal Components Analysis
• Multidimensional Scaling
05/03/2023 17
Principal Components Analysis
• Given … with mean 0
• Find such that
• And
• is leading eigenvectors of
05/03/2023 18
Multidimensional Scaling
• MDS: exploring similarities or dissimilarities in data.
• Given data points with distance function is defined as:
• The dissimilarity matrix can be defined as:
Find … such that
05/03/2023 19
Nonlinear Manifold Learning
• ISOMAP (Tenenbaum, et al, 00)
• LLE (Roweis, Saul, 00)
• Laplacian Eigenmaps (Belkin, Niyogi, 01)
05/03/2023 20
Algorithmic framework
• Neighborhood graph common to all methods.
05/03/2023 21
Isomap: Motivation
• PCA/MDS see just the Euclidean structure
• Only geodesic distances reflect the true low-dimensional geometry of the manifold
• The question: – How to approximate geodesic distances?
05/03/2023 22
Isomap
1. Construct neighborhood graph using Euclidean distance– -Isomap: neighbors within a radius
– -Isomap: nearest neighbors
2. Compute shortest path as the approximation of geodesic distance
1.
2. For , replace all by
3. Construct -dimensional embedding using MDS
05/03/2023 23
Isomap: results
Face varying in pose and illumination Hand varying in finger extension & wrist rotation
05/03/2023 24
Isomap: estimate the intrinsic dimensionality
05/03/2023 25
Locally Linear Embedding
• Intuition: each data point and its neighbors are expected to lie on or close to a locally linear patch of the manifold.
05/03/2023 26
Locally Linear Embedding
1. Assign neighbours to each data point (-NN)
2. Reconstruct each point by a weighted linear combination of its neighbors.
3. Map each point to embedded coordinates.
S T Roweis, and L K Saul Science 2000;290:2323-2326
05/03/2023 27
Steps of locally linear embedding
• Suppose we have data points in a dimensional space.
• Step 1: Construct neighborhood graph– -NN neighborhood
– Euclidean distance or normalized dot products
05/03/2023 28
Steps of locally linear embedding
• Step 2: Compute the weights that best linearly reconstruct from its neighbors by minimizing
where
05/03/2023 29
Steps of locally linear embedding
• Step 3: Compute the low-dimensional embedding best reconstructed by by minimizing
• Note: is a sparse matrix, and -th row is barycentric coordinates (center of mass) of in the basis of its nearest neighbors.
• Similar to PCA, using lowest eigenvectors of to embed.
05/03/2023 30
LLE (Comments by Ruimao Zhang)
算法优点• LLE 算法可以学习任意维的局部线性的低维流形 .
• LLE 算法中的待定参数很少 , K 和 d.
• LLE 算法中每个点的近邻权值在平移 , 旋转 , 伸缩变换下是保持不变的 .
• LLE 算法有解析的整体最优解 , 不需迭代 .
• LLE 算法归结为稀疏矩阵特征值计算 , 计算复杂度相对较小 , 容易执行 .
算法缺点• LLE 算法要求所学习的流形只能是不闭合的且在局部是线性的 .
• LLE 算法要求样本在流形上是稠密采样的 .
• LLE 算法中的参数 K, d 有过多的选择 .
• LLE 算法对样本中的噪音很敏感 .
05/03/2023 31
Laplacian Eigenmaps
• Using the notion of the Laplacian of a graph to compute a low-dimensional representation of the data– The laplacian of a graph is analogous to the Laplace Beltrami operator
on manifolds, of which the eigenfunctions have properties desirable for embedding (See M. Belkin and P. Niyogi for justification).
05/03/2023 32
Laplacian matrix (discrete Laplacian)
• Laplacian matrix is a matrix representation of a graph
– is the Laplacian matrix
– is the degree matrix
– is the adjacent matrix
05/03/2023 mlss09us_niyogi_belkin_gmml 33
Laplacian Eigenmaps
05/03/2023 mlss09us_niyogi_belkin_gmml 34
Laplacian Eigenmaps
05/03/2023 mlss09us_niyogi_belkin_gmml 35
Laplacian Eigenmaps
: Degree matrix
: (Weighted) adjacent matrix: Laplacian matrix
𝐷− 1𝐿𝑓=𝜆 𝑓
05/03/2023 36
Justification of optimal embedding
• We have constructed a weighted graph
• We want to map to a line so that connected points stay as close together as possible
• This can be done by minimizing the objective function
• It incurs a heavy penalty if neighboring points are mapped far apart.
05/03/2023 37
Justification of optimal embedding (Cont.)
05/03/2023 38
Justification of optimal embedding (Cont.)
The minimization problem reduces to finding
Note the constraint removes an arbitrary scaling factor in the embedding.
Using Lagrange multiplier and setting the derivative with respect to equal to zero, we obtain
The optimum is given by the minimum eigenvalue solution to the generalized eigenvalue problem (trivial solution: ).
05/03/2023 39
More methods for non-linear manifold learning
05/03/2023 40
Applications
• Super-resolution
• Laplacianfaces
05/03/2023 41
Super-Resolution Through Neighbor Embedding
• Intuition: small patches in the low- and high-resolution images form manifolds with similar local geometry in two distinct spaces.
• X: low-resolution image Y: target high-resolution image
• The algorithm is extremely analogous to LLE!– Step 1: construct neighborhood of each patch in X
– Step 2: compute the reconstructing weights of the neighbors that minimize the reconstruction error
– Step 3: perform high-dimensional embedding to (as opposed to the low-dimensional embedding of LLE)
– Step 4: Construct the target high-resolution image Y by enforcing local compatibility and smoothness constraints between adjacent patches obtained in step 3.
Chang, Hong, Dit-Yan Yeung, and Yimin Xiong. "Super-resolution through neighbor embedding." CVPR 2004.
05/03/2023 42
Super-Resolution Through Neighbor Embedding
• Training parameters– The number of nearest neighbors K
– The patch size
– The degree of overlap
Chang, Hong, Dit-Yan Yeung, and Yimin Xiong. "Super-resolution through neighbor embedding." CVPR 2004.
05/03/2023 43
Super-Resolution Through Neighbor Embedding
Chang, Hong, Dit-Yan Yeung, and Yimin Xiong. "Super-resolution through neighbor embedding." CVPR 2004.
05/03/2023 44
Laplacianfaces
• Mapping face images in the image space via Locality Preserving Projections (LPP) to low-dimensional face subspace (manifold), called Laplacianfaces.
• LLP is analogous to Laplacian Eigenmaps except the objective function– Laplacian Eigenmaps:
– LLP:
He, Xiaofei, et al. "Face recognition using laplacianfaces." Pattern Analysis and Machine Intelligence, IEEE Transactions on 27.3 (2005): 328-340.
05/03/2023 45
Laplacianfaces
Learning Laplacianfaces for Representation
1. PCA projection (kept 98 percent information in the sense of reconstruction error)
2. Constructing the nearest-neighbor graph
3. Choosing the weights
4. Optimize
The k lowest eigenvectors of
are choosing to form
is the so-called Laplacianfaces.
05/03/2023 46
Laplacianfaces
Two-dimensional linear embedding of face images by Laplacianfaces.
05/03/2023 47
Reference and Resources
• 浅谈流形学习 . Pluskid. 2010-05-29. http://blog.pluskid.org/?p=533&cpage=1
• Wikipedia: MDS, Manifold, Laplacian Matrix
• PCA: M. Bishop, PRML
• Eigenvalue decomposition and SVD: 机器学习中的数学(5)-强大的矩阵奇异值分解(SVD)及其应用• General Eigenvalue Problem: wolfram, tutorial
• Video lecture: Geometric Methods and Manifold Learning. Mikhail Belkin, Partha Niyogi (author of Laplacian eigenmap)
• MANI fold Learning Matlab Demo: http://www.math.ucla.edu/~wittman/mani/index.html
05/03/2023 48
Thank you.