PRML 10.1 ~ 10.3 7/31

Yuki Soma

10.1 Variational Inference . . . . . . . . . . . . . . . . . . . . . . 462 ◦ 10.1.1 Factorized distributions . . . . . . . . . . . . . . . . . . . . 464 ◦ 10.1.2 Properties of factorized approximations . . . . . . . . . . . 466 ◦ 10.1.3 Example: The univariate Gaussian . . . . . . . . . . . . . . 470 ◦ 10.1.4 Model comparison . . . . . . . . . . . . . . . . . . . . . . 473

10.2 Illustration: Variational Mixture of Gaussians . . . 474 ◦ 10.2.1 Variational distribution . . . . . . . . . . . . . . . . . . . . 475 ◦ 10.2.2 Variational lower bound . . . . . . . . . . . . . . . . . . . 481 ◦ 10.2.3 Predictive density . . . . . . . . . . . . . . . . . . . . . . . 482 ◦ 10.2.4 Determining the number of components . . . . . . . . . . . 483 ◦ 10.2.5 Induced factorizations . . . . . . . . . . . . . . . . . . . . 485

10.3 Variational Linear Regression . . . . . . . . .. . . . . . 486 ◦ 10.3.1 Variational distribution . . . . . . . . . . . . . . . . . . . . 486 ◦ 10.3.2 Predictive distribution . . . . . . . . . . . . . . . . . . . . 488 ◦ 10.3.3 Lower bound . . . . . . . . . . . . . . . . . . . . . . . . . 489

𝐗 𝐙 𝑝(𝐙|𝐗) 𝐙

◦

◦ 11 MCMC

◦

◦ EP 10.7

𝐙

1.

◦ 𝑞 𝐙 = 𝑞(𝐙|𝛚) 𝛚

2. 𝐙

◦

◦ 𝑞𝑖 𝐙𝑖 = 𝑞𝑖

◦

◦ 𝐙

𝐗

𝑝(𝐗, 𝐙)

𝑝(𝐙|𝐗) 𝑝(𝐗)

◦

ℒ(𝑞) 𝐾𝐿(𝑞| 𝑝 = 0 𝑞 𝐙 = 𝑝(𝐙|𝐗)

10.5 ℒ(𝑞)

𝒒𝒊

10.5 ℒ(𝑞)

KL

𝑞𝑗 = 𝑝 𝐗, 𝐙𝐣

ℒ(𝑞) 𝑞𝑗∗ 𝑗 = 1,… ,𝑀

𝑞𝑖 𝑖 ≠ 𝑗

1. 𝑞𝑗

2.

foreach 𝑞𝑖 :

𝑞𝑖 𝑞𝑗 𝑞𝑖

𝑞𝑖

◦ ℒ(𝑞) 𝑞𝑖

𝐙

KL

KL 𝑝(𝐙) 0 𝑞(𝐙)0

KL 𝑞(𝐙) 𝑝(𝐙)

KL 1

1

𝑝(𝑚) 𝑚 ◦ 𝑝(𝑚|𝐗)

𝑞 𝐙,𝑚 = 𝑞 𝐙 𝑞(𝑚) ◦ 𝐙

⇒ 𝑞 𝐙,𝑚 = 𝑞 𝐙|𝑚 𝑞(𝑚)

10.10

ln 𝑝 𝐗 = ℒ − 𝑞 𝑍 𝑚 𝑞 𝑚 ln𝑝 𝑍,𝑚 𝑋

𝑞 𝑍 𝑚 𝑞 𝑚𝐙𝑚

◦ ℒ = 𝑞 𝑍 𝑚 𝑞 𝑚 ln𝑝(𝑍,𝑋,𝑚)

𝑞 𝑍 𝑚 𝑞 𝑚𝐙𝑚

ℒ 𝑞(𝑚)

◦ 𝑞(𝑚) ∝ 𝑝(𝑚)𝑒ℒ𝑚 (10.36)

ℒ𝑚 = 𝑞 𝑍 𝑚 𝑞 𝑚 ln𝑝(𝑍,𝑋|𝑚)

𝑞 𝑍 𝑚𝐙

ℒ𝑚 𝑞(𝑍|𝑚) (10.36)𝑞(𝑚)

◦ 𝐾

𝐾

𝛑 1

◦ 𝐳𝑖 1-of-𝐾 𝐾

𝐙

𝜋

𝜇, Λ

◦ 𝐦0 = 𝟎

◦ 𝑝 𝑞

◦ PRML

1. E 𝑧𝑛𝑘 = 𝑟𝑛𝑘

2. 𝑞∗(𝜋, 𝜇, Λ)

3. 𝑞∗ 𝐙 ◦ 𝑟𝑛𝑘

4. 2.

0 ◦

𝛼0 < 1 ◦ 𝛼0 = 10−3

◦

◦

◦

◦

t

10.81 ◦ 𝑁

𝑞 𝜋 𝑞(𝜇, Λ)

𝐾 𝐾!

K

ln 𝐾!10.22

𝜋

𝜋 𝑞 …

0

◦ RVM 7.22

3.3

𝛼, 𝛽

◦ 𝛽

10.6

…

10.9 𝛼

𝛼

𝑞