1. PRML#14 (ver. 2.1) 2013/10/05 Mixtures of Gaussians 9.2, 9.2.1, 9.2.2 @takuya_fukagai

2. 9.2 (Mixtures of Gaussians) 2.3.9(1) 2 2.21 Old Faithful272 (:)(:) 3. 9.2 (Mixtures of Gaussians) 2.3.9(2) 31 2.223()3 () p(x) = k N(x | k,k 2 ) k=1 3 4. 9.2 (Mixtures of Gaussians) 2.3.9(3) 23 2.23 (a) 3 (b) p(x) (c) p(x) p(x) = k N(x | k,k ) k=1 3 5. 9.2 (Mixtures of Gaussians) 2.3.9 9.2(latent variable) p(x) = k N(x | k,k ) k=1 K 6. z K p(z) p(x|z)p(x,z) 9.2 (Mixtures of Gaussians) K2 z z10 1-of-K z = z1 z2 zK ! " # # # # # $ % & & & & & z = 0 1 0 0 ! " # # # # # # $ % & & & & & & z zk {0,1} zk k=1 K =1 9.4 p(x,z)=p(z)p(x|z) 7. 9.2 (Mixtures of Gaussians) p(zk =1) = k z k k z1-of-K 9.4 p(x,z)=p(z)p(x|z) 0 k 1 k k=1 K =1 p(z) = k zk k=1 K z = z1 zk1 zk zk+1 zK " # $ $ $ $ $ $ $ $ $ % & ' ' ' ' ' ' ' ' ' z = 0 0 1 0 0 ! " # # # # # # # # $ % & & & & & & & & 1-of-K 8. p(x | z) = N(x | k,k )zk k=1 K 9.2 (Mixtures of Gaussians) p(x | zk =1) = N(x | k,k ) 9.4 p(x,z)=p(z)p(x|z) p(x) = p(z)p(x | z) = z k zk k=1 K N(x | k,k )zk k=1 K $ % & ' ( )k=1 K = k N(x | k,k ) k=1 K z = z1 zk1 zk zk+1 zK " # $ $ $ $ $ $ $ $ $ % & ' ' ' ' ' ' ' ' ' z = 0 0 1 0 0 ! " # # # # # # # # $ % & & & & & & & & 1-of-K zx x z 9. xz () kzk=1 (zk)xzk=1 (zk)kx (responsibility) 9.2 (Mixtures of Gaussians) (zk ) p(zk =1| x) = p(zk =1)p(x | zk =1) p(zj =1)p(x | zj =1) j=1 K = k N(x | k,k ) j N(x | j,j ) j=1 K 9.4 p(x,z)=p(z)p(x|z) z = z1 zk1 zk zk+1 zK " # $ $ $ $ $ $ $ $ $ % & ' ' ' ' ' ' ' ' ' z = 0 0 1 0 0 ! " # # # # # # # # $ % & & & & & & & & 1-of-K (9.13) 10. 9.2 (Mixtures of Gaussians) 8.1.2(ancestral sampling) 1. z p(z) 2. x z p(x | z) 11. 9.2 (Mixtures of Gaussians) 2.3.9 9.53 500 .9.5 : (a), (b), (c) x (a) 3(z) (b) (c) p(x) = k N(x | k,k ) k=1 3 xn (znk ) (znk ) 12. (9.5(c)) 9.5(c) 500 9.2 (Mixtures of Gaussians) p(x) = k N(x | k,k ) k=1 3 (k=1) (k=2)(k=3) xn (znk ) (znk ) i j () xi (zi1 ) = 1,(zi2 ) = 0,(zi3 ) = 0 xj (zj1 ) = 0, (zj 2 ) = 0.5, (zj 3 ) = 0.5 13. 9.2.1 ND N x DX x1,, xN{ } X = x1 T xn T xN T ! " # # # # # # # $ % & & & & & & & xn T =[xn1, xn2 ,, xnD ] xN T =[xN1, x N 2 ,, xND ] x1 T =[x11, x12 ,, x1D ] 14. 9.2.1 X NK z N x K Z Z = z1 T zn T zN T ! " # # # # # # # $ % & & & & & & & zn T =[zn1, zn2 ,, znK ] zN T =[zN1, z N 2 ,, zNK ] z1 T =[z11, z12 ,, z1K ] 15. 9.2.1 9.2 9.6 p(x) = k N(x | k,k ) k=1 K 9.6 xn zn 16. 9.2.1 Nxn(n=1,...,N) NX p(xn | ,,) = k N(xn | k,k ) k=1 K ln p(X | ,,) = ln { k N(xn | k,k ) n=1 K } n=1 N = ln{ k N(xn | k,k ) k=1 K } n=1 N N (9.14) 17. 9.2.1 jj 1xn N(xn | j, j 2 I) = 1 (2)D/2 j 2 I 1/2 exp{ 1 2 (xn j )T j 2 I(xn j )} = 1 (2)D/2 (D j 2 )1/2 = 1 (2)D/2 D1/2 j k =k 2 I j = xn 18. 9.2.1 jj 1xn N(xn | j, j 2 I) = 1 (2)D/2 j 2 I 1/2 exp{ 1 2 (xn j )T j 2 I(xn j )} = 1 (2)D/2 (D j 2 )1/2 = 1 (2)D/2 D1/2 j k =k 2 I j = xn (j0) 19. 9.2.1 9.7: p(xn) (j0) 9.7 xn 20. 9.2.1 1 1. 1() 2. 0 ( ) 21. 9.2.1 () (10.1)() 1 K K! K!-1((identifiability)) 12 22. 9.2.2 EM EM(Expectation-Maximization Algorithm) EM 10.1EM EM 23. 9.2.2 EM (9.14) 1. k0 ( 9.17) 2. k0 ( 9.19) 3. kk(k)=1(9.9) ( 9.22) (9.17), (9.19), (9.22) ( 9.13) ( 9.13) (E step)(9.17), (9.19), (9.22)(M step) EM (znk ) 24. 9.2.2 EM ( 9.17) ( 9.19) ( 9.22) (znk ) = k N(xn | k,k ) j N(xn | j,j ) j=1 K k = 1 Nk (znk )xn n=1 N k = Nk N k = 1 Nk (znk )(xn k )(xn k )T n=1 N Nk = (znk ) n=1 N kxn (responsibility) N = Nk k=1 K " # $ % & ' # ( 9.17), ( 9.19), ( 9.22) 29.3 p.77-78 ( 9.18) (9.13) 25. 9.2.2 EM /* EM PRML() p.154-p.155 */ ln p(X|, , ) 1. kkk 2. E step: (znk), (n=1,...,N, k=1,...,K) 3. M step: (znk)kk k(k=1,...,K) 4. ln p(X|, , ) 2. 26. 9.2.2 EM 2.3.9Old FaithfulEM 2 2.21 Old Faithful272 ()() 27. 9.2.2 EM .9.8 : 2.3.9Old Faithful2 EM (a) 21 (b) E step (c) M stepkk 28. 9.2.2 EM .9.8 : () EM kk (d) 2EM (e) 5EM (f) 20EM 29. EM ( p.154 ) K-means() K-means EM K-means EM 9.2.2 EM