Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu ›...

52
物理フラクチュオマティクス論 Physical Fluctuomatics 応用確率過程論 A li d St h ti P Applied Stochastic Process 5 回グラフィカルモデルによる確率的情報処理 5th Probabilistic information processing by means of 5th Probabilistic information processing by means of graphical model 東北大学 大学院情報科学研究科 応用情報科学専攻 田中 和之(Kazuyuki Tanaka) k @ i ithk j kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ 1 物理フラクチュオマティクス論(東北大)

Transcript of Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu ›...

Page 1: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

物理フラクチュオマティクス論Physical Fluctuomatics

応用確率過程論A li d St h ti PApplied Stochastic Process

第5回グラフィカルモデルによる確率的情報処理5th Probabilistic information processing by means of5th Probabilistic information processing by means of

graphical model

東北大学 大学院情報科学研究科 応用情報科学専攻田中 和之(Kazuyuki Tanaka)k @ i i t h k [email protected]

http://www.smapip.is.tohoku.ac.jp/~kazu/

1物理フラクチュオマティクス論(東北大)

Page 2: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

今回の講義の講義ノート

田中和之著:確率モデルによる画像処理技術入門,

森北出版,2006.

2物理フラクチュオマティクス論(東北大)

Page 3: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

ContentsContents

11 IntroductionIntroduction1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model44 Statistical Performance AnalysisStatistical Performance Analysis4.4. Statistical Performance AnalysisStatistical Performance Analysis5.5. Concluding RemarksConcluding Remarks

物理フラクチュオマティクス論物理フラクチュオマティクス論((東北大東北大)) 33

Page 4: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

ContentsContents

11 IntroductionIntroduction1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model44 Statistical Performance AnalysisStatistical Performance Analysis4.4. Statistical Performance AnalysisStatistical Performance Analysis5.5. Concluding RemarksConcluding Remarks

物理フラクチュオマティクス論物理フラクチュオマティクス論((東北大東北大)) 44

Page 5: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Markov Random Fields for Image Processing

Markov Random Fields

S G d D G (1986) IEEE T ti PAMIS G d D G (1986) IEEE T ti PAMI

Markov Random Fields are One of Probabilistic Methods for Image processing.

S. Geman and D. Geman (1986): IEEE Transactions on PAMIS. Geman and D. Geman (1986): IEEE Transactions on PAMIImage Processing for Image Processing for Markov Random Fields (MRF)Markov Random Fields (MRF)(Simulated Annealing Line Fields)(Simulated Annealing Line Fields)(Simulated Annealing, Line Fields)(Simulated Annealing, Line Fields)

J Zhang (1992): IEEE Transactions on Signal ProcessingJ Zhang (1992): IEEE Transactions on Signal ProcessingJ. Zhang (1992): IEEE Transactions on Signal ProcessingJ. Zhang (1992): IEEE Transactions on Signal ProcessingImage Processing in EM algorithm for Image Processing in EM algorithm for Markov Markov Random Fields (MRF)Random Fields (MRF) (Mean Field Methods)(Mean Field Methods)Random Fields (MRF)Random Fields (MRF) (Mean Field Methods)(Mean Field Methods)

物理フラクチュオマティクス論(東北大) 5

Page 6: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Markov Random Fields for Image ProcessingIn Markov Random Fields, we have to consider not only the states with high probabilities but also ones with low probabilities.I M k R d Fi ld

Hyperparameter EstimationIn Markov Random Fields, we

have to estimate not only the image but also hyperparameters

Estimation

image but also hyperparameters in the probabilistic model.We have to perform the

Statistical Quantitiesp

calculations of statistical quantities repeatedly.

Estimation of Image

We can calculate statistical quantities by adopting the Gaussian graphical model as a prior probabilistic model

d b i G i i t l f l物理フラクチュオマティクス論(東北大) 6

and by using Gaussian integral formulas.

Page 7: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Purpose of My TalkR i f f l ti f b bili tiReview of formulation of probabilistic model for image processing by means of conventional statistical schemes.Review of probabilistic image processing by using Gaussian graphical model (Gaussian Markov Random Fields) as the(Gaussian Markov Random Fields) as the most basic example.

K Tanaka: StatisticalK Tanaka: Statistical Mechanical ApproachMechanical ApproachK. Tanaka: StatisticalK. Tanaka: Statistical--Mechanical Approach Mechanical Approach to Image Processing (Topical Review), J. Phys. to Image Processing (Topical Review), J. Phys. A: Math. Gen., vol.35, pp.R81A: Math. Gen., vol.35, pp.R81--R150, 2002.R150, 2002.A: Math. Gen., vol.35, pp.R81A: Math. Gen., vol.35, pp.R81 R150, 2002.R150, 2002.

Section 2 and Section 4 are summarized in the present talk.

物理フラクチュオマティクス論(東北大) 7

Page 8: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

ContentsContents

11 IntroductionIntroduction1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model44 Statistical Performance AnalysisStatistical Performance Analysis4.4. Statistical Performance AnalysisStatistical Performance Analysis5.5. Concluding RemarksConcluding Remarks

物理フラクチュオマティクス論物理フラクチュオマティクス論((東北大東北大)) 88

Page 9: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Image Representation in Computer Vision

Digital image is defined on the set of points arranged on a square lattice.qThe elements of such a digital array are called pixels.We have to treat more than 100,000 pixels even in the , pdigital cameras and the mobile phones.

xx )1,1( )1,2( )1,3(

)2,1( )2,2( )2,3(

y y )3,1( )3,2( )3,3(Pixels200307480640

物理フラクチュオマティクス論(東北大) 9

y y),( yx

Pixels200,307480640

Page 10: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Image Representation in Computer Vision

x yxfyx ,),(

0, yxf 255yxf

Pixels 65536256256 yAt each point the intensity of light is represented as an

0, yxf 255, yxf

At each point, the intensity of light is represented as an integer number or a real number in the digital image datadata.A monochrome digital image is then expressed as a two-dimensional light intensity function and the value isdimensional light intensity function and the value is proportional to the brightness of the image at the pixel.

物理フラクチュオマティクス論(東北大) 10

Page 11: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Noise Reduction by Conventional FiltersNoise Reduction by Conventional FiltersConventional FiltersConventional Filters

The function of a linear filter is to take the sumfilter is to take the sum of the product of the mask coefficients and the

Smoothing Filters

mask coefficients and the intensities of the pixels.

173120219202190202192

A

192 202 190

202 219 120

192 202 190

202 173 120

Smoothing Filters

173110218100120219202Average

100 218 110 100 218 110

It is expected that probabilistic algorithms for p p gimage processing can be constructed from such aspects in the conventional signal processing.

物理フラクチュオマティクス論(東北大) 11

Markov Random Fields Probabilistic Image ProcessingAlgorithm

Page 12: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Bayes Formula and Bayesian Network

}Pr{}|Pr{}|P { AABBAPrior

}Pr{}Pr{}|Pr{}|Pr{

BAABBA Probability

Posterior Probability

}{

Bayes Rule AData-Generating Process

Posterior Probability Bayes RuleEvent A is given as the observed data.Event B corresponds to the original

A

BEvent B corresponds to the original information to estimate. Thus the Bayes formula can be applied to the

BBayesianThus the Bayes formula can be applied to the

estimation of the original information from the given data.

Bayesian Network

物理フラクチュオマティクス論(東北大) 12

Page 13: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Image Restoration by Probabilistic Model

NoiseAssumption 1: The degraded image is randomly generated

Transmission

Noise g y gfrom the original image by according to the degradation

Original Image

Degraded Image

Transmission process. Assumption 2: The original image is randomly generatedImage Image

Posterior

image is randomly generated by according to the prior probability.

PriorLikelihood

}Image Degraded|Image OriginalPr{

}ageDegradedImPr{}Image OriginalPr{}Image Original|Image DegradedPr{

物理フラクチュオマティクス論(東北大) 13

Likelihood Marginal Bayes Formula

Page 14: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Image Restoration by Probabilistic ModelThe original images and degraded images areThe original images and degraded images are represented by f = {fi} and g = {gi}, respectively.

DegradedOriginal

ImageImage

),( iii yxr

Position Vectori

Position Vector of Pixel i i

fi: Light Intensity of Pixel iin Original Image

gi: Light Intensity of Pixel iin Degraded Image

物理フラクチュオマティクス論(東北大) 14

in Original Image in Degraded Image

Page 15: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Probabilistic Modeling of Image Restoration

Posterior

PriorLikelihood

}Image Degraded|Image OriginalPr{

}ImageOriginalPr{}ImageOriginal|ImageDegradedPr{

Assumption 1: A given degraded image is obtained from the original image by changing the state of each pixel to another state by the same probability, independently of the other pixels.

}ImageOriginal|ImageDegradedPr{ fggi gi

Vi

ii fg ),(}|Pr{ }gg|gg{

fFgGfg

Random Fieldsfi fi

or

物理フラクチュオマティクス論(東北大) 15

Random Fields

Page 16: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Probabilistic Modeling of Image RestorationImage Restoration

Posterior

}ImageDegraded|ImageOriginalPr{

PriorLikelihood

}ImageOriginalPr{}ImageOriginal|ImageDegradedPr{

}ImageDegraded|ImageOriginalPr{

}ImageOriginalPr{}ImageOriginal|ImageDegradedPr{

Assumption 2: The original image is generated according to a prior probability Prior Probability consists of a product of

}IO i i lP { f

prior probability. Prior Probability consists of a product of functions defined on the neighbouring pixels.

Eij

ji ff ),(}Pr{ }ImageOriginalPr{

fFf

i jEij

Random FieldsProduct over All the Nearest Neighbour Pairs of Pixels

物理フラクチュオマティクス論(東北大) 16

Product over All the Nearest Neighbour Pairs of Pixels

Page 17: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Prior Probability for Binary ImageIt i i t t h h ld th f tiIt is important how we should assume the function (fi,fj) in the prior probability. 1,0if

Eijji ff ),(}Pr{ fF

i jj

We assume that every nearest-neighbour pair of pixels take the same state of each other in the prior probability.

)0,1()1,0()0,0()1,1(

== >p p p

21 p

21i j Probability of

Neigbouring Pi l

物理フラクチュオマティクス論(東北大) 17

> Pixel

Page 18: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Prior Probability for Binary Image

== >p p p

21 p

21i j Probability of

Nearest Neigbour Pair of Pixels

Which state should the center ?

Pair of Pixels

pixel be taken when the states of neighbouring pixels are fixed g g pto the white states?

>

Prior probability prefers to the configuration with the least number of red lines

物理フラクチュオマティクス論(東北大) 18

with the least number of red lines.

Page 19: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Prior Probability for Binary ImagePrior Probability for Binary Imagey y gy y g

== >p p

Which state should the center

?-?== >

Which state should the center pixel be taken when the states of neighbouring pixels are fixed as g g pthis figure?

> >=

Prior probability prefers to the configuration with the least number of red lines

物理フラクチュオマティクス論(東北大) 19

with the least number of red lines.

Page 20: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

What happens for the case of large umber of pixels?large umber of pixels?

1.0

Co ariance Patterns with

0 4

0.6

0.8Covariance between the nearest neghbour

Patterns with both ordered statesand disordered states

p 0.0

0.2

0.4

0 0 0 2 0 4 0 6 0 8 1 0

nearest neghbour pairs of pixels are often generated

near the critical point. 0.0 0.2 0.4 0.6 0.8 1.0lnpsmall p large p

Sampling by Marko chain

Disordered State Critical Point(Large fluctuation)

Marko chain Monte Carlo

Ordered State

物理フラクチュオマティクス論(東北大) 20

(Large fluctuation)

Page 21: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Pattern near Critical Point f P i P b biliof Prior Probability

1.0Covariance We regard that patterns

0 4

0.6

0.8Covariance between the nearest neghbour

g pgenerated near the critical point are similar to the

0.0

0.2

0.4

0 0 0 2 0 4 0 6 0 8 1 0l

pairs of pixels local patterns in real world images.

0.0 0.2 0.4 0.6 0.8 1.0ln psmall p large p

similar

物理フラクチュオマティクス論(東北大) 21

similar

Page 22: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

ContentsContents

11 IntroductionIntroduction1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model44 Statistical Performance AnalysisStatistical Performance Analysis4.4. Statistical Performance AnalysisStatistical Performance Analysis5.5. Concluding RemarksConcluding Remarks

物理フラクチュオマティクス論物理フラクチュオマティクス論((東北大東北大)) 2222

Page 23: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Bayesian Image Analysis by Gaussian Graphical ModelGaussian Graphical Model

11

,if

Ejiji ff

Z },{

2

PR 21exp

)(1|Pr

fFPriorProbability Probability

V:Set of all the pixels

||21},{

2PR 2

1exp)( VEji

ji dzdzdzzzZ

0005.0 0030.00001.0

Patterns are generated by MCMC.the pixels

Markov Chain Monte Carlo Method

E:Set of all the nearest-neighbour pairs of pixels

物理フラクチュオマティクス論(東北大) 23

Markov Chain Monte Carlo Methodp p

Page 24: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Bayesian Image Analysis by Gaussian Graphical ModelGaussian Graphical Model

,, ii gfDegradation Process is assumed to be the additive white Gaussian noise.

Viii gf 2

22 21exp

2

1,Pr

fFgGVi 2

V: Set of all the pixels

Original Image f Gaussian Noise n Degraded Image g

2,0~ Nfg ii Degraded image is obtained by adding a ,Nfg ii

Hi f G i R d N b

y gwhite Gaussian noise to the original image.

物理フラクチュオマティクス論(東北大) 24

Histogram of Gaussian Random Numbers

Page 25: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Bayesian Image Analysis

f fF Pr fFgG Pr gfg

fF Pr fFgG Pr gOriginalImage

Degraded Image

Prior Probability Degradation Process

PrPrPr

fFfFgGgGfF

Image g

Posterior Probability

PrPr

gGgGfF

Ejiji

Viii ffgf

},{

),(),(

E:Set of all

V:Set of All the pixels

Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability

E:Set of all the nearest neighbour

i f i l

物理フラクチュオマティクス論(東北大) 25

posterior probability.pairs of pixels

Page 26: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Estimation of Original Image

We have some choices to estimate the restored image from posterior probability. p p y

In each choice, the computational time is generally exponential order of the number of pixels.

}|Pr{maxargˆ gGzFfz

Maximum A Posteriori (MAP) estimation

(1)

}|Pr{maxargˆ gG iiz

i zFfi

Maximum posterior marginal (MPM) estimation

(2)

if

ii fF\

}|Pr{}|Pr{f

gGfFgG

2}|Pr{minargˆ zgGzF dzf iiii

Thresholded Posterior Mean (TPM) estimation

(3)

物理フラクチュオマティクス論(東北大) 26

i Mean (TPM) estimation

Page 27: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Statistical Estimation of HyperparametersHyperparameters are determined so as to maximize the marginal likelihood Pr{G=g|,} with respect to

},|Pr{max arg)ˆ,ˆ( gG

respect to ,

zzFzFgGgG d}|Pr{},|Pr{},|Pr{

,

gg }|{},|{},|{

Original Image Degraded ImageVy

f g

Marginalized with respect to F

}|Pr{ fF },|Pr{ fFgG gy

x Marginalized with respect to F},|Pr{ gG

物理フラクチュオマティクス論(東北大) 27

Marginal Likelihood

Page 28: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Bayesian Image Analysis ,, ji gf

A Posteriori Probability

},|Pr{

}|Pr{},|Pr{},,|Pr{gG

fFfFgGgGfF

}{

222

POS 21

21exp

,,1g

ffgfZ Eji

jiVi

ii

TT2

},{POS

21))((

21exp1

,,

fCfgfgfg

g

Z

EjiVi

POS 22,,g Z

||21

TT2POS

21))((

21exp,, VdzdzdzZ zCzgzgzg

Gaussian Graphical Model

22

物理フラクチュオマティクス論(東北大) 28

Page 29: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Average of Posterior Probability

gGzFzF ||21},,|Pr{,

Vdzdzdz

CI

IzCICI

Izz ||21

T

22

22 2

1exp

Vdzdzdzgg

I

CIIzCI

CIIz ||21

T

22

22 2

1exp

Vdzdzdzgg

gCI

I2

)0,00())()((21exp)(

||

||21T2

V

V ,,dzdzdz

mzCI mz mz

Gaussian Integral formula

物理フラクチュオマティクス論(東北大) 29

Page 30: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Bayesian Image Analysis by Gaussian Graphical Model by Gauss a G ap ca ode ,, ii gf

fff 22 11}|{ Gf

Posterior Probability

Ejiji

Viii ffgf

},{

222 2

12

1exp},,|Pr{

gGfF

V S t f llAverage of the posterior probability can be calculated by using the multi- E:Set of all the

V:Set of all the pixels

zgGzFzf },,|Pr{ˆ d

dimensional Gauss integral Formula

|V|x|V| matrix

nearest-neghbour pairs of pixels

gCI

I

gf

2

},,|{

},{1

4Eji

jiji C

|V|x|V| matrix

CI otherwise0

Multi Dimensional Gaussian Integral Formula

物理フラクチュオマティクス論(東北大) 30

Multi-Dimensional Gaussian Integral Formula

Page 31: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Statistical Estimation of Hyperparameters

)()2(

),,(}|Pr{},|Pr{},|Pr{ 2/||2

POS

Z

Zd V

gzzFzFgGgG )()2( PR

2/||2 ZV

POS ,,Z g

||21

TT2

21))((

21exp Vdzdzdz zCzgzgz

O i i l I D d d I

||21

TPR

21exp VdzdzdzZ zCz

f g}|Pr{ fF },|Pr{ fFgG gOriginal Image Degraded ImageVy

Marginalized with respect to Fx

},|Pr{ gG

物理フラクチュオマティクス論(東北大) 31

Marginal Likelihood},|Pr{ gG

Page 32: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

C l l ti f P titi F tiCalculations of Partition Function

Czz )2(1exp)(||

||21V

VPR dzdzdzZ

T

POS ),,(g Z

CCzz

det2exp)( ||||21 VVPR dzdzdzZ

T

||21T

2

T

22

22

1exp

21

21exp

gCg

gCI

CggCI

IzCIgCI

Iz

Vdzdzdz

||21

T

22

22

2

2

1exp

2exp

gCI

IzCIgCI

Iz

gCI

g

Vdzdzdz

T22

||2

21exp

)det()2(

2

gCI

CgCI

CICI

V

AmzA mz

det)2()( )(

21exp

||||21

TV

Vdzdzdz

物理フラクチュオマティクス論(東北大) 32

(A is a real symmetric and positive definite matrix.)Gaussian Integral formula

Page 33: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Exact expression of Marginal Likelihood in G i G iGaussian Graphical Model

)()2(

),,(}|Pr{},|Pr{},|Pr{

PR2/||2

POS

Z

Zd V

gzzFzFgGgG

Multi-dimensional Gauss integral formula

T

22 21exp

det2

det},|Pr{ gCI

CgCI

CgG

V det2 CICI

y }|P {ˆˆ GWe can construct an exact EM algorithm.

If̂

Vy

},|Pr{max arg,

,

gG

物理フラクチュオマティクス論(東北大) 33

gCI

f 2 x

Page 34: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Bayesian Image Analysis by Gaussian Graphical Modelby Gaussian Graphical Model

1

Iteration Procedure in Gaussian Graphical Modelg

1T

22

2

11||1

111Tr

||1

g

CICg

CIC

ttVttt

Vt

2422

g

T22

242

2

2

11

11||

111

1Tr||

1 gCI

CgCI

I

tt

ttVtt

tV

t

I 2ˆ f̂gCI

Im 2)()()(

ttt

2)(minarg)(ˆ tmztf ii

zi

i f

0.0006

0.0008

0.001 t

0

0.0002

0.0004

tg ˆ物理フラクチュオマティクス論(東北大) 34

0 20 40 60 80 100 tg f

Page 35: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Image Restoration by Markov Random Field Model and Conventional FiltersField Model and Conventional Filters

MSEStatistical Method 315

Lowpass Filter

(3x3) 388(5x5) 413

Original ImageOriginal Image Degraded ImageDegraded Image

Filter (5x5) 413

Median Filter

(3x3) 486

(5x5) 445

2ˆ||

1MSE

Vi

ii ffV

( )

RestoredRestoredImageImage

V:Set of all the pixels

(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) MedianMRFMRF

物理フラクチュオマティクス論(東北大) 35

(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) MedianMRFMRF

Page 36: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

ContentsContents

11 IntroductionIntroduction1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model44 Statistical Performance AnalysisStatistical Performance Analysis4.4. Statistical Performance AnalysisStatistical Performance Analysis5.5. Concluding RemarksConcluding Remarks

物理フラクチュオマティクス論物理フラクチュオマティクス論((東北大東北大)) 3636

Page 37: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Performance Analysis

Sample Average of Mean Square Error

1y

1x̂

lity

e e

2y

2x̂

ˆroba

bil

e W

hit

an N

ois

x 3y

4y3x

4x̂erio

r Pr

Add

itiv

Gau

ssia

4

5y

4

5x̂Post

e

SignalA G

Estimated ResultsObserved

Data 5

1

2||ˆ||51][MSE nxx

物理フラクチュオマティクス論(東北大) 37

15 n

Page 38: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Statistical Performance AnalysisStatistical Performance Analysis

ydxXyYxyhV

x },|Pr{),,(1),|(MSE2

V

Original Image Degraded Image

gy

Additive White Gaussian Noise

},|Pr{ xXyY x

),,( yh },,|Pr{ yYxX

Restored Image

},|Pr{ xXyY

g

Posterior ProbabilityAdditive White Gaussian Noise

物理フラクチュオマティクス論(東北大) 38

Page 39: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Statistical Performance Analysis

ydxXyYxyhV

x Pr,,1),|MSE(2

ydxXyYxyV

V

Pr1 212 CI

V

otherwise,0

},{,1,4

EjiVji

ji C

xdyxPxyh

12

)|(,,

CI

1exp,Pr 2yxxXyY ii

,

y2 CI

)2

1exp(

2exp,Pr

22

2

yx

yxxXyYVi

ii

物理フラクチュオマティクス論(東北大) 39

2

Page 40: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Statistical Performance Estimation for Gaussian Markov Random Fields

21

22

||2

2

2

21exp

211

},|Pr{),,(1),|(MSE

ydyxxyV

ydxXyYxyhV

x

V

CII

otherwise,0

},{,1,4

EjiVji

ji C

2||22

22

||2

22

111

21exp

21)(1

22

ydxyxxxyV

V

V

V

CI

CII

CII

CI

22

||

2

2

2

T

2

2

2

22

||

2

2

2

21exp

21)()(1

21exp

21)(1

ydxyxxyxxyV

ydxyxxyV

V

CIC

CII

CIC

CII

CIC

CII

||2

22

||

22T

111

21exp

21)(

)()(1

22

ydxyxyxyV

V

V

V

C

CII

CICICICI

22

||

22

2T

22

||

22

2T

21exp

21)(

)(1

21exp

21

)()(1

ydxyxyxV

ydxyxxyV

V

CIC

CIC

= 0

2422

22

||

22

242T

11

21exp

21

)(1

22)(

ydxyxxV

VV

CI

CIC

CI

物理フラクチュオマティクス論(東北大) 40

T2222 )(||

1)(

Tr1 xI

xVV

CC

CII

Page 41: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Statistical Performance Estimation for Gaussian Markov Random Fields

ydxXyYxyhV

x 2 },|Pr{),,(1),|(MSE

ydyxxyV

VV

22

||

2

2

2 21exp

211

CII

xI

xVV

V

22

242T

22

2

)(||1

)(Tr1

22

CC

CII

CI

},{,1,4

EjiVji

ji C otherwise,0

600=40

600=40

),|(MSE x ),|(MSE x

200

400=40

200

400=40

0

200

0 0 001 0 002 0 0030

200

0 0 001 0 002 0 003

物理フラクチュオマティクス論(東北大) 41

0 0.001 0.002 0.0030 0.001 0.002 0.003

Page 42: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

ContentsContents

11 IntroductionIntroduction1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model44 Statistical Performance AnalysisStatistical Performance Analysis4.4. Statistical Performance AnalysisStatistical Performance Analysis5.5. Concluding RemarksConcluding Remarks

物理フラクチュオマティクス論物理フラクチュオマティクス論((東北大東北大)) 4242

Page 43: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Summary

Formulation of probabilistic model for image processing by means ofimage processing by means of conventional statistical schemes has been summarized.Probabilistic image processing by usingProbabilistic image processing by using Gaussian graphical model has been shown as the most basic exampleas the most basic example.

物理フラクチュオマティクス論(東北大) 43

Page 44: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

ReferencesReferences

K. Tanaka: Introduction of Image Processing by P b bili ti M d l M ikit P bli hi CProbabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) .

K T k St ti ti l M h i l A h tK. Tanaka: Statistical-Mechanical Approach to Image Processing (Topical Review), J. Phys. A, 35 (2002)35 (2002).

A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing Proceedings ofSignal and Image Processing, Proceedings of IEEE, 90 (2002).

物理フラクチュオマティクス論(東北大) 44

Page 45: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Problem 5-1: Derive the expression of the posterior probability Pr{F=f|G=g,,} by using p p y { f| g } y gBayes formulas Pr{F=f|G=g,,}=Pr{G=g|F=f,}Pr{F=f,}/Pr{G=g|,}. Here P {G |F f } d P {F f } d t bPr{G=g|F=f,} and Pr{F=f,} are assumed to be as follows:

gf 21exp1Pr fFgG

Vi

ii gf22 2exp

2,Pr

fFgG

ji ff

Z2

21exp

)(1|Pr

fF

Eji

jZ },{PR 2)(

||212

PR 21exp)( Vji dzdzdzzzZ

},{2 Eji

[Answer]

Ejiji

Viii ffgf

Z },{

222

POS 21

21exp

,,1},,|Pr{

ggGfF

22 11 ddd

物理フラクチュオマティクス論(東北大) 45

||21

},{

222PR 2

12

1exp VEji

jiVi

ii dzdzdzzzgzZ

Page 46: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Problem 5-2: Show the following equality.

},{

222 2

12

1

Eji

jiVi

ii ffgf

TT2

},{

21))((

21 fCfgfgf

j

22

4 ji

otherwise0},{1 Eji

jji C

物理フラクチュオマティクス論(東北大) 46

otherwise0

Page 47: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Problem 5-3: Show the following equality.

TT2

21))((

21 zCzgzgz

T

22

22 1 gIzCIgIz

T

222

12

gCg

CICI

22g

CIg

物理フラクチュオマティクス論(東北大) 47

Page 48: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Problem 5-4: Show the following equalities by using theProblem 5 4: Show the following equalities by using the multi-dimensional Gaussian integral formulas.

||2

||21},{

222POS

1)2(

21

21exp,,

C

g

V

VEji

jiVi

ii dzdzdzffgfZ

T

22

||2

21exp

)det()2( g

CICg

CI

V

Cdet

)2(21exp)( ||

||

||21},{

2V

V

VEji

jiPR dzdzdzffZ

物理フラクチュオマティクス論(東北大) 48

Page 49: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

P bl 5 5 D i h di i f hProblem 5-5: Derive the extremum conditions for the following marginal likelihood Pr{G=g} with respect to the hyperparameters and the hyperparameters and .

T

221expdet},|Pr{ gCgCgG

V

22 2

pdet2

},|{ gCI

gCI

g V

T2 111 CC

[Answer]

T

22 ||1Tr

||11 g

CICg

CIC

VV

2422 11 CI

T22

242

2

22

||1Tr

||1 g

CI

CgCI

I

VV

物理フラクチュオマティクス論(東北大) 49

Page 50: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

P bl 5 6 D i h di i f hProblem 5-6: Derive the extremum conditions for the following marginal likelihood Pr{G=g} with respect to the hyperparameters and the hyperparameters and .

T

221expdet},|Pr{ gCgCgG

V

22 2

pdet2

},|{ gCI

gCI

g V

T2 111 CC

[Answer]

T

22 ||1Tr

||11 g

CICg

CIC

VV

2422 11 CI

T22

242

2

22

||1Tr

||1 g

CI

CgCI

I

VV

物理フラクチュオマティクス論(東北大) 50

Page 51: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Problem 5-7: Make a program that generate a degraded image by the additive white Gaussian noise. Generate some degraded images from a given standard images by settingdegraded images from a given standard images by setting =10,20,30,40 numerically. Calculate the mean square error (MSE) between the original image and the degraded image.( ) g g g g

21MSE gf

Original Image Gaussian Noise Degraded Image

||

MSE

Vi

ii gfV

Original Image Degraded Image

K.Tanaka: Introduction of Image Processing by Probabilistic Models

Hi t f G i R dSample Program:

Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 .

物理フラクチュオマティクス論(東北大) 51

Histogram of Gaussian Random Numbers Fi Gi~N(0,402)

http://www.morikita.co.jp/soft/84661/

Page 52: Tanaka-Ohzeki Research Group - AlidSthtiPApplied ... › ~kazu › PhysicalFluctuo...今回の講義の講義ノート 田中和之著: 確率モデルによる画像処理技術入門,森北出版,2006.

Problem 5-8: Make a program of the following procedure in Problem 5 8: Make a program of the following procedure in probabilistic image processing by using the Gaussian graphical model and the additive white Gaussian noise.

Algorithm: Repeat the following procedure until convergence

1T

22

2

11||1

111Tr

||1

g

CICg

CIC

ttVttt

Vt

T22

242

2

2

11

11||

111

1Tr||

1 gCI

CgCI

I

tt

ttVtt

tV

t

gCI

Im 2)()()(

ttt

2)(minarg)(ˆ tmztf ii

zi

i

K.Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 .

物理フラクチュオマティクス論(東北大) 52

Sample Program: http://www.morikita.co.jp/soft/84661/g