Architecture and Equilibra 结构和平衡

Architecture and EquilibraArchitecture and Equilibra结构和平衡结构和平衡

Chapter 6

2002.12.4

2

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and EquilibriaPerface

lyaoynov stable theorem

I

I

S

L

I : 整个系统集合

S : 稳定系统集合

L : 可由李亚普诺夫函数判定稳定的系统集合

2002.12.4

3

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria6.1 6.1 Neutral Network As Stochastic Gradient syNeutral Network As Stochastic Gradient systemstem

Classify Neutral network model By their synaptic connection topolgies and by how learning modifies their connection topologies

pathwaysfeedbackorloopssynapticclosediffeedback

loopssynapticclosedNoifdfeedforwar

..2

..1

samplingstrainingunlabelleduselearningervisedUn

samplingstraining

oformationmembershipclassuselearningSupervised

:sup.2

inf:.1

synaptic connection topolgies

how learning modifies their connection topologies

2002.12.4

4

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.1 6.1 Neutral Network As Stochastic Gradient sNeutral Network As Stochastic Gradient systemystem

Gradi edescent

LMSBackPropagati on

Rei nforcement Leari ng

Recurrent BackPropagati on

Vetor Quanti zati on

Sel f -Organi zati on MapsCompeti tve l earni ngCounter-propagati on

RABAMBroeni an anneal i ng

ABAMART-2

BAM-Cohen-Grossberg ModelHopfi el d ci rcui t

Brai n-state- I n_BoxAdapti ve-Resonance

ART-1ART-2

Feedforward Feedback

Decode

Supervi sed

Unsupervi sed

Encode

Neural NetWork Taxonomy

2002.12.4

5

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.2 6.2 Global Equilibra:convergence and stabilitGlobal Equilibra:convergence and stabilityy

Neural network :synapses , neurons

three dynamical systems:

synapses dynamical systems

neuons dynamical systems

joint synapses-neurons dynamical systems

Historically,Neural engineers study the first or second neural network.They usually study learning in feedforward neural networks and neural stability in nonadaptive feedback neural networks. RABAM and ART network depend on joint equilibration of the synaptic and neuronal dynamical systems.

'

M'

X

),(''

MX

2002.12.4

6

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.2 6.2 Global Equilibra:convergence and stabilitGlobal Equilibra:convergence and stabilityy

Equilibrium is steady state (for fixed-point attractors)

Convergence is synaptic equilibrium.

Stability is neuronal equilibrium.

More generally neural signals reach steady state even though the activations still change.

We denote steady state in the neuronal field

Stability - Equilibrium dilemma :Neuron fluctuate faster than synapses fluctuate.

Convergence undermines stability

6.10M

6.20X

xF

36.0Fx

2002.12.4

7

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgorithSynaptic convergence to centroids:AVQ Algorithmsms

We shall prove that:

Competitve AVQ synaptic vector converge to pattern-class centroid. They vibrate about the centroid in a Browmian motion

jm

Competitve learning adpatively qunatizes the input pattern space

charcaterizes the continuous distributions of pattern.

nR

)(xp

centroidXX

AVQ^

2002.12.4

8

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgoritSynaptic convergence to centroids:AVQ Algorithmshms

7.6,

6.6....321

jiifjDiDKDDDDnR

nRPattern

The Random Indicator function

Supervised learning algorithms depend explicitly on the indicator functions.Unsupervised learning algorthms don’t require this pattern-class information.

Centriod

KDDDDIIII ,......,,

321

860

1)(

j

j

D Dxif

DxifxI

j

96)(

)(^

jD dxxpjD dxxxp

jx

Comptetive AVQ Stochastic Differential Equations

2002.12.4

9


The Stochastic unsupervised competitive learning law:

106])[(

jjjjj nmxySm

We want to show that at equilibrium jjjj xmxm )E(or

116)( xISjDj

We assume

The equilibrium and convergence depend on approximation (6-11) ,so 6-10 reduces :

126])[(

jjDj nmxxIm j

2002.12.4

10


Competitive AVQ Algorithms

miixmi ,,......1,)()0( 1. Initialize synaptic vectors:

2.For random sample ,find the closet(“winning”)synaptic

vector

)(tx

)(tm j

221

2.......

136)()(min)()(

m

iij

xxxwhere

txtmtxtm

3.Update the wining synaptic vectors by the UCL ,SCL,or DCL learning algorithm.

)(tm j

2002.12.4

11


Unsupervised Competitive Learning (UCL)

156)()1(

146)]()([)()1(

jiiftmtm

tmtxctmtm

ii

jijj

}{ ic defines a slowly deceasing sequence of learning coefficient

)(samples10,000for000,10

11.0,instanceFor txt

ci

Supervised Competitive Learning (SCL)

176)]()([)(

)]()([)(

166)()())(()()1(

Djxiftmtxctm

Djxiftmtxctm

tmtxtxrctmtm

jij

jij

jjijj

2002.12.4

12


Differential Competitive Learning (DCL)

196)()1(

186)]()())[(()()1(

jiiftmtm

tmtxtySctmtm

ii

jjjtjj

))1(( tyS jj denotes the time change of the jth neuron’s competitive

signal . In practice we only use the sign of (6-20)206))(())1(())1(( tyStyStyS jjjjjj

Stochastic Equilibrium and Convergence

Competitive synaptic vector coverge to decsion-class centrols.

May coverge to locally mixima.

2002.12.4

13

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgorithmSynaptic convergence to centroids:AVQ Algorithmss

AVQ centroid theorem:

if a competitive AVQ system converges,it converge to the centroid of the sampled decision class.

2161)^

(Pr mequilibriuatxmob jj

Proof. Suppose the jth neuron in Fy wins the actitve competition.

Suppose the jth synaptic vector codes for decision class jm jD

2260

jm

Suppose the synaptic vector has reached equilibrium

2002.12.4

14


mean-zero is singalnoise

236

jj nmxImofbecause jDj

jj

j

D

Dj

Dj

D

Dj

jR

jD

j

mEx

concludestheoremcentroidAVQthe

xdxxp

dxxxpm

dxxpmdxxxp

dxxpmx

nEdxxpmxxI

mEo

nExpectatioTake

j

j

jj

j

n j

^

:

)(

)(

246)()(

)()(

)())((

:

^^

2002.12.4

15


Arguments:

• The sptial and temporal integrals are approximate equal.

•The AVQ centriod theorem assumes that convergence occurs.

•The AVQ centroid convergence theorem ensure :

exponential convergence

2002.12.4

16

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem6.4 AVQ Convergence Theorem

AVQ Convergence Theorem:

Competitive synaptic vectors converge exponentially quikly to pattern-class centroids.

Proof.Consider the random quadratic form L

2562

1

0 0

2

n

i

m

j

iji )m(xL

The pattern vectors x do not change in time.

(still valid if the pattern vector x change in time)

2002.12.4

17


296)(2))((

126)(

286)(

276

266

i jnmx

i jmxxI

nmxImofbecause

mmx

mm

L

mm

Lx

x

LL

ijijiijiD

ijijiDij

ij

i j

iji

ij

i j ij

ij

i j iji

i

i

j

j

The average E[L] as Lyapunov function for the sochastic competitice dynamical system.

Assume: Noise process is zero-mean and independence of the noise process with “signal”process ijm-x

2002.12.4

18


316)()(

306][2

jDj

iji dxxpmx

LELE

gives

So ,on average by the learning law 6-12, 0)(

LE

If any synaptic vector move along its trajetory.

So, the competitive AVQ system is asymtotically stabel,and in gereral converges exponentially quickly to a locally equilibrium.

Suppose 0)(

LE If 0

jm Then every synaptic vector has

Reached equilibrium and is constant .

2002.12.4

19


Since p(x) is a nonnegative weigth function.

The weighted integral of the learning difference

must equal zero :

iji mx

326)()( odxxpmxDj

ji

So equilibrium synaptic vector equal centroids.Q.E.D

2002.12.4

20

Chapter 6 Architecture and EquilibraChapter 6 Architecture and Equilibra 6.4 AVQ Convergence Theorem6.4 AVQ Convergence Theorem

Argument

• Total mean-squared error of vector quantization for the partition

• So the AVQ convergence theorem implies that

the class centroid, and asymptotically ,competitive synaptic vector-total mean-squared error.

kDD ,.....1

126])[(

jjDj nmxxIm jBy

The Synaptic vectors perform stochastic gradient desent on the mean-squared-error in pettern-plus-error space 1nR

In the sense :competitive learning reduces to stochostic gradient descent

2002.12.4

21

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.5 6.5 Global stability of feedback neural networGlobal stability of feedback neural networksks

Global stability is jointly neuronal-synaptics steady state.

Global stability theorems are powerful but limited.

Their power:

•their dimension independence

•nonlinear generality

•their exponentially fast convergence to fixed points.

Their limitation:

•do not tell us where the equilibria occur in the state space.

2002.12.4

22

Chapter 6 Architecture and EquilibraChapter 6 Architecture and Equilibra 6.5 6.5 Global stability of feedback neural networGlobal stability of feedback neural networksks

Stability-Convergence Dilemma

Stability-Convergence Dilemma arise from the asymmetry in neounal and synaptic fluctuation rates.

Neurons change faster than synapses change.

Neurons fluctuate at the millisecond level.

Synapses fluctuate at the second or even minute level.

The fast-changing neurons must balance the slow-changing synapses.

2002.12.4

23

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.5 6.5 Global stability of feedback neural networGlobal stability of feedback neural networksks

Stability-Convergence Dilemma

1.Asymmetry:Neurons in and fluctuate faster than the synapses in M.

2.stability: (pattern formation).

3.Learning:

4.Undoing:

the ABAM theorem offers a general solution to stability-convergence dilemma.

00

yx FandF

.000

MFandF yx

.000

yx FandFM

xF yF

2002.12.4

24

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.6 The ABAM Theorem6.6 The ABAM Theorem

The ABAM TheoremThe Hebbian ABAM and competitive ABAM models are globally stabel.

356

346)i()()(

336)()()(

1

1

jiijij

n

i

ijijjjjj

p

j

ijjjiiiii

SSmm

mxSybyay

mySxbxax

Hebbian ABAM model:

Competitive ABAM model , replacing 6-35 with 6-36

366

ijijij mSSm

2002.12.4

25


If the positivity assumptions 0000 '' jiji SSaa

Then, the models are asymptotically stable,

and the squared activation and synaptic velocities decrease exponentially quickly to their equilibrium values:

0,0,0222

ijji myx

Proof. the proof uses the bounded lyapunov function L

3762

1)()()()( 2

0'

0'

i jijjj

j

yjjjii

i

xiii

i j

ijji mdbSdbSmSSL ji

2002.12.4

26


386))((: dt

dx

dx

dFtxF

dt

dgivesationdifferentiofrulechainthe

i

ii

i j

jiij

i

ijij

j

jji j

ijjiii

i j

ijjiij

j i

ijijjji j

ijjiii

jijijjjjiii

ij

i j

ji

i

ijij

jjj

ijji

ii

SSm

mSbbSmSbaS

throughby

mSSm

mSbySmSbxS

mmybSxbS

mSSmSySmSxSL

iji

416)(

)()(

366346

406)(

)()(

396

2

2'2'

''

''

''

Make the difference to 6-37:

2002.12.4

27


,0000 '' jiji SSaaofbecause .,0, estrajectorisystemalongLSo

To prove global stability for the competitve learning law 6-36

i jijjiijij

i

ijij

j

jji j

ijjiii

mSSmSS

mSbbSmSbaSL

426))((

)()( 2'2'

.0

1)(

00

))(()(

2

estrajectorialongL

SjifmS

Sjif

mSSmSSmSSm

iji

ijjiijijijjiij

We prove the stronger asymptotic stable of the ABAM models

with the positivity assumptions. 0000 '' jiji SSaa

2002.12.4

28


4360

40622'2'

i jij

jj

j

j

ii

i

i myb

Sx

a

SL

Along trajectories for any nonzero change in any neuronal activation or any synapse.

Trajectories end in equilibrium points.

Indeed 6-43 implies:

4560

44600222

ijji

ijji

myxiff

myxiffL

The squared velocities decease exponentially quickly because of the strict negativity of (6-43) and ,to rule out pathologies .

Q.E.D

2002.12.4

29

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.7 structural stability of unsuppervised learning and RABA6.7 structural stability of unsuppervised learning and RABAMM

Is unsupervised learning structural stability?

Structural stability is insensivity to small perturbations

•Structural stability ignores many small perturbations.

•Such perturbations preserve qualitative properties.

Basins of attractions maintain their basic shape.

2002.12.4

30

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM

Random Adaptive Bidirectional Associative Memories RABAM

Browian diffusions perturb RABAM model.

The differential equations in 6-33 through 6-35 now become stochastic differential equations, with random processes as solutions.

ijij

yj

xi

msynapsetheinprocessmotionbrowianB

FinneuroniththeinprocessmotionbrowianB

FinneuroniththeinprocessmotionbrowianB

:

:

.:

The diffusion signal hebbian law RABAM model:

2002.12.4

31


486()(

476)i()()(

466)()()(

)

1

1

ijjiijij

n

i

ijijjjjj

p

j

ijjjiiiii

dByjSxiSmdm

dbjdtmxSybyady

dBidtmySxbxadx

With the stochastic competitives law:

496])()[( ijijiijjij dBdtmxSySdm

2002.12.4

32


486()(

476)i()()(

466)()()(

)

1

1

ijjiijij

n

i

ijijjjjj

p

j

ijjjiiiii

dByjSxiSmdm

dbjdtmxSybyady

dBidtmySxbxadx

With the stochastic competitives law:

496])()[( ijijiijjij dBdtmxSySdm

2002.12.4

33


With noise (independent zero-mean Gaussian white-noise process).

the signal hebbian noise RABAM model:

546)(,)(,)(

5360)()()(

526)()(

516)i()()(

506)()()(

222

1

1

ijijjjii

ijji

ijjjiiijij

jn

i

ijijjjjj

i

p

j

ijjjiiiii

nVnVnV

nEnEnE

nySxSmm

nmxSybyay

nmySxbxax

2002.12.4

34

Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM

RABAM Theorem.

The RABAM model (6-46)-(6-48) or (6-50)-(6-54),

is global stable.if signal functions are strictly increasing and ampligication functions and are strictly postive, the RABAM model is asympotically stable.

ia jb

Proof. The ABAM lyapunov function L in (6-37) now defines

a random process. At each time t,L(t) is a random variable.

The expected ABAM lyapunov function E(L) is a lyapunov function for the RABAM.

556),,(....)( dxdydMMyxpLLELRABAM

2002.12.4

35


586]}[{}[{

]}[{][

576]}[{}[{

]}[{}

][][{

566

)()(

'

'

'

'2

2'2'

j'

i'

i j

jiijij

j i

ijijjj

i j

jjiii

i j

jiijij

j i

ijijjj

i j

jjiii

i j

jiij

j i

ijijjji j

ijjiii

i j

jiijij

j i

ijijji j

ijjii

SSmnEmSbSnE

miSbSnELE

SSmnEmSbSnE

miSbSnESSm

mSbaSmSbaSE

SSmm

mSbySmSbxSE

LELE

ABAM

2002.12.4

36


Q.E.N

000)(0)(,

606)(

596]}{[)(}[{)(

]}[{)(][

model""

'

'

ABAMABAM

ABAM

ABAM

LorLasaccordingestrajectorialongLEorLESo

LE

noisemeanzero

SSmEnEmSbSEnE

miSbSEnELE

RABAMtheintermnoiseadditiveandsignaltheofindepence

i j

jiijij

j i

ijijjj

i j

jjiii

【 Reference 】[1] “Neural Networks and Fuzzy Systems -Chapter 6” P.221-26

1 Bart kosko University of Southern California.

Architecture and Equilibra 结构和平衡

Documents

Transcript of Architecture and Equilibra 结构和平衡