NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

NEURAL NETWORK THEORY

NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS

欢迎大家提出意见建议！ 2003.10.15

2


NEURONS AS FUNCTIONS

Neurons behave as functions.

Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).

3

NEURONS AS FUNCTIONS

The transduction description: a sigmoidal or S-shaped curve

the logistic signal function:

cxexS

1

1)(

)()(' 001 cScSdx

dSS

The logistic signal function is sigmoidal and strictly increases for positive scaling constant c >0.


4

NEURONS AS FUNCTIONSS(x)

x

0-∞ - + +∞

Fig.1 s(x) is a bounded monotone-nondecreasing function of x

If c→+∞ ， we get threshold signal function (dash line),Which is piecewise differentiable


5

SIGNAL MONOTONICITY

In general, signal functions are monotone nondecreasing S’>=0. This means signal functions have an upper bound or saturation value.The staircase signal function is a piecewise-differentiableMonotone-nondecreasing signal function.


6

SIGNAL MONOTONICITYAn important exception: bell-shaped signal function or Gaussian signal functions

02

cexS cx)(

xScxeS cx ',2'2

The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise.


7

SIGNAL MONOTONICITYGeneralized Gaussian signal function define potential or radial basis function :

])(2

1exp[)( 2

2 n

j

ijj

ii xxS

n

n Rxxx ),,( 1

),,( in

ii 1

input activation vector:

variance:

mean vector:


2i

)(xSi

we shall consider only scalar-input signal functions:

)( ii xS

8

SIGNAL MONOTONICITYneurons are nonlinear but not too much so ---- a property as semilinearity

Linear signal functions - make computation and analysis comparatively easy - do not suppress noise - linear network are not robustNonlinear signal functions - increases a network’s computational richness - increases a network’s facilitates noise suppression - risks computational and analytical intractability - favors dynamical instability


9

SIGNAL MONOTONICITYSignal and activation velocities

the signal velocity: =dS/dt

Signal velocities depend explicitly on action velocities

S

xSdt

dx

dx

dSS '


10

BIOLOGICAL ACTIVATIONS AND SIGNALS

Fig.2 Neuron anatomy

神经元 (Neuron) 是由细胞核 (cell nucleus) ，细胞体 (soma) ，轴突 (axon) ，树突 (dendrites) 和突触 (synapse) 所构成的


11

X= （ x1 ， x2 ，…， xn ）W= （ w1 ， w2 ，…， wn ）net=∑xiwinet=XW

x2 w2 ∑ f o=f （ net

）

xn wn

…

net=XW

x1 w1



12


Competitive Neuronal Signal

11

2

1

1

cxcx exS

exS )()(

logical signal function ( Binary Bipolar )

The neuron “wins” at time t if , “loses” ifand otherwise possesses a fuzzy win-loss status between 0 an 1.

a. Binary signal functions : [0,1]

b. Bipolar signal functions : [-1,1]

McCulloch—Pitts (M—P) neurons


1))(( txS 1))(( txS

13

NEURON FIELDS

Neurons within a field are topologically ordered, often by

proximity.

zeroth-order topology : lack of topological structure

Denotation: , ,

neural system samples the function m times to generate the associated pairs , ... ,

The overall neural network behaves as an adaptive filter and sample data changed network parameters.

XF YF ZF },{ YX FF },,{ ZYX FFF

),( mm yx),( 11 yx


pn RRf :YX FF

f

14

NEURONAL DYNAMICAL SYSTEMS

Description:a system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentials

Activation differential equations: niFFgx YXii ,,),,( 21

piFFhy YXii ,,),,( 21

in vector notation:

niFF YXi ,,),,( 21gx

piFF YXi ,,),,( 21hy

(1)

(2)

(3)

(4)


15

NEURONAL DYNAMICAL SYSTEMSNeuronal State spaces

nn Rtxtxt ))(,),(()( 1X

pp Rtytyt ))(,),(()( 1Y

So the state space of the entire neuronal dynamical system is: pn RR

Augmentation:

ZYX FFF ]|[pn

z RF


},{ YX FF

16

NEURONAL DYNAMICAL SYSTEMSSignal state spaces as hyper-cubes

The signal state of field at time t:

XF

)))((,)),((())(( txStxStXS nXn

X 11

The signal state space: an n-dimensional hypercube

The unit hypercube : or

,

The relationship between hyper-cubes and the fuzzy set :

, subsets of correspond to the

vertices of


nI n]1,0[

pnyx IIFF , pn

yx IFF | py

nx IFIF ,

nxxxX ,,, 21 n2 X n2nI

17

NEURONAL DYNAMICAL SYSTEMS

Neuronal activations as short-term memory

Short-term memory(STM) : activation

Long-term memory(LTM) : synapse


18

1 、 Liner Function

S(x) = cx + k , c>0

SIGNAL FUNCTION (ACTIVATION FUNCTION)

x

S

o

k


19

2. Ramp Function r if x≥θS(x)= cx if |x|<θ

-r if x≤-θr>0, r is a constant.


r

-r

θ -θ x

S


20


3 、 threshold linear signal function: a special Ramp Function

Another form:

else

cxif

cxif

cx

xS 0

1

0

1

)(

)),max(,min()( cxxS 01

0cS '


21


4 、 logistic signal function:

xc

xc

xc

cx

ee

e

exS

22

2

1

1)(

Where c>0.

01 )(' ScSS

So the logistic signal function is monotone increasing.


22


5 、 threshold signal function:

Where T is an arbitrary real-valued threshold,and k indicates the discrete time step.

Txif

Txif

Txif

xSxSk

k

k

kk

1

1

1

1

0

)(

1

)(


23


6 、 hyperbolic-tangent signal function:

Another form:

)tanh()( cxxS

01 2 )(' ScS

cxcx

cxcx

ee

eecx

)tanh(


24


7 、 threshold exponential signal function:

When ,

),min()( cxexS 1

1cxe

0 cxceS '

02 cxecS ''

0 cxnn ecS )(


25


8 、 exponential-distribution signal function:

When ,

),max()( cxexS 10

0x

0 cxceS '

0'' 2 cxecS


26


9 、 the family of ratio-polynomial signal function:

An example

For ,

),max()(n

n

xc

xxS

0

1n

02

1

)('

n

n

xc

cnxS


27



28



29

PULSE-CODED SIGNAL FUNCTION

Definition:

(5)

(6)

t

tsii dsesxtS )()(

t

tsjj dsesytS )()(

tatpulsenoif

tatoccurspulseaiftxi

0

1)(

where

(7)


30


Pulse-coded signals take values in the unit interval [0,1].

Proof:

when 0)(txi

0

t

tsii dsesxtS )()(

1)( txi

1

ts

s

tst

tsi eedsetS lim)(

when


31

PULSE-CODED SIGNAL FUNCTION Velocity-difference property of pulse-coded signals

The first-order linear inhomogenous differential equation: (8)

(9)

)()( tqxtpx The solution to this differential equation:

t

tst dsesqextx0

0 )()()(

A simple form for the signal velocity:

)()()( tStxtS iii

)()()( tStytS jjj

(10)

(11)


t

tsii dsesxtS )()( (5)

32


The central result of pulse-coded signal functions:

The instantaneous signal-velocity equals the current pulse minus the current expected pulse frequency.

------------- the velocity-difference property of pulse-coded signal functions


)()()( tStxtS iii (10)

NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS

Documents

Transcript of NEURAL NETWORK THEORY NEURONAL DYNAMICS Ⅰ : ACTIVATIONS AND SIGNALS