Post on 05-Feb-2016
description
NEURAL NETWORK THEORY
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
欢迎大家提出意见建议! 2003.10.15
2
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
NEURONS AS FUNCTIONS
Neurons behave as functions.
Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).
3
NEURONS AS FUNCTIONS
The transduction description: a sigmoidal or S-shaped curve
the logistic signal function:
cxexS
1
1)(
)()(' 001 cScSdx
dSS
The logistic signal function is sigmoidal and strictly increases for positive scaling constant c >0.
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
4
NEURONS AS FUNCTIONSS(x)
x
0-∞ - + +∞
Fig.1 s(x) is a bounded monotone-nondecreasing function of x
If c→+∞ , we get threshold signal function (dash line),Which is piecewise differentiable
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
5
SIGNAL MONOTONICITY
In general, signal functions are monotone nondecreasing S’>=0. This means signal functions have an upper bound or saturation value.The staircase signal function is a piecewise-differentiableMonotone-nondecreasing signal function.
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
6
SIGNAL MONOTONICITYAn important exception: bell-shaped signal function or Gaussian signal functions
02
cexS cx)(
xScxeS cx ',2'2
The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise.
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
7
SIGNAL MONOTONICITYGeneralized Gaussian signal function define potential or radial basis function :
])(2
1exp[)( 2
2 n
j
ijj
ii xxS
n
n Rxxx ),,( 1
),,( in
ii 1
input activation vector:
variance:
mean vector:
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
2i
)(xSi
we shall consider only scalar-input signal functions:
)( ii xS
8
SIGNAL MONOTONICITYneurons are nonlinear but not too much so ---- a property as semilinearity
Linear signal functions - make computation and analysis comparatively easy - do not suppress noise - linear network are not robustNonlinear signal functions - increases a network’s computational richness - increases a network’s facilitates noise suppression - risks computational and analytical intractability - favors dynamical instability
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
9
SIGNAL MONOTONICITYSignal and activation velocities
the signal velocity: =dS/dt
Signal velocities depend explicitly on action velocities
S
xSdt
dx
dx
dSS '
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
10
BIOLOGICAL ACTIVATIONS AND SIGNALS
Fig.2 Neuron anatomy
神经元 (Neuron) 是由细胞核 (cell nucleus) ,细胞体 (soma) ,轴突 (axon) ,树突 (dendrites) 和突触 (synapse) 所构成的
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
11
X= ( x1 , x2 ,…, xn )W= ( w1 , w2 ,…, wn )net=∑xiwinet=XW
x2 w2 ∑ f o=f ( net
)
xn wn
…
net=XW
x1 w1
BIOLOGICAL ACTIVATIONS AND SIGNALS
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
12
BIOLOGICAL ACTIVATIONS AND SIGNALS
Competitive Neuronal Signal
11
2
1
1
cxcx exS
exS )()(
logical signal function ( Binary Bipolar )
The neuron “wins” at time t if , “loses” ifand otherwise possesses a fuzzy win-loss status between 0 an 1.
a. Binary signal functions : [0,1]
b. Bipolar signal functions : [-1,1]
McCulloch—Pitts (M—P) neurons
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
1))(( txS 1))(( txS
13
NEURON FIELDS
Neurons within a field are topologically ordered, often by
proximity.
zeroth-order topology : lack of topological structure
Denotation: , ,
neural system samples the function m times to generate the associated pairs , ... ,
The overall neural network behaves as an adaptive filter and sample data changed network parameters.
XF YF ZF },{ YX FF },,{ ZYX FFF
),( mm yx),( 11 yx
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
pn RRf :YX FF
f
14
NEURONAL DYNAMICAL SYSTEMS
Description:a system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentials
Activation differential equations: niFFgx YXii ,,),,( 21
piFFhy YXii ,,),,( 21
in vector notation:
niFF YXi ,,),,( 21gx
piFF YXi ,,),,( 21hy
(1)
(2)
(3)
(4)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
15
NEURONAL DYNAMICAL SYSTEMSNeuronal State spaces
nn Rtxtxt ))(,),(()( 1X
pp Rtytyt ))(,),(()( 1Y
So the state space of the entire neuronal dynamical system is: pn RR
Augmentation:
ZYX FFF ]|[pn
z RF
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
},{ YX FF
16
NEURONAL DYNAMICAL SYSTEMSSignal state spaces as hyper-cubes
The signal state of field at time t:
XF
)))((,)),((())(( txStxStXS nXn
X 11
The signal state space: an n-dimensional hypercube
The unit hypercube : or
,
The relationship between hyper-cubes and the fuzzy set :
, subsets of correspond to the
vertices of
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
nI n]1,0[
pnyx IIFF , pn
yx IFF | py
nx IFIF ,
nxxxX ,,, 21 n2 X n2nI
17
NEURONAL DYNAMICAL SYSTEMS
Neuronal activations as short-term memory
Short-term memory(STM) : activation
Long-term memory(LTM) : synapse
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
18
1 、 Liner Function
S(x) = cx + k , c>0
SIGNAL FUNCTION (ACTIVATION FUNCTION)
x
S
o
k
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
19
2. Ramp Function r if x≥θS(x)= cx if |x|<θ
-r if x≤-θr>0, r is a constant.
SIGNAL FUNCTION (ACTIVATION FUNCTION)
r
-r
θ -θ x
S
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
20
SIGNAL FUNCTION (ACTIVATION FUNCTION)
3 、 threshold linear signal function: a special Ramp Function
Another form:
else
cxif
cxif
cx
xS 0
1
0
1
)(
)),max(,min()( cxxS 01
0cS '
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
21
SIGNAL FUNCTION (ACTIVATION FUNCTION)
4 、 logistic signal function:
xc
xc
xc
cx
ee
e
exS
22
2
1
1)(
Where c>0.
01 )(' ScSS
So the logistic signal function is monotone increasing.
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
22
SIGNAL FUNCTION (ACTIVATION FUNCTION)
5 、 threshold signal function:
Where T is an arbitrary real-valued threshold,and k indicates the discrete time step.
Txif
Txif
Txif
xSxSk
k
k
kk
1
1
1
1
0
)(
1
)(
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
23
SIGNAL FUNCTION (ACTIVATION FUNCTION)
6 、 hyperbolic-tangent signal function:
Another form:
)tanh()( cxxS
01 2 )(' ScS
cxcx
cxcx
ee
eecx
)tanh(
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
24
SIGNAL FUNCTION (ACTIVATION FUNCTION)
7 、 threshold exponential signal function:
When ,
),min()( cxexS 1
1cxe
0 cxceS '
02 cxecS ''
0 cxnn ecS )(
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
25
SIGNAL FUNCTION (ACTIVATION FUNCTION)
8 、 exponential-distribution signal function:
When ,
),max()( cxexS 10
0x
0 cxceS '
0'' 2 cxecS
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
26
SIGNAL FUNCTION (ACTIVATION FUNCTION)
9 、 the family of ratio-polynomial signal function:
An example
For ,
),max()(n
n
xc
xxS
0
1n
02
1
)('
n
n
xc
cnxS
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
27
SIGNAL FUNCTION (ACTIVATION FUNCTION)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
28
SIGNAL FUNCTION (ACTIVATION FUNCTION)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
29
PULSE-CODED SIGNAL FUNCTION
Definition:
(5)
(6)
t
tsii dsesxtS )()(
t
tsjj dsesytS )()(
tatpulsenoif
tatoccurspulseaiftxi
0
1)(
where
(7)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
30
PULSE-CODED SIGNAL FUNCTION
Pulse-coded signals take values in the unit interval [0,1].
Proof:
when 0)(txi
0
t
tsii dsesxtS )()(
1)( txi
1
ts
s
tst
tsi eedsetS lim)(
when
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
31
PULSE-CODED SIGNAL FUNCTION Velocity-difference property of pulse-coded signals
The first-order linear inhomogenous differential equation: (8)
(9)
)()( tqxtpx The solution to this differential equation:
t
tst dsesqextx0
0 )()()(
A simple form for the signal velocity:
)()()( tStxtS iii
)()()( tStytS jjj
(10)
(11)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
t
tsii dsesxtS )()( (5)
32
PULSE-CODED SIGNAL FUNCTION
The central result of pulse-coded signal functions:
The instantaneous signal-velocity equals the current pulse minus the current expected pulse frequency.
------------- the velocity-difference property of pulse-coded signal functions
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
)()()( tStxtS iii (10)