Chapter 1 Introduction to Neurocomputing

Chapter 1Introduction to Neurocomputing

國立雲林科技大學資訊工程研究所張傳育 (Chuan-Yu Chang ) 博士Office: ES 709

TEL: 05-5342601 ext. 4337

E-mail: [email protected]

What is Neurocomputing?

Neurocomputing approach Involves a learning process within an ANN 一旦類神經網路訓練完成，該類神經網路即可進行特定的工

作。例如 pattern recognition Associated ( 聯想 )

Why can we (human) perform certain tasks much better than a digital computer? Our brain is organized

大腦神經 (nerve) 的傳導速度比電子的速度慢 (106) 倍，但腦神經具有大量平行的計算能力。 ( 約 1011個 neuron)

大腦是一個 adaptive, nonlinear, parallel 的計算機。

What is Neurocomputing? (cont.) 類神經網路的主要能力：

由範例學習 (learn by example) 歸納 (generalize)

The NN can classify input patterns to an acceptable level of accuracy even if they were never used during the training process.

大部分的類神經網路具有類似的特徵： Parallel computational architecture Highly interconnected Nonlinearity output

What is Neurocomputing? (cont.) Applications of neural networks:

Image processing Prediction and forecasting Associative memory Clustering Speech recognition Combinatorial optimization Feature extraction …

What is Neurocomputing? (cont.) 使用 neurocomputing 解決特定問題的優點：

Fault tolerance Nonlinear Adaptive

Course Introduction

Text book Fredric M. Ham, Principles of Neurocomputing for

Science & Engineering, McGRAW-HILL, 2001. Simon Haykin, Neural Networks-A comprehensive

foundation, 2nd Edition. Prentice Hall, 1999

Contents Introduction to Neurocomputing Fundamental Neurocomputing Concepts

Activation Function Adaline and Madaline Perceptron Multilayer perceptron Leaning Rules

Mapping Networks Associative memory Backpropagation Learning Algorithm Counterpropagation Radial Basis Function Neural Networks

Contents (con’t)

Self-Organizing Networks Kohonen Self-Organizing Map Learning Vector Quantization Adaptive Resonance Theory Neural network

Recurrent Networks and Temporal feedforward Networks Hopfield Associative Memory Simulated Annealing Boltzmann Machine Time-Delay Neural Networks

Contents (con’t)

Statistical Methods Using Neural Networks Principal Component Analysis Independent Component Analysis

Course Requirements

Homeworks Supervised learning: Digits Recognition Unsupervised learning: Traveling Salesperson Pr

oblem (TSP) Final Project

Presentation Implementation and Demonstration

Final Examination

請思考這門課對各位的研究是否有幫助？

Does this course helpful to your research?

Historical notes

McCulloch and Pitts (1943) No training to neurons. Act as certain logic functions.

Hebb (1949) Based on a neurobiological viewpoint, describes a learning

process. Hebb stated that information is stored in the connections of

neurons and postulated a learning strategy for adjustment of the connection weight.

Rosenblatt (1958) Proposed the concept of the perceptron

Historical notes (cont.)

Widrow and Hoff (1960) The Adaline (adaptive linear element) trained by the LMS le

arning rule. Minsky and Papert (1969)

Slowed down neural network research in 1969. Perceptrons have limited capabilities, the XOR problem.

Kohonen and Anderson (1972) Content-addressable associative memories.

Werbos (1974) The first description of the backpropagation algorithm for tr

aining multilayer feed-forward perceptrons.


Little and Shaw (1975) Use a probabilistic model of a neuron instead of a determini

stic one. Lee (1975)

Presented the fuzzy McCulloch-Pitts neuron mode. Hopfield (1982)

Proposed a recurrent neural network The network can store information in a dynamically stable s

torage and retrieval. Kohonen (1982)

Presented the self-organizating feature map. It is an unsupervised, competitive learning, clustering netw

ork in which only one neuron is “on” at a time.


Oja (1982) Presented a single linear neuron trained by a normalized Hebbia

n learning rule that acts as a principal-component analyzer. The neuron is capable of adaptive extracting the 1st principal eig

nvector from the input data. Carpenter and Grossberg (1987)

Developed self-organizing neural networks based adaptive resonance theory (ART)

Sivilotti, Mahowald, and Mead (1987) The first VLSI realization of neural networks.

Broomhead and Lowe (1988) First exploitation of radial basis function in designing neural netw

orks.

McCulloch-Pitts neuron

The McCulloch-Pitts neuron is a two-state device. (on/off, binary) The output of a particular neuron cannot coalesce with output of

another neuron; it can branch to another neuron and terminate as an input to that neuron.( 個別神經元的輸出無法與其他神經元的輸出合併，但可當成其他神經元的輸入。 )

Two types of terminations Excitatory input Inhibitory input

There can be any number of inputs to a neuron. .( 一個神經元可有任意多個輸入 )

Whether the neuron will be on (firing) or off depends on the threshold of the neuron.( 但其輸出是由 threshold 所決定。 )


Physical assumptions:( 基本假設 ) The neuron activity is an all-or-nothing process, ie., the acti

vation of the neuron is binary. A certain fixed number of synapses (>1) must be excited wi

thin a period of latent addition for a neuron to be excited. The only significant delay within the nervous system is syn

aptic delay. The activity of any nonzero inhibitory synapse absolutely pr

events excitation of the neuron at that time. The structure of the network does not change with time.


Architecture of a McCulloch-Pitts neuron N excitatory input M inhibitory input 1 個由 threshold 決定的輸出 y 。

為滿足前面的假設 4，

ufor

ufory

0

1

cnw

u= xi w


Example 1 The AND logic function can be realized using the

McCulloch-Pitts neuron having two synaptic connections with equal weights of 1 and a threshold =2.


Example 2: The OR logic function can be realized using the McCulloch-

Pitts neuron having two synaptic connections with equal weights of 2 and a threshold =2.


Example 3: The XOR logic function can be realized using three McC

ulloch-Pitts neuron having two synaptic connections with equal weights of 2 and a threshold =2.

Neurocomputing and Neuroscience Biological neural networks

神經系統 (nervous system) 是一個巨大且複雜的神經網路 (neural network) ，大腦是神經系統的中樞，神經系統和各感官器官相連結，傳遞感官訊息給大腦，並且傳送行為命令給反應器官 (effectors) 。

大腦是由大約 1011個神經元所組成，而這些神經元相互連接成子網路，組成所謂的細胞核 (nuclei) ，細胞核是由一連串的神經元所組成，具有特定的功能。

在大腦中的感官系統 (sensory system) 和它的子網路，擅於將複雜的感官資訊分解 (decompose) 成知覺的基本特性。對於不同的知覺會有不同的分解。

Neurocomputing and Neuroscience

Fausett 更有效的建構生物的 (biological) 神經元 (neuron)成類神經元 (artificial neurons) 。大腦和神經系統的其他部分是由許多不同種類的神經元組成，有不同的電子特性、數量、大小、及連接方式。

樹狀突

軸突

Small cell Large cell


一個生物神經元包含有三個部分：樹狀突 (dendrites) ：收取

來自其他神經元的訊號。 Cell body ：用來彙總來自樹

狀突及其他神經鍵的訊號。當收到的刺激大於 threshold時，神經元將產生一個電能(potential) ，也就是 fire ，並將此電能沿著 axon 傳給其他神經元或是目的細胞，例如肌肉。

軸突 (axon) ：藉由神經鍵(synapses) 和其他神經元的軸突 (axon) 相連結。神經鍵(synapses) 的數目可能從數百到上萬。

（體幹）


A biological neuron consists of three main components ： Dendrites ： receive signals fro

m other neurons 。 Cell body (soma) ： sums the in

coming signals from the dendrites and sums the signals from the numerous synapses on its surface.

Axon ： the axons of other neurons connect to the dendrite and cell body surfaces by means of connectors called synapses 。 The number of synaptic connection from other neurons may range from a few hundred to 10,000.

（體幹）

Neurocomputing and NeuroscienceSimplified drawing of the synapses


由於神經元的外表有神經薄膜，因此訊號到達樹狀突時會因時間和距離而迅速衰減。而喪失刺激神經元的能力，除非它們收到來自附近神經元的訊號強化了衰減的訊號。

Neurocomputing and Neuroscience 如果神經元的輸入無法達到 threshold ，則輸入訊號將快速

衰減，而且不會產生行動電能 (potential) 。因此， action potential 的產生原則是 all-or-nothing 。

輸入強度是表現在每秒產生的 action potential 數量，而不是action potential 的大小。較強的輸入信號會比較弱的輸入信號產生較多的 action potential 。


Synapses are the points of contact that connect the axon terminals to their targets.

Synapse 由下列所組成： Nerve terminal Synaptic cleft or gap Postsynaptic membrane

Classification of neural networks Supervised learning vs. unsupervised learning

Chapter 1 Introduction to Neurocomputing

Documents

Transcript of Chapter 1 Introduction to Neurocomputing