Convolutional Neural Network (CNN) presentation from theory to code in Theano
87
M&S Convolutional Neural Network from Theory to Code Seongwon Hwang
-
Upload
seongwon-hwang -
Category
Data & Analytics
-
view
600 -
download
0
Transcript of Convolutional Neural Network (CNN) presentation from theory to code in Theano
M&S
Convolutional Neural Network from Theory
to Code
Seongwon Hwang
M&S
Preliminaries
M&S
Tensor :
(Index Notation)
ky klijy
ijklA
kja
ijA iA
kijjB
M&S
Order (Rank)
M&S
Range
[𝑎 𝑏𝑐 𝑑]
[𝑎 𝑏 𝑐𝑑 𝑒 𝑓𝑔 h 𝑖 ] [ 𝑎 𝑏
𝑒 𝑓𝑐 𝑑𝑔 h
𝑖 𝑗𝑚 𝑛
𝑘 𝑙𝑜 𝑝 ]
𝐴𝑖𝑗
i = 2, j = 2
i = 3, j = 3 i = 4, j = 4
[𝑎 𝑏𝑐 𝑑𝑒 𝑓 ]
i = 3, j = 2
M&S
Free index : unrepeated index
+
+
+, k = 1,2
2
1iikik xay
) ,( 21 yyy
M&S
Dummy index : repeated index
1,2kixay ikik ,
Free index Dummy index
2
1iikik xay
+
+
M&S
Example
iji BA jj BABA 2211 =
212111 ,1 BABAj
222121 ,2 BABAj
) ,( 222121212111 BABABABA iji BA =
i, j = 1,2
M&S
Example
ijjA 332211 iii AAA
) , ,( 333322311233222211133122111 AAAAAAAAA
133122111 ,1 AAAi
233222211 ,2 AAAi
333322311 ,3 AAAi
=i, j = 1,2,3
M&S
e-Permutation
ijke
M&S
Example – Determinant
bcaddcba
aaaa
A detdetdet2221
1211
bcadaaaa
aaeaaeaaeaae
aae jiij
00 21122211
221222211221221112211111
21
M&S
Example – Determinant
) ( ...det 21... matrixNNaaaeA Nkjikij
M&S
Kronecker delta
M&S
Derivatives
kjjkjj xxyy
xyy
2
,, ,
1,2,3jφxφ
zφ
yφ
xφφgrad jj
) , ,( ,
Example – Gradient
M&S
CNN Tensor notation in Theano
- Input Images -
4D tensor
1D tensor
[number of feature maps at layer m, number of feature maps at layer m-1, filter height, filter width]
ijklx
opqrW
mb
[ i, j, k, l ] =
[ o, p, q, r ] =
[ m ] =
- Weight -
- Bias -
[n’th feature map number]
[mini-batch size, number of input feature maps, image height, image width]
M&S
Convolutional Neural Network Theory I
M&S
Convolution?
daatwax
twxty
)()(
))(()(
a
anwaxny ][][][
- Continuous Variables -
- Discrete Variables -
M&S
Convolution?
a
anwaxny ][][][
- Discrete Variables -
][][ awax ][][ awax
)]([][ nawax n ste
p move
Y-axis transformation
M&S
Y축
X축
Convolution?
X축
X축
M&S
Cross-Correlation?
a
nawaxnwxny ][][])[ (][
- Discrete Variables (In real number) -
][][ awax ][][ nawax n step move
★
M&S
Convolution VS. Cross-Correlation
Cross-CorrelationConvolution
M&S
Cross-Correlation in 2D
Output (y) Kernel (w) Input (x)
n m
nmwjnimx
jiwxjiy
],[],[
],)[(],[
M&S
Intuition for Cross-Correlation
wx
★
M&S
wx
★
Intuition for Cross-Correlation
M&S
wx
★
Intuition for Cross-Correlation
M&S
wx
★
Intuition for Cross-Correlation
M&S
wx
★
Intuition for Cross-Correlation
Input
) (
mapFreatureneuronHidden
fieldrecptiveLocal
filteror kernel
M&S
Convolutional Neural Network Theory II
M&S
CNN overall procedures
Input Image Convolutional Layer Pooling MLP Softmax Output
M&S
CNN overall procedures
Input Image
Convolutional Layer
Pooling
MLP
Softmax Output
M&S
Input Image
Input Image
1D - Input neurons
2D - Input neurons
M&S
Convolutional Layer
Input Image
Convolutional Layer
Input neurons
Hidden neuron
M&S
Traditional Neural Network
Input layer Hidden layer Output layer
M&S
CNN - Sparse Connectivity
Input layer Hidden layer(Feature map)
Output layer
M&S
CNN – Dimension shrinkage ( + Pooling)
Input layer Output layerHidden layer(Feature map)
M&S
Cross-Correlation Input (or Hidden) layer with weight
Input layer Output layer
1W2W
3W
'3W'2W'1W
Hidden layer(Feature map)(Receptive field)
M&S
Shared Weight representation
Input layer Output layer
1W2W
3W
'3W'2W'1W
Hidden layer(Feature map)
M&S
Shared Weight representation
Input layer Output layer
1W2W
3W
'3W'2W'1W
Hidden layer(Feature map)
M&S
Shared Weight representation
Input layer Output layer
1W2W
3W
'3W'2W'1W
Hidden layer(Feature map)
M&S
Shared Weight representation
Input layer Output layer
1W2W
3W
'3W'2W'1W
Hidden layer(Feature map)
M&S
Shared Weight representation
Input layer Output layer
1W2W
3W
'3W'2W'1W
Hidden layer(Feature map)
M&S
Shared Weight representation
Input layer Output layer
1W2W
3W
'3W'2W'1W
Hidden layer(Feature map)
M&S
Multiple Feature maps
Input Image
Convolutional Layer
Input neurons First hidden layer
M&S
Max Pooling
Input Image
Convolutional Layer
PoolingMax pooling with 2x2 filters and stride 2
M&S
Why pooling?
Input Image
Convolutional Layer
Pooling
1. Low computation 2. Translation invariance
3. Transformation invariance
4. Scaling invariance
M&S
Several types of Pooling
Input Image
Convolutional Layer
Pooling
M&S
Transform data dimension before MLP
Input Image
Convolutional Layer
Pooling
MLP
1D - Output neurons
2D - Output neurons
M&S
Multilayer Perceptron (MLP)
Input Image
Convolutional Layer
Pooling
MLP
Input layer Hidden layer 1 Hidden layer 2
M&S
Softmax Output
Input Image
Convolutional Layer
Pooling
MLP
Softmax Output
Hidden layer 2 Output layer
M&S
Several types of CNN
M&S
Intuition for CNN
Input Image
Convolutional Layer
Pooling
MLP
Softmax Output
M&S
Convolutional Neural Network
Code in Theono
M&S
CNN overall procedures
Input Image
Convolutional Layer
Pooling
MLP
Softmax Output
M&S
Input Image
Input Image
- Input Images -
4D tensor
[mini-batch size, number of input feature maps, image height, image width]ij
klx
5...
28
28
500
7
[ i, j, k, l ] =
Mini batch 1
5...
28
28
500
8
Mini batch 100
. . .
50,000 images in the training data
M&S
Weight tensor
Input Image
Convolutional Layer
4D tensor
[number of feature maps at layer m, number of feature maps at layer m-1, filter height, filter width]
opqrW [ o, p, q, r ] =
- Weight -
M&S
Exercise for Input and Weight tensor
1111x
1111W
Input layer Convolutional layer 1 Convolutional layer 2
[ 1, 1, 1, 1 ]
[ 1, 1, 1, 1 ]
M&S
Code for Convolutional Layer
28
28 8
def evaluate_lenet5(learning_rate=0.1, n_epochs=2, dataset=‘minist.pkl.gz’, nkerns=[20, 50], batch_size=500):
LeNetConvPoolLayer
image_shape=(batch_size, 1, 28, 28)filter_shape=(nkerns[0], 1, 5, 5)poolsize=(2, 2)
image_shape=(batch_size, nkerns[0], 12, 12)filter_shape=(nkerns[1], nkerns[0], 5, 5)poolsize=(2, 2)
Layer0 – Convolutional layer 1
Layer1 – Convolutional layer 2
5
5
20
24
24
20
5
5
12
12
8
8
44
20 50 50 50
28 – 5 + 1 = 24Convolution
24 / 2 = 12Pooling
12 – 5 + 1 = 8Convolution
8 / 2 = 4Pooling
Class
M&S
Zero-Padding Input layer Output layerHidden layer(Feature map)
Input Image
Convolutional Layer
Pooling
M&S
Zero-Padding Input layer Output layerHidden layer(Feature map)
0
0
M&S
Zero-Padding Input layer Output layerHidden layer(Feature map)
0
0
M&S
Zero-Padding Input layer Output layerHidden layer(Feature map)
0
0
M&S
Zero-Padding Input layer Output layerHidden layer(Feature map)
0
0
M&S
Zero-Padding Input layer Output layerHidden layer(Feature map)
0
0
0
0
M&S
Zero-Padding Input layer Output layerHidden layer(Feature map)
0
0
0
0
M&S
Zero-Padding
0
0
M&S
Zero-Padding
0
0
M&S
Zero-Padding
0
0
M&S
Zero-Padding
0
0
0
0
M&S
Zero-Padding
0
0
0
0
M&S
Zero-Padding
0
0
0
0
M&S
Zero-Padding
No zero-padding Zero-padding 1 Zero-padding 2
Dimension Reduction Dimension Equality Dimension Increase
Zero-pad-ding in Theano
Default in Theano
M&S
Code for Zero-Padding setting in Theano Library
conv.py (In Library File)
def conv2d(input, filters, image_shape=None, filter_shape=None, border_mode=‘valid’, subsample=(1, 1), **kargs):
‘valid’ ‘full’
We should modify dimension changes in code by ourselves!
Image_shape + filter_shape – 1Image_shape - filter_shape + 1
28 + 5 – 1 = 3228 - 5 + 1 = 24
M&S
1. Border in Pooling
No problem in Border Ingore_border = False Ingore_border = True
M&S
1. Border in Pooling (Code)
No problem in Border
Ingore_border = False Ingore_border = True
pooled_out = downsample.max_pool_2d(input=conv_out, ds=poolsize, ignore_border=True)
LeNetConvPoolLayerClass
Default in Theano library is False!
M&S
2. Stride in Pooling
Default in Theano stride size = poolsize
Unduplicated!
M&S
2. Stride in Pooling2
Default in Theano stride size = poolsize
Unduplicated!
M&S
2. Stride in Pooling2
Default in Theano stride size = poolsize = (2,2)
Unduplicated!
M&S
2. Stride in Pooling
2
Default in Theano stride size = poolsize = (2,2)
Unduplicated!
M&S
2. Stride in Pooling
2
Default in Theano stride size = poolsize = (2,2)
Unduplicated!
M&S
2. Stride in Pooling
stride size = (1, 1)
6
M&S
2. Stride in Pooling1
76
M&S
2. Stride in Pooling (Code)
76 8
6 7 8
3 3 4
pooled_out = downsample.max_pool_2d(input=conv_out, ds=poolsize, ignore_border=True)
pooled_out = downsample.max_pool_2d(input=conv_out, ds=poolsize, ignore_border=True, st = (1,1))
M&S
Activation Function in Convolutional Layer
self.output=T.tanh(pooled_out + self-.b.dimshuffle(‘x’,0,‘x’,‘x’)
Convolution Pooling Activation
LeNetConvPoolLayerClass
M&S
Dimension Reduction 2D 1D
Input Image
Convolutional Layer
Pooling
MLP
layer2_input = layer1_input.output.flatten(2) 1D - Output neurons
2D - Output neurons
M&S
Code for MLP in Theano
Input Image
Convolutional Layer
Pooling
MLP layer2 = HiddenLayer( rng, input=layer2_input, n_in-nkerns[1] * 4 * 4, n_out=500, activation = T.tanh)
HiddenLayerClass
Last output size for C+P
Number of node at Hidden layer
Activation function at Hidden layer
***In order to extend the number of Hidden Layer in MLP, We need to make layer3 by copying this code***
M&S
Code for Softmax Output in Theano
Input Image
Convolutional Layer
Pooling
MLP
Softmax Output
layer3 = LogisticRegression( input=layer2.output, n_in=500, n_out=10)
LogisticRegressionClass
cost = layer3.negative_log_likelihood(y)
Number of node at previous Hidden layer
Final Output sizeEx) 0, 1, 2, ,,, 9
Softmax activation function
M&S
CNN application in bioinformatics problem
