Transfer LearningOleksandr Baiev, PhD

ProblemsML problems

Computer Vision

Natural Language Processing

Signal processing

Predicting

Decision making

MODEL

INPUTS

INFERENCE

DS

Conventional approaches

Domain specific featuresIN

PU

TS

Model fitted to specific

features set INFE

REN

CE

Assumes domain specific

research, conducted by

appropriate expert

(feature engineering)

Looks like hacks for

practical solutions

Neural Networks SolutionIN

PU

TS

INFE

REN

CE

Simple pre- and postprocessing:

scaling, shifting, etc.

Is Neural Networks is simple solution?

SVM

- Kernel type

- Smoothness

RANDOM FOREST

- Number of trees

- Trees’ depth

NEURAL NETWORKS

Is Neural Networks simple solution?

• A few dozens of thousands of samples – is not bad dataset

• A few hundreds of thousands of samples – is good dataset

• Overfitting is problem even for datasets with several millions samples

Transfer learning

Learning system

Learning system

Knowledge

Source taskTarget task

Storing knowledge gained while

solving one problem and applying it

to a different but related problem

What is knowledge in Neural Networks

Transfer learning for Neural Networks

1) Train on Source Task

2) Remove last layers

3) Add new last layers

4) Train net for Target Task

Known topologyAlexNet

GoogleNet

VGG-16

ResNet-50

Weights transfer

Source task:For example, train VGG-16 for ImageNet classification (1.2*10^6 samples, 1000 classes)

Target task: Remove last layer with 1000 outputsPut your own last layer for N classes=> same topology, weights initialized by weights from source task’s net

When such flow can be used

Low amount of data Source and Target

tasks are similar

Transfer learning types

• New classifier (for example, SVM)

• Layer by Layer pretrain (for example, early autoencoders)

• Finetuning (with lower LR)

Examples. “Plankton” and AlexNet

“Plankton”

National Data Science Bowl

kaggle.com

Classes: 121 with hierarchy

Samples: ~30 000

AlexNet: 8 layers

inputs: 227x227

Examples. “Pankton” and Cifar10

“Plankton”

National Data Science Bowl

kaggle.com

Classes: 121 with hierarchy

Samples: ~30 000

AlexNet: 4 layers

inputs: 32x32

Example. “Drivers” and GoogleNet“Drivers”

State Farm Distracted

Driver Detection

kaggle.com

Classes: 121 with hierarchy

Samples: ~30 000

AlexNet: 4 layers

inputs: 32x32

Model Zoo

Model ZooCAFFE : ~ 30 models for different problems

https://github.com/BVLC/caffe/wiki/Model-Zoo

MatConvNet: ~10 models for different problemshttp://www.vlfeat.org/matconvnet/pretrained/

TensorFlow: https://github.com/tensorflow/models

Torch:https://github.com/facebook/fb.resnet.torchhttps://github.com/szagoruyko/loadcaffe