MACHINE LEARNING FROM OPTIMIZATION PERSPECTIVEZheng Han

June 15th, 2017

6/14/17

1

谢绝转载，转载请联系WSFC

Wall Street Fintech Club (WSFC), © All Rights Reserved ,www.wallstreetfintechclub.com, 公共号：华尔街 Fintech俱乐部，禁止未授权转载

• Machine Learning (ML)• Case Study• Optimization in ML• - Learning Theory• - Regularization• - Learning Algorithm

2

CONTENT

Wall Street Fintech Club (WSFC), © All Rights Reserved ,www.wallstreetfintechclub.com, 公共号：华尔街 Fintech俱乐部，禁止未授权转载

ML – A FAVORABLE PERSPECTIVE

ML – ANALOGY

ML – THREE COMPONENTS

Machine Learning = Representation + Evaluation + Optimization

CASE STUDY – DECISION TREE

CASE STUDY – DECISION TREE

● Representation: x = (Sunny Hot, High, Strong), tree structure to represent boolean function

● Evaluation: false positive rate, false negative rate, etc..

● Optimization: efficiently construct a tree that gives relatively low predictive error

Optimization:Stochastic Gradient Descent(SGD)s:Momentum / Nesterov accelerated gradientAdagrad / Adadelta / RMSprop / Adam

Backpropagation

CASE STUDY – NEURAL NETWORK

Representation: images -> pixels -> matrices

ML – AUTOMATE AUTOMATION

OPTIMIZATION

OPTIMIZATION IN ML

Representation

Evaluation

Optimization

Parameterized Optimization

OPTIMIZATION IN ML

Expected Risk

Empirical Risk

Learning Theory

OPTIMIZATION IN ML

Expected Risk

Empirical Risk

Structural Risk Minimization

Model Complexity

STRUCTURAL RISK MINIMIZATION BY REGULARIZATION

OPTIMIZATION ALGORITHM IN ML

Gradient Descent Method:

Second-order Method:

OPTIMIZATION ALG FOR DEEP LEARNING

• Reinforcement Learning: dynamic programming• Online Learning: online convex optimization• Evolutionary Algorithms: generic algorithms • Big Data: distributed optimization, sparse optimization

17

OTHER RELEVANT TOPICS

Wall Street Fintech Club (WSFC), © All Rights Reserved ,www.wallstreetfintechclub.com, 公共号：华尔街 Fintech俱乐部，禁止未授权转载

ML IN REAL-WORLD

• A Few Useful Things to Know about Machine Learning. Pedro Domingos• The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake

Our World. Pedro Domingos• Scikit-Learn: http://scikit-learn.org• An Extended Version Of The Scikit-Learn Cheat Sheet. Christophe Bourguignat• Deep Learning. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton• Optimization Methods for Large-Scale Machine Learning. Leon Bottou, Frank E. Curtis,

and Jorge Nocedal• The LION Way: Machine Learning plus Intelligent Optimization. Roberto Battiti, Mauro

Brunato

19

REFERENCES

Wall Street Fintech Club (WSFC), © All Rights Reserved ,www.wallstreetfintechclub.com, 公共号：华尔街 Fintech俱乐部，禁止未授权转载