MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning....
Transcript of MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning....
![Page 1: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/1.jpg)
MACHINE LEARNING FROM OPTIMIZATION PERSPECTIVEZheng Han
June 15th, 2017
6/14/17
1
谢绝转载,转载请联系WSFC
Wall Street Fintech Club (WSFC), © All Rights Reserved ,www.wallstreetfintechclub.com, 公共号:华尔街 Fintech俱乐部,禁止未授权转载
![Page 2: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/2.jpg)
• Machine Learning (ML)• Case Study• Optimization in ML• - Learning Theory• - Regularization• - Learning Algorithm
2
CONTENT
Wall Street Fintech Club (WSFC), © All Rights Reserved ,www.wallstreetfintechclub.com, 公共号:华尔街 Fintech俱乐部,禁止未授权转载
![Page 3: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/3.jpg)
ML – A FAVORABLE PERSPECTIVE
![Page 4: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/4.jpg)
ML – ANALOGY
![Page 5: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/5.jpg)
ML – THREE COMPONENTS
Machine Learning = Representation + Evaluation + Optimization
![Page 6: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/6.jpg)
CASE STUDY – DECISION TREE
![Page 7: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/7.jpg)
CASE STUDY – DECISION TREE
● Representation: x = (Sunny Hot, High, Strong), tree structure to represent boolean function
● Evaluation: false positive rate, false negative rate, etc..
● Optimization: efficiently construct a tree that gives relatively low predictive error
![Page 8: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/8.jpg)
Optimization:Stochastic Gradient Descent(SGD)s:Momentum / Nesterov accelerated gradientAdagrad / Adadelta / RMSprop / Adam
Backpropagation
CASE STUDY – NEURAL NETWORK
Representation: images -> pixels -> matrices
![Page 9: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/9.jpg)
ML – AUTOMATE AUTOMATION
![Page 10: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/10.jpg)
OPTIMIZATION
![Page 11: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/11.jpg)
OPTIMIZATION IN ML
Representation
Evaluation
Optimization
Parameterized Optimization
![Page 12: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/12.jpg)
OPTIMIZATION IN ML
Expected Risk
Empirical Risk
Learning Theory
![Page 13: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/13.jpg)
OPTIMIZATION IN ML
Expected Risk
Empirical Risk
Structural Risk Minimization
Model Complexity
![Page 14: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/14.jpg)
STRUCTURAL RISK MINIMIZATION BY REGULARIZATION
![Page 15: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/15.jpg)
OPTIMIZATION ALGORITHM IN ML
Gradient Descent Method:
Second-order Method:
![Page 16: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/16.jpg)
OPTIMIZATION ALG FOR DEEP LEARNING
![Page 17: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/17.jpg)
• Reinforcement Learning: dynamic programming• Online Learning: online convex optimization• Evolutionary Algorithms: generic algorithms • Big Data: distributed optimization, sparse optimization
17
OTHER RELEVANT TOPICS
Wall Street Fintech Club (WSFC), © All Rights Reserved ,www.wallstreetfintechclub.com, 公共号:华尔街 Fintech俱乐部,禁止未授权转载
![Page 18: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/18.jpg)
ML IN REAL-WORLD
![Page 19: MACHINE LEARNING FROM OPTIMIZATION …...• A Few Useful Things to Know about Machine Learning. Pedro Domingos Pedro Domingos • The Master Algorithm: How the Quest for the Ultimate](https://reader033.fdocument.pub/reader033/viewer/2022052613/5f29f7cd738757282d0d7b83/html5/thumbnails/19.jpg)
• A Few Useful Things to Know about Machine Learning. Pedro Domingos• The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake
Our World. Pedro Domingos• Scikit-Learn: http://scikit-learn.org• An Extended Version Of The Scikit-Learn Cheat Sheet. Christophe Bourguignat• Deep Learning. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton• Optimization Methods for Large-Scale Machine Learning. Leon Bottou, Frank E. Curtis,
and Jorge Nocedal• The LION Way: Machine Learning plus Intelligent Optimization. Roberto Battiti, Mauro
Brunato
19
REFERENCES
Wall Street Fintech Club (WSFC), © All Rights Reserved ,www.wallstreetfintechclub.com, 公共号:华尔街 Fintech俱乐部,禁止未授权转载