Post on 24-Feb-2016
description
Overfitting and Its AvoidanceChapter 5
指導教授: 徐立群 教授學生: R16014101 陳怡齊
R16011234 吳年鑫
Overfitting
即「過適」、「超適」或稱「過度擬合」 意指在調適一個 model時,使用過多參數。對比於可取得的資料總量來說,一個荒謬的 model只要足夠複雜,是可以完美地適應 (fit)資料。 不合乎一般化 (Generalization)
違反奧卡姆剃刀( Occam’s Razor ) 原則
Overfitting & Generalization
A extreme example –
Customer churn or non-churn
Training data & Holdout data
Overfitting Examined
• Holdout Data and Fitting Graphs -
Figure 1. A typical fitting graph.
A fitting graph shows the accuracy of a model as a function of complexity .
Overfitting Examined
Base rate - What would b be ?
Figure 2. A fitting graph for the customer churn (table) model.
Overfitting in Tree Induction
Decision tree induction overfitting starts to the “sweet spot” in the graph .
Figure 3. A typical fitting graph for tree induction.
Overfitting in Mathematical Functions
We add more Xi, the function becomes more and more complicated.
Each Xi has a corresponding Wi, which is a learned parameter of the model .
Two dimensions you can fit a line to any two points and in three dimensions you can fit a plane to any three points .
This concept generalizes: as you increase the dimensionality, you can perfectly fit larger and larger sets of arbitrary points .
Example: Overfitting Linear Functions
Data: sepal width, petal widthTypes : Iris Setosa, Iris Versicolor
Two different separation lines:a. Logistic regressionb. Support vector machine Figure 4
Example: Overfitting Linear Functions
Figure 4 Figure 5
Example: Overfitting Linear Functions
Figure 6 Figure 7
From Holdout Evaluation to Cross-Validation
Holdout Evaluation Splits the data into only one training and one holdout set.
Cross-validation computes its estimates over all the data by performing multiple splits and systematically swapping out samples for testing. ( k folds, typically k would be 5 or 10. )
The Churn Dataset Revisited
Average accuracy of the folds with classification trees is 68.6%—significantly lower than our previous measurement of 73%. ( the standard deviation of the fold accuracies is 1.1 )
“Example: Addressing the Churn Problem with Tree Induction” in Chapter 3. The logistic regression models
show slightly lower average accuracy (64.1%) and with higher variation ( standard deviation of 1.3 )
Classification trees may be preferable to logistic regression because of their greater stability and performance.
Learning Curves The generalization performance of data-driven
modeling generally improves as more training data become available.
Overfitting Avoidance & Complexity Control
Concept in Tree Induction : Tree induction commonly uses two techniques to avoid overfitting. These
strategies are : (i) to stop growing the tree before it gets too complex, and (ii) to grow the tree until it is too large, then “prune” it back, reducing its size (and
thereby its complexity).
Methods in Tree Induction : To limit tree size is to specify a minimum number of instances that must be
present in a leaf. Hypothesis test ( P-value )
Overfitting Avoidance & Complexity Control
General Method for Avoiding Overfitting Compare the best model we can build from one family (say, classification
trees) against the best model from another family (say, logistic regression).
Training set
Test set( hold out )
Training subset
Validation set
Nested holdout testing Select the best model by assess by
having a complexity of 122 nodes ( the sweet spot).
Induce a new tree with 122 nodes from the whole, original training data.
Final hold out
Overfitting Avoidance & Complexity Control
Original data
Training set
Test set
Nested Cross-ValidationSequential Forward Selection