12008 Chingchun

Bootstrap

Chingchun Huang (黃敬群 )

Vision Lab, NCTU

22008 Chingchun

Introduction

A data-based simulation method For statistical inference

– finding estimators of the parameter in interest – Confidence of the parameter in interest

32008 Chingchun

An example• Two statistics definition for a random variable

• Average: sample mean • Standard error: The standard deviation of the sample means

• Calculation of two statistics • Carry out measurement “many” times

• Observations from these two statistics • Standard error decreases as N increases• Sample mean becomes more reliable as N increases

42008 Chingchun

Central limit theoremCentral limit theorem

• Averages taken from Averages taken from any any distribution distribution (your experimental data) will have a normal (your experimental data) will have a normal distributiondistribution• The error for such an statistic will The error for such an statistic will decrease slowly as the number of decrease slowly as the number of observations increaseobservations increase

52008 Chingchun

Normal distributionNormal distribution Averages of N.D.Averages of N.D.

distributiondistribution Averages of Averages of distribution distribution

62008 Chingchun

Uniform distributionUniform distribution Averages of U.D.Averages of U.D.

72008 Chingchun

Consequences of central limit theoremConsequences of central limit theorem

Bootstrap --- the technique to the rescueBootstrap --- the technique to the rescue

• But nobody tells you how big the sample has to But nobody tells you how big the sample has to be..be..• Should we believe a measurement of Should we believe a measurement of “Average”?“Average”? •How about other objects rather than “Average”How about other objects rather than “Average”

82008 Chingchun

Basic idea of bootstrap• Originally, from some list of data, one

computes an object (e.g.: statistic).• Create an artificial list by randomly drawing

elements from that list. Some elements will be picked more than once. • Nonparametric mode (later)• Parametric mode (later)

• Compute a new object.• Repeat 100-1000 times and look at the

distribution of these objects.

92008 Chingchun

A simple example• Data available comparing grades before and after

leaving graduate school

• Some linear correlation between grades =0.776

• But how reliable is this result (=0.776)?

102008 Chingchun

A simple example

112008 Chingchun

122008 Chingchun

132008 Chingchun

A simple example

142008 Chingchun

Confidence intervalsConsider the similar situation as before The parameter of interest is (e.g. Mean) is an estimator of based on the sample . We are interested in finding the confidence

interval for the parameter.

x

152008 Chingchun

The percentile algorithmInput the level=2 for the confidence interval.Generate B number of bootstrap samples.Compute for b = 1,…, BArrange the new data set with ‘s in order.Compute and percentile for the

new data.C.I. is given by ( th , th )Percentile 5% 10% 16% 50% 84% 90% 95%

Percentile 49.7 56.4 62.7 86.9 112.3 118.7 126.7

of

*b

*b

(1 )

(1 )

*b

*b

*b

162008 Chingchun

How many bootstraps ?• No clear answer to this.

• Rule of thumb : try it 100 times, then 1000 times, and see if your answers have changed by much.

172008 Chingchun

How many bootstraps ?

B 50 100 200 500 1000 2000 3000

Std.

Error

0.0869 0.0804 0.0790 0.0745 0.0759 0.0756 0.0755

182008 Chingchun

Convergence

This histogram is showing the distribution of the correlation coefficient for the bootstrap sample . Here B=200, B=500

0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.750

5

10

15

20

25

30

35

40

45

0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.750

20

40

60

80

100

120

192008 Chingchun

Contd…

B=1000, B=2000

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

50

100

150

200

250

300

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

100

200

300

400

500

600

202008 Chingchun

Contd…..

B=3000 B=4000

Now it can be seen the sampling distributions of correlation coefficient are more or less identical.

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

100

200

300

400

500

600

700

800

900

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

200

400

600

800

1000

1200

212008 Chingchun

Contd…..

The above graph is showing the similarity in the distribution of the bootstrap distribution and the direct enumeration from random samples from the empirical distribution

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

200

400

600

800

1000

1200

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

200

400

600

800

1000

1200

222008 Chingchun

Is it reliable ?• Observations

• Good agreement for Normal (Gaussian) distributions• Skewed distributions tend to more problematic,

particularly for the tails• A tip: For now nobody is going to shoot you down

for using it.

232008 Chingchun

Schematic representation of bootstrap procedure

1( , , )nx x x

*1x*2x *Bx

*1s*2s *Bs

*s

242008 Chingchun

Bootstrap

The bootstrap can be used either non-parametrically or parametrically

In nonparametric mode, it avoids restrictive and sometimes dangerous parametric assumptions about the form of the underlying population .

In parametric mode it can provide more accurate estimates of errors than traditional methods.

252008 Chingchun

Parametric Bootstrap

Real World

Statistic of interest

Bootstrap World

Estimated Bootstrapprobability sample model

Bootstrap Replication

unknown

probability

Model

1 2( , ,... )nP x x x x

Observed data

ˆ ( )s x

* * * *1 2

ˆ ( , ,... )nP x x x x

* *ˆ ( )s x

(distribution) P x (samples)

262008 Chingchun

Bootstrap

The technique was extended, modified and

refined to handle a wide variety of problems

including:– (1) confidence intervals and hypothesis tests, – (2) linear and nonlinear regression, – (3) time series analysis and other problems

2626

272008 Chingchun

• Fit a cubic spline: (N=50 training data)

7

1

T

j jj

μ x E Y X x β h x h x β

Example: one-dimensional smoothing

TY h x β n 2~ (0, )n N

282008 Chingchun

• Least squares

yHHH TT 1ˆ βxhxμ T ˆˆ where

21ˆ)ˆr(av

HH T

ijij xhH

where

N

i

iN

xy

1

22 ˆˆ

1

1 2ˆˆ ˆ ˆ( )T T Tse x std h x β h x H H h x

xesx ˆˆ96.1ˆ

The bootstrap and maximum likelihood method

292008 Chingchun

• Nonparametric bootstrap

Repeat B=200 times:

- draw a dataset of N=50 with replacement from the training data zi=(xi,yi)

- fit a cubic spline Construct a 95% pointwise confidence interval:

At each xi compute the mean and find the 2,5% and 97,5% percentiles

The bootstrap and maximum likelihood method

302008 Chingchun

• Parametric bootstrap

We assume that the model errors are Gaussian:

Repeat B=200 times:

- draw a dataset of N=50 with replacement from the training data zi=(xi,yi)

- fit a cubic spline on zi : and estimate

- simulate new responses : zi*=(xi,yi

*)

- fit a cubic spline on zi*:

Construct a 95 pointwise confidence interval:

At each xi compute the mean and find the 2,5% and 97,5% percentiles

20,σε~N;εXμY

2ˆ0ˆ σ,~Nε;εxμy *i

*ii

*i

ixμ

*1*ˆ iTTT

i yHHHxhx

21ˆˆˆ σxhHHx,hxμ~Nxμ TT*

2σ

The bootstrap and maximum likelihood method

312008 Chingchun

• Parametric bootstrap

Conclusion: least squares = parametric bootstrap as B

(only because of Gaussian errors)

The bootstrap and maximum likelihood method

322008 Chingchun

Some notations

The Bootstrap is – A computer-based method for assigning measures

of accuracy to statistical estimates. – The basic idea behind bootstrap is very simple, and

goes back at least two centuries.– The bootstrap method is not a way of reducing the

error ! It only tries to estimate it.– Bootstrap methods depend only on the Bootstrap

samples. It does not depend on the underlying distribution.

332008 Chingchun

A general data set-upWe have dealt with

– The standard error – The confidence interval – With the assumption that distribution is either

unknown or very complicated.

The situation can be more general – Like regression , – Sometimes using maximum likelihood estimation.

342008 Chingchun

ConclusionThe bootstrap allow the data analyst to

– Asses the statistical accuracy of complicated procedures, by exploiting the power of the computer.

The use of the bootstrap either– Relief the analyst from having to do complex

mathematical derivation or– Provide an answer where no analytical answer can be

obtained.

352008 Chingchun

Addendum : The Jack-knife• Jack-knife is a special kind of bootstrap.

• Each bootstrap subsample has all but one of the original elements of the list.

• For example, if original list has 10 elements, then there are 10 jack-knife subsamples.

362008 Chingchun

Introduction (continued) Definition of Efron’s nonparametric bootstrap. Given a sample of n independent identically

distributed (i.i.d.) observations X1, X2, …, Xn from a distribution F and a parameter of the distribution F with a real valued estimator

(X1, X2, …, Xn ), the bootstrap estimates the

accuracy of the estimator by replacing F with Fn,

the empirical distribution, where Fn places

probability mass 1/n at each observation Xi.

3636

372008 Chingchun

Introduction (continued)Let X1

*, X2*, …, Xn

* be a bootstrap sample, that is a sample of size n taken with replacement from Fn .

The bootstrap, estimates the variance of

(X1, X2, …, Xn ) by computing or approximating the variance of

* = (X1*, X2

*, …, Xn* ).

3737

382008 Chingchun

Introduction (continued)The bootstrap is similar to earlier techniques

which are also called resampling methods:– (1) jackknife, – (2) cross-validation, – (3) delta method, – (4) permutation methods, and – (5) subsampling..

3838

392008 Chingchun

Bootstrap RemediesIn the past decade many of the problems where

the bootstrap is inconsistent remedies have been found by researchers to give good modified bootstrap solutions that are consistent.

For both problems describe thus far a simple procedure called the m-out-n bootstrap has been shown to lead to consistent estimates .

3939

402008 Chingchun

The m-out-of-n Bootstrap This idea was proposed by Bickel and Ren (1996) for handling

doubly censored data. Instead of sampling n times with replacement from a sample of

size n they suggest to do it only m times where m is much less than n.

To get the consistency results both m and n need to get large but at different rates. We need m=o(n). That is m/n→0 as m and n both → ∞.

This method leads to consistent bootstrap estimates in many cases where the ordinary bootstrap has problems, particularly (1) mean with infinite variance and (2) extreme value distributions.

4040

Don’t know why.

412008 Chingchun

Examples where the bootstrap fails Athreya (1987) shows that the bootstrap

estimate of the sample mean is inconsistent when the population distribution has an infinite variance.

Angus (1993) provides similar inconsistency results for the maximum and minimum of a sequence of independent identically distributed observations.

4141