USAID PAT Philippines

8/3/2019 USAID PAT Philippines

1/24

1

Grameen Foundation submission to USAID:Progress out of Poverty Index (PPI) for the Philippines

Accuracy

1.

Overall approach to tool developmentThe GFUSA PPI, or scorecard, for the Philippines is an easy-to-use, objective tool that estimates theprobability that a client is among the poorest half of those whose income is below the national povertyline (Schreiner, 2006a). This annual per capita extreme poverty line is, on average, 9,060 pesos perperson per year. The PPI can help development projects to report on participants poverty rates, targetservices, and track changes in poverty over time.

The tool estimates a poverty likelihood, that is, the probability that a client is very poor. It does notdirectly estimate income. This approach builds on decades of experience in credit scoring (where lendersestimate the probability of falling intox days of arrears) and the World Banks proxy means tests thatestimate poverty likelihoods.

While the tool is accurate, its design emphasizes practicality. In particular, it is:

Short and simple for both staff and clients Easy to collect and analyze data Low risk of misreporting or manipulationThis is accomplished via a scorecard (Figure 1, Annex) with the following features:

10 indicators, all observable in a short visit to the clients home All categorical indicators (Does you own a sala set?, not What is the value of your sala set?) Only math is addition (no ratios, squares, logarithms, or exponents) All points are either zero or positive integers (no negatives or decimals) All scores between 0 (most-likely very poor) and 100 (least-likely very poor)Field workers can compute scores by hand, in the field, in real time, without calculators or software.Scores are converted to poverty probabilities via a simple table (Figure 2).

Overall, this design facilitates both predictive accuracy and organizational buy-in.

2. Data sourceThe tool is constructed using the 38,014 households in the 2002 Annual Poverty Indicators Survey(APIS). The 2002 APIS measures only income, not expenditure.

3. Process of indicator selectionIndicators were selected to be:

Inexpensive to collect, easy to answer, and simple to verify Strongly correlated with poverty Liable to change as poverty status changes over time


2/24

2

About 500 indicators were tested, with categories including (Schreiner, 2006a):

Household and housing characteristics (such as cooking fuel and type of floor) Individual characteristics (such as age and highest grade completed) Household consumption (such as spending on non-alcoholic drinks) Household durable goods (such as electric fans and telephones)

About 100 indicators were selected after a basic screen for predictive power using the uncertaintycoefficient of Goodman and Kruskal (1979). These were narrowed to 10 in a stepwise Logit regression.At each step, selection was based on the judgment of an analyst with expertise in both microfinance andscoring who considered both raw predictive power (as measured by c, the logit analog to R2) as well as:

Simplicity and cost of collection Verifiability and susceptibility to strategic falsification Contrast/variety vis--vis previously selected indicators Likelihood of varying as poverty status changes over time Face validity for users (experience, theory, and common sense)Expert judgment is part of all statistical analysis; its use here is not ad hoc, nor are indicators selected apriori. By documenting the use of judgment, the process is more rigorous (susceptible to replication,critique, and improvement) than if there were no acknowledgement of the presence of judgment.

Measures of accuracyPlease see Additional Issues and Schreiner (2005 and 2006b).

Philippines(extreme poverty line is9,060 pesos per month per person, seeSchreiner 2006a)Poverty Rate: 17.0, year 2002

TotalAccuracy

PovertyAccuracy

Under-coverage

Leakage

PIE BPAC

Logit 83.8 61.2 38.8 47.9 9.1 43.7The score cut-off is 3034.

Predicted Very Poor Predicted Not Very Poor

True Very Poor 10.4 9.6

True Not Very Poor 6.6 73.4The score cut-off is 3034. Figures normalized by the population of the Philippines so that the sum of the four cells is 100.

4. Estimating proverty likelihoodsPPI weights (Figure 1) come from logit coefficients (Figure 3). Logit is the scoring industry standardbecause it is simple, easy-to-explain, and robust. For ease-of-use, the logit coefficients were transformedto non-negative integers such that scores range from 0 to 100. The scores are converted to povertylikelihoods via a simple look-up table (Figure 2). It bears repeating that scores can be computed by hand,

in the field, in real time.

Figure 4 is from an out-of-sample bootstrap (Efron and Tibshirani, 1993). It is out-of-sample because ituses households not used to build the scorecard. It is a bootstrap because 10,000 samples were drawnwith replacement. The poverty likelihood associated with a given score is the average (across 10,000samples) share of households with a given score who are poor.


3/24

3

To understand this, consider an example with two bootstrapped samples. The first sample has twohouseholds with scores of 04, one of them very poor. The second sample has three households withscores of 04, all of them very poor. Overall, the poverty likelihood for scores of 04 is then 75 percent(the average of 50 percent in sample 1 and 100 percent in sample 2).

The logit regression and the out-of-sample bootstrap ensure that the estimated poverty likelihoods areobjective; they are derived directly from data in the Philippines 2002 APIS.

Misuse and misreporting is inhibited by transparency. For example, the scorecard is freely available athttp://www.microfinance.com/English/Papers/Scoring_Poverty_Philippines.pdf.If an employee tried to manipulate points, it could be detected because the true points are publicknowledge. Furthermore, clients are unlikely to misreport because the indicators are easy to verify. Ofcourse, employees can cook indicator values regardless of transparency; the only training and follow-upcan mitigate this.

5. Decision rule for classification as very poor or not very poorThe tool associates each household with a poverty probability. In this sense, it does not classify a

household as either 100-percent very poor or 100-percent not very poor.

The portfolio poverty rate (a.k.a. head count index or share of clients who are very poor) is theaverage of all clients poverty likelihoods. Suppose 100 clients have scores of 10 (and poverty likelihoodsof 61.9 percent, Figure 2), 100 have scores of 30 (poverty likelihoods of 35.2 percent), and 100 havescores of 50 (poverty likelihoods of 12.6 percent). The portfolio poverty rate is the average povertylikelihood of 36.6 percent.

This approach estimates that 36.6 percent of clients are very poor, althoughlike the direct-incomeapproachit does not reveal with certainty exactly who is very poor.

If a program wants to use scores for targeting, it could select a cut-off score, for example, 3034 or below

(poverty likelihoods of 35.2 percent or more). For program purposes, these applicants with scores belowthe selected cut-off are labeled qualified (rather than very poor).

Of course, this does not mean that programs set their own poverty lines. Rather, it means that programschoosebased on their own mission and valuestheir desired mix of very poor and not very poorclients. The poverty line used to derive poverty likelihoods in Figure 2 is fixed by the government of thePhilippines and does not depend on how a program uses poverty likelihoods for its own purposes.

For example, suppose a program has 100 applicants with scores of 10 (poverty likelihoods of 61.9percent), 100 with scores of 30 (poverty likelihoods of 35.2 percent), and 100 with scores of 50 (povertylikelihoods of 12.6 percent). If the score cut-off is 30, then the program accepts 200 applicants with anaverage poverty likelihood of 48.6 percent. That is, it accepts 91 very poor clients (income below the

extreme poverty line), and 103 not very poor clients (income above the extreme poverty line). If thescore cut-off were lowered to 10, then the program would accept 100 applicants with an average povertylikelihood of 61.9 percent, 62 of them with income below the extreme poverty line, and 38 with incomeabove the extreme poverty line.


4/24

4

Implementation

We have chosen the household survey and intake survey methodologies for implementing the PPI.

Attached as an annex to this document is GFs training manual for using the PPI.

1. Implementation strategy and data collection methodology: Sampling strategy is outlined in the training manual on page 2 in the section: Whom to Sample:

Defining the Population, Sample Size and Confidence Intervals

Delivering the survey is outlined on p. 4 in the section: Guidelines for Training and Deliveringthe survey.

Adapting the survey to the institutions is covered in the manual on p.6 in the section: Adaptingthe survey to the institution.

Additional issues related to implementation strategy and data collection include the importance ofplanning for doing the work and pre-testing the PPI on a sample of clients. The details for this areoutlined in the manual on p.1 Preparing to Use the PPI and p.6 Pre-test.

2.

Costs in time and money to utilize the tool: Staff time required for delivering the tool, entering data and analysis, auditing and management is

covered in the section on p. 7 bginning with the table: Time required to administer PPI.

Costs incurred to use the tool are outlined in the section: Cost of administering the PPI (excludingsalaries) on p. 8.

The time required of project beneficiaries (MFI clients) is outlined in the table on p. 7 Timerequired to administer PPI and in the paragraphs following the table.

Other costs are required or may be incurred to utilize this tool are outlined in the section on p.8Cost of administering the PPI (excluding salaries).

The costs for data entry are outlined in the section on p. 7 Time required to administer PPI and inthe following sections.

3. Data processing and analysis process: The software and equipment requirements, quality control measures, data entry and cleaning

processes and calculation of aggregate poverty level are outlined in the section on p.8 Datacollection, processing and analysis.

4. Risk Management and Mitigation (covered in the manual on p.8 in the section Risk Managementand Mitigation).

5. On-going issues (covered in the manual on p.9 in the section On-going issues).


5/24

5

Additional Issues

As explained in Schreiner (2005 and 2006b), PIE and BPAC are new measures of accuracy, previouslyunknown in the scoring industry.

How to un-reinvent the wheel? There are three types of accuracy that are relevant:

Accuracy for portfolio poverty rates Accuracy for poverty likelihoods Concentration of the very poor in low scoresThe standard indicator of accuracy for statistics (such as the portfolio poverty rate) is the confidenceinterval (http://en.wikipedia.org/wiki/Confidence_interval). Confidence intervals are well-understood bylay people and are relevant for both the poverty-likelihood and direct-income approach.

For the Philippines tool, out-of-sample bootstrap tests show that, with 99-percent confidence, theestimated portfolio poverty rate is within +/ 1.0 percentage points of the true poverty rate (Schreiner,2007).

Confidence intervals are also the standard indicator of accuracy of poverty likelihoods. The average 99-percent interval for the Philippines is +/3.6 percentage points (Figure 4).

Finally, accuracy increases as the very poor are more concentrated among low scores. The standardmeasure for this is the area under the Receiver Operator Characteristic (ROC) curve(http://en.wikipedia.org/wiki/Receiver_operating_characteristic). The greater the area, the better theconcentration. For the Philippines, this area is 73.4 (out of a possible 100, Figure 5).

Use of the area under ROC as a measure of accuracy assumes equal costs for errors of undercoverage andleakage. BPAC makes precisely this assumption (IRIS, 2005). If these costs are not equal (and in generalthey are not equal), then the standard technique involves benefit-cost matrices, as described for the

Philippines in Schreiner (2006a).


6/24

6

AnnexFigure 1: GFUSA Poverty Tool for the Philippines


7/24

7

In d ica t or

1. 5 3 or 4 1 or 2 Zer

0 7 16 27

2. Does the family own a gas stove or gas

range?

No Ye

0 13

3. How many television sets does the family

own?

Zero 1 2

0 9 18

4. What are the house' s outer walls made of? Strong (iron,

tile, concrete,

wood, as

4

5. How many radios does the family own? Zero 1 2

0 3 106. Does the family own a sala set? No Ye

0 9

7. What is the house' s roof made of? Strong (Galv

aluminum til

brick, stone,

2

8. What kind of toilet facility does the family have? Water

3

9. Do all children in the family of ages 6 to 11 go to school? No Yes No children

0 4 6

10. Do any family members have salaried employment? No Ye

0 6

Source: Calculations based on the 2002 APIS.

Values

Light (cogon , nipa , or sawali , ba mboo,

anahaw)

Light ( Salvaged, makeshift, cogon , nipa , or

anahaw)

None, open pit, closed pit, or other

0

0

0

How many people in the family are aged 0 to

17?


8/24

8

Figure 2: Scores and corresponding poverty likelihoods

Score

0-4 85.0 85.0 83.55-9 79.7 81.3 84.4

10-14 61.9 71.3 85.6

15-19 70.5 71.0 87.0

20-24 53.2 65.4 88.4

25-29 42.4 58.3 90.0

30-34 35.2 52.1 91.7

35-39 23.8 44.8 93.2

40-44 22.2 40.1 94.9

45-49 16.5 36.3 96.1

50-54 12.6 32.4 97.5

55-59 8.4 30.5 98.0

60-64 4.7 26.8 98.6

65-69 2.5 23.5 99.0

70-74 1.7 21.4 99.2

75-79 1.6 19.8 99.6

80-84 0.7 18.1 100.0

85-89 0.0 17.5 100.0

90-94 0.0 17.3 100.0

95-100 0.0 17.0 0.0Total: 17.0

Surveyed cases weighted to represent t he Ph ilippines' populat ion.

Source: Calculations based on the 2002 APIS.

P over t y likelihood

for people with

score in range (%)

% of people

< = scor e

who are poor

% of people

> sco r e

who are non-poor


9/24

9

Figure 3: Raw results from Logit regressionT h e S A S S y s t e m 0 1 : 2 3 W e d n e s d a y , M a r c h 1 4 , 2 0 0 7 1 3

T h e L O G I S T I C P r o c e d u r e

R e s p o n s e P r o f i l e

O r d e r e d T o t a l T o t a l

V a l u e x 2 7 F r e q u e n c y W e i g h t

1 1 5 9 3 6 6 9 6 0 . 6 6 2

2 0 1 2 9 1 0 1 1 8 8 5 . 3 3 8

P r o b a b i l i t y m o d e l e d i s x 2 7 = ' 0 ' .

M o d e l F i t S t a t i s t i c s

I n t e r c e p t

I n t e r c e p t a n d

C r i t e r i o n O n l y C o v a r i a t e s

A I C 2 4 8 2 6 . 1 6 8 1 5 2 2 4 . 8 8 6

S C 2 4 8 3 4 . 0 1 3 1 5 3 5 0 . 3 9 1

- 2 L o g L 2 4 8 2 4 . 1 6 8 1 5 1 9 2 . 8 8 6

R - S q u a r e 0 . 4 0 0 1 M a x - r e s c a l e d R - S q u a r e 0 . 5 4 6 5

A n a l y s i s o f M a x i m u m L i k e l i h o o d E s t i m a t e s

S t a n d a r d W a l d

P a r a m e t e r D F E s t i m a t e E r r o r C h i - S q u a r e P r > C h i S q L a b e l

I n t e r c e p t 1 - 4 . 0 3 4 2 0 . 1 1 5 6 1 2 1 6 . 9 2 1 5 < . 0 0 0 1 I n t e r c e p t : x 2 7 = 1

x 7 1 1 0 . 9 2 0 7 0 . 0 5 6 9 2 6 2 . 2 6 6 8 < . 0 0 0 1 P e o p l e i n t h e f a m i l y o f a g e s 0 t o 1 7 ; 1



x 1 8 1 1 1 . 1 6 2 3 0 . 0 4 8 7 5 6 9 . 0 2 5 2 < . 0 0 0 1 F a m i l y o w n s a g a s s t o v e ; 1

x 1 6 1 1 0 . 8 3 5 5 0 . 0 5 1 3 2 6 5 . 1 7 6 4 < . 0 0 0 1 T V s o w n e d b y f a m i l y ; 1

x 1 6 2 1 1 . 6 5 8 3 0 . 1 4 9 1 1 2 3 . 6 3 6 0 < . 0 0 0 1 T V s o w n e d b y f a m i l y ; 2

x 1 4 1 1 0 . 4 2 8 3 0 . 0 5 9 1 5 2 . 5 7 5 7 < . 0 0 0 1 T y p e o f c o n s t r u c t i o n m a t e r i a l s f o r w a l l s ; 1

x 1 5 1 1 0 . 3 3 0 3 0 . 0 5 0 4 4 2 . 9 1 2 9 < . 0 0 0 1 R a d i o s o w n e d b y f a m i l y ; 1

x 1 5 2 1 1 . 0 2 5 7 0 . 1 0 7 8 9 0 . 5 9 2 6 < . 0 0 0 1 R a d i o s o w n e d b y f a m i l y ; 2

x 1 7 1 1 0 . 8 4 2 3 0 . 0 4 7 3 3 1 7 . 3 5 2 4 < . 0 0 0 1 F a m i l y o w n s a s a l a s e t ; 1

x 1 3 1 1 0 . 1 7 4 7 0 . 0 6 2 3 7 . 8 6 2 7 0 . 0 0 5 0 T y p e o f c o n s t r u c t i o n m a t e r i a l s f o r r o o f ; 1

x 2 1 1 1 0 . 3 2 7 1 0 . 0 5 1 1 4 0 . 8 9 7 9 < . 0 0 0 1 T y p e o f t o i l e t f a c i l i t y ; 1

x 1 0 1 1 0 . 3 8 4 5 0 . 0 9 8 3 1 5 . 2 8 8 2 < . 0 0 0 1 S c h o o l a t t e n d a n c e o f c h i l d r e n o f t h e f a m i l y o f a g e s 6 t o 1 1 ; 1

x 1 0 2 1 0 . 5 5 3 9 0 . 1 0 7 6 2 6 . 4 9 9 1 < . 0 0 0 1 S c h o o l a t t e n d a n c e o f c h i l d r e n o f t h e f a m i l y o f a g e s 6 t o 1 1 ; 2

x 1 2 1 1 0 . 6 3 9 4 0 . 0 4 3 1 2 2 0 . 2 5 4 4 < . 0 0 0 1 N u m b e r o f h o u s e h o l d m e m b e r s w i t h s a l a r i e d e m p l o y m e n t ; 1

A s s o c i a t i o n o f P r e d i c t e d P r o b a b i l i t i e s a n d O b s e r v e d R e s p o n s e s

P e r c e n t C o n c o r d a n t 8 7 . 8 S o m e r s ' D 0 . 7 5 9

P e r c e n t D i s c o r d a n t 1 1 . 9 G a m m a 0 . 7 6 2

P e r c e n t T i e d 0 . 3 T a u - a 0 . 3 2 8

P a i r s 7 6 6 3 3 7 6 0 c 0 . 8 8 0


10/24

1

Figure 4: Confidence intervals for poverty likelihoods

0

10

20

30

40

50

60

70

80

90

100

0-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 7

Range of score

Differenceb

etweenpredictedandestima

ted

pov

ertylikelihood(%pts.)

Upper 99% bound

Upper 95% bound

Upper 90% bound

Mean

Lower 90% bound

Lower 95% bound

Lower 99% bound

Average confidence intervals

across scores, weighted bypersons (+ / - percentage poin


11/24

2

Figure 5: ROC curve indicating concentration of very poor amon

0.00

0.05

0.10

0.15

0.20

0.250.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.800.85

0.90

0.95

1.00

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

P ercentage of all cases, ordered by score

Proportionof

verypoorwithscores

USAID PAT Philippines

Documents

Transcript of USAID PAT Philippines