Smart urban Facilities monitoring for U-City performance evaluation methods
M&E Frameworks How frameworks guide program monitoring and evaluation.
-
Upload
clark-lancey -
Category
Documents
-
view
219 -
download
2
Transcript of M&E Frameworks How frameworks guide program monitoring and evaluation.
M&E Frameworks
How frameworks guide program monitoring and evaluation
Learning Objectives
Describe function of Conceptual, Logical and Results frameworks
Apply Proximate Determinants Framework to M&E Planning
Use logical frameworks to guide M&E of HIV programs
Apply achievements to results frameworks
Session Overview Function of frameworks Conceptual Frameworks
Proximate Determinants Logic Frameworks
PMTCT program implementation, GFATM Results Frameworks
PEPFAR GFATM M&E Reporting Framework Activity
Developing a logic framework for an HIV program
Discussion
Why do we use frameworks in M&E?
Functions of Frameworks for M&E
Roadmap to program planning, monitoring and evaluation
Delineate clear pathways to program goals & objectives
Define relationships between program inputs, processes, outputs, and outcomes
Describe how program factors interact with external context (environmental factors)
Lead to sound implementation and good M&E plans
Types of Frameworks - Conceptual
Definition Illustrate relationships among all factors (systemic,
organizational, individual) that influence program operation & successful achievement of goals
Purpose Show how program components will operate to
influence outcomes within wider context Guide identification of indicators Guide impact analysis (causal pathways)
Conceptual Frameworks
Makes explicit connections among relevant contextual (environmental) factors and your program
Helps clarify the “why” and “how” questions of program operation and design:
• Assumptions that underlie chosen activities
• Factors activities are expected to affect
• Causal linkages leading to achievement of program objectives
Conceptual Framework Examples:Proximate Determinants
Premise of “proximate determinants” frameworks: Factors leading to health or demographic outcome
operate through an ordered path of determinants Determinants grouped in proximity to outcome
Underlying (socioeconomic) factors farthest away, biological closest
“Proximate” determinants take intermediate position between underlying and biological Proximate determinants are affected by underlying
factors & affect the biological determinants Proximate determinants are what most programs seek to
influence
Proximate Determinants Example: HIV (STI) Incidence1
Underlying determinants Socioeconomic and socio-cultural context (personal and
community) Health programs environment: VCT, ARV, STD control, condom
promotion, IEC Proximate determinants (interventions)
Partner acquisition, mixing patterns, concurrency, abstinence Condom use, STI present, risky practices, STI treatment Treatment
Biological Determinants Rate of contact between susceptible and infected Efficiency of transmission Duration of activity
Outcomes: HIV incidence/mortality, STI incidence
1 Boerma and Weir, 2005
1Boerma and Weir, 2005
Proximate Determinants Framework for HIV/STI1
1 Boerma and Weir, 2005
UnderlyingDeterminants
ProximateDeterminants
BiologicalDeterminants
HealthOutcome
Demog.Outcome
ContextSocioculturalSocioeconomicDemographic
InterventionsVCTSTI controlCondom prom.IECBlood safetySafe injections
Partner acquis.Mixing patternsConcurrencyIDU
Condom useOther STIType of sexNeedle safety
ARV TreatmentSTI Treatment
Exposure ofsusceptible toinfected
Efficacy oftransmissionper contact
Duration of infectivity
HIVinfection Mortality
Disease
1Boerma and Weir, 2005
Ecological Framework:Links between violence against women and HIV1
1 WHO 2010, Addressing violence against women and HIV AIDS: What works?
More risky male partners:
More controlling & violent masculinities
More sexual risk taking
More likely HIV/STIs
Gender & relationship inequality
Rape, child sex abuse &
IPV
Psychological Distress:
Chronic AnxietyDepression
PTSDSubstance use
Reduced protective powers:
More acquiescent femininities
More frequent sexLess condom use
More risky sex:
More partnersConcurrency
Transact. Sex
Sex while intox.
HIV
Rape
Direct transmission
Indirect transmission
Types of Frameworks -Logic Definition
Pictorial diagram that summarizes essential program information by describing its main elements (inputs, activities, outputs, outcomes) & how they work together to affect a desired health outcome
Purpose Make intended program outcomes explicit Show internal logic of program Reveal assumptions of how program leads to outcomes Guide indicator selection Promote communication among stakeholders Guide impact evaluation
Source: CDC Evaluation Guidance Handbook at: http://www.cdc.gov/hiv/aboutdhap/perb/guidance/chapter4.htm
Logic Frameworks - Components Problem statement
The “why” of program implementation Statement what puts a population at risk for HIV
Inputs Program resources like staff, curricula, money
Activities What the program does like outreach, training, testing
Outputs Deliverables of program like people trained or tested
Outcomes Program results like change in knowledge, service use
Impact Long term change in health status like decreased HIV incidence
Source: CDC Evaluation Guidance Handbook at: http://www.cdc.gov/hiv/aboutdhap/perb/guidance/chapter4.htm
Logic Framework Example: PMTCT Program1
Problem Statement HIV rates rising in pregnant women & infants in
Thailand. Transmission risk from mom to child significant during pregnancy & delivery, with additional risk pp via breastfeeding
Inputs Staff, funding from CDC GAP & gov’t, PMTCT
counseling/protocol, HIV test kits, ARV drugs, breast milk substitutes
Activities ANC services include counseling, testing & referral &
referral, distribute ARV prophylaxis, provide infant feeding counseling & support
1CDC M&E Guide, Version 1, Appendix 1
Logic Framework Example: PMTCT Program, cont’d.1
Outputs (per quarter) 500 women participate in a 2 hr. ANC session 250 of these will get HIV test results All (of 500) who test positive will be referred for ARV prophylaxis, infant
feeding C&S Half of all positive women will receive prophylaxis and infant feeding
counseling and breast milks substitutes Outcomes
Increase in access to HIV testing Increase in knowledge of status Increase in knowledge about prophylaxis Increase in ARV prophylaxis use by pregnant HIV+ Increase in rapid infant weaning/use breast milk substitutes
Impact: ↓ HIV incidence in infants, ↓ morbidity/mortality among people living with HIV/AIDS
1CDC M&E Guide, Version 1, Appendix 1
Developing a Logic Model1
Input from all levels of stakeholders Program staff, program participants and others to analyze
intended program outcomes
A good problem statement leads to a good logic model Analyze the problem, program resources, desired
outputs, outcomes and impact Design activities to achieve outputs and outcomes Usually expressed in diagram or table
1CDC, ETA Evaluation Briefs, Logic models, No. 2, Dec. 2005
Types of Frameworks – Results1
USAID funded projects, like a logic framework Definition
A tabular/graphic description of a program’s strategy for achieving a specific strategic objective (SO) through intermediate results, with a narrative supplement.
Purpose Like a logic framework, planning & management tool for programs Specifies out exactly how, through a set of intermediate results, the
program SO will be achieved Provides a basis of consensus for all program stakeholders Conveys the development hypothesis—plausible linkages between
the intermediate results and SO
1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947
Results Frameworks - Components1
Strategic Objective The ultimate end point of the Results Framework—the goal
of the program “Standard by which the operational unit (performance) is
willing to be judged” Appropriate level of ambition means the SO should be
achievable, clear and measurable Intermediate results
Achievable in the short term, lower level than SO An intermediate step leading to achieving the SO Lower level of impact than the SO; cumulative effect leads to
the SO
1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947
Results Frameworks - Components1
Clarification of causal linkages Narrative that shows how intermediate results are linked to
the SO
Links can be between results which then leads to SO
Critical Assumptions Set of conditions necessary under which the development
hypothesis (strategy for achieving the SO) will hold true
Not under project/USAID control, but will affect SO/results
1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947
Results Framework Example – PEPFAR funded ART Program
SO: Utilization of ART services
IR-1: Availability of quality services
IR-2: Demand for services
IR-1.1: Increase ART sites
IR-1.2: Supply sites with ARVs
IR-1.3: Training for providers
IR-2.1: Increase knowledge ofART
IR-2.2: Increase referral from VCT
Developing a Results Framework Set an appropriate strategic objective
Identify intermediate results
Clarify causal linkages between results
Identify critical assumptions
Spend time on the narrative portion so that all stakeholders understand What is trying to be achieved
How to report verifiable achievements
GFATM M&E Framework1
Key Principles Country ownership, harmonization, consistency, balance between
routine & survey data, avoid duplication of efforts Small set of indicators
Focus on multi-agency M&E Toolkit (http://www.theglobalfund.org/en/performance/monitoring_evaluation/)
12-18 indicators per grant Relies on UNAIDS MERG documents/indicators No “Global Fund” indicators (country ownership) Mix of short- & long-term indicators
Quarterly or 6-monthly for some 15-18months for others, important for next Phase
1The Global Fund Monitoring and Evaluation KEY MESSAGES,http://www.theglobalfund.org/en/files/about/structures/lfa/background/LFATrainingMaterials/ME/ME_Key_Principles.pdf
GFATM M&E Framework, contd.1
Avoid duplication Use existing in-country indicators, data systems, planned
surveys
Strengthen existing systems and budget for data gaps
Indicator targets need to match grant proposals
Existing tools for performance evaluation M&E Toolkit, M&E Systems Strengthening Tool, Data Quality
Assessment Tool
Tools developed for GFATM grants, but applicable to many contexts, as we’ll see
1The Global Fund Monitoring and Evaluation KEY MESSAGES,http://www.theglobalfund.org/en/files/about/structures/lfa/background/LFATrainingMaterials/ME/ME_Key_Principles.pdf
New Donor Demands: Gender
PEPFAR, UNAIDS, GFATM, WB require gender integration into programs
Addressing gender in meaningful way = measureable
All agencies have strategies to address gender equality
Many resources out there (we’ll examine some at the end of the week)
Activity: Developing a Logic Framework
Assemble workshop group-work members Use provided CDC-GAP activity sheet:
“Logic Model Worksheet for Planned Implementation and Outcomes”
Take about an hour to complete sheet Resulting logic models to be presented back (5
minutes for each group) by group representative This draft logic model will guide your group work and
form the basis for indicator selection; you will have the opportunity to revise based on feedback
MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for Health,
and Tulane University. Views expressed in this presentation do not
necessarily reflect the views of USAID or the U.S. Government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.