Final final collaborating on outcomes 30 5 12
-
Upload
bonrosie -
Category
Health & Medicine
-
view
83 -
download
1
description
Transcript of Final final collaborating on outcomes 30 5 12
30 May 2012 Belfast Castle
Philip McDonagh
Chair
Bill OsborneBuilding Change Trust
The Building Change Trust is
a 10 year £10 million charitable fund
endowed by the Big Lottery Fund
to support change and transformation in the Northern Ireland Community and Voluntary sector
Our vision
“A strong, vibrant, independent
and relevant community and
voluntary sector in NI”
CENI - one of the 5 original Trust bid partners along with CFNI, RCN, BITC and VNOW
CENI - significant work on outcome and impact
Measuring Up – identifying needs
Making Reporting more Effective
Use of Quality Standards
Measuring Change approach piloted with BIG, Belfast City Council and Neighbourhood Renewal
Trust actively considering its further and future
role with respect to impact measurement
May appear an esoteric and abstract issue
but
Our vision of a strong, independent and
relevant sector that changes people and places
for the better requires that:
• More cvs organisations are helped to focus on impact
• Cvs organisations understand and use
appropriate tools to set out achievements
Context is
• Evidence based policy making and outcomes based commissioning
• Identification of and investment in what works
• Organisations are striving to make best use of limited resources for maximum benefit
Anticipate continuing to work with CENI
• Exploring use of impact measurement process with government
• Will also be working in partnership with NPC and others to bring benefits of Inspiring Impact to NI
Plan to invest some resources focusing on impact in order to support change for positive development in sector
Not something we can do alone and want to work with government and the sector to take this initiative forward
Keen to hear your views on what action needs to be taken here.
In the first instance contact Trust Administrator, Nigel McKinney
Visit our website
www.buildingchangetrust.org
@changetrust
Will HairePermanent Secretary, DSD
Challenges and issues: a UK perspective from the Inspiring Impact programme
Tris Lumley, Head of Development, NPC
Collaborating on Outcomes: Funders and the Sector Working Together30th May 2012
What’s the context for impact measurement?
• Charities and social enterprises under pressure to demonstrate impact• What should they do to improve?• Driven by funders or for own benefit?• What’s proportionate for them?• Working in isolation, reinventing the wheel
• Funders may want to help• What should they do to support improvement?• Who should pay to increase charities’ capacity?• How should funders think about their own impact?
And impact’s not just about measurement…
If impact measurement is driven by funders…
…it will probably fail to become embedded and really help charities learn and increase impact
We need structural solutions to structural barriers
Why am I here?
• NPC trying to help create structural solutions:• Inspiring Impact—10 year collaborative programme
• Focusing on barriers, solutions and incentives
• At this conference:• To share a UK perspective on impact
• To be part of discussion in NI context
19
What are the barriers to progress?
Providers
Incentives weak (with few penalties for poor impact reporting).
Cultural resistance (at leadership and frontline)
Insufficient resources (money, expertise and capacity)
Technical challenges (don’t have skills and systems to measure; lack of consensus on indicators, methods etc)
Practice (inability to attribute (or predict) an organisation’s contribution to outcomes with confidence)
Funders
Incentives highly variable (and often inconsistent and poorly communicated)
Cultural resistance (to prioritise impact, to spend money on measurement)
Insufficient resources (money and capacity for them and provider)
Technical challenges (skills and systems to identify high impact providers, and collect and synthesise learning and evidence)
Practice: (challenge of building the evidence base over the long term)
Commissioners
Incentives skewed (towards outputs and ‘cost to serve’. Focus on accountability over evidence / learning.)
Cultural resistance (cuts are both a positive and a negative factor)
Insufficient resources (money and capacity for them and provider)
Technical challenges (skills and systems to identify high impact providers, and collect and synthesise evidence)
Practice: (inability to use impact data to drive cost reduction / budget reallocation)
Cross-cutting barriers
What are Inspiring Impact’s solutions?
• We’ve identified five key strands of work
• Leadership and culture
• Coordinating support
• Data, tools and systems
• Shared measurement
• Funder, commissioner & investor practice
Charities & social enterprises
Our initial focus for charities and social enterprises…
What does good practice look like?What’s proportionate?
How do we stack up?How can we improve?
What approach is right for us?Which tools/systems should we use?
How do others do it?What works in our field?
Answers to key questions
What does good practice look like?What’s proportionate?
How do we stack up?How can we improve?
What approach is right for us?Which tools/systems should we use?
How do others do it?What works in our field?
Code of Good Impact Practice
ImpactDiagnostic
Coordinated resourcesMarketplace for tools/systems?
Shared measurement
Concrete deliverables
And for funders themselves…
Capturing different ‘types’ of funder impact
Strategic impact
Practice impact
Funding impact
Strategy for deploying resources to achieve impact
How behaviour as a funder impacts on grantees
The impact on beneficiaries and the community achieved through funding
Plans with funders will emerge dynamically
• Working group of foundations• Identify key aspects
• Explore potential research, actions
• Scope out plans and campaigns for short and medium term
What’s next for Inspiring Impact?
We’re just getting started
• Programme launching June 2012• Establishing links/partnerships across UK• Building advisory groups for key projects• Gathering intelligence on existing initiatives
• Want to hear from you about things we should be aware of and opportunities to engage…
Questions for Speakers
Brendan McDonnell
Director CENI
Collaborating on Outcomes
Introducing
‘Measuring Change’
Recent CENI research shows that:
existing systems focus on counting activities and ensuring financial
probity - not designed to measure outcomes
approaches not standardised so difficult to aggregate diverse project
outcomes to show overall programme impact
prohibitive cost of comprehensive, robust outcome
measurement systems can ‘crowd out’ resources
.
Climate of austerity and increasing social need - challenge for public
funders and projects to measure the outcome of their interventions.
Concordat and PAC report - need for collaboration between Govt & Sector
Context
What are the things you want to Change?
Outcomes
Where are you starting from? Baseline
What difference has been made? Change
Measuring Change
Practical, robust and cost-effective approach to capturing
the hard-to-measure qualitative outcomes of
community-based programmes.
Two distinct but connected components:
1. Developing an overarching framework of programme-
level outcomes
2. Applying an innovative data collection method to
measure project baseline and change against these
outcomes. Underpinned by involvement of both programme and
project stakeholders in a collaborative process
facilitated by CENI.
How it Works
Key Elements
Strategic Focus: Facilitated sessions with Funder to
rationalise a ‘theory of change’ for the programme and
articulate specific change outcomes.
Structure/Themes: Identify key themes which give shape
and structure to an outcomes framework - common format
to locate diverse project-level outcomes.
Stakeholder Involvement: Process informed by input
from both projects and programme funder - fosters shared
ownership of the outcomes framework.
Develop Outcomes Framework
Theme Change Outcomes
People • Enhanced Individual Capacity• Educational Development• Better Healthy lifestyle Choices• Positive Mental Health• Improved Family Cohesion• Improved Personal safety
Community • Better Engagement with Hard to Reach Groups • More Active Involvement in the Community • Enhanced Volunteering • Improved Access to Community Facilities
Organisation • Better Partnership working• More Strategic influence• Enhanced Practice development
Relationship • Better Bonding• More Bridging• Improved Linking
Sample Programme Outcomes Framework
Multiple Perspectives: involves stakeholders (partners,
management, staff, volunteers, users) in a facilitated ‘Expert
Panel’ - maximises involvement and enables ‘triangulation’ of
evidence.
Facilitation: CENI evaluator as a ‘critical friend’ ensures
equality of input across all stakeholders, challenging and
testing their evidence.
Measurement Scale: uses the Rickter numerical scale (0 - 10)
to rate project baseline position and change. Scale helps
neutralise contentious debate and reach consensus.
Critical Debate: facilitates candid discussion and critical peer
review - challenges stakeholders to prioritise what outcomes
are
important and be realistic about their contribution to change.
Data Collection: ‘Nominal Group Technique’
1. Baseline – panels rate their project on each outcome at the start
2. Change – panels rate their project’s progress on each outcome (after
a set time)
Panel discussion is recorded to give context and rationale for the scores
Projects can also rate importance and difficulty of achieving outcomes
Putting a Metric on Qualitative Change
0 1 2 3 4 5 6 7 8 9 10
Low High
An effective way of extracting large amounts of data from
diverse perspectives within a project and distilling this into an
agreed estimate.
Applied systematically across multiple projects it enables the
generation of robust evidence in a standardised format.
This can be analysed:
Horizontally: to baseline and measure the progress of
individual projects against the
programme outcomes
Vertically: to indicate overall programme impact by
aggregating data from individual projects
Data Analysis
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
A B C D E F G H I J
Baseline
Change
Projects
Baseline and Change Scores for Funded Projects
Big Lottery Fund - support grant holders to baseline and measure change.
Neighbourhood Renewal Programme - identify and baseline Community Renewal outcomes.
Belfast City Council - inform review of strategic outcomes and help support change management.
Measuring Change has been successfully piloted by CENI across a number of funding programmes including:
Case Studies
Still a work in progress – ‘Another Tool in the Kit’
Successfully piloted – case studies
Flexible – readily transferable to variety of situations (Age NI, IFA)
Provides mechanism to aid collaboration in both identifying and measuring outcomes
Contribute to wider body of knowledge on evaluation and outcomes measurement
Shift from Counting Activities to Measuring Change
Conclusion
Big Lottery Fund Norrie Breslin
Neighbourhood Renewal Alison Chambers
Belfast City Council Catherine Taggart
Measuring Change Case Studies
Norrie Breslin,
Head of Policy and Learning
Big Lottery Fund NI
Supporting grant holders to
baseline and measure change
Mission and Values
Our Mission
• Bringing improvements to communities and the
lives of people most in need.
Our Values
Making best use of lottery money
Using Knowledge and evidence
Being supportive and helpful
Funding approach
Strategic programmes
Outcomes funder
Partnerships
Working across outcomes
2 stage application process
Development grants
Self evaluation
5 year projects
Programmes
Live and Learn 17 Projects, totalling over £16.2million
Learning and well being
Safe and Well 18 Projects, totalling nearly £17million
Safety and well being
Wide range of projects and beneficiaries
Across geographic and specific communities
The Issue
Varied response to self evaluation
Support for grant holders
Demonstrate impact
Move from programme specific evaluation
The Journey
Change matrix
Communicating to grant holders
Communicating to staff
Encouraging participation
The Outcome
Participation 27/35
Impact on grant holders
Involving wider staff team
NI Committee
BIG UK wide
Next steps
Neighbourhood Renewal
Identify and Baseline
Community Renewal Outcomes
Alison Chambers
Neighbourhood Renewal Neighbourhood Renewal
Strategy launched 2003
36 Neighbourhoods
Over 102,000 households - around 280,000 people
Neighbourhood Partnerships established in each area
Representative of public/private/political and voluntary and community sectors
Neighbourhood Action Plans developed detailing the priority issues to be addressed in each area
NEIGHBOURHOOD RENEWAL – NEIGHBOURHOOD RENEWAL – STRATEGIC OBJECTIVESSTRATEGIC OBJECTIVES
• Community Renewal developing confident communities
• Economic Renewal developing economic activity
• Social Renewal improving social conditions
• Physical Renewal creating attractive, safe environments
Key Outcome Areas
Community Development
Worklessness
Demography
Education
Health
Crime
Physical
Community Renewal
Mid Term Review findings
Lack of Definition and Baseline Position
Inputs, activities, outputs no outcomes
Community Renewal
CENI – Social Assets Research
Outcomes Framework
Measuring Change
Community Renewal
14 outcomes – 5 themes
3 Areas
Expert Panels
Community Renewal
Programme Evaluation
Profile Areas
Next Steps
Capacity Grant: reviewing outcomesCate Taggart
Community Development Manager
Context
• BCC Investment programme• Economic Environment• BCC Community Development Framework• RPA – planned transfer of Regeneration function• Pilot – single CD fund for city
Community Grant Programme
• BCC more than just a grant administrator: both financial and officer support
• Programme across 6 grant categories:- Advice & Information
- Capacity Support- Revenue Support for Community Buildings- C&YP: summer schemes and Ur City 2- Project grants- Community Chest
• £2.6m across 412 grants last year
Service grant programme
Small grants: community development projects Grants for Summer schemes appropriate for children & young people aged 5-
14 years Community chest grant (in support of, for example, small scale seasonal
activity)
Large grants:• Advice services: core funding to cover revenue and other operating costs;
supplementary funding (with DSD) for advice outreach costs• Revenue (financial support towards running costs of community buildings)
• Capacity building (grant funding to support the building of skills and knowledge of community organisations and local people).
Capacity Building Grant recipientswith Neighbourhood Renewal areas
Community Development Framework
• define needs, issues and solutions for their community; and
• influence or take decisions about issues that matter to them and that affect their lives;
• take action to help themselves and make a difference.
Community development activity is the main means by which we can be better engaged with local people and support their involvement in improving the city and its neighbourhoods. It enables people to come together to:
It is a long-term, value-based process which targets positive social change.’
66
Community Development Strategy
Align our resources to achieve new vision and strategy
• Core Community Development work
• Engagement that works
• Building Effective Partnerships
• Shared Service Design & Delivery
CENI pilot (& beyond .........)
Capacity Building Grant elected to pilot the application of the Measuring Change model within a funding programme designed to build the assets of local communities
Ambition to review and agree the strategic outcomes of the programme in light of the CD framework and
To develop an outcomes framework based on Social Assets
To inform the design of related assessment criteria (for a future version of the programme)
To inform the design of a performance management framework (compliance and change)
68
Capacity Review
1. Link supported grant activity to identified programme outcomes
2. Produce Standard Framework of Programme Outcomes;
3. Apply Standard Measurement Tool to:
• Baseline projects against programme outcomes
• Evidence individual contribution to programme
• Enable comparison or relative change across projects, themes
• Aggregate to demonstrate programme impact
4. Support Projects to Use Framework and Measurement Tool
Why ................
Funder Funded organisation
Demonstrate the services we commission are having a positive change on individuals and society and therefore strengthen the case for renewed funding.
Demonstrate potential to funders, board, user groups, recipients .....
Monitor performance / compliance Know project / service is on track
Identify and disseminate practice learning
Contribute to an evidence base of practice
Communicate impact of investment Raise profile (communications)
Motivate staff, elected members, ... Motivate staff and volunteers
69
How...............
• CENI trawl of current applications to list outcomes as stated by funded groups
• Group these outcomes into common themes, using the original SA framework to help shape the approach
• Staff consultation (NGT): clarify purpose and focus and agree priorities
• Iterative process to produce refined strategic aim and draft outcomes framework
What ............
Theme (4) Desired Outcome (11)
Core: Developing Support Group core competency
Enhance Capacity of funded Support GroupStronger PartnershipIncreased leverage
Operational:Developing Local Groups (to be capable, representative and resilient)
Improved local infrastructure of groupsImproved group capacityIncreased group resilience
Operational:Developing Individuals (to be more involved and better skilled)
Increased individual participationImproved individual skillsEnhance volunteering
Strategic:Developing communities (Relationships, services and wellbeing)
Improved Social CapitalImproved Quality of Life
Next steps
• Process of consultation and testing of the framework
• Incorporate revised framework into transition year of CDIP
• Assess framework (evaluate)
• Assess capacity of groups and design and delivery support programme
72
Challenge
• The right strategy
• Engaging and supporting community groups and staff team
• Practicality Test (simple, clear, proportional, transparent, accountable, verifiable, affordable, etc)
o Design assessment model (criterion based)
o Design PMF (practicality test)
• Communication throughout the process
THANKS
What do you see as the main issues / challenges in relation
to funders and the sector collaborating to develop outcome
measures?
How do you think the Measuring Change approach
contributes to collaboration between funders and the sector
around outcomes?
How does Measuring Change complement other
approaches to collaboration on outcomes measurement
currently available to the sector?
Roundtable discussions
Panel Discussion/QuestionsMoving on: Next Steps