Regional PolicyRegional Policy
How are evaluations used in the EU? How to make them more usable?
Stockholm, 8 October 2015
Kai Stryczynski, DG Regional and Urban Policy
Unit Evaluation and European Semester
Cohesion Policy
1
Regional PolicyRegional Policy
Evaluations in a system of shared management - responsibilities
• Ex ante – Member States (MS)
• During the programming period – MS (and Commission)
• Ex post – Commission with MS
2
Regional PolicyRegional Policy
Lessons from 2007-2013: Evaluation
• Great diversity in practice and quality among 800 evaluations of Regional and Cohesion Fund supported interventions carried out between 2007-2012 and 721 evaluations of the ESF up to December 2013
• Limited but growing numbers of evaluations dealing with effects and impact
Cohesion Policy
4
Regional PolicyRegional Policy
Types of Use of Evaluation
• Instrumental (implementing recommendations)• Conceptual (helping policy makers to
understand)• Enlightenment (accumulating knowledge)• Process use (organizational changes)• Persuasive or Symbolic (accountability or
legitimization)
Saunders 2012
5
Regional PolicyRegional Policy
Maximising the Impact of Evaluation
• Research uncertain – no one way to maximise impact
• Use is diverse, unpredictable and can be political
• But, we can expect maximum influence when it is relevant, timely and of high quality
6
Regional PolicyRegional Policy
We are becoming better – but still have a quality problem
• Insufficient attempt to examine logic of interventions.• Deficiencies in the design and use of methods.• Terms of Reference often poorly specified, Sometimes
bias in the Terms of Reference, with indications given in the Terms of Reference of the conclusions to be drawn.
• A lack of public discussion of findings• (Gaffey, Riché, 2014)
•
7
Regional PolicyRegional Policy
Objectives in OPs – what do you want to change?
• Each priority axis shall articulate what it wants to change. What motivates the policy? Specific Objective
• Observe whether the change is taking place. How to know if policy succeeds? Result Indicator
In other words: the subject of future evaluations
Regional PolicyRegional Policy
9
..because too often it was not clear
"Lack of clarity in policy documents …on the precise objectives which programmes …were intended to achieve and how their achievement was to be verified"
Ex post evaluation 2007-2013, synthesis report, p157
One of the triggers of result orientation 2014 - 2020
Cohesion Policy
Regional PolicyRegional Policy
10
Link to policy monitoring
• Result indicators capture the development of the problem to be fixed – in a region,…
• …and should be monitored
• Interventions are set to contribute to solution – need for evaluation to estimate contribution
• Final step for evaluations – look at contribution and overall picture
Note: Values for beneficiaries are also needed!
Regional PolicyCohesion Policy
Regional PolicyRegional Policy
Impact evaluations
What is required by Regulation?- Evaluate contribution of OP to objectives of each
Priority Axis
What does this mean?- Each Specific Objective should be covered- To differentiate across specific objectives the rigour –
be demanding for new - important objectives - Theory-based and / or counterfactual evaluations, CBA,
…
Cohesion Policy
11
Regional PolicyRegional Policy
Evaluation plans
• Legal requirement – decided by monitoring committee
• Recommendation on content by Commission
• What we hope for from advanced regions:• Innovations• Counterfactual evaluations• Build evaluability into programmes
Regional PolicyRegional Policy
Evaluation plans (2) think about Usability
• Built into the design of the evaluation• Requires reflection on
Reasons and purposes of evaluation Potential users Evaluation objectives The nature of data and evidence Audience for the output Timing of release of evaluation output Agency undertaking the evaluation
Saunders 2012
13
Regional PolicyRegional Policy
14
Steps towards meta-evaluations (?)
•Legal requirement on COM to summarise evaluations from MS
•How to do it?
•Quality assurance?
•Use for future thematic work – ex post evaluation
Cohesion Policy
15
Further EC Support
REGIO/EMPL have put in place a Helpdesk:
•To review evaluation plans
•To provide targeted methodological support for selected evaluations/Member States who request support
•To organise trainings for managing authorities
Cohesion Policy
Further EU support (2)
CRIE: Center for Research on Impact Evaluation
-Focus on counterfactual impact evaluation-Capacity building of MS: training and practical advice-Regional Workshops on CIE Initiative -Accompanying pilot projects in different MS-https://crie.jrc.ec.europa.eu
16
Ex ante evaluations – some question marks
• Are ex ante evaluations still equally justified given the emphasis of impact evaluations during the programming period?
• Did ex ante evaluations deliver value for money? Was result orientation understood / explained well enough? Were evaluators critical enough? Were findings used by programmers?
Governance (responsibility) optimal?
17
2007-2013 ex post evaluations
Cohesion Policy
18
• Ambitious excercise- series of thematic evaluations and synthesis until spring 2016
• Aim of ex post evaluations- analyse the extent to which resources were used, effectiveness,
efficiency and impacts of funding
• Expected use- for accountability purposes to the Court of Auditors, European
Parliament, etc. - for learning purposes for '14-'20 and beyond.
Open data – transparency
• Open Data pilot project https://cohesiondata.ec.europa.eu/
• Data for researchhttp://ec.europa.eu/regional_policy/en/policy/
evaluations/data-for-research/
19
20
Evaluation Conference 2016EU Cohesion Policy at work: a results-oriented policy
Zagreb, 9-10 June 2016
Cohesion Policy
21
Competition for best evaluations and evaluation proposals
Completed evaluations by any individual or company of the effectiveness of EU Cohesion Policy interventions supported by the European Regional Development Fund, the Cohesion Fund or the European Social Fund.
Use at least one of 3 methods:Ex post cost benefit analysisTheory based evaluationCounterfactual evaluation
Deadline for the entries: 31 December 2015
For regular updates on the competition and the 2016 conference, visit the page: http://ec.europa.eu/regional_policy/impact/evaluation/index_en.cfm
Or contact:
Proposal for an impact evaluation of Cohesion Policy
Evaluation not started
Preferably using data in a new way – data on the DG REGIO or EMPL websites or data from a national or regional authority
Open to all individuals and participation of young researchers and PhD students is strongly encouraged
Top Related