Eval Research
-
Upload
chesleybeaver -
Category
Documents
-
view
221 -
download
0
Transcript of Eval Research
-
8/7/2019 Eval Research
1/30
Robert Tijssen
Centre for Science and Technology Studies (CWTS)Leiden University
Netherlands
What isWhat is evaluationevaluation ofof
research achievements and impactsresearch achievements and impacts
taking stock of current general issuestaking stock of current general issues
Presentation at the brainstorming meeting on Evaluating Research Outcomes and Impacts
EFC - European Forum on Philanthropy and Research Funding, Brussels 17 April 2008
CWTS
-
8/7/2019 Eval Research
2/30
For Eur opean foun da t ions , eva lua t ion i s an ins t r um ent
[ an d a p r o ce ss ] w h i ch t e ach e s i m p o r t a n t l esso n s t o b e u se d b y
f oun da t i ons , t he i r p a r t n e r o r gan i sa t i ons , o th e r c i v i l societ y
o rgan i sat i ons , and gove rn m en ts .
I n t h i s ro l e , evalua t i on has inc reasing l y been recogn i sed as hav ing
t h e ca p aci t y t o st r e n g t h e n f o u n d a t i o n s a cco u n t a b i l i t y ,
m anagem en t , unde rs tand ing o f t h e i r w o rk s resu l t s, and t h e ir
cred ib i l i t y , by d i ssem ina t i ng pow er fu l l essons abou t t hose resu l t s.
Source: The role of evaluation in the 21th century foundation / Edward Pauly. - Gtersloh : Bertelsmann Foundation, 2005
-
8/7/2019 Eval Research
3/30
Research evaluation
[ Research ] ev al ua t i on i s t h e sy st em a t i c det e rm i na t i on
o f m e r i t , w o r t h , and s ign i f i cance o f som e th i ng o r som eone
[ w he re researc h ( - r e la ted ) act i v i t i es m ade a con t r i bu t i on ]
W ha t k i n d o f resear ch ?
How sys tem at i c ?
W h a t k i n d o f co n t r i b u t i o n ?
Ho w si g n if i ca n t w a s t h e co n t r i b u t i o n ( i f an y ) ?
Ho w d i d t h e g r an t i m p a ct o n t h e co n t r i b u t i o n ?
-
8/7/2019 Eval Research
4/30
Impacts of research grants
Adapted from Lewison (2000)
-
8/7/2019 Eval Research
5/30
Pressure from (government) accounting agencies
towards more detailed financial and legal record
keeping and towards stricter and more detailed
auditing of financial and legal procedures
Similar pressure from in-house administrators,
financial departments, controllers and control units toprovide more (quantitative) information on (interim)
results of research funding
Expected outcomes may have to be specified before
funding is released, monitored and evaluated
Result: bureaucracy, paperwork and the need for(more) performance indicators and evaluation reports
Financial-administrative environment
http://images.google.com/imgres?imgurl=http://bp3.blogger.com/_mJmwQtPmusk/Rqd0f6Tv1vI/AAAAAAAAAus/6nFtJ6-kNr4/s400/red%2Btape.jpg&imgrefurl=http://burningourmoney.blogspot.com/2007/07/red-tape-industry.html&h=290&w=300&sz=17&hl=en&start=8&tbnid=sZes79erVZ2eTM:&tbnh=112&tbnw=116&prev=/images%3Fq%3Dred%2Btape%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
6/30
Objectives-based/evidence-based
approachOr g a n i si n g p r i n ci p les
To meet a priori objectives (goals, targets, milestones, impacts)
Improve effectiveness and efficiency of funding
Purpose
To relate inputs to the attainment of objectives
Methodo log ies
Qualitative (detecting and interpreting value judgments)
Quantitative (detecting and interpreting quantities)
-
8/7/2019 Eval Research
7/30
Objectives-based/evidence-based
approachKey st reng t hs
Common sense appeal
Widely used approach
Focus on explicit objectives
Uses established methods and (measurement) techniques
Key w eakn esses
Objectives are not clearly specifiedNarrow basis for judging the level of attainment of a project or program
Focuses primarily on (short term) results
Elements are (not sufficiently) amenable to quantitative measures
-
8/7/2019 Eval Research
8/30
-
8/7/2019 Eval Research
9/30
Improving and adaptingevaluation methodologies
Introducing pillars of practical wisdom
http://images.google.com/imgres?imgurl=http://www.cortelimprovement.com/Pillars-of-Success-Org-Exc-.jpg&imgrefurl=http://www.cortelimprovement.com/body_organizational_excellence.html&h=353&w=527&sz=44&hl=en&start=1&tbnid=RJ3tmF1xmmVvEM:&tbnh=88&tbnw=132&prev=/images%3Fq%3Dpillars%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
10/30
-
8/7/2019 Eval Research
11/30
Evaluation objectives should focus on
the production, dissemination and
utilization of research-basedknowledge, skills and capabilities
http://images.google.com/imgres?imgurl=http://senatesite.com/blog/uploaded_images/dc1-796917.JPG&imgrefurl=http://senatesite.com/blog/&h=768&w=1024&sz=160&hl=en&start=34&tbnid=MjZwFMiKSgvDoM:&tbnh=113&tbnw=150&prev=/images%3Fq%3D%2522policy%2Bissues%2522%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DNhttp://images.google.com/imgres?imgurl=http://www.mcbassi.com/uploaded_graphics/sub.pic.4.250.300.jpg&imgrefurl=http://www.mcbassi.com/freeresources_guides.html&h=300&w=250&sz=60&hl=en&start=13&tbnid=lcreIjWD8uyPhM:&tbnh=116&tbnw=97&prev=/images%3Fq%3D%2522Measurement%2Bmethodologies%2522%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DG -
8/7/2019 Eval Research
12/30
Focus on the defining characteristicsof scientific research
People
Creativity and ideas
Risk taking
Long term agendas
International competition and/or cooperation
-
8/7/2019 Eval Research
13/30
Trust, but verify
http://images.google.com/imgres?imgurl=http://www.dba-oracle.com/t_oracle_myths_performance_testing_files/image016.jpg&imgrefurl=http://www.dba-oracle.com/t_oracle_myths_performance_testing.htm&h=183&w=251&sz=9&hl=en&start=18&tbnid=jOhB5G1-KVCIIM:&tbnh=81&tbnw=111&prev=/images%3Fq%3DTrust,%2Bbut%2Bverify%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
14/30
Include measurable objectives
http://images.google.com/imgres?imgurl=http://jan.ucc.nau.edu/~d-ctel/OLG/images/GS-Objectives-page_01.jpg&imgrefurl=http://afroditiapostolidou.blogspot.com/2005_12_01_archive.html&h=321&w=258&sz=12&hl=en&start=1&tbnid=1gCZYJfXMMbpcM:&tbnh=118&tbnw=95&prev=/images%3Fq%3Dmeasurable%2Bobjectives%2B%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
15/30
Numbers matter
Measurable criteria and numerical performanceindicators drives behaviors and results
Goodharts Law: when a measure becomes a target, it ceases to be agood measure
http://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://i3.photobucket.com/albums/y99/ElaineSupkis/clnfeb/trade-deficit-disaster-smal.gif&imgrefurl=http://culturelifenewsii.blogspot.com/2006_02_01_archive.html&h=147&w=146&sz=3&hl=en&start=35&tbnid=yJYqOsN0ZNZEDM:&tbnh=95&tbnw=94&prev=/images%3Fq%3DNumbers%2Bmatter%2B%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN -
8/7/2019 Eval Research
16/30
Evaluation toolbox
Use whatever tools and methods
most appropriate to reach all stakeholders
http://images.google.com/imgres?imgurl=http://www.arb.ca.gov/as/laborrelations/supkit/Images/toolbox.gif&imgrefurl=http://wearethebusinessmen.com/%3Fp%3D45&h=480&w=504&sz=13&hl=en&start=3&tbnid=8NOFyaFqzf4YjM:&tbnh=124&tbnw=130&prev=/images%3Fq%3Dtoolbox%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
17/30
Evidence-based evaluations
require well-acceptable metrics
and performance indicators
http://images.google.com/imgres?imgurl=http://www.peter.com.au/photos/photos/2003-03-30-Workshop_14-Measurements.jpg&imgrefurl=http://www.peter.com.au/photos/sungroper.html&h=480&w=640&sz=57&hl=en&start=26&tbnid=qNfMFhvWAeZ98M:&tbnh=103&tbnw=137&prev=/images%3Fq%3D%2522measurements%2522%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN -
8/7/2019 Eval Research
18/30
Past performance
is an acceptable predictor of
current capabilities
and future success
http://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN -
8/7/2019 Eval Research
19/30
Degrees of success and failure
Specify key objectives and performance criteria as clearly as
possible in advance
Apply uniform definitions, measurements, metrics and (inter)national
scales
http://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DNhttp://images.google.com/imgres?imgurl=http://wcm.pavliks.com/WCMAdmin/Images/wwwleanvelocityca/images/bigstockphoto_Past_Performance_766574.jpg&imgrefurl=http://www.leanvelocity.ca/Content.cfm%3FC%3D6973%26SC%3D1%26SCM%3D0%26MI%3D4618%26L1M%3D4368&h=800&w=1200&sz=220&hl=en&start=14&tbnid=niRk8I7P-dCMyM:&tbnh=100&tbnw=150&prev=/images%3Fq%3Dpast%2Bperformance%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN -
8/7/2019 Eval Research
20/30
Keep it simple
Grant Output ImpactActivities
but dont ignore realities !
Other inputs Other outputs Other impacts
-
8/7/2019 Eval Research
21/30
Basket of methods and measures
There is no single best method for researchperformance evaluation
http://images.google.com/imgres?imgurl=http://www.networkworld.com/best/2006/newhedBEST06.gif&imgrefurl=http://www.thetotalhealthshop.com/&h=307&w=436&sz=20&hl=en&start=17&tbnid=Opx2WrVNfQGFiM:&tbnh=89&tbnw=126&prev=/images%3Fq%3Dsingle%2Bbest%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
22/30
Garbage in, garbage out
Check the information value and qualityof the required data for monitoring and evaluation
http://images.google.com/imgres?imgurl=http://www.gocarlo.com/lagalerie/images/alleylamp-512.jpg&imgrefurl=http://mastmalang.wordpress.com/2007/04/05/i-feel-like-a-lamp-post/&h=383&w=512&sz=68&hl=en&start=8&tbnid=-8Be0bUUCX4YeM:&tbnh=98&tbnw=131&prev=/images%3Fq%3Dlamppost%2Band%2Blost%2Bkeys%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
23/30
Include external sources
of independent information
Opinions of subject experts and expert panels
Statistical information (bibliometric data)
Peer review based
http://images.google.com/imgres?imgurl=http://senatesite.com/blog/uploaded_images/dc1-796917.JPG&imgrefurl=http://senatesite.com/blog/&h=768&w=1024&sz=160&hl=en&start=34&tbnid=MjZwFMiKSgvDoM:&tbnh=113&tbnw=150&prev=/images%3Fq%3D%2522policy%2Bissues%2522%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN -
8/7/2019 Eval Research
24/30
Peer review basedquality control systems are not infallible
Results of peer review are highly dependent on theselection of experts and expert panels/committees
Knowledgeable and impartial evaluators are difficult to
find (in small countries/scientific communities)
Critical aspects of research, research environment,technical infrastructure, or details of researchfindings, are difficult to grasp by external experts
Experts lack expertise across all relevant and relatedknowledge domains (interdisciplinary andmultidisciplinary areas)
Experts lacking in newly emerging fields (with manyyoung researchers and newcomers)
Lack of consensus, lack of common frame of reference
Bibli t i i di t
http://images.google.com/imgres?imgurl=http://senatesite.com/blog/uploaded_images/dc1-796917.JPG&imgrefurl=http://senatesite.com/blog/&h=768&w=1024&sz=160&hl=en&start=34&tbnid=MjZwFMiKSgvDoM:&tbnh=113&tbnw=150&prev=/images%3Fq%3D%2522policy%2Bissues%2522%26start%3D20%26ndsp%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DNhttp://images.google.com/imgres?imgurl=http://www.mcbassi.com/uploaded_graphics/sub.pic.4.250.300.jpg&imgrefurl=http://www.mcbassi.com/freeresources_guides.html&h=300&w=250&sz=60&hl=en&start=13&tbnid=lcreIjWD8uyPhM:&tbnh=116&tbnw=97&prev=/images%3Fq%3D%2522Measurement%2Bmethodologies%2522%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DG -
8/7/2019 Eval Research
25/30
Bibliometric indicators
Bibliometric data are based on a large number ofexpert opinions and peer judgements
(peer-review accepted publications and citations
from peers)
Bibliometrics is non-reactive measurement
methodology
(decisions to accept or cite papers are not or
much less - affected by the possibility of future
evaluation)
Due to the large number of peer votes, and their
international scope, the information base is
expanded
(biases and misconceptions of local experts are also
countered)
-
8/7/2019 Eval Research
26/30
Partnerships
Researchers and research organizations
should submit high-quality
(cleaned and harmonized) performance data
Evaluation fatigue and
http://images.google.com/imgres?imgurl=http://www.digilore.com/images/partnersImage.jpg&imgrefurl=http://www.digilore.com/partners.shtml&h=300&w=369&sz=16&hl=en&start=2&tbnid=SDSeV5glWo4VyM:&tbnh=99&tbnw=122&prev=/images%3Fq%3Dpartners%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN -
8/7/2019 Eval Research
27/30
Evaluation fatigue andadministrative burdens
Collect essential information only
Tailor the information need to the relevant issues and contexts
(e.g. fields of science, types of activity, timescales)
http://images.google.com/imgres?imgurl=http://rockycha.files.wordpress.com/2007/09/fatigue.jpg&imgrefurl=http://rockycha.wordpress.com/2007/09/&h=392&w=281&sz=23&hl=en&start=2&tbnid=B6ZIuOwU2cF7yM:&tbnh=123&tbnw=88&prev=/images%3Fq%3Dfatigue%26gbv%3D2%26hl%3Den%26sa%3DGhttp://images.google.com/imgres?imgurl=http://www.antique-hardware.com/key.jpg&imgrefurl=http://www.polyvore.com/cgi/set%3Fid%3D323488&h=315&w=306&sz=13&hl=en&start=2&tbnid=uMmfl9_TVoavCM:&tbnh=117&tbnw=114&prev=/images%3Fq%3Dkey%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
28/30
Learning is the key
Accept trade-offs between accountability andeffectiveness/efficiency of administrative
structures and procedures
Mismatches between information needs and supply
Shift focus from one-off ex-post evaluations ofreturns to ongoing monitoring of capability
development, scientific progress and(sustainable) added values
http://images.google.com/imgres?imgurl=http://www.antique-hardware.com/key.jpg&imgrefurl=http://www.polyvore.com/cgi/set%3Fid%3D323488&h=315&w=306&sz=13&hl=en&start=2&tbnid=uMmfl9_TVoavCM:&tbnh=117&tbnw=114&prev=/images%3Fq%3Dkey%26gbv%3D2%26hl%3Den%26sa%3DG -
8/7/2019 Eval Research
29/30
It takes two to tango
Involve stakeholders and users and in all stagesof the evaluation process
Share tasks and responsibilities
From jury judgment to coaching?
L i
-
8/7/2019 Eval Research
30/30
Learning curves
Focus evaluation on determining the key successfactors (and causes of failures)
Responsiveness to science-related developments
Improving evaluation methods and tools
http://images.google.com/imgres?imgurl=http://www.pms.ac.uk/pphi/institute/masters/images/learningJourneyBig.gif&imgrefurl=http://www.pms.ac.uk/pphi/institute/masters/index.html&h=384&w=581&sz=31&hl=en&start=23&tbnid=NNe7loVVviK1NM:&tbnh=89&tbnw=134&prev=/images%3Fq%3Dlearning%26start%3D20%26gbv%3D2%26ndsp%3D20%26hl%3Den%26sa%3DN