8/6/2019 Hgu Ness 2009 Paper
1/26
Theuseandinfluenceofindicators Aconceptualframeworkforresearch
HenrikGudmundsson
Technical University of Denmark, Department of Transport, Bygningstorvet 116 Vest, DK2800
Lyngby,[email protected]
NordicEnvironmentalSocialSciences(NESS) 912June2009,London
Workinggroup9:Knowledgetechnologiesandchanginginstitutions
Abstract
The use of information in policy making has been critically examined in research areas such as
evaluation,planningstudies,andpolicysciencesmorebroadly.Recentresearchhasalsosoughtto
addressthesubsequent influenceof the information,atthe levelof individuals,processesand
institutions.Indicatorsisaparticularknowledgetechnologywithhighpretenceofrelevanceand
usability. While they are sometimes purported as the very essence of needed information,
indicators are hardly immune to noninstrumental roles, or nonuse experienced for other
knowledge technologies. However, to detect and possibly explain indicator influence or lack
thereof inpractice isnosimple task.TheEuropeanFP7 researchprojectPOINThas its focuson
howindicatorsareusedinpractice,andaimstostudytowhatextentandhowindicatorsactually
influence policy processes and outcomes. The main areas of empirical study in the project are
indicators of sustainable development and environmental policy integration in selected sectors.
This paper presents how an analytical framework is designed for the subsequent empirical
research inthePOINTproject,drawing fromadiverserangeof literature. Themainaimofthe
paper is to present and discuss propositions regarding conceptualization and typology for the
studyofuseandinfluenceofindicatorsinpolicymaking.
The research leading to these results has received funding from the European Commission's Seventh
Framework Programme (FP7/20072013) under the grant agreement n 217207 (POINT
project),www.point.pbworks.com
.
1
http://www.point.pbworks.com/http://www.point.pbworks.com/8/6/2019 Hgu Ness 2009 Paper
2/26
1. IntroductionIndicators isa formofknowledgetechnology that iswidelyapplied indifferentareasofpolicy
making. The indicator literature has generally been quite occupied with technical aspects
concerningappropriatemeasurement,representation,andapplicationofindicators,andlesswith
the actual uses, or impacts of indicators in policy making, or governance (Rosenstrm and
Lyytimki2006).
The purpose of the project POINT (POlicy Influence of IndicaTors), supported by the European
Unions7th
FrameworkProgram, istoconductresearchthatcanhelpunderstandhow indicators
areactuallyused,andhowtheybecomeinfluentialornot indifferentmanifestationsofpolicy
making. The thematic focus is on indicators in support of sustainable development and
environmental policy integration, but the aim is really to understand more about indicator
influencegenerallywithinandbeyondthoseparticularempiricalareas.
Thebackgroundfortheprojectiswhatseemslikeawideninggapinthefieldofindicators:
Ontheonehandeffortstoestablishindicatorsystemsareproliferatingatalllevelsofgovernance
fromthelocaltotheglobal,withgeneralaimssuchashelpingtofocusattentiontosalientpolicy
problems,keepingplansontracktowardsfulfilmentofgoalsandtargets,enablingevidencebased
policy processes, holding public managers accountable for results, and sustaining wider public
awarenessofimportantsocietalproblems.
Ontheotherhandtherearegrowingconcernsoverwhethertheseindicatorsystemsareactually
deliveringontheirpromises;iftheymeasuretherightthingsintherightways;ifdecisionmakers
are paying sufficient attention to them, and if policies and communities are becoming
substantiallybetter informed andmanagedasa result. There arealso darkersuspicions about
potentialmanipulativeusesofindicators,wasteofresourcestobuilduselesssystems,andrisks
offosteringperversebehavioursamongsubjectsorentitieswhoseperformanceisbeingmeasured.
2
8/6/2019 Hgu Ness 2009 Paper
3/26
Nevertheless, relatively few broader studies have been made so far of actual indicator use and
influence (with notable exceptions such as Innes 1990; Rydin 2002; Hezri 2006). An important
question concerns how essentially intangible phenomena such as use, and influence of
indicatorscanbemeaningfullydetected,systematizedandstudied, inotherwordshowresearch
inthisfieldcanbemadeoperationalforrecommendingwaystoenhancetheroleofindicatorsin
supporting policies. Published work to date suggests this to be a challenging endeavour with
multipledimensionsandseveralpitfalls,buttheliteraturealsopointstoarichpaletteofconcepts,
approachesandresults,offeredbyavarietyofscientificdisciplines,frommeasurement theory,to
environmentalassessment,tomanagement, topoliticalscience,andbeyond.
ThispaperpresentsworkinprogressbythePOINTprojecttodesignananalyticalframeworkto
guide subsequent empirical research into a number of areas of indicator application. To define
what is understood by framework here it is useful to consider Elinor Ostroms typology of
researchapproaches,distinguishingbetweenframeworks,theoriesandmodels,andaccordingto
which Frameworks organize diagnostic and prescriptive inquiry () They provide the most
general list of variables Frameworks provide metatheoretical language that can be used to
comparetheories ().Itcanprovide..anythingfromaskeletalsetofvariablestosomethingas
extensiveasaparadigm(Ostrom1999,citedfromSabatier1999p262).Analyticalframeworks
have been conceived in related areas such as evaluation use research (Kikhart 2000) and
environmentalassessment(Cashetal2002),butseemstobelessdevelopedintheindictorsarea.
We see themainjustification of the POINT project as seeking tobroaden, deepen and connect
current understandings and approaches of research in and near the field of indicator use and
influence.The framework istoprovide the metatheoretical language,and the general listsof
variables mentioned by Ostrom, which will allow parallel theories to be applied and possibly
compared.Aparadigmisnotonthedrawingboard,norisadistincttheoryormodel.
Theempiricalstudymethodstobeinformedbytheframeworkencompassdocumentanalysis(of
policy documents, indicator reports, meeting minutes etc), facetoface semistructured
interviewswithindicatordevelopersandusers,andsoftsystemsinspired,interactiveworkshops
involvingindicatoruseexperts,practitioners,sceptics,andotherstakeholders.
3
8/6/2019 Hgu Ness 2009 Paper
4/26
Theworkontheframeworkitselfinvolvesaliteraturereview,expert/peerconsultation,typology
building,andtheformationofhypothesisconcerningindicatorinfluencedeterminants.Insection
2ofthepaperanoverviewoftheconsideredliteratureisgiven.Sections35developkeyconcepts
andterminologies.Section6looksbrieflyatasetupofexplanatoryfactors,andsection7provides
outlookforresearchanddiscussion.
2. MainliteraturesreviewedTwokeyquestionshavebeenconsideredwithregardtoliteraturereview:First,whattoaskfrom
theliterature,andsecondly,whichliteraturestoconsult. Concerningthefirstquestionfocushas
beensetonthefollowingtwoissues:
How doesexisting literatureconceptualize indicators with a view to identify their
real useandinfluence?Howarethesetermsdefined,typologised,andconnected
inwhatwewillcalluse influencepathways?
What does the literature suggest as critical to understand or explain use and
influence or lack thereof? Which are operational explanatory factors or dynamics
proposedforindicatorinfluenceindifferentfunctionsorareasofapplication?
A broad literature search and review process has been undertaken looking for references that
couldhelptoaddressthesequestions.Thesearchhasextendedwaybeyondindicatorresearch
as such into broader fields such as of knowledge utilization, evaluation, performance
management and policy studies more broadly. Figure 1 illustrate one way to depict the main
areasofliteratureandhowtheygenerallyinformthearea.
4
8/6/2019 Hgu Ness 2009 Paper
5/26
Figure1.Researchfieldsinformingthestudyofindicatoruseandinfluence.
Wecannotethatinparticularthenorth,east,andsouthpartoftheinnersetofliteraturesin
figure1presentrewardingbutdifferentcontributionstoconceptualizeterminologyandpathways,
while inspiration to develop explanatory causalities etc can be found throughout the whole
landscape. However, as noted by James and Jorgensen (2009) there is an unfortunate divide
betweengeneraltheoriesofpolicychangeandthe knowledgeutilization literature.Theformer
tendtoundervaluethepowerofknowledgeconceptsforpolicychange,whilethelatterrisksto
overextendor fetishize it. Inpracticeseveralreferencesoverlaptwoormoreofthesegroups,
andthedistinctionsarepragmaticandsomewhatarbitrary.
3. Definingtheobject:IndicatorsandframeworksEvenifonlycommonlyrecognized indicatorswithincertainpracticefieldsweretobestudied,an
indicatordefinitionisneverthelessnecessarytospecifytheobjectofwhichanyuseandinfluence
5
8/6/2019 Hgu Ness 2009 Paper
6/26
istobedetected.Thedefinitionshouldespeciallyhelptodistinguish,ifpossible,useandinfluence
ofindicatorsfromthatofothertypesofinformation(suchasstatistics,evaluations,modelsetc).
Thereisaconsiderableliteratureaboutindicators,wheremanydefinitionsareproposed(Moldan
etal2005,Hammondetal1995;Morse2004;OECD2003;Franceschinietal2008;Gallopin1997;
1996). However, many indicator definitions are concerned with ideal indicators, and are
thereforephrasedinlanguagealreadyloadedwithapretenceofindicatorsfunctionalityanduse.
Wewishtostayclearoftoostrongimplicitassumptionsinthisregard.
Asaworkingdefinitionweconsider indicatorssimplyasvariables(Gallopin1997),butonesthat
are constructed or selected to operationally represent properties of more or less well defined
representationtargets(Franceschinietal2008),withanaimtoallowsimplifiedcommunication
about them. In operational applications indicators assume numbers orvalues for the variables.
Judgments,orevaluationsoftheobservedrepresentationmayalsobeincludedwithanindicator.
Indicatorscaninthiswaysbeseenasdistinctfromotherknowledgeconcepts,suchasdataand
statistics,(fromwhichtheymaybebuilt)andfrominformationandknowledge(towhichthey
maycontribute)asdepictedinfigure2.
Communication
Data
Information
Knowledge
Statistics
Indicators(variables + values)
Figure2.Indicatorslogicalpositioninasetofbasicknowledgerelatedconcepts
It is troublesome for the research that the same content may be present in several different
knowledge forms or technologies in the policy processes. Thecontent may pass from data, to
statistics, to general information, at the same time as being contained in indicators. To
6
8/6/2019 Hgu Ness 2009 Paper
7/26
distinguishinfluenceofindicatorsfromthatofotherknowledgeconcepts,twofurthernotionscan
beintroduced,thatofrepresentationaltechniques,andthatofindicatorframeworks.
Examples of typical representational techniques are illustrated in figure 3. They may include
visualisationaswellasnarratives.Eachpartcanemphasizethevariable,thevalue,or(ifapplicable)
the evaluation, or all at once. Reference to particular representational forms of the knowledge
content in, say, policy documents, may help to distinguish when observed use or influence is
genuinelyindicatorrelated.
XXX is decreasing
Variable is about
Value says
Evaluation shows
Headline
Table
Graph
Signifier
Text ImageXXX is decreasing
Variable is about
Value says
Evaluation shows
XXX is decreasing
Variable is about
Value says
Evaluation shows
XXX is decreasing
Variable is about
Value says
Evaluation shows
Headline
Table
Graph
Signifier
Text Image
HeadlineHeadline
Table
GraphGraph
SignifierSignifier
Text ImageImage
Figure3.Differentwaystoexpressindicatorvariable,valueandevaluation.
Policyindicatorsdonotstandalonebutaretypicallyusedaspartofsets,systemsorframeworks
of indicators. Indicatorframeworksembody severaldifferent indicators,structuredaccordingto
one or more principlesorconcepts (Becker 2007; Gudmundsson 2004;Lyytimki &Rosenstrm
2008).Frameworksare importantherebecausetheyobviously frame,anddelimitthewaythe
individualindicatorsarereported andpossiblyhowtheyareperceivedandapplied.Frameworks
may therefore direct, change, or even dominate the use and influence of individual indicators.
Evidence of the presence of an indicator framework in policy communication could therefore
suggest(setsof)indicatorsbeinginfluential. However,thereisnostandarddefinitionofwhatan
indicator framework consists of or how it is identified, let alone how it influences policy.
Gudmundsson
(2004)
proposes
to
distinguish
between
conceptual
(inner),
and
application
(outer)
7
8/6/2019 Hgu Ness 2009 Paper
8/26
partsof indicatorframeworks.Theinner,conceptualpartorganizethecontent(representingthe
dimensionsofthetargetwithappropriatevariablesetc).Examplesincludecausalframeworks(P
SR,DSR,DPSIR);domainframeworks(environmental,social,economic),sectoralframeworks,
etc(Mclaren1996).Applicationpartslinks indicatorstodifferenttypesoffunctionorpurposeof
theindicators.Distinctionsinpurposecanbemadesuchasbetweeninformationversus control
oriented frameworks (Gudmundsson 2004); or according to different stages in a policyprocess,
suchasexanteassessmentorexpostevaluation(EEA1999;Hezri2005).
As an example on can contrast systemoriented, versus performance oriented frameworks.
System frameworksmeasure,and informpolicyabout, relevantpropertiesof thesystems that
thepoliciesaresupposedto intervene inorprotect (economy,transport,environmentetc).The
causal DPSIR framework used for environmental reporting is an example (EEA 1999).
Performance frameworks are different in the sense that they seek to measure what or how
someone,e.g.anorganizationisdoing(itassumesagency).Focusisinmeasuresofinputtoand
output form the organization, and its efficiency and effectiveness. An example is the Policy
evaluation framework used to evaluate European expenditure programmes (Nagarajan &
Vanheukelen, 1997). It COULD be expected that a strong system framework conceptually
influencesthegeneralproblemperceptionofthe indicatedtarget,while it is less likelytoassert
directinfluenceonagencybehaviour;converselytheperformanceframeworkmay putpressure
onspecificactions,whilegeneratinglessintermsofnewideas.
Table 1 suggests a range of dimensions and categories to typologise characterise indicator
frameworks.Whattheymeanfor indicator influenceremainstobestudied,andtheirusefulness
for empirical research is untested. Table 2 depicts a typology of individual indicators extended
fromonemadebytheEuropeanEnvironmentAgency(EEA1999).Thetypologymaybeusedto
suggestifeitherasystemoraperformanceorientedframeworkispresent.Inpractice,however,
theframeworksareinfactmixed
8
8/6/2019 Hgu Ness 2009 Paper
9/26
Table1Fourdimensionsofindicatorframeworks,withexamplecategories1.Conceptualisations 2.Applications
Causal(DPSIR,PSR) Domainbased(env,soc,econ) Sectoral(transport,agriculture,energy) Issuebased(differentissues) Goalbased(setofgoals) Performancebased(organisation; program)
Information(generalinformation) Appraisal(exante) Monitoring(duringprogram) Evaluation(expost) Assessment(comprehensive) Control(steeringtowardsgoal)
3.Aggregations 4.Logistics
Disaggregate(multipleindicators) Semiaggregate(aggregationwithinatheme) Aggregate(aggregationacrossthemes)
Adhoc(Custombuiltforoneevent) Partlyorganized(someroutines) Highlyorganized(extensiveroutinesforregularcyclic
reporting
Table2ExtendedIndicatortypology
Generic types
Systemindicators Performanceindicators
Descriptive Disaggregatedescriptionin
time/spaceInput
Normative Referencetosystem
limits/thresholds
Effectiveness(targetfulfillment)
Ratio Decomposition Organisationalefficiency
(input/output)
Composite Overallsystemconditionin
time/space
Aggregateoutcome
4. UseandinfluenceTraditionallyindicatorshavebeendiscussedmostlyinwhatcanbecalledarationalpositivistor
instrumental mode (Boulanger 2007), with assumed policy functions related to target setting,
decisionmakingandevaluationofresults.However,overthelastdecadeorsoalternativeviews
onthepolicyfunctionsofindicatorshaveemerged,especiallyinresearchareaslikesustainability
indicatorsandperformancemanagement(Cobb&Rixford1998;Meadows1998;Innes&Booher
2000, Hammond et al 1995; Eckerberg & Mineur 2003; Behn 2003). Innes (1998) emphasize
9
8/6/2019 Hgu Ness 2009 Paper
10/26
especiallythecommunicativerolesofindicatorsinpolicyandplanning,whileRydin(2002)discuss
broader roles such as raising awareness, educating the public and motivating civic action.
Meanwhile performance measurement scholars have highlighted cultural, legitimizing or even
ritualisticfunctionsofperformanceindicators(Collier1996;Feldman&March1981).
Essentialinthisnewindicatorliteratureisapropositionthatindicatorsarenotbestunderstood
as tools in themselves, but must be studied rather in terms of their role in broader policy
processesthey inform.Boulanger(2002) introducetwoalternativepolicymodels,whichhecalls
theDiscursiveconstructivistandtheStrategicmodel.Intheformermodelindicatorsareusedin
processestoreframeproblems, identifysharedvaluesandbuildacommondiscourse,exactlyas
depicted by Innes, Rydin and others above. Indicator qualities such as communicability and
dramatization and resonance (Mitchell 1996), become important, even if this may mean that
indicators loose accuracy and utility in amore strict positivist sense (Cobb &Rixford1998). In
Boulangers third model indicators are part of a conflictual context with perpetual competition
betweeninterests.Theuseofindicatorsisreducedtothatofmanipulationandplayalimitedrole.
Tosomedegreethesenewperspectivesonindicatorusabilitydrawfromearlierdevelopmentsin
broaderfieldofstudiesofknowledgeandevaluationutilization(Weiss&Bucuvalas1980;Weiss
1979,Shulha&Cousins1997).Theearlyresearchliteratureinthisareafocussedprimarilyonthe
potentialfordirectuseofresultsfordecisionmakingandproblemsolving.However,oftenlittle
evidenceofsuchusewasfoundinpractice(Weiss1998;Vedung1997);instead,theattentionof
researchers gradually became alerted to other unintended ways in which the evidence was
treated. A prominent expression was Carol Weiss identification of six forms of research
utilization(Weiss1979) seeTable3.Thestudiesgenerallyshowed,notthatknowledgedriven
orproblemsolvingusesofresearchdidnotoccuratall,butweretypicallyfoundin.relatively
lowlevel,narrowgaugedecisions.(Weiss1979,p428).Theseobservationsrecognizesthatpolicy
anddecisionmakingareessentiallysocialprocesses,whicharelikelytostronglyshapeandframe
any knowledge use taking place, rather than the other way around. Certain know ledges are
already so strongly embedded in the minds, or discourses, of policy actors that major new
informationcannotbedirectlyabsorbed(Sabatier1987;Schn&Rein1994).
10
8/6/2019 Hgu Ness 2009 Paper
11/26
Table3.FormsofResearchUtilization(Weiss1979,hereadaptedfromBowen&Zwi2005)
TheKnowledgeDrivenModel:Thismodelsuggeststhatemergentresearchaboutasocialproblemwill leadto
directapplicationtopolicy;itreliesoneffectivestrategiesforthetransferofresearchevidenceintopractice.
TheProblemSolvingModel: This model expects research to provide empirical evidence and conclusions that
help solve a policy problem; it assumes that evidence is systematically gathered and applied in the policy
process.
The Interactive Model:Thismodelsuggeststhatthesearchforknowledgemovesbeyondresearchtoincludea
varietyofsourcessuchaspoliticsandinterests;itaimstoreflectthecomplexityofthepolicymakingprocess.
ThePoliticalModel:Inthismodel,decisionmakersarenotreceptivetoresearchunlessitservespoliticalgain,
thatis,demonstratesproofforapredetermineddecision;evidenceissoughttojustifytheproblem.
TheTacticalModel:Thismodelseesevidenceusedtosupportandjustifygovernment inaction,orrejectionof
anddelayincommitmenttoapolicyissue
TheEnlightenment Model:Thismodelsuggeststhatcumulativeresearchshapesconceptsandperspectivesthat
permeatethepolicyprocessovertime,influencinghowpeoplethinkaboutsocialissues.
It seems plausible that similar ways to use indicators, compared to evaluations or research
resultscouldapply,eveniftheindicators,asdefinedpreviously,areseemlikedistinctknowledge
technologies. The enlightenment model appears for example to be in line with observations
about inthebroadly communicativeusesof indicators found instudiescitedabove.Thisseem
especially applicable to understand influence of indicators as variables, and in the form of
indicator frameworks, both of which may be assumed to shape concepts in policy making,
particularlyinconnectionwithrecurringsystemsoverseveralcycles(Innes1998).Theknowledge
driven and problem solving models of Weiss fits on the other hand perfectly with the
instrumentalfunctionsofindicators,andtheuseofthevalueslayer.Thesemayhoweverbeless
influentialthanexpected,unlessbackedbystrong institutional frameworks.Wewilldiscuss the
use concepts in relation to indicators a little closer in the following, adopting the modified
typology which is now most commonly applied (Amara et al 2004; Shulha & Cousins 1997;
Romsdahl2005).
11
8/6/2019 Hgu Ness 2009 Paper
12/26
1)The Instrumentalmodel involves applying knowledge in specific, direct ways, for indicatorscentringonthevaluesandevaluations.Bydiscoveringtheresultsofgovernmentinterventions,an
indicatorcould(1)helpgovernmentsdecidewhetherornotcontinuewithparticularpolicies;(2)
expandandinstitutionalisesuccessfulprogramsandpoliciesandcutbackunsuccessfulones;and
(3)findoutwhichprogrammesorpoliciestomodifyand inwhichmanner. Incaricature,policy
makers aiming at clearly established policy goals would react to timely published indicators,
adjustingpoliciesaccordingtotheinformationprovidedwiththevalueoftheindicators.
2)TheConceptualmodelinvolvesproducedknowledgebeingusedinamoregeneralform,moreindirectly and less specifically than in instrumental use. The indicator variable, or possibly the
indicator framework,hereprovidesgeneralbackground information, leadingto enlightenment,
thepercolationofnewinformation,ideasandperspectivesintothearenasinwhichdecisionsare
made(Weiss1999,471).Theindicators(or,acontinuouslyoperatedindicatorset), maythereby
affectdecisionmakersproblemdefinitions,andprovidenewperspectivesonandinsightsintothe
problem area, instead of providing information for a single moment of decision, or to a
hypothetical single decisionmaker. An effective indicator framework may contribute to re
conceptualizeorredefineanareofpolicymaking,orevenestablishanew fieldof intervention.
Indicator use becomes a learning process, and the knowledge created through the process
becomesutilisedwhiletheprogramisrunning.
3) The Politicalmodel encompasses three related subcategories: Legitimisation, which hasfrequentlybeenseeninanegativelight,knowledgeprovisionservingasarationalisticritualaimed
atjustifyingdecisionsthathavealreadybeentakenorpoliciesalready inplace(Lampinen1992,
3037;Weiss1999,477).ThiscorrespondstothethirdmodelofBoulanger(2007),wheretherole
ofindicatorsisnegligible. However,explicitlegitimisationcanalsobeseenanessentialelementto
secureacceptanceofpolicy inademocracyand thereforenotpurely bad (e.g.Feinstein2002,
434;Valovirta2002).Tacticaluse,isaboutknowledgebeingcommissioned,forinstance,primarilyinordertopostponeoravoidunwanteddecisionmakingbyreferringtoongoingorplannedwork.
(Vedung 2001, p 141). It seems plausible that such a use could be made of an indicator set,
particularlyasthecommissioningmay involveconceptualisationand logisticalsupport,requiring
12
8/6/2019 Hgu Ness 2009 Paper
13/26
considerableeffortsandtime. Symbolicusemeansknowledgeresearchbeingusedsymbolically,toconveyanimageoramessageandnothingelse(Weiss1999,477). Indicatorswouldinsucha
caseconstituteasimulacrumintendedtogivetheimpressionofarationalorganisationthatsets
goals,hasaseriousandcompetentmanagement,andtakesrationaldecisionsonthebasisofdata.
Thisscenarioisnotunlikelyeither(Pollitt1998;Weiss1999,472473;Vedung2001,141).
In subsequent studies and debates several further types of use have been proposed. For
exampletheobservationthatanknowledgegeneratingprocessitself(ratherthantheresults)may
bethemostimportantfortheactors;leadingtosocalledprocessuse(Shulhaetal1997).Again
this seems like a plausible effect with regard to indicator frameworks, in the case where this
develops over an extended period, in a collaborative process, involving both producers and
varioususersoftheindicators,asreportedbye.g.Innes(1998);Holden(2008;andRydin(2002).
Processuse of an indicator framework may lead toa new practiceof referring to the topic in
measurementterms,evenifthereisnodirectagreementovervalues,variablesorevaluations.
Imposeduse isyetanothertypeinventedinWeissetal(2005).Imposedusereferstosituations
inwhichtheuseofresearchknowledge ismadeobligatory,throughreportingrequirements,for
instance.Thereareseveralsuchexamplesinthecaseofindicators,forexamplelegallymandated
monitoringandevaluationtasksinconnectionwithEuropeantransportandagriculturalpolicies.
Ofinterestarefinallyalsodistinctionsbetweenintended(orlegitimate)useandunintendeduseor
misuse. Cousins (2004) proposes a 4 field typology. He places all the three canonized uses,
instrumental,politicalandconceptual,underthefieldofideal(legitimate)use.Healsoenvisage
legitimatenonuse.Onthenegativesideheputs abuse(illegitimatesuppression)andmisuse
(mistaken use and deliberate manipulation) as three other main categories of use. Similar
concepts have been applied in indicator research. Feller (2002) and Halachmi (2002) discuss
examples of misuse of in performance measurement. Arndt & Oman (2006) reveal extensive
abuseintheareaofgoodgovernanceindicators,meaningopinionbasedmeasuresofissuessuch
as corruptionand politicalstability, inwaysnotdeemedrobustortransparent.However, it is
likelytoinvolvedifficultiestodistinguishclearlybetweenwhatisuseandmisuseofanindicator,
asthismaydependonwhosperspectiveisadopted.
13
8/6/2019 Hgu Ness 2009 Paper
14/26
Sofarwecannoticeawildlyexpandingnotionoftheuseofindicators,startingfromarelatively
clear idea of the indicator as a representational knowledge technology with instrumental
functions, moving gradually into a wide field of different more or less operational uses and
misuses of indicators. Substantial critique of this blurred use notion has been raised by
knowledgeandevaluationutilizationscholarssuchasRich(1997),Kickhart(2000)andHenryand
Mark(2003;2004).AccordingtoRich,usehasbecomemuchtooambiguousaterm,whileitdoes
notevenencompassalltherelevantaspectsofhowknowledgecanmakeadifferenceforpolicy.
Heproposestoconsideruseassomethingmuchsimplerlikeifinformationhasbeenreceivedand
read.AccordingtoHenryandMark(2003)theparalleluseofnormativeanddescriptivemeanings
ofuseishighlyproblematic.Theyalsopositthat,theideaofenlightenmentorconceptualuse
does not provide reasonable specificity to guide measurement of evaluations outcomes or to
insure that casual observers or researchers are talking about the same outcomes. (Henry and
Mark2003,p311). Theysuggestinsteadtostudythevariousconsequencesthatmaystemfrom
evaluations/knowledgeuse,includingeffectssuchasattitudechange,andpolicyorientedlearning.
Hencetheysuggesttofocusonevaluationinfluenceratherthanuse.Theycharacteriseinfluence
asthesubsetofevaluationconsequencesthatcouldplausibly leadtowardorawayfromsocial
betterment (Henry and Mark, 2003, 295). The real test of knowledge influence is thus if
contributestoantheimprovementofsocialconditions.
We sympathise strongly with this shift of attention from use to influence, also as regards
indicators. We agree that the overloading of use would make it almost impossible to
comprehendwhat itmeans thatan indicator issomehow usedornot.Wealsoagree that the
consequencesaremoreimportantthantheuseitself.Howeverwewouldnotabandonthenotion
of use completely. Rather, with Rich (1997) we would consider these notions as successive
elementsinlogicalsequenceorpathway,whereusemayleadtoinfluence,whichagainmaylead
toimpacts.
We reserve the notion of use as a term for immediate actions of persons or organizations,
reflectingtheoriginalintentionalmeaningofusingsomething.Wedefinepolicyuseofindicators
as: adoption and operation of an indicator value, evaluation, or variable by a person or group
14
8/6/2019 Hgu Ness 2009 Paper
15/26
involved in a policy situation or process. With inspiration from Rich (1997); Hezri (2006) and
others we propose to distinguish policy use of indicators at four different levels. Each level
representsasteptowardmoreextensiveuse.Eachlevelissplitfurtherintotwosublevels,which
maybeappliedinresearch ifmoredetailiswishedfor. ThelevelsofuseareshowninTable4.
Table4Policyuseofindicators1.Reception
1.1Receive,notice,observe
1.2Forwardtoothers(nochange)
2.Internalapplication
2.1Forownwork(calculation/text)
2.2Useininternal communication
3.Externalapplication
3.1Communication withotherpolicyinstitutions
3.2.Communication withstakeholders
4.Decisionsupport
4.1Useinofficialpolicyplan/report/document
4.2Useformakingajudgmentanddecision
This does not mean we wish to loose the highly relevant insights concerning instrumental
conceptual,politicaluseetc,butratherthatseseeaneedtorephrasethoseeffectsfrombeing
understood as uses to be depicted rather as analytical categories requiring considerable
interpretation.Weredefinethoseas potentialrolesindicatorsmayhave,andwillconsiderthem
aspossiblefinalstepintheresearch,ratherthananimmediateone.
Asregards influencewesimplyadopttheextensivenotionsofHenry&Mark(2003),whichhave
developedforresearchonevaluations,asweassumecouldbeapplicableforindicatorframeworks
as well. They characterize influence as processes that can occur through a variety of pathways
takingplaceatthreedistinctlevels:individual,interpersonalandcollective(seefurthertable5),
individual level: changes in attitudesand individualbehaviour.Six subcategoriesof
change are defined at this level (e.g. perceived salience of an issue; acquisition of new skills
through the evaluation etc). The influence on this level depends on expectations, interests,
informationalbackgroundsandworldviewsofindividuals;
15
8/6/2019 Hgu Ness 2009 Paper
16/26
interpersonal: concerns the role of evaluations in the processes of argumentation
anddialogueamongactors,e.g.persuasion,legitimisation,criticismordefence,
collective: outcomes in terms of policy decisions and actions, shared beliefs and
understandings,legitimacyofpoliciesandactors,agendasetting,andnetworkformation.
Table5PolicyinfluenceofindicatorsIndividuallevel
ExamplesadaptedfromHenry&Mark(2003)
Attitudechange Changeopinionaboutfeasibilityofa program
Salience Increaseawarenessaboutanissue
Elaboration Stimulatenewexpectations
Priming Makeanissueappearmoreimportantbyhighlighting
Skillacquisition Learningnewwaystomeasure aproblemBehaviorchange Adoptingnewpracticefollowingshowingpositiveresults
Interpersonallevel
Persuasion Moreconvincingargumentationviaofficial report
Justification Reinforcingarguments infavor ofsolution
Changeagent Mobilizestakeholderstoactivelypursuechange
Minorityopinioninfluence Informationprovidesminoritywithdecisivearguments
Socialnorms Informationmakesnewbehavior seemappropriate
Collectivelevel
Agendasetting Informationviamediabringsanissuetotopofagenda
Policyorientedlearning Advocacycoalitionmodifiesrecommendations
Policychange Newpolicyreformadoptedaftergrowingevidence
Diffusion Evidenceofpolicysuccessstimulatesadoptionelsewhere
However,wedonotsetheneedtoacceptHenry&Marksrestrictionof influenceassomething
leadingtosocialbetterment.Thisnotionexcludesinfluenceleadingtonegativeresults,whichare
however also interesting. And it may be hard to determine if social betterment in fact occurs
formtheuseof indicators.Still,weare interested in socialbettermentbutsuggesttoconsider
thisasonepossibleoutcomeorimpact.
WethereforealsoproposetokeepinfluencedistinctfromImpacts.ImpactcanbeseenwithRich
as isdifferentas it refers to thatinformationwasusedand it leddirectly toadecisionorto
action(Rich1997p15).SEDL(2008) definesitasBroad,societallevelchangesorimprovements
thattypicallyoccurbeyondtheimmediateinfluenceofprogramactivities.Inbothcasesitrefers
tosomethingmorestrongthaninfluence.Impactwethusunderstandasaresult(intermediate
16
8/6/2019 Hgu Ness 2009 Paper
17/26
or final)of some influence (immediate,near,ordistant, remote);while the influence isprocess
oriented,leadingonlypotentiallytosomeimpact(likeasuseonlypotentiallyleadingtoinfluence).
A further distinction could be made between Direct intentional impacts and indirect impacts
(anything else). Proposed categories are shown in table 6. Despite attempt of clarity, the
distinctionsmightbedifficulttomaintain,andcouldberevisedaspartoftheempiricalresearch.
Table6PolicyimpactsofindicatorsDirectintentionalimpacts
Awarenessofmeasuredaims
Compliancewithdesiredbehaviour
Fulfilmentoftargets
Efficiencyofpolicy
Social/environment/economicconditionsimprovement
Indirectimpacts
Consensus/conflictaboutpolicyaims
Innovationsofsectorstakeholders
Attentiontononmeasuredpolicyaims
Resourcesofmeasuringorganisations
Impactsonotherpolicies
5. PathwaysIt would be possible, according to Henry & Mark, to start almost anywhere in their influence
model,andchartpathwaysbackandforthandacrosslevelsovertime.Inthisway,theysuggest,
many,manypathwaysarepossible(Henry&Mark 2003,p307).Thisisnolesstrueifwe,as
proposed,connecttheinfluencecategorybackwardstouseandforwardtoimpactandfurther
toapotential roleoftheindicator. Intheliteraturegenerallythereareamultitudeofwaysto
depictinteractionsbetweenknowledgeconceptsandpolicyprocesses.Indicatorsthemselvescan
forexamplebefollowedthroughstagessuchasconception,production,application,andrevision
(Keebleetal2002;McLaren1996),andtheflowsdividedintotypessuchasdatadriven,versus
theorydriven processes (Niemeier 2002). At the policy side the heuristic of the policy cycle
providesthemostwellknowlogicwithstageslikeagendasetting,selectionofalternativesand
implementation (Ilner 1984). Scholars such as Sabatier (1999; 1987) have proposed more
advanced ideas about how knowledge input and policy processes generally interrelate or
disconnect over time, as a result of the stability of certain core beliefs. Others again like Innes
17
8/6/2019 Hgu Ness 2009 Paper
18/26
(1990) andHezri (2004; 2005) have zoomed in on how interactionshaveplayedout in cases of
indicators. While there is not much firm knowledge in this research territory, inspiration to
envisagevariousindicatorpathwayscancertainlybefound.
ToPOINTapathwayisnotsimplyasetofstagesinanindicatorslife,orasetofstagesinapolicy,
eventhoughbothcanbepartsofpathways.Pathways illustrate interactionsbetweenpolicyand
indicator(orindicatorframework).Itcanbeproposedtodefineapathwayasaseriesofeventsor
processes through which indicators or indicator frameworks interact with policy processes,
outputs,outcomesorsubsystems.Thelogicallowstoconnectindicatorwithpolicythroughaset
ofterms(use,influence,impactetc)andassumesasequenceoftheseterms,althoughfeedbacks
and deadendsareobviouslyconceivable. Thegeneral idea is illustrated In figure4,whilethe
specificcategoriesusedforpolicyandindicatorstagesherearerandomexamples.
Figure4.Thegeneralideaofinfluencepathways
6. InfluentialordeterminingfactorsThe paper has been occupied with defining core terminology and typologies of indicators,
frameworks,uses,influencesandpathwaysasguidanceforempiricalresearch. Amajorchallenge
will be to understand what drives indicator development into certain pathways; which factors
mayexplaintheinfluenceoccurringornot. Theproblemisnottofindinspirationintheliteratures,
itisrathertodelimitthewide rangeofoptionsofferedtosomethingmanageable.Theinfluence
Policy agenda
setting
Policy trejectory
Policy Design Policy
le
Policy
imgitimization plementation
Indicator
Conception
Indicator
Production
Indicator
Dissemination
Indicator
Application
Pathways of use - influence- impact -
Indicator trajectory
18
8/6/2019 Hgu Ness 2009 Paper
19/26
setupadoptedfromHenryandMark(2003)entailssomeriskofoverconnectingasisoperates
atsuchdifferentprocesslevels(individualstosocieties),digressofchangeandtimehorizons.Each
empiricalanalysiswillneedtospecifyitsowndelimitationsintheseregards,andcustomizeaset
ofpotentialexplanatoryfactorstopursue. SofarPOINThasorganizedthefieldintoatypologyof
domainsoffactors,definedasfollows
The firstdomain iscalledthe indicatorfactors.Thesehavetodowith indicatorsconsideredas
instrumentsoftechnicalrepresentationandcontrol.Indicatorfactorsincludecharacteristicsofthe
information such as relevance, validity, reliability , comprehensibility timeliness,
comprehensiveness, coherence, , and concreteness of policy recommendations. (Niemeier & de
Groot 2008; Vedung, 1991; Van der Meer, 1999; Weiss, 1999, 479;). In some cases indicator
factorsseemtobeconsideredtheonlyimportantonesforitsuse:iftheindicatoriscorrectina
computationalsenseandpresentedcorrectlythenitwillbeusedmore(or,whatgetsmeasured,
gets done as a popular saying goes). As we have noted, even perfect measures could provoke
completelydifferentreactionsoruses.
Theseconddomainiscalleduserfactors.Thisreferstothepeopleandorganizationsinvolvedin
policyprocesses,andhowtheyareinclinedtocallfor,respondtoorapplyindicatorsinparticular
situations. This could again depend on a host of elements such as the users backgrounds,
positions, group affiliations and motivations. For example, some administrators may have
statistical training which makes them more ready to apply certain indicators; other participants
may mistrust certain information sources because is conflicts with their pregiven worldviews;
others again may learn by shifting positions. The users of indicators typically operate in a
complexandnuancedenvironment.Anindicator,nomatterhowwellcraftedandpresented,will
bejustoneelementinamaelstromofcallsforattention,andusersmaybedifferentlyinclinedin
thisrespect.
Thethirddomain is labelled policyfactors.Thisreferstothetypesofpolicydomainandpolicy
task the indicators are supposed to inform (e.g. in terms of what level of complexity or
controversyisinvolved;Hoppe2005),inparticularalsotheinstitutionalframeworksinwhichthe
19
8/6/2019 Hgu Ness 2009 Paper
20/26
users operate the indicators. Institutional frameworks consist of formal and informal rules and
proceduresthatgoverntheprocesses,andtherolesofdifferenttypesof information inagiven
situation.Forexample,theremaybelegalrequirementstoapplyacertainindicatortobenchmark
an organisation against a target, whereas informal codes of conduct may delegitimize this
information in favour of other concerns in a later step in the process. Differences in legal and
administrative cultures among policy sectors or between institutional arrangements in different
countriesareamongthepotentialpolicyfactors(Pollitt2005).
Figure5illustratesthegeneralsetup.ContinuedresearchinPOINTwillexplorehowtothethree
domainsoffactorscontributeandinteracttoshapeinfluence.
Pathways of Use Influence Impact
Figure5.Preliminaryconceptualframeworkofindicatoruseandinfluence
20
8/6/2019 Hgu Ness 2009 Paper
21/26
8/6/2019 Hgu Ness 2009 Paper
22/26
Table7Exampleofanalyticalframeworksetupsub analyis
Background analysis Influence analysis
Policy
background
Indicator
Emergence/
Function
Use Influence Impact Roles
Policycontext Originofsystem Reception Individual Directintended Instrumental
Policyframework Official
function/appli
cation
Internal
application
Interpersonal Indirect Political
Users Typesof
indicatorsExternal
applicationCollective Conceptual
Typesof
framework(e.g
systemor
performance);
logisticalaspects
Decisionsupport
Process
SinglestepMultiplestep
FullstepSinglestep
Multiplestep
Truncated
22
8/6/2019 Hgu Ness 2009 Paper
23/26
References
Arndt, Christiane; Oman, Charles 2006. Uses and Abuses of Governance Indicators. OECD
DevelopmentCentreStudies.OrganisationofEconomicCooperationandDevelopment,Paris.
Becker, Joanna 2007. How Frameworks can help operationalize Sustainable Development
Indicators.WorldFutures,63:137150,2007
Behn, R. D. 2003. Why measure performance? Different purposes require different measures.
PublicAdministrationReview,63,5,pp.588606.
Boulanger, 2007. Political uses of social indicators: overview and application to sustainable
developmentindicators.Int.J.SustainableDevelopment,Vol.10,Nos.12pp1432.
BowenS;ZwiA.B.2005Pathways toEvidenceInformedPolicyandPractice:AFramework for
Action.
PLoS
Med
2(7).
Cash, D., Clark, W.; Alcock, F.; Dickson, N.; Eckley, N and Jaeger, J. 2002. Salience, Credibility,
Legitimacy and boundaries: linking research, assessment and decision making, John F Kennedy
schoolofgovernment,HarvardUniversity.
Cobb, C. & Rixford, G. 1998. Competing Paradigms in the Development of Social and Economic
Indicators.CSLSConferenceontheStateofLivingandthe qualityofLifeinCanada,October3031
1998,Ottawa,Ontario.
Collier,PaulM 2006.InSearchofPurposeandPriorities:PolicePerformanceIndicatorsinEngland
andWales.PublicMoney&ManagementJune2006,pp165172.
Cousins, J.B 2004. Commentary: Minimizing Evaluation Misuse as Principled Practice. American
JournalofEvaluation2004;25;p391397.
Eckerberg,Katerina&Mineur,Eva2003.TheuseoflocalSustainabilityIndicators:casestudiesin
twoSwedishmunicipalities.LocalEnvironment,Vol.8,No.6,591614,December2003.
EEA 1999. Environmental indicators: Typology and overview. European Environment Agency,
Copenhagen.
Eurostat 2009. ESS Handbook for Quality Reports. European Commission. Office for Official
Publications of the European Communities, Luxembourg. URL
http://epp.eurostat.ec.europa.eu/cache/ITY_OFFPUB/KSRA08016/EN/KSRA08016EN.PDF
Feinstein, Osvaldo N. 2002. Use of Evaluations and Evaluation of their Use. Evaluation 8(4) pp.
433439.
Feldman,MarthaS&March,JamesD.1981.InformationinOrganisationsasSignalsandSymbols.
AdministrativeScienceQuarterly,Vol.22,Issue2,pp.June1981pp.171186.
Feller,I.2002.PerformanceMeasurementRedux.AmericanJournalofEvaluation23(4):435452.
Franceschini,F.;Galetto,M.;Maisano,D.;andMastrogiacomo,L.2008.Propertiesofperformance
indicators in operations management. A reference framework. International Journal of
ProductivityandPerformanceManagement,Vol.57No.2pp.137155.
23
8/6/2019 Hgu Ness 2009 Paper
24/26
Gallopin, Gilberto Carlos 1997. Indicators and Their Use: Information for Decision Making.
Introduction.pp1327in:Moldan,B&Billharz,S.:SustainabilityIndicators.Reportontheproject
onIndicatorsofSustainableDevelopment.Wiley,Chichester.
Gudmundsson, H. 2003. The Policy Use of Environmental Indicators. Learning from Evaluation
Research.Vol.2,No.2,2003:112.
Halachmi, Arie 2002. Performance measurement: a look at some possible dysfunctions. Work
Study.Volume51.Number5.2002.pp.230239.
Hammond et al 1995. Environmental Indicators: A Systematic Approach to Measuring and
Reporting on Environmental Policy Performance in the Context of Sustainable Development.
WorldResourcesInstitute,WashingtonDC.
Henry,GaryT. andMark,MelvinM.2003.BeyondUse:UnderstandingEvaluationsInfluenceon
AttitudesandActions,AmericanJournalofEvaluation,Vol.24,No.3,2003,pp.293314
Hezri,A.A2006ConnectingSustainabilityIndicatorsToPolicySystems.Athesissubmittedforthe
degreeofDoctorofPhilosophy.TheAustralianNationalUniversity,OCTOBER2006.
Hezri AA 2005. Utilisation of sustainability indicators and impact through policy learning in the
Malaysian policy processes. Journal of Environmental Assessment Policy and Management 7(4),
575595.
HezriAA.2004.SustainabilityindicatorsystemandpolicyprocessesinMalaysia:aframeworkfor
utilisationandlearning.JournalofEnvironmentalManagement73,357371.
Holden, Meg 2008. Social learning in planning: Seattles sustainable. development codebooks.
ProgressinPlanning69(2008)140.
Hoppe, R 2005. Rethinking the sciencepolicy nexus: from knowledge utilization and science
technologystudiestotypesofboundaryarrangements.PoiesisPrax3,pp199215
Illner, M 1984. On functional types of indicators in social planning. Social Indicators
Research,Volume14,Number3,pp.275285.
Innes, J.E.; Booher, D.E. 2000. Indicators for sustainable communities: A strategy building on
complexitytheoryanddistributedintelligence.PlanningTheory&Practice,1(2),pp.173186.
Innes, J. E. 1990. Knowledge and Public Policy. The search for meaningful indicators. Second
expandededition.Transactionpublishers,NewBrunswick.
IISD 2007. Compendium of Sustainable Development Indicator Initiatives. International Institute
forSustainableDevelopment,Winnipeg.URL:www.iisd.org/measure/compendium/
James, Thomas E.; Jorgensen, Paul D. 2009. Policy Knowledge, Policy Formulation, and Change:
RevisitingaFoundationalQuestion.ThePolicyStudiesJournal,Vol.37,No.1pp141162.
Keeble et al 2002 Using Indicators to Measure Performance at a Corporate and Project level.
JournalofBusinessEthics44,pp.149158
Kirkhart,KarenE2000.ReconceptualizingEvaluationUse: AnintegratedTheoryofInfluence.Pp2
24in:Caracelli,V.J;Preskill,H(eds.):TheExpandingScopeofEvaluationUse.APublicationofthe
AmericanEvaluationAssociation,JosseyBass,SanFrancisco.
24
http://www.iisd.org/measure/compendium/http://www.iisd.org/measure/compendium/http://www.iisd.org/measure/compendium/8/6/2019 Hgu Ness 2009 Paper
25/26
Lampinen, Osmo. 1992. The utilization of social science research in public policy. Doctoral
Dissertation.SuomenAkatemianjulkaisuja4.Helsinki.VAPKkustannus.
Maclaren, Virginia W. 1996. Urban Sustainability Reporting. Journal of the American Planning
Association,Vol62,No2,Spring1996.
Mark,M.M.;Henry,G.T.2004.TheMechanismsandOutcomesofEvaluationInfluence.Evaluation.
Vol10(1)pp.3557
Meadows, Donella 1998. Indicators and Information Systems for Sustainable Development. A
ReporttotheBalatonGroup.TheSustainabilityInstitute,Hartland,VT.
Moldan, B & Billharz, S. 1997. Sustainability Indicators. Report on the project on Indicators of
SustainableDevelopment.Wiley,Chichester.
Morse, S. 2004. Indices and indicators in development. An unhealthy obsession with numbers.
Earthscan,London.
Niemeijer,D.2002.Developingindicatorsforenvironmentalpolicy:datadrivenandtheorydriven
approachesexaminedbyexample.EnvironmentalScience&Policy5 pp.91103.
Niemeijer, David; de Groot, Rudolf S. 2008. Conceptual framework for selecting environmental
indicatorsets.EcologicalIndicators8pp1425.
OECD 2003 Environmental Indicators Development, Measurement and Use. Reference paper,
OECD,Paris.
Ostrom,Elinor1999 InstitutionalRationalChoice:AnAssessmentoftheIADFramework.In:Paul
Sabatier,(ed).TheoriesofthePolicyProcess.WestviewPress,Boulder.
Pollitt,Christopher2005.PerformanceManagementinPractice:AComparativeStudyofExecutive
Agencies.JournalofPublicAdministrationResearchandTheory,16:2544.
Rich,R.F.1997.Measuringknowledgeutilization:Processandoutcomes.KnowledgeandPolicy:
TheInternationalJournalofKnowledgeTransferandUtilization10.pp1125
Romsdahl, Rebecca J. 2005. When Do Environmental Decision Makers Use Social Science?
AppendixAin:Brewer,GarryD. andStern,PaulC.Ed.2005.DecisionMakingfortheEnvironment:
SocialandBehavioralScienceResearchPriorities.NationalResearchCouncil,WashingtonD.C.
Rosenstrm, Ulla; Lyytimki, Jari 2006 The Role of Indicators in Improving Timeliness of
InternationalEnvironmentalReports.EuropeanEnvironment16,pp3244.
Rydin, Y. 2002. Indicators Into Action: Local Sustainability Indicator Sets in Their Context. The
PastilleConsortium,LondonSchoolofEconomics,London.
Sabatier,P.A(ed)1999.TheoriesofthePolicyprocess.WestviewPress,Boulder.
Sabatier, Paul A.1987. Knowledge, policyoriented learning, and policy change, Knowledge:
Creation,Diffusion,Utilization8,pp.649692.
SEDL 2008. Research Utilization Support and Help (RUSH). Southwest Educational Development
Laboratory, Austin. URL: //www.researchutilization.org/learnru/glossary.html. Last Updated:
Thursday,10April2008at12:21PM
25
8/6/2019 Hgu Ness 2009 Paper
26/26
Shulha, Lyn M.; Cousins, J. Bradley, 1997. Evaluation Use: Theory, Research, and Practice Since
1986.AmericanJournalofEvaluation,Feb1997;vol.18:pp.195208.
Valovirta,V2002.EvaluationUtilizationasArgumentation.EvaluationVol8(1),pp. 6080.
VanderMeer,F.B.1999.EvaluationandtheSocialConstructionofImpacts.Evaluation5,387406.
Vedung,E.1997.Publicpolicyandprogramevaluation.TransactionPublishers,NewBrunswick.
Weiss, Carol H. 1999. The Interface Between Evaluation and Public Policy. Evaluation 5 (4), pp.
468486.
Weiss,CarolH.1998.HaveWeLearnedAnythingNewAbout theUseofEvaluations?American
JournalofEvaluation,Vol19,No1,pp2133.
Weiss, C. H., 1987, The Circuitry of Enlightenment: Diffusion of Social Science Research to
Policymakers,Knowledge:Creation,Diffusion,Utilisation8(2):274281.
WeissCarolH.1979.TheManyMeaningsofResearchUtilization.PublicAdministrationReview,
Vol.39,No.5.,pp.426431
Weiss,CarolH;BucuvalasMichaelJ1980.TruthTestsandUtilityTests:DecisionMakers'Frames
ofReferenceforSocialScienceResearch.AmericanSociologicalReview,Vol.45,No.2.(Apr.,1980),
pp.302313.