Hgu Ness 2009 Paper

8/6/2019 Hgu Ness 2009 Paper

1/26

Theuseandinfluenceofindicators Aconceptualframeworkforresearch

HenrikGudmundsson

Technical University of Denmark, Department of Transport, Bygningstorvet 116 Vest, DK2800

Lyngby,[email protected]

NordicEnvironmentalSocialSciences(NESS) 912June2009,London

Workinggroup9:Knowledgetechnologiesandchanginginstitutions

Abstract

The use of information in policy making has been critically examined in research areas such as

evaluation,planningstudies,andpolicysciencesmorebroadly.Recentresearchhasalsosoughtto

addressthesubsequent influenceof the information,atthe levelof individuals,processesand

institutions.Indicatorsisaparticularknowledgetechnologywithhighpretenceofrelevanceand

usability. While they are sometimes purported as the very essence of needed information,

indicators are hardly immune to noninstrumental roles, or nonuse experienced for other

knowledge technologies. However, to detect and possibly explain indicator influence or lack

thereof inpractice isnosimple task.TheEuropeanFP7 researchprojectPOINThas its focuson

howindicatorsareusedinpractice,andaimstostudytowhatextentandhowindicatorsactually

influence policy processes and outcomes. The main areas of empirical study in the project are

indicators of sustainable development and environmental policy integration in selected sectors.

This paper presents how an analytical framework is designed for the subsequent empirical

research inthePOINTproject,drawing fromadiverserangeof literature. Themainaimofthe

paper is to present and discuss propositions regarding conceptualization and typology for the

studyofuseandinfluenceofindicatorsinpolicymaking.

The research leading to these results has received funding from the European Commission's Seventh

Framework Programme (FP7/20072013) under the grant agreement n 217207 (POINT

project),www.point.pbworks.com

.

1
http://www.point.pbworks.com/http://www.point.pbworks.com/


2/26

1. IntroductionIndicators isa formofknowledgetechnology that iswidelyapplied indifferentareasofpolicy

making. The indicator literature has generally been quite occupied with technical aspects

concerningappropriatemeasurement,representation,andapplicationofindicators,andlesswith

the actual uses, or impacts of indicators in policy making, or governance (Rosenstrm and

Lyytimki2006).

The purpose of the project POINT (POlicy Influence of IndicaTors), supported by the European

Unions7th

FrameworkProgram, istoconductresearchthatcanhelpunderstandhow indicators

areactuallyused,andhowtheybecomeinfluentialornot indifferentmanifestationsofpolicy

making. The thematic focus is on indicators in support of sustainable development and

environmental policy integration, but the aim is really to understand more about indicator

influencegenerallywithinandbeyondthoseparticularempiricalareas.

Thebackgroundfortheprojectiswhatseemslikeawideninggapinthefieldofindicators:

Ontheonehandeffortstoestablishindicatorsystemsareproliferatingatalllevelsofgovernance

fromthelocaltotheglobal,withgeneralaimssuchashelpingtofocusattentiontosalientpolicy

problems,keepingplansontracktowardsfulfilmentofgoalsandtargets,enablingevidencebased

policy processes, holding public managers accountable for results, and sustaining wider public

awarenessofimportantsocietalproblems.

Ontheotherhandtherearegrowingconcernsoverwhethertheseindicatorsystemsareactually

deliveringontheirpromises;iftheymeasuretherightthingsintherightways;ifdecisionmakers

are paying sufficient attention to them, and if policies and communities are becoming

substantiallybetter informed andmanagedasa result. There arealso darkersuspicions about

potentialmanipulativeusesofindicators,wasteofresourcestobuilduselesssystems,andrisks

offosteringperversebehavioursamongsubjectsorentitieswhoseperformanceisbeingmeasured.

2


3/26

Nevertheless, relatively few broader studies have been made so far of actual indicator use and

influence (with notable exceptions such as Innes 1990; Rydin 2002; Hezri 2006). An important

question concerns how essentially intangible phenomena such as use, and influence of

indicatorscanbemeaningfullydetected,systematizedandstudied, inotherwordshowresearch

inthisfieldcanbemadeoperationalforrecommendingwaystoenhancetheroleofindicatorsin

supporting policies. Published work to date suggests this to be a challenging endeavour with

multipledimensionsandseveralpitfalls,buttheliteraturealsopointstoarichpaletteofconcepts,

approachesandresults,offeredbyavarietyofscientificdisciplines,frommeasurement theory,to

environmentalassessment,tomanagement, topoliticalscience,andbeyond.

ThispaperpresentsworkinprogressbythePOINTprojecttodesignananalyticalframeworkto

guide subsequent empirical research into a number of areas of indicator application. To define

what is understood by framework here it is useful to consider Elinor Ostroms typology of

researchapproaches,distinguishingbetweenframeworks,theoriesandmodels,andaccordingto

which Frameworks organize diagnostic and prescriptive inquiry () They provide the most

general list of variables Frameworks provide metatheoretical language that can be used to

comparetheories ().Itcanprovide..anythingfromaskeletalsetofvariablestosomethingas

extensiveasaparadigm(Ostrom1999,citedfromSabatier1999p262).Analyticalframeworks

have been conceived in related areas such as evaluation use research (Kikhart 2000) and

environmentalassessment(Cashetal2002),butseemstobelessdevelopedintheindictorsarea.

We see themainjustification of the POINT project as seeking tobroaden, deepen and connect

current understandings and approaches of research in and near the field of indicator use and

influence.The framework istoprovide the metatheoretical language,and the general listsof

variables mentioned by Ostrom, which will allow parallel theories to be applied and possibly

compared.Aparadigmisnotonthedrawingboard,norisadistincttheoryormodel.

Theempiricalstudymethodstobeinformedbytheframeworkencompassdocumentanalysis(of

policy documents, indicator reports, meeting minutes etc), facetoface semistructured

interviewswithindicatordevelopersandusers,andsoftsystemsinspired,interactiveworkshops

involvingindicatoruseexperts,practitioners,sceptics,andotherstakeholders.

3


4/26

Theworkontheframeworkitselfinvolvesaliteraturereview,expert/peerconsultation,typology

building,andtheformationofhypothesisconcerningindicatorinfluencedeterminants.Insection

2ofthepaperanoverviewoftheconsideredliteratureisgiven.Sections35developkeyconcepts

andterminologies.Section6looksbrieflyatasetupofexplanatoryfactors,andsection7provides

outlookforresearchanddiscussion.

2. MainliteraturesreviewedTwokeyquestionshavebeenconsideredwithregardtoliteraturereview:First,whattoaskfrom

theliterature,andsecondly,whichliteraturestoconsult. Concerningthefirstquestionfocushas

beensetonthefollowingtwoissues:

How doesexisting literatureconceptualize indicators with a view to identify their

real useandinfluence?Howarethesetermsdefined,typologised,andconnected

inwhatwewillcalluse influencepathways?

What does the literature suggest as critical to understand or explain use and

influence or lack thereof? Which are operational explanatory factors or dynamics

proposedforindicatorinfluenceindifferentfunctionsorareasofapplication?

A broad literature search and review process has been undertaken looking for references that

couldhelptoaddressthesequestions.Thesearchhasextendedwaybeyondindicatorresearch

as such into broader fields such as of knowledge utilization, evaluation, performance

management and policy studies more broadly. Figure 1 illustrate one way to depict the main

areasofliteratureandhowtheygenerallyinformthearea.

4


5/26

Figure1.Researchfieldsinformingthestudyofindicatoruseandinfluence.

Wecannotethatinparticularthenorth,east,andsouthpartoftheinnersetofliteraturesin

figure1presentrewardingbutdifferentcontributionstoconceptualizeterminologyandpathways,

while inspiration to develop explanatory causalities etc can be found throughout the whole

landscape. However, as noted by James and Jorgensen (2009) there is an unfortunate divide

betweengeneraltheoriesofpolicychangeandthe knowledgeutilization literature.Theformer

tendtoundervaluethepowerofknowledgeconceptsforpolicychange,whilethelatterrisksto

overextendor fetishize it. Inpracticeseveralreferencesoverlaptwoormoreofthesegroups,

andthedistinctionsarepragmaticandsomewhatarbitrary.

3. Definingtheobject:IndicatorsandframeworksEvenifonlycommonlyrecognized indicatorswithincertainpracticefieldsweretobestudied,an

indicatordefinitionisneverthelessnecessarytospecifytheobjectofwhichanyuseandinfluence

5


6/26

istobedetected.Thedefinitionshouldespeciallyhelptodistinguish,ifpossible,useandinfluence

ofindicatorsfromthatofothertypesofinformation(suchasstatistics,evaluations,modelsetc).

Thereisaconsiderableliteratureaboutindicators,wheremanydefinitionsareproposed(Moldan

etal2005,Hammondetal1995;Morse2004;OECD2003;Franceschinietal2008;Gallopin1997;

1996). However, many indicator definitions are concerned with ideal indicators, and are

thereforephrasedinlanguagealreadyloadedwithapretenceofindicatorsfunctionalityanduse.

Wewishtostayclearoftoostrongimplicitassumptionsinthisregard.

Asaworkingdefinitionweconsider indicatorssimplyasvariables(Gallopin1997),butonesthat

are constructed or selected to operationally represent properties of more or less well defined

representationtargets(Franceschinietal2008),withanaimtoallowsimplifiedcommunication

about them. In operational applications indicators assume numbers orvalues for the variables.

Judgments,orevaluationsoftheobservedrepresentationmayalsobeincludedwithanindicator.

Indicatorscaninthiswaysbeseenasdistinctfromotherknowledgeconcepts,suchasdataand

statistics,(fromwhichtheymaybebuilt)andfrominformationandknowledge(towhichthey

maycontribute)asdepictedinfigure2.

Communication

Data

Information

Knowledge

Statistics

Indicators(variables + values)

Figure2.Indicatorslogicalpositioninasetofbasicknowledgerelatedconcepts

It is troublesome for the research that the same content may be present in several different

knowledge forms or technologies in the policy processes. Thecontent may pass from data, to

statistics, to general information, at the same time as being contained in indicators. To

6


7/26

distinguishinfluenceofindicatorsfromthatofotherknowledgeconcepts,twofurthernotionscan

beintroduced,thatofrepresentationaltechniques,andthatofindicatorframeworks.

Examples of typical representational techniques are illustrated in figure 3. They may include

visualisationaswellasnarratives.Eachpartcanemphasizethevariable,thevalue,or(ifapplicable)

the evaluation, or all at once. Reference to particular representational forms of the knowledge

content in, say, policy documents, may help to distinguish when observed use or influence is

genuinelyindicatorrelated.

XXX is decreasing

Variable is about

Value says

Evaluation shows

Headline

Table

Graph

Signifier

Text ImageXXX is decreasing

Variable is about

Value says

Evaluation shows

XXX is decreasing

Variable is about

Value says

Evaluation shows

XXX is decreasing

Variable is about

Value says

Evaluation shows

Headline

Table

Graph

Signifier

Text Image

HeadlineHeadline

Table

GraphGraph

SignifierSignifier

Text ImageImage

Figure3.Differentwaystoexpressindicatorvariable,valueandevaluation.

Policyindicatorsdonotstandalonebutaretypicallyusedaspartofsets,systemsorframeworks

of indicators. Indicatorframeworksembody severaldifferent indicators,structuredaccordingto

one or more principlesorconcepts (Becker 2007; Gudmundsson 2004;Lyytimki &Rosenstrm

2008).Frameworksare importantherebecausetheyobviously frame,anddelimitthewaythe

individualindicatorsarereported andpossiblyhowtheyareperceivedandapplied.Frameworks

may therefore direct, change, or even dominate the use and influence of individual indicators.

Evidence of the presence of an indicator framework in policy communication could therefore

suggest(setsof)indicatorsbeinginfluential. However,thereisnostandarddefinitionofwhatan

indicator framework consists of or how it is identified, let alone how it influences policy.

Gudmundsson

(2004)

proposes

to

distinguish

between

conceptual

(inner),

and

application

(outer)

7


8/26

partsof indicatorframeworks.Theinner,conceptualpartorganizethecontent(representingthe

dimensionsofthetargetwithappropriatevariablesetc).Examplesincludecausalframeworks(P

SR,DSR,DPSIR);domainframeworks(environmental,social,economic),sectoralframeworks,

etc(Mclaren1996).Applicationpartslinks indicatorstodifferenttypesoffunctionorpurposeof

theindicators.Distinctionsinpurposecanbemadesuchasbetweeninformationversus control

oriented frameworks (Gudmundsson 2004); or according to different stages in a policyprocess,

suchasexanteassessmentorexpostevaluation(EEA1999;Hezri2005).

As an example on can contrast systemoriented, versus performance oriented frameworks.

System frameworksmeasure,and informpolicyabout, relevantpropertiesof thesystems that

thepoliciesaresupposedto intervene inorprotect (economy,transport,environmentetc).The

causal DPSIR framework used for environmental reporting is an example (EEA 1999).

Performance frameworks are different in the sense that they seek to measure what or how

someone,e.g.anorganizationisdoing(itassumesagency).Focusisinmeasuresofinputtoand

output form the organization, and its efficiency and effectiveness. An example is the Policy

evaluation framework used to evaluate European expenditure programmes (Nagarajan &

Vanheukelen, 1997). It COULD be expected that a strong system framework conceptually

influencesthegeneralproblemperceptionofthe indicatedtarget,while it is less likelytoassert

directinfluenceonagencybehaviour;converselytheperformanceframeworkmay putpressure

onspecificactions,whilegeneratinglessintermsofnewideas.

Table 1 suggests a range of dimensions and categories to typologise characterise indicator

frameworks.Whattheymeanfor indicator influenceremainstobestudied,andtheirusefulness

for empirical research is untested. Table 2 depicts a typology of individual indicators extended

fromonemadebytheEuropeanEnvironmentAgency(EEA1999).Thetypologymaybeusedto

suggestifeitherasystemoraperformanceorientedframeworkispresent.Inpractice,however,

theframeworksareinfactmixed

8


9/26

Table1Fourdimensionsofindicatorframeworks,withexamplecategories1.Conceptualisations 2.Applications

Causal(DPSIR,PSR) Domainbased(env,soc,econ) Sectoral(transport,agriculture,energy) Issuebased(differentissues) Goalbased(setofgoals) Performancebased(organisation; program)

Information(generalinformation) Appraisal(exante) Monitoring(duringprogram) Evaluation(expost) Assessment(comprehensive) Control(steeringtowardsgoal)

3.Aggregations 4.Logistics

Disaggregate(multipleindicators) Semiaggregate(aggregationwithinatheme) Aggregate(aggregationacrossthemes)

Adhoc(Custombuiltforoneevent) Partlyorganized(someroutines) Highlyorganized(extensiveroutinesforregularcyclic

reporting

Table2ExtendedIndicatortypology

Generic types

Systemindicators Performanceindicators

Descriptive Disaggregatedescriptionin

time/spaceInput

Normative Referencetosystem

limits/thresholds

Effectiveness(targetfulfillment)

Ratio Decomposition Organisationalefficiency

(input/output)

Composite Overallsystemconditionin

time/space

Aggregateoutcome

4. UseandinfluenceTraditionallyindicatorshavebeendiscussedmostlyinwhatcanbecalledarationalpositivistor

instrumental mode (Boulanger 2007), with assumed policy functions related to target setting,

decisionmakingandevaluationofresults.However,overthelastdecadeorsoalternativeviews

onthepolicyfunctionsofindicatorshaveemerged,especiallyinresearchareaslikesustainability

indicatorsandperformancemanagement(Cobb&Rixford1998;Meadows1998;Innes&Booher

2000, Hammond et al 1995; Eckerberg & Mineur 2003; Behn 2003). Innes (1998) emphasize

9


10/26

especiallythecommunicativerolesofindicatorsinpolicyandplanning,whileRydin(2002)discuss

broader roles such as raising awareness, educating the public and motivating civic action.

Meanwhile performance measurement scholars have highlighted cultural, legitimizing or even

ritualisticfunctionsofperformanceindicators(Collier1996;Feldman&March1981).

Essentialinthisnewindicatorliteratureisapropositionthatindicatorsarenotbestunderstood

as tools in themselves, but must be studied rather in terms of their role in broader policy

processesthey inform.Boulanger(2002) introducetwoalternativepolicymodels,whichhecalls

theDiscursiveconstructivistandtheStrategicmodel.Intheformermodelindicatorsareusedin

processestoreframeproblems, identifysharedvaluesandbuildacommondiscourse,exactlyas

depicted by Innes, Rydin and others above. Indicator qualities such as communicability and

dramatization and resonance (Mitchell 1996), become important, even if this may mean that

indicators loose accuracy and utility in amore strict positivist sense (Cobb &Rixford1998). In

Boulangers third model indicators are part of a conflictual context with perpetual competition

betweeninterests.Theuseofindicatorsisreducedtothatofmanipulationandplayalimitedrole.

Tosomedegreethesenewperspectivesonindicatorusabilitydrawfromearlierdevelopmentsin

broaderfieldofstudiesofknowledgeandevaluationutilization(Weiss&Bucuvalas1980;Weiss

1979,Shulha&Cousins1997).Theearlyresearchliteratureinthisareafocussedprimarilyonthe

potentialfordirectuseofresultsfordecisionmakingandproblemsolving.However,oftenlittle

evidenceofsuchusewasfoundinpractice(Weiss1998;Vedung1997);instead,theattentionof

researchers gradually became alerted to other unintended ways in which the evidence was

treated. A prominent expression was Carol Weiss identification of six forms of research

utilization(Weiss1979) seeTable3.Thestudiesgenerallyshowed,notthatknowledgedriven

orproblemsolvingusesofresearchdidnotoccuratall,butweretypicallyfoundin.relatively

lowlevel,narrowgaugedecisions.(Weiss1979,p428).Theseobservationsrecognizesthatpolicy

anddecisionmakingareessentiallysocialprocesses,whicharelikelytostronglyshapeandframe

any knowledge use taking place, rather than the other way around. Certain know ledges are

already so strongly embedded in the minds, or discourses, of policy actors that major new

informationcannotbedirectlyabsorbed(Sabatier1987;Schn&Rein1994).

10


11/26

Table3.FormsofResearchUtilization(Weiss1979,hereadaptedfromBowen&Zwi2005)

TheKnowledgeDrivenModel:Thismodelsuggeststhatemergentresearchaboutasocialproblemwill leadto

directapplicationtopolicy;itreliesoneffectivestrategiesforthetransferofresearchevidenceintopractice.

TheProblemSolvingModel: This model expects research to provide empirical evidence and conclusions that

help solve a policy problem; it assumes that evidence is systematically gathered and applied in the policy

process.

The Interactive Model:Thismodelsuggeststhatthesearchforknowledgemovesbeyondresearchtoincludea

varietyofsourcessuchaspoliticsandinterests;itaimstoreflectthecomplexityofthepolicymakingprocess.

ThePoliticalModel:Inthismodel,decisionmakersarenotreceptivetoresearchunlessitservespoliticalgain,

thatis,demonstratesproofforapredetermineddecision;evidenceissoughttojustifytheproblem.

TheTacticalModel:Thismodelseesevidenceusedtosupportandjustifygovernment inaction,orrejectionof

anddelayincommitmenttoapolicyissue

TheEnlightenment Model:Thismodelsuggeststhatcumulativeresearchshapesconceptsandperspectivesthat

permeatethepolicyprocessovertime,influencinghowpeoplethinkaboutsocialissues.

It seems plausible that similar ways to use indicators, compared to evaluations or research

resultscouldapply,eveniftheindicators,asdefinedpreviously,areseemlikedistinctknowledge

technologies. The enlightenment model appears for example to be in line with observations

about inthebroadly communicativeusesof indicators found instudiescitedabove.Thisseem

especially applicable to understand influence of indicators as variables, and in the form of

indicator frameworks, both of which may be assumed to shape concepts in policy making,

particularlyinconnectionwithrecurringsystemsoverseveralcycles(Innes1998).Theknowledge

driven and problem solving models of Weiss fits on the other hand perfectly with the

instrumentalfunctionsofindicators,andtheuseofthevalueslayer.Thesemayhoweverbeless

influentialthanexpected,unlessbackedbystrong institutional frameworks.Wewilldiscuss the

use concepts in relation to indicators a little closer in the following, adopting the modified

typology which is now most commonly applied (Amara et al 2004; Shulha & Cousins 1997;

Romsdahl2005).

11


12/26

1)The Instrumentalmodel involves applying knowledge in specific, direct ways, for indicatorscentringonthevaluesandevaluations.Bydiscoveringtheresultsofgovernmentinterventions,an

indicatorcould(1)helpgovernmentsdecidewhetherornotcontinuewithparticularpolicies;(2)

expandandinstitutionalisesuccessfulprogramsandpoliciesandcutbackunsuccessfulones;and

(3)findoutwhichprogrammesorpoliciestomodifyand inwhichmanner. Incaricature,policy

makers aiming at clearly established policy goals would react to timely published indicators,

adjustingpoliciesaccordingtotheinformationprovidedwiththevalueoftheindicators.

2)TheConceptualmodelinvolvesproducedknowledgebeingusedinamoregeneralform,moreindirectly and less specifically than in instrumental use. The indicator variable, or possibly the

indicator framework,hereprovidesgeneralbackground information, leadingto enlightenment,

thepercolationofnewinformation,ideasandperspectivesintothearenasinwhichdecisionsare

made(Weiss1999,471).Theindicators(or,acontinuouslyoperatedindicatorset), maythereby

affectdecisionmakersproblemdefinitions,andprovidenewperspectivesonandinsightsintothe

problem area, instead of providing information for a single moment of decision, or to a

hypothetical single decisionmaker. An effective indicator framework may contribute to re

conceptualizeorredefineanareofpolicymaking,orevenestablishanew fieldof intervention.

Indicator use becomes a learning process, and the knowledge created through the process

becomesutilisedwhiletheprogramisrunning.

3) The Politicalmodel encompasses three related subcategories: Legitimisation, which hasfrequentlybeenseeninanegativelight,knowledgeprovisionservingasarationalisticritualaimed

atjustifyingdecisionsthathavealreadybeentakenorpoliciesalready inplace(Lampinen1992,

3037;Weiss1999,477).ThiscorrespondstothethirdmodelofBoulanger(2007),wheretherole

ofindicatorsisnegligible. However,explicitlegitimisationcanalsobeseenanessentialelementto

secureacceptanceofpolicy inademocracyand thereforenotpurely bad (e.g.Feinstein2002,

434;Valovirta2002).Tacticaluse,isaboutknowledgebeingcommissioned,forinstance,primarilyinordertopostponeoravoidunwanteddecisionmakingbyreferringtoongoingorplannedwork.

(Vedung 2001, p 141). It seems plausible that such a use could be made of an indicator set,

particularlyasthecommissioningmay involveconceptualisationand logisticalsupport,requiring

12


13/26

considerableeffortsandtime. Symbolicusemeansknowledgeresearchbeingusedsymbolically,toconveyanimageoramessageandnothingelse(Weiss1999,477). Indicatorswouldinsucha

caseconstituteasimulacrumintendedtogivetheimpressionofarationalorganisationthatsets

goals,hasaseriousandcompetentmanagement,andtakesrationaldecisionsonthebasisofdata.

Thisscenarioisnotunlikelyeither(Pollitt1998;Weiss1999,472473;Vedung2001,141).

In subsequent studies and debates several further types of use have been proposed. For

exampletheobservationthatanknowledgegeneratingprocessitself(ratherthantheresults)may

bethemostimportantfortheactors;leadingtosocalledprocessuse(Shulhaetal1997).Again

this seems like a plausible effect with regard to indicator frameworks, in the case where this

develops over an extended period, in a collaborative process, involving both producers and

varioususersoftheindicators,asreportedbye.g.Innes(1998);Holden(2008;andRydin(2002).

Processuse of an indicator framework may lead toa new practiceof referring to the topic in

measurementterms,evenifthereisnodirectagreementovervalues,variablesorevaluations.

Imposeduse isyetanothertypeinventedinWeissetal(2005).Imposedusereferstosituations

inwhichtheuseofresearchknowledge ismadeobligatory,throughreportingrequirements,for

instance.Thereareseveralsuchexamplesinthecaseofindicators,forexamplelegallymandated

monitoringandevaluationtasksinconnectionwithEuropeantransportandagriculturalpolicies.

Ofinterestarefinallyalsodistinctionsbetweenintended(orlegitimate)useandunintendeduseor

misuse. Cousins (2004) proposes a 4 field typology. He places all the three canonized uses,

instrumental,politicalandconceptual,underthefieldofideal(legitimate)use.Healsoenvisage

legitimatenonuse.Onthenegativesideheputs abuse(illegitimatesuppression)andmisuse

(mistaken use and deliberate manipulation) as three other main categories of use. Similar

concepts have been applied in indicator research. Feller (2002) and Halachmi (2002) discuss

examples of misuse of in performance measurement. Arndt & Oman (2006) reveal extensive

abuseintheareaofgoodgovernanceindicators,meaningopinionbasedmeasuresofissuessuch

as corruptionand politicalstability, inwaysnotdeemedrobustortransparent.However, it is

likelytoinvolvedifficultiestodistinguishclearlybetweenwhatisuseandmisuseofanindicator,

asthismaydependonwhosperspectiveisadopted.

13


14/26

Sofarwecannoticeawildlyexpandingnotionoftheuseofindicators,startingfromarelatively

clear idea of the indicator as a representational knowledge technology with instrumental

functions, moving gradually into a wide field of different more or less operational uses and

misuses of indicators. Substantial critique of this blurred use notion has been raised by

knowledgeandevaluationutilizationscholarssuchasRich(1997),Kickhart(2000)andHenryand

Mark(2003;2004).AccordingtoRich,usehasbecomemuchtooambiguousaterm,whileitdoes

notevenencompassalltherelevantaspectsofhowknowledgecanmakeadifferenceforpolicy.

Heproposestoconsideruseassomethingmuchsimplerlikeifinformationhasbeenreceivedand

read.AccordingtoHenryandMark(2003)theparalleluseofnormativeanddescriptivemeanings

ofuseishighlyproblematic.Theyalsopositthat,theideaofenlightenmentorconceptualuse

does not provide reasonable specificity to guide measurement of evaluations outcomes or to

insure that casual observers or researchers are talking about the same outcomes. (Henry and

Mark2003,p311). Theysuggestinsteadtostudythevariousconsequencesthatmaystemfrom

evaluations/knowledgeuse,includingeffectssuchasattitudechange,andpolicyorientedlearning.

Hencetheysuggesttofocusonevaluationinfluenceratherthanuse.Theycharacteriseinfluence

asthesubsetofevaluationconsequencesthatcouldplausibly leadtowardorawayfromsocial

betterment (Henry and Mark, 2003, 295). The real test of knowledge influence is thus if

contributestoantheimprovementofsocialconditions.

We sympathise strongly with this shift of attention from use to influence, also as regards

indicators. We agree that the overloading of use would make it almost impossible to

comprehendwhat itmeans thatan indicator issomehow usedornot.Wealsoagree that the

consequencesaremoreimportantthantheuseitself.Howeverwewouldnotabandonthenotion

of use completely. Rather, with Rich (1997) we would consider these notions as successive

elementsinlogicalsequenceorpathway,whereusemayleadtoinfluence,whichagainmaylead

toimpacts.

We reserve the notion of use as a term for immediate actions of persons or organizations,

reflectingtheoriginalintentionalmeaningofusingsomething.Wedefinepolicyuseofindicators

as: adoption and operation of an indicator value, evaluation, or variable by a person or group

14


15/26

involved in a policy situation or process. With inspiration from Rich (1997); Hezri (2006) and

others we propose to distinguish policy use of indicators at four different levels. Each level

representsasteptowardmoreextensiveuse.Eachlevelissplitfurtherintotwosublevels,which

maybeappliedinresearch ifmoredetailiswishedfor. ThelevelsofuseareshowninTable4.

Table4Policyuseofindicators1.Reception

1.1Receive,notice,observe

1.2Forwardtoothers(nochange)

2.Internalapplication

2.1Forownwork(calculation/text)

2.2Useininternal communication

3.Externalapplication

3.1Communication withotherpolicyinstitutions

3.2.Communication withstakeholders

4.Decisionsupport

4.1Useinofficialpolicyplan/report/document

4.2Useformakingajudgmentanddecision

This does not mean we wish to loose the highly relevant insights concerning instrumental

conceptual,politicaluseetc,butratherthatseseeaneedtorephrasethoseeffectsfrombeing

understood as uses to be depicted rather as analytical categories requiring considerable

interpretation.Weredefinethoseas potentialrolesindicatorsmayhave,andwillconsiderthem

aspossiblefinalstepintheresearch,ratherthananimmediateone.

Asregards influencewesimplyadopttheextensivenotionsofHenry&Mark(2003),whichhave

developedforresearchonevaluations,asweassumecouldbeapplicableforindicatorframeworks

as well. They characterize influence as processes that can occur through a variety of pathways

takingplaceatthreedistinctlevels:individual,interpersonalandcollective(seefurthertable5),

individual level: changes in attitudesand individualbehaviour.Six subcategoriesof

change are defined at this level (e.g. perceived salience of an issue; acquisition of new skills

through the evaluation etc). The influence on this level depends on expectations, interests,

informationalbackgroundsandworldviewsofindividuals;

15


16/26

interpersonal: concerns the role of evaluations in the processes of argumentation

anddialogueamongactors,e.g.persuasion,legitimisation,criticismordefence,

collective: outcomes in terms of policy decisions and actions, shared beliefs and

understandings,legitimacyofpoliciesandactors,agendasetting,andnetworkformation.

Table5PolicyinfluenceofindicatorsIndividuallevel

ExamplesadaptedfromHenry&Mark(2003)

Attitudechange Changeopinionaboutfeasibilityofa program

Salience Increaseawarenessaboutanissue

Elaboration Stimulatenewexpectations

Priming Makeanissueappearmoreimportantbyhighlighting

Skillacquisition Learningnewwaystomeasure aproblemBehaviorchange Adoptingnewpracticefollowingshowingpositiveresults

Interpersonallevel

Persuasion Moreconvincingargumentationviaofficial report

Justification Reinforcingarguments infavor ofsolution

Changeagent Mobilizestakeholderstoactivelypursuechange

Minorityopinioninfluence Informationprovidesminoritywithdecisivearguments

Socialnorms Informationmakesnewbehavior seemappropriate

Collectivelevel

Agendasetting Informationviamediabringsanissuetotopofagenda

Policyorientedlearning Advocacycoalitionmodifiesrecommendations

Policychange Newpolicyreformadoptedaftergrowingevidence

Diffusion Evidenceofpolicysuccessstimulatesadoptionelsewhere

However,wedonotsetheneedtoacceptHenry&Marksrestrictionof influenceassomething

leadingtosocialbetterment.Thisnotionexcludesinfluenceleadingtonegativeresults,whichare

however also interesting. And it may be hard to determine if social betterment in fact occurs

formtheuseof indicators.Still,weare interested in socialbettermentbutsuggesttoconsider

thisasonepossibleoutcomeorimpact.

WethereforealsoproposetokeepinfluencedistinctfromImpacts.ImpactcanbeseenwithRich

as isdifferentas it refers to thatinformationwasusedand it leddirectly toadecisionorto

action(Rich1997p15).SEDL(2008) definesitasBroad,societallevelchangesorimprovements

thattypicallyoccurbeyondtheimmediateinfluenceofprogramactivities.Inbothcasesitrefers

tosomethingmorestrongthaninfluence.Impactwethusunderstandasaresult(intermediate

16


17/26

or final)of some influence (immediate,near,ordistant, remote);while the influence isprocess

oriented,leadingonlypotentiallytosomeimpact(likeasuseonlypotentiallyleadingtoinfluence).

A further distinction could be made between Direct intentional impacts and indirect impacts

(anything else). Proposed categories are shown in table 6. Despite attempt of clarity, the

distinctionsmightbedifficulttomaintain,andcouldberevisedaspartoftheempiricalresearch.

Table6PolicyimpactsofindicatorsDirectintentionalimpacts

Awarenessofmeasuredaims

Compliancewithdesiredbehaviour

Fulfilmentoftargets

Efficiencyofpolicy

Social/environment/economicconditionsimprovement

Indirectimpacts

Consensus/conflictaboutpolicyaims

Innovationsofsectorstakeholders

Attentiontononmeasuredpolicyaims

Resourcesofmeasuringorganisations

Impactsonotherpolicies

5. PathwaysIt would be possible, according to Henry & Mark, to start almost anywhere in their influence

model,andchartpathwaysbackandforthandacrosslevelsovertime.Inthisway,theysuggest,

many,manypathwaysarepossible(Henry&Mark 2003,p307).Thisisnolesstrueifwe,as

proposed,connecttheinfluencecategorybackwardstouseandforwardtoimpactandfurther

toapotential roleoftheindicator. Intheliteraturegenerallythereareamultitudeofwaysto

depictinteractionsbetweenknowledgeconceptsandpolicyprocesses.Indicatorsthemselvescan

forexamplebefollowedthroughstagessuchasconception,production,application,andrevision

(Keebleetal2002;McLaren1996),andtheflowsdividedintotypessuchasdatadriven,versus

theorydriven processes (Niemeier 2002). At the policy side the heuristic of the policy cycle

providesthemostwellknowlogicwithstageslikeagendasetting,selectionofalternativesand

implementation (Ilner 1984). Scholars such as Sabatier (1999; 1987) have proposed more

advanced ideas about how knowledge input and policy processes generally interrelate or

disconnect over time, as a result of the stability of certain core beliefs. Others again like Innes

17


18/26

(1990) andHezri (2004; 2005) have zoomed in on how interactionshaveplayedout in cases of

indicators. While there is not much firm knowledge in this research territory, inspiration to

envisagevariousindicatorpathwayscancertainlybefound.

ToPOINTapathwayisnotsimplyasetofstagesinanindicatorslife,orasetofstagesinapolicy,

eventhoughbothcanbepartsofpathways.Pathways illustrate interactionsbetweenpolicyand

indicator(orindicatorframework).Itcanbeproposedtodefineapathwayasaseriesofeventsor

processes through which indicators or indicator frameworks interact with policy processes,

outputs,outcomesorsubsystems.Thelogicallowstoconnectindicatorwithpolicythroughaset

ofterms(use,influence,impactetc)andassumesasequenceoftheseterms,althoughfeedbacks

and deadendsareobviouslyconceivable. Thegeneral idea is illustrated In figure4,whilethe

specificcategoriesusedforpolicyandindicatorstagesherearerandomexamples.

Figure4.Thegeneralideaofinfluencepathways

6. InfluentialordeterminingfactorsThe paper has been occupied with defining core terminology and typologies of indicators,

frameworks,uses,influencesandpathwaysasguidanceforempiricalresearch. Amajorchallenge

will be to understand what drives indicator development into certain pathways; which factors

mayexplaintheinfluenceoccurringornot. Theproblemisnottofindinspirationintheliteratures,

itisrathertodelimitthewide rangeofoptionsofferedtosomethingmanageable.Theinfluence

Policy agenda

setting

Policy trejectory

Policy Design Policy

le

Policy

imgitimization plementation

Indicator

Conception

Indicator

Production

Indicator

Dissemination

Indicator

Application

Pathways of use - influence- impact -

Indicator trajectory

18


19/26

setupadoptedfromHenryandMark(2003)entailssomeriskofoverconnectingasisoperates

atsuchdifferentprocesslevels(individualstosocieties),digressofchangeandtimehorizons.Each

empiricalanalysiswillneedtospecifyitsowndelimitationsintheseregards,andcustomizeaset

ofpotentialexplanatoryfactorstopursue. SofarPOINThasorganizedthefieldintoatypologyof

domainsoffactors,definedasfollows

The firstdomain iscalledthe indicatorfactors.Thesehavetodowith indicatorsconsideredas

instrumentsoftechnicalrepresentationandcontrol.Indicatorfactorsincludecharacteristicsofthe

information such as relevance, validity, reliability , comprehensibility timeliness,

comprehensiveness, coherence, , and concreteness of policy recommendations. (Niemeier & de

Groot 2008; Vedung, 1991; Van der Meer, 1999; Weiss, 1999, 479;). In some cases indicator

factorsseemtobeconsideredtheonlyimportantonesforitsuse:iftheindicatoriscorrectina

computationalsenseandpresentedcorrectlythenitwillbeusedmore(or,whatgetsmeasured,

gets done as a popular saying goes). As we have noted, even perfect measures could provoke

completelydifferentreactionsoruses.

Theseconddomainiscalleduserfactors.Thisreferstothepeopleandorganizationsinvolvedin

policyprocesses,andhowtheyareinclinedtocallfor,respondtoorapplyindicatorsinparticular

situations. This could again depend on a host of elements such as the users backgrounds,

positions, group affiliations and motivations. For example, some administrators may have

statistical training which makes them more ready to apply certain indicators; other participants

may mistrust certain information sources because is conflicts with their pregiven worldviews;

others again may learn by shifting positions. The users of indicators typically operate in a

complexandnuancedenvironment.Anindicator,nomatterhowwellcraftedandpresented,will

bejustoneelementinamaelstromofcallsforattention,andusersmaybedifferentlyinclinedin

thisrespect.

Thethirddomain is labelled policyfactors.Thisreferstothetypesofpolicydomainandpolicy

task the indicators are supposed to inform (e.g. in terms of what level of complexity or

controversyisinvolved;Hoppe2005),inparticularalsotheinstitutionalframeworksinwhichthe

19


20/26

users operate the indicators. Institutional frameworks consist of formal and informal rules and

proceduresthatgoverntheprocesses,andtherolesofdifferenttypesof information inagiven

situation.Forexample,theremaybelegalrequirementstoapplyacertainindicatortobenchmark

an organisation against a target, whereas informal codes of conduct may delegitimize this

information in favour of other concerns in a later step in the process. Differences in legal and

administrative cultures among policy sectors or between institutional arrangements in different

countriesareamongthepotentialpolicyfactors(Pollitt2005).

Figure5illustratesthegeneralsetup.ContinuedresearchinPOINTwillexplorehowtothethree

domainsoffactorscontributeandinteracttoshapeinfluence.

Pathways of Use Influence Impact

Figure5.Preliminaryconceptualframeworkofindicatoruseandinfluence

20


21/26


22/26

Table7Exampleofanalyticalframeworksetupsub analyis

Background analysis Influence analysis

Policy

background

Indicator

Emergence/

Function

Use Influence Impact Roles

Policycontext Originofsystem Reception Individual Directintended Instrumental

Policyframework Official

function/appli

cation

Internal

application

Interpersonal Indirect Political

Users Typesof

indicatorsExternal

applicationCollective Conceptual

Typesof

framework(e.g

systemor

performance);

logisticalaspects

Decisionsupport

Process

SinglestepMultiplestep

FullstepSinglestep

Multiplestep

Truncated

22


23/26

References

Arndt, Christiane; Oman, Charles 2006. Uses and Abuses of Governance Indicators. OECD

DevelopmentCentreStudies.OrganisationofEconomicCooperationandDevelopment,Paris.

Becker, Joanna 2007. How Frameworks can help operationalize Sustainable Development

Indicators.WorldFutures,63:137150,2007

Behn, R. D. 2003. Why measure performance? Different purposes require different measures.

PublicAdministrationReview,63,5,pp.588606.

Boulanger, 2007. Political uses of social indicators: overview and application to sustainable

developmentindicators.Int.J.SustainableDevelopment,Vol.10,Nos.12pp1432.

BowenS;ZwiA.B.2005Pathways toEvidenceInformedPolicyandPractice:AFramework for

Action.

PLoS

Med

2(7).

Cash, D., Clark, W.; Alcock, F.; Dickson, N.; Eckley, N and Jaeger, J. 2002. Salience, Credibility,

Legitimacy and boundaries: linking research, assessment and decision making, John F Kennedy

schoolofgovernment,HarvardUniversity.

Cobb, C. & Rixford, G. 1998. Competing Paradigms in the Development of Social and Economic

Indicators.CSLSConferenceontheStateofLivingandthe qualityofLifeinCanada,October3031

1998,Ottawa,Ontario.

Collier,PaulM 2006.InSearchofPurposeandPriorities:PolicePerformanceIndicatorsinEngland

andWales.PublicMoney&ManagementJune2006,pp165172.

Cousins, J.B 2004. Commentary: Minimizing Evaluation Misuse as Principled Practice. American

JournalofEvaluation2004;25;p391397.

Eckerberg,Katerina&Mineur,Eva2003.TheuseoflocalSustainabilityIndicators:casestudiesin

twoSwedishmunicipalities.LocalEnvironment,Vol.8,No.6,591614,December2003.

EEA 1999. Environmental indicators: Typology and overview. European Environment Agency,

Copenhagen.

Eurostat 2009. ESS Handbook for Quality Reports. European Commission. Office for Official

Publications of the European Communities, Luxembourg. URL

http://epp.eurostat.ec.europa.eu/cache/ITY_OFFPUB/KSRA08016/EN/KSRA08016EN.PDF

Feinstein, Osvaldo N. 2002. Use of Evaluations and Evaluation of their Use. Evaluation 8(4) pp.

433439.

Feldman,MarthaS&March,JamesD.1981.InformationinOrganisationsasSignalsandSymbols.

AdministrativeScienceQuarterly,Vol.22,Issue2,pp.June1981pp.171186.

Feller,I.2002.PerformanceMeasurementRedux.AmericanJournalofEvaluation23(4):435452.

Franceschini,F.;Galetto,M.;Maisano,D.;andMastrogiacomo,L.2008.Propertiesofperformance

indicators in operations management. A reference framework. International Journal of

ProductivityandPerformanceManagement,Vol.57No.2pp.137155.

23


24/26

Gallopin, Gilberto Carlos 1997. Indicators and Their Use: Information for Decision Making.

Introduction.pp1327in:Moldan,B&Billharz,S.:SustainabilityIndicators.Reportontheproject

onIndicatorsofSustainableDevelopment.Wiley,Chichester.

Gudmundsson, H. 2003. The Policy Use of Environmental Indicators. Learning from Evaluation

Research.Vol.2,No.2,2003:112.

Halachmi, Arie 2002. Performance measurement: a look at some possible dysfunctions. Work

Study.Volume51.Number5.2002.pp.230239.

Hammond et al 1995. Environmental Indicators: A Systematic Approach to Measuring and

Reporting on Environmental Policy Performance in the Context of Sustainable Development.

WorldResourcesInstitute,WashingtonDC.

Henry,GaryT. andMark,MelvinM.2003.BeyondUse:UnderstandingEvaluationsInfluenceon

AttitudesandActions,AmericanJournalofEvaluation,Vol.24,No.3,2003,pp.293314

Hezri,A.A2006ConnectingSustainabilityIndicatorsToPolicySystems.Athesissubmittedforthe

degreeofDoctorofPhilosophy.TheAustralianNationalUniversity,OCTOBER2006.

Hezri AA 2005. Utilisation of sustainability indicators and impact through policy learning in the

Malaysian policy processes. Journal of Environmental Assessment Policy and Management 7(4),

575595.

HezriAA.2004.SustainabilityindicatorsystemandpolicyprocessesinMalaysia:aframeworkfor

utilisationandlearning.JournalofEnvironmentalManagement73,357371.

Holden, Meg 2008. Social learning in planning: Seattles sustainable. development codebooks.

ProgressinPlanning69(2008)140.

Hoppe, R 2005. Rethinking the sciencepolicy nexus: from knowledge utilization and science

technologystudiestotypesofboundaryarrangements.PoiesisPrax3,pp199215

Illner, M 1984. On functional types of indicators in social planning. Social Indicators

Research,Volume14,Number3,pp.275285.

Innes, J.E.; Booher, D.E. 2000. Indicators for sustainable communities: A strategy building on

complexitytheoryanddistributedintelligence.PlanningTheory&Practice,1(2),pp.173186.

Innes, J. E. 1990. Knowledge and Public Policy. The search for meaningful indicators. Second

expandededition.Transactionpublishers,NewBrunswick.

IISD 2007. Compendium of Sustainable Development Indicator Initiatives. International Institute

forSustainableDevelopment,Winnipeg.URL:www.iisd.org/measure/compendium/

James, Thomas E.; Jorgensen, Paul D. 2009. Policy Knowledge, Policy Formulation, and Change:

RevisitingaFoundationalQuestion.ThePolicyStudiesJournal,Vol.37,No.1pp141162.

Keeble et al 2002 Using Indicators to Measure Performance at a Corporate and Project level.

JournalofBusinessEthics44,pp.149158

Kirkhart,KarenE2000.ReconceptualizingEvaluationUse: AnintegratedTheoryofInfluence.Pp2

24in:Caracelli,V.J;Preskill,H(eds.):TheExpandingScopeofEvaluationUse.APublicationofthe

AmericanEvaluationAssociation,JosseyBass,SanFrancisco.

24
http://www.iisd.org/measure/compendium/http://www.iisd.org/measure/compendium/http://www.iisd.org/measure/compendium/


25/26

Lampinen, Osmo. 1992. The utilization of social science research in public policy. Doctoral

Dissertation.SuomenAkatemianjulkaisuja4.Helsinki.VAPKkustannus.

Maclaren, Virginia W. 1996. Urban Sustainability Reporting. Journal of the American Planning

Association,Vol62,No2,Spring1996.

Mark,M.M.;Henry,G.T.2004.TheMechanismsandOutcomesofEvaluationInfluence.Evaluation.

Vol10(1)pp.3557

Meadows, Donella 1998. Indicators and Information Systems for Sustainable Development. A

ReporttotheBalatonGroup.TheSustainabilityInstitute,Hartland,VT.

Moldan, B & Billharz, S. 1997. Sustainability Indicators. Report on the project on Indicators of

SustainableDevelopment.Wiley,Chichester.

Morse, S. 2004. Indices and indicators in development. An unhealthy obsession with numbers.

Earthscan,London.

Niemeijer,D.2002.Developingindicatorsforenvironmentalpolicy:datadrivenandtheorydriven

approachesexaminedbyexample.EnvironmentalScience&Policy5 pp.91103.

Niemeijer, David; de Groot, Rudolf S. 2008. Conceptual framework for selecting environmental

indicatorsets.EcologicalIndicators8pp1425.

OECD 2003 Environmental Indicators Development, Measurement and Use. Reference paper,

OECD,Paris.

Ostrom,Elinor1999 InstitutionalRationalChoice:AnAssessmentoftheIADFramework.In:Paul

Sabatier,(ed).TheoriesofthePolicyProcess.WestviewPress,Boulder.

Pollitt,Christopher2005.PerformanceManagementinPractice:AComparativeStudyofExecutive

Agencies.JournalofPublicAdministrationResearchandTheory,16:2544.

Rich,R.F.1997.Measuringknowledgeutilization:Processandoutcomes.KnowledgeandPolicy:

TheInternationalJournalofKnowledgeTransferandUtilization10.pp1125

Romsdahl, Rebecca J. 2005. When Do Environmental Decision Makers Use Social Science?

AppendixAin:Brewer,GarryD. andStern,PaulC.Ed.2005.DecisionMakingfortheEnvironment:

SocialandBehavioralScienceResearchPriorities.NationalResearchCouncil,WashingtonD.C.

Rosenstrm, Ulla; Lyytimki, Jari 2006 The Role of Indicators in Improving Timeliness of

InternationalEnvironmentalReports.EuropeanEnvironment16,pp3244.

Rydin, Y. 2002. Indicators Into Action: Local Sustainability Indicator Sets in Their Context. The

PastilleConsortium,LondonSchoolofEconomics,London.

Sabatier,P.A(ed)1999.TheoriesofthePolicyprocess.WestviewPress,Boulder.

Sabatier, Paul A.1987. Knowledge, policyoriented learning, and policy change, Knowledge:

Creation,Diffusion,Utilization8,pp.649692.

SEDL 2008. Research Utilization Support and Help (RUSH). Southwest Educational Development

Laboratory, Austin. URL: //www.researchutilization.org/learnru/glossary.html. Last Updated:

Thursday,10April2008at12:21PM

25


26/26

Shulha, Lyn M.; Cousins, J. Bradley, 1997. Evaluation Use: Theory, Research, and Practice Since

1986.AmericanJournalofEvaluation,Feb1997;vol.18:pp.195208.

Valovirta,V2002.EvaluationUtilizationasArgumentation.EvaluationVol8(1),pp. 6080.

VanderMeer,F.B.1999.EvaluationandtheSocialConstructionofImpacts.Evaluation5,387406.

Vedung,E.1997.Publicpolicyandprogramevaluation.TransactionPublishers,NewBrunswick.

Weiss, Carol H. 1999. The Interface Between Evaluation and Public Policy. Evaluation 5 (4), pp.

468486.

Weiss,CarolH.1998.HaveWeLearnedAnythingNewAbout theUseofEvaluations?American

JournalofEvaluation,Vol19,No1,pp2133.

Weiss, C. H., 1987, The Circuitry of Enlightenment: Diffusion of Social Science Research to

Policymakers,Knowledge:Creation,Diffusion,Utilisation8(2):274281.

WeissCarolH.1979.TheManyMeaningsofResearchUtilization.PublicAdministrationReview,

Vol.39,No.5.,pp.426431

Weiss,CarolH;BucuvalasMichaelJ1980.TruthTestsandUtilityTests:DecisionMakers'Frames

ofReferenceforSocialScienceResearch.AmericanSociologicalReview,Vol.45,No.2.(Apr.,1980),

pp.302313.

Hgu Ness 2009 Paper

Documents

Transcript of Hgu Ness 2009 Paper