The CZ example of evaluating and drafting of drug strategy

Lucia KiššováJindřich Vobořil

Drug policy formulated in strategic documents• 1990 – 1992 Commission of CSFR for Drugs• December 1992 – Christmas memorandum to the Government• 1993 – set up of the Interministerial Drug Commission• 1993 – 1996 – First Drug Policy Program• 1998 – 2000 – Second Drug Policy Program• 2001 – 2004 – National Drug Strategy• 2005 – 2009 – National Drug Strategy and 2 Action plans

– 2005-2006 action plan– 2007-2009 action plan

• 2010 – 2018 – National Drug Strategy and 3 Action Plans– 2010-2012 action plan– 2013-2015 action plan– 2016-2018 action plan

• National strategy:– More general, defines main goals and basic principles of drug

policy for the covered period, drug policy intervention areas • Action plans:

– More specific, define activities, responsible institutions and deadlines

Evaluation of drug policy 2005-2009• Subject of evaluation

– National strategy 2005-2009– Action plan 2007-2009

• Time frame– 05/2009 – 03/2010

• Internal evaluation – National Drug Committee secretariat – body responsible for drug policy coordination

• Aim of the evaluation – to learn:– The extend to which strategic goals where achieved

(How has the drug situation changed)– What were the main achievements and failures of drug policy in 5 years

Concept of evaluation

Based on the (simplified) policy cycle model

Findings and recommendationsWhat have we learnt in connection to drafting

new drug strategy

• Formal structure of strategic documents is convenient, most of the strategic goals are relevant for the future

• The strategy and action plan were not fully cohesive/linked-up– One strategic goal had no activity that could lead (directly or indirectly) to its

fulfillment – Even though strategic goals were mostly focused on changing the drug

situation, the majority of activities were focused on coordination and organizational framework

– In some areas majority of planed activities had limited potential to change the situation, so even if implemented the strategic goal could not be achieved

– Only small no. of activities were essential to achieve strategic goals

• The character and potential of planned activities defined in Action plan 2007-2009 and also the level of successful implementation corresponds with the success in achieving strategic goals and areas

Incidention matrix – all activities that contribute to the achievement of goals

Incidention matrix – activities that are essential to achieve goals

eg. - preventionStrategy goal – decrease the level of experimental use

Not achieved:• high level of experimental use – cannabis, alcohol …• The consumption of illegal drugs increased and of legal drug stabilized on a high

level

Prevention in Action plan 2007-2009• No activity leaded directly to specific strategic goals• High % (70) of activities focused on organizational-coordination frame – only 30 %

of activities were focused on solving a problem• Only 35 % - implemented activities in 2007-2009 and 50 % in 2005-2006• No improvement was noticed in this area (experts opinion); number of strong

points from 2004 were weakened (SWOT)

Strategy goal – Decrease the risks of drug use

Achieved:• Infection diseases and other health consequences on a low level• Stabilization of problem drug users• Relatively stable network of drug services

Harm-reduction in Action plan 2007-2009• Had the highest no. of activities leading directly to achieving most of strategic goals• Highest % (65) activities, which were specifically focused on problem solving,• Fulfilled/implemented 78 % of activities 2005-2006 (most from all 4 pillars), 41 % in

2007-2009

• Therefore more attention should be paid to better link action plan and strategy – planned activities with strategic goals

• While planning check weather planed activities contribute to the achievement of goals and weather the activities are realistic

• Define limited no. of priorities for specific time period in order to concentrate attention, personal and financial resources

• Formulate activities clearly – unclear vague definition causes variety of interpretation, it complicated implementation and evaluation– Positive correlation was find between clear formulation of activity and the

level of achievement

• Czech drug policy can rely on the professional potential of experts, availability of relevant information and on the institutional framework (network of services for drug users, institutions responsible for drug policy implementation. – We should rely on this factors in implementing future drug policy

• The level of coordination of relevant institutions plays positive but also a negative role in drug policy implementation.– The future drug policy should pay attention to strengthen the area of

cooperation and coordination• Negative factors that complicate the goal achievement is formality in

activity fulfillment, not realistic activities defined in action plan. – In the future this should be eliminated by paying attention to set realistic

activities, that is possible to implement and more accurate definition of how the activity could be achieved

• The Czech drug policy is negatively influenced by limited financial resources, given priority to other public policies, low interest in drug field by politicians and finally high level of social tolerance to use of legal drugs and underestimation of risks related to drug use.

Drafting the new drug strategy and action plan

• The process of drafting new strategic documents and the process of evaluation were conducted in parallel, involved the same people therefore the transfer of knowledge was easier

• The process of drafting new documents and evaluation was coordinated by National Drug Committee Secretariat

• Strategy:

– We decided to keep the strategy and its structure more-less the same

– According to findings from drug situation monitoring we updated strategic goals

• Action plan:

– Expert working groups were established

– Priorities of drug policy for the period covered by action plan (3 years) were set by the secretariat of NDC

Establishment of working groups• Working group for each intervention drug policy areas (7 areas)

– 7 areas – Prevention, Treatment, Harm reduction, Law enforcement, Coordination and financing, Information, research, evaluation and International cooperation

• Identify all relevant stakeholders for each area

• Each working group had to ideally consist of:

– ministries, NGOs, professional organization, individual experts, university (Centre of adictology)

• 15-20 people per one group

• 60 experts all together involved in the process

Work of working groups• Participated in the evaluation and worked with outputs of evaluation• Met cca. 3x• Conducted SWOT analysis of situation in relevant drug policy area • Identified main problems (W) and strengths (S) – experts ranked W,S

according to their importance• Experts than tried to find solutions to identified main problems by method

of brainstorming– There was a possibility to send ideas also by e-mail after the meeting of

working group– There was a criterion that proposed activities should cover defined priorities

of drug policy for the period of three years. • Proposed solutions/ideas created the basis for defining activities for action

plan• Secretariat of National Drug Committee made the selection of proposed

activities/ideas and drafted the new action plan.

The process of negotiation/commenting on action plan• Draft of action plan was discussed:

– in the working groups that participated in the process of evaluation and drafting

– in the Committee of regions – Committee of Department representatives (ministries, institution)– Then in the National Drug Committee (members are ministers and other high

level representatives of other institution)

• After that the draft action plan was submitted to the Government

Drug Policy Coordination

• Pros and cons of this process (-) More work and more effort needed (-) More time consuming (-) Demands good negotiations and facilitation skills (+) Involves all relevant stakeholders and experts from the low level to

the high level management (+) Helps to steer the discussion, exchange views and therefore helps

to find commonly acceptable drug policy (+) Supports the feeling of shared responsibility for drug policy

• Still keep in mind:– Drug policy is still a public policy– Even if you try to be it as evidence based and objective as possible

there will still be different points of views, individual interests of ministries or individuals, political influence

Thank you and

Kissova.lucia@vlada.czVoboril.jindrich@vlada.cz

Detailed information on the evaluation of CZ drug strategy and

action plan

Background information for BC participants

National Drug Strategy 2005-2009

Specific goals:I. Stabilize / decrease the no. of

problem drug usersII. Stop the increase of experimental

drug useIII. Stabilize / decrease drug

consumptionIV. Decrease the risks of drug useV. Guarantee availability of drug

servicesVI. Decrease the accessibility of

drugs

Technical-organizational goals:VII. Effective financingVIII. CoordinationIX. Information to general publicX. International cooperationXI. Evaluation of activities

• Decrease the level of any drug use prevalence and the potential risks

• To combat organized crime

Main goals

Action plan 2007-20097 drug areas: • Prevention• Treatment• Harm reduction• Law enforcement

Action plan 2007-2009 had 43 objectives and 170 activities

• Coordination and financing• Information, research, evaluation• International cooperation

Example of a structure of action plan

Aim of evaluation• To learn :

– The extend to which strategic goals where achieved (How has the drug situation changed)

– What were the main achievements and failures of drug policy in 5 years

• Organizational aspects:– internal evaluation – National Drug Committee secretariat (a body responsible

for coordination and implementation of drug policy)– 1,5 employee, no budget– The Board of Drug Strategy Evaluation (5 experts) was established:

• Provided methodological guidance of the evaluation process and evaluation tools

• Supervision of final outcome/findings and final version of evaluation report

Concept of evaluation

Based on the (simplified) policy cycle model

Stakeholders perception on the structure and formulation of evaluated drug strategy and

action plan

We were about to draft new strategic documents so in the frame of evaluation we asked stakeholders on their perception of current strategy and action plan (We wanted to find out weather we should keep the structure for the new strategy or change it)

• Method used– Online questionnaire

• Target group:– Representants of relevant ministries (33 people)– Regions (regional drug coordinators - 14)– Members of professional organizations (8 people)

• Questions asked in the questionnaire:– Were the strategic goals realistic?– Did strategic documents clearly define a way (measurements) for reaching

strategic goals?– Were the priorities clearly defined?– Do you perceive a current action plan as an effective tool for reaching the

strategic goals?– How well did the action plan reflected current drug situation and trends?– How useful you find for your work the drug strategy and action plan?– Do you find the structure of current strategy and action plan convenient for

you?

Example

Evaluation of the process of formulation

How the strategic documents were formulated

• During the process of drafting and adopting strategic documents in 2004 there was a personal change in the position of the main actor/stakeholder responsible for drafting strategic documents.

• National strategy 2005-2009 was drafted by the previous stakeholder, the following action plan 2005-2007 by new one.

• In those two documents some inconsistencies were found – drug policy areas in the action plan and its activities did not fully corresponded to the National Strategy. With the change of the main responsible stakeholder original purpose/structure and content of action plan has been affected/changed.

• Therefore we decided to focus our evaluation also on the process of formulation of strategic documents.

Q1: Have changes occurred within the formulation process? Q2: To what extend is the Action plan and Strategy cohesive/linked-up? Q3: What is the character/type of activities planed in Action plan?

FormulationEvaluation questions

Have strategic documents changed compared to their planed design?• Methods used

– Comparison of official (and internal) documents – what structure and contend was planed in 2004 what was approved in 2005

– Semi-structure interview with the former stakeholder• Main finding

– There was a change in the original purpose and content of action plan – Originally the action plan should contain the area Use of legal drugs, but it

was abandoned – alcohol and tobacco is still not integrated part of drug policy (a failure of drug policy)

– There was also a change in the originally planed area Public relation for Information, research, evaluation – therefore no links existed between activities in the action plan and a specific strategic goal

FormulationQ1: Have changes occurred within the formulation process?

For Q2 we wanted to know how many activities really lead to strategic goals.For Q3 we assessed the activities from the point of their formulation, their

potential to solve a problem and their significance for drug situation

• Methods used:– Analysis of documents based on the idea of logframe analysis – it helps to link

and clarify mutual relationship among goals, objectives, outputs and activities– Internal working group was set up (5 people) for this purpose – the working

group assessed each activity of action plan in 5 different criteria

Q2: To what extend is the Action plan and Strategy cohesive/linked up?Q3: What is the character/type of activities planed in Action plan?

• Criteria assessed by the working group for each activity:– formulation – was the activity clear and understandable? (yeas/no)– context of problem solving:

• is the activity focused on solving a problem? (eg. Implementation of specific prevention program)

• or is the activity focused on creating conditions/framework for solving the problem? (e.g. Regular meetings of prevention committee)

– significance for drug situation:• Activity with high potential to change the drug situation (eg. To adopt stricter

regulations for free selling of medicine containing pseudoefedrine) • Activity with partial potential to change the drug situation (eg. To publish a

methodological manual for prevention)• Activity with no potential to change the drug situation (eg. To realize a questionnaire

survey among teachers)– relevance to strategic goals:

• Each activity was then assessed from the point of their relevance to individual strategic goals (we had 11 strategic goals). The group assessed weather the activity was crucial for reaching the strategic goal or contributed to the strat. goal only partially or did not contribute at all.

• This way we found out that activities formulated in the area of prevention were rather formal, had no direct link with strategic goals and very low potential to change the situation.

Example

Process of Implementation evaluation

Q1: How successful was the implementation of Action Plan 2007-2009?Q2: What factors influenced the process of implementation (positively and

negatively)?Q3: What are the conditions for implementation of drug policy on the

regional level?

Evaluation questions

• Methods used:– Top-down approach - Objectives set on the central level are assessed from

the point of – how these have been achieved, what hindered their achievement, and what is the consistency of outputs and outcomes with set objectives.

– A questionnaire was sent to relevant stakeholders/actors – (16 ministries and institutions directly responsible for activity implementation, 6 collaborating subjects – professional organizations, 14 regions)

– An internal working group of the body responsible for evaluation assessed the information submitted

Q1: How successful was the implementation of Action Plan? (monitoring of implementation)

• Structure of the questionnaire – asked for each activity (eg.):– Was the activity implemented? (yeas-no-partially)– Was it implemented on time? (yeas-no)– Describe the process of activity implementation (how it was implemented)– Describe problems in implementation/reasons for not implementing the

activity – Describe the outcome of activity implementation (weather the activity met

its original purpose)– If applicable, describe alternative outcomes of activity implementation or

non intended outcomes.– Provide data for defined indicators (indicators were set for each activity)– If possible, provide data of financial expenditures needed for activity

implementation– Do you plan to continue with this activity in the future action plan?

ImplementationExample

• Working group than assessed the information submitted and compared it with real outputs/objectives planed in Action plan

• Eg. A stakeholder informed us that data on alcohol are regularly collected and published in annual report therefore the implementation of an activity is fulfilled. However the working group found that in reality there is no such annual report and therefore marked the activity as not implemented.

• This way we decreased the formality in fulfillment of activities and made the evaluation more objective (eg. In the area of prevention declared implemented activities decreased from cca 45 % down to 35 %)

• Methods used:– expert groups for 7 drug areas – according to areas of action plan (cca. 60

people) (15-20 people per one expert group). Expert groups conducted SWOT analysis for each area of action plan (in 2009).

– Comparison of SWOT analysis conducted in 2004 and 2009 (comparison of opportunities and threads). Experts assessed weather the opportunity identified in 2004 was taken or not and weather the thread identified in 2004 occurred within 5 years or not)

– On-line anonymous questionnaire (35 respondents) – state organizations, local government, professional organizations – asking respondents to mark the level of influence of selected factors (positively, negatively)

ImplementationQ2: What factors influenced the process of implementation (positively and negatively)?

Example• The questionnaire asked stakeholders to mark the level of influence to the

implementation of factors listed bellow (-3 very negatively, +3 very positively):– Formulation of activity– Realistic planning– Allocated financial resources– Institutional framework– Personal resources– Available information needed for activity achievement– Level of motivation of professionals and institutions in drug field– Level of professional knowledge of people responsible for activity

implementation– Level of cooperation of relevant stakeholders/institutions– Level of coordination– Level of flexibility of responsible actors to act and react on new trends in drug

situation– Other – please mention

• Methods used:– On-line questionnaire among regional drug coordinators (14 regions)

• Questions focused on the area of:– Extend to which goals of national drug strategy has been introduced into

regional strategic documents (how many and which goals became a part of regional strategic documents)

– Cooperation with state organizations– Position of regional drug coordinator/possibilities to influence drug policy

implementation (where is the regional coordinator situated within organization structure, what other areas except drug policy is drug coordinator responsible for, weather regional drug coordinator is a member of all relevant regional coordination bodies and working groups/committees etc.)

– Development of conditions for implementation of regional drug policy in the last 5 years- personal, political, financial, institutional (indicate weather the situation in listed conditions worsened or got better)

ImplementationQ3: What are the conditions for implementation of drug policy on the regional level?

Outcome evaluationof CZ drug policy 2005-2009

Evaluation questions

Q2: What was the development in each drug area in the last 5 years? Q1: What is the level of achievement of specific strategic goals?Q3: What is the level of achievement of technical-organizational strategic

goals?

• 7 drug areas – Prevention, Treatment, Harm reduction, Law enforcement, Coordination and financing, Information, research, evaluation and International cooperation

• Methods used:– Expert working groups for each drug area (7 groups), 60 experts (15-20

experts in each working group)– Each expert group assesses elements of SWOT analysis from 2004

• Is the strength still strong point or has it become week point?• have the week points been eliminated?• have we used identified opportunities or have the threads occurred or

not…?– Expert groups than identified new elements of SWOT for 2009

Outcome evaluationQ1: What was the development in each drug area in the last 5 years?

Q2: What is the level of achievement of specific strategic goals?

• 6 specific strategic goals - focused on the change of drug situation:I. Stabilize / decrease the no. of problem drug usersII. Stop the increase of experimental drug useIII. Stabilize / decrease drug consumptionIV. Decrease the risks of drug useV. Guarantee availability of drug servicesVI. Decrease the accessibility of drugs

• Methods used:– Analysis of quantitative data/indicators of drug situation (data provided by

NFP)

Example• We assessed the development in the following indicators:

– Goal I. – no. of problem drug users and their characteristics (average age, injecting use, primary drug), no.of problem drug users in treatment (in-out-patient, substitution treatment)

– Goal II. – prevalence of drug use in general population, in school population, development in the institutional framework and support to primary prevention programmers

– Goal III. – alcohol and tobacco consumption, consumption of other illegal drugs

– Goal IV. – drug related death (direct/indirect), drug related infection diseases, no. of distributed needles and syringes, no.of tests for infection diseases, and other intervention of low threshold facilities.

– Goal V. – availability and accessibility of services for drug users, socio-economic characteristics of drug users in treatment

– Goal VI. – drug seizures, seizures of drug precursors (medicines containing pseudoephedrine), laboratories for pervitin production, drug related law offences, subjectively perceived availably of drugs in the marked

• 5 Technical-organizational goals:VII. Effective financingVIII. CoordinationIX. Information to general publicX. International cooperationXI. Evaluation of activities

• Methods used:– Combination of results and findings from all parts of evaluation (evaluation

of formulation process, implementation and from expert working groups)

Q3: What is the level of achievement of technical-organizational strategic goals?

Success and failures of drug policy• Finally the evaluation report concludes the success and failures of drug

policy within the years 2004-2009. • For this we used synthesis of information and findings gained from each

step of the whole evaluation process.

Lucia KiššováKissova.lucia@vlada.czTel: 00420 725 721 467