1Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

PHC Progression Model: a novel mixed- methods tool for measuring primary health care system capacity

Hannah L Ratcliffe ,1 Dan Schwarz,1 Lisa R Hirschhorn ,2 Cintia Cejas,3 Abdoulaye Diallo,4 Ezequiel Garcia- Elorrio,5 Jocelyn Fifield ,1 Diane Gashumba,6 Lucy Hartshorn,1 Nicholas Leydon,7 Mohamed Mohamed,8 Yoriko Nakamura,9 Youssoupha Ndiaye,4 Jacob Novignon,10 Anthony Ofosu,11 Sanam Roder- DeWan ,12 Angelique Rwiyereka,13 Federica Secci,14 Jeremy H Veillard,14 Asaf Bitton 1

Practice

To cite: Ratcliffe HL, Schwarz D, Hirschhorn LR, et al. PHC Progression Model: a novel mixed- methods tool for measuring primary health care system capacity. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

Handling editor Seye Abimbola

► Additional material is published online only. To view please visit the journal online (http:// dx. doi. org/ 10. 1136/ bmjgh- 2019- 001822).

Received 4 July 2019Revised 20 August 2019Accepted 25 August 2019

For numbered affiliations see end of article.

Correspondence toHannah L Ratcliffe; hratcliffe@ ariadnelabs. org

© Author(s) (or their employer(s)) 2019. Re- use permitted under CC BY. Published by BMJ.

AbsTrACTHigh- performing primary health care (PHC) is essential for achieving universal health coverage. However, in many countries, PHC is weak and unable to deliver on its potential. Improvement is often limited by a lack of actionable data to inform policies and set priorities. To address this gap, the Primary Health Care Performance Initiative (PHCPI) was formed to strengthen measurement of PHC in low- income and middle- income countries in order to accelerate improvement. PHCPI’s Vital Signs Profile was designed to provide a comprehensive snapshot of the performance of a country’s PHC system, yet quantitative information about PHC systems’ capacity to deliver high- quality, effective care was limited by the scarcity of existing data sources and metrics. To systematically measure the capacity of PHC systems, PHCPI developed the PHC Progression Model, a rubric- based mixed- methods assessment tool. The PHC Progression Model is completed through a participatory process by in- country teams and subsequently reviewed by PHCPI to validate results and ensure consistency across countries. In 2018, PHCPI partnered with five countries to pilot the tool and found that it was feasible to implement with fidelity, produced valid results, and was highly acceptable and useful to stakeholders. Pilot results showed that both the participatory assessment process and resulting findings yielded novel and actionable insights into PHC strengths and weaknesses. Based on these positive early results, PHCPI will support expansion of the PHC Progression Model to additional countries to systematically and comprehensively measure PHC system capacity in order to identify and prioritise targeted improvement efforts.

InTroduCTIonThe 2018 Astana Declaration reaffirmed the global community’s commitment to strength-ening primary health care (PHC) and high-lighted the central role that PHC must play in the achievement of universal health coverage.1 However, too often PHC is weak,

underprioritised, and unable to deliver on this potential, particularly in low- income and middle- income countries (LMICs).2–4

In many LMICs, the ability to improve PHC is limited by a lack of relevant data to accu-rately measure and diagnose performance to inform policies and set priorities. The Primary Health Care Performance Initiative (PHCPI) was formed to accelerate improve-ments in PHC through better measurement and knowledge sharing.5 To guide its work, PHCPI developed a conceptual framework outlining the core systems, inputs, and service delivery elements necessary to produce strong PHC outputs and outcomes (figure 1)6 and conducted an extensive scoping process to identify the best globally available metrics to assess each component captured in the conceptual framework. Through these

summary box

► Effective primary health care (PHC) is essential for achieving the promise of quality universal health coverage, but PHC performance is weak in many low- income and middle- income countries (LMICs).

► The ability to improve PHC systems is limited by a lack of relevant metrics and data for assessing crit-ical areas of PHC performance and system capacity.

► The Primary Health Care Performance Initiative (PHCPI) undertook a structured, participatory pro-cess to design a new mixed- methods assessment tool to measure PHC capacity in LMICs more sys-tematically and comprehensively.

► PHCPI partnered with governments in five LMICs to pilot the assessment tool in 2018 and found that the assessment yielded novel and actionable informa-tion on PHC strengths and weaknesses and was fea-sible, acceptable, and effective in generating local ownership of the results.

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

2 Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

BMJ Global Health

Figure 1 The Primary Health Care Performance Initiative conceptual framework. Focal areas for the PHC Progression Model are highlighted in yellow; other areas of the framework are also included in the Vital Signs Profile but assessed using different a methodology, reflecting available quantitative data. PHC, primary healthcare; NCDs, non- communicable diseases; RMNCH, reproductive, maternal, newborn, and child health.

efforts, PHCPI was able to identify adequate measures for many components of the framework, including topics such as PHC spending, access, quality, service coverage, and health outcomes.7

However, PHCPI’s scoping process failed to identify robust, globally available indicators of the underlying capacities of PHC that impact overall performance and outcomes (figure 1).7 As described further in box 1, PHC capacities include topics such as:

► How well the PHC system is governed and led. ► Whether PHC systems have the ability to detect and

adjust to changing population health needs. ► Whether key inputs are available, equitably distrib-

uted, and of sufficiently high quality to meet popula-tion health needs.

► Whether PHC systems know and engage with the populations they serve.

► Whether healthcare providers work as teams in well- managed facilities, using data to drive improvement efforts.

Many of these are complex, interrelated topics whose performance varies over a continuum that is difficult to quantify and requires more nuanced exploration than a single quantitative measure may allow.

Through engagement and partnership with repre-sentatives from more than 30 LMICs over several years, PHCPI recognised that the collective inability to fully measure PHC capacity severely curtailed countries’ ability to develop a holistic understanding of their system’s strengths and weaknesses and hence their ability to identify and implement needed improvements. To address this measurement gap and ensure that essen-tial information on PHC capacity could be comprehen-sively assessed, PHCPI developed a novel tool—the PHC

Progression Model—to systematically assess the complex, foundational capacities of PHC. The results of the PHC Progression Model assessment are incorporated into the PHC Vital Signs Profile, a PHCPI measurement tool that summarises a country’s performance across the concep-tual framework and is designed to support countries in identifying priority areas for improvement in PHC, track progress over time, and promote accountability for results by making essential performance information transparent and publicly available.8

In 2018, PHCPI partnered with five LMIC to pilot PHC Progression Model assessments. This paper describes the methods used to develop the PHC Progression Model as well as the process and lessons learnt from implementation.

A new wAy of meAsurIng PHC CAPACITyTo identify a means of systematically assessing PHC capacity, PHCPI built on the recent increase in use of mixed- methods, rubric- based assessment tools in the health landscape. These tools are designed to capture performance across a range of different levels of system maturity, as defined by prespecified performance cate-gories described in a series of rubrics. As a first step in conceptualising the PHC Progression Model, we used literature reviews, internet searches, expert recommen-dations, and snowball sampling to search for rubric- based tools in the healthcare domain. We identified a multi-tude of relevant tools, and six in particular that substan-tially influenced the conceptualisation and design of our work.9–14 Through a review of publicly available materials and interviews with the developers and/or implementers of these tools, we extracted relevant key learnings and

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822 3

BMJ Global Health

box 1 Key questions assessed by the primary health care (PHC) Progression model

governance and leadership ► Do countries have evidence- based PHC policies and strategies in place?

► Are there effective governance structures to implement and enforce these PHC policies?

► Is there a robust quality management infrastructure for PHC, includ-ing quality policies and strategies, legislation and regulation, quality standards, and use of continuous quality improvement programmes and methods?

► Does the country have a system that formalises and ensures strong social accountability mechanisms, including the systematic engagement of private sector, civil society, non- governmental or-ganisations, and non- health actors in the integrated planning and governance of PHC and public disclosure of performance?

Adjustment to population health needs ► Do countries have comprehensive and reliable surveillance sys-tems in place to detect and respond to changing disease burden and emerging outbreaks?

► Are national health priorities set based on disease burden, health outcomes, and user needs?

► Does the PHC sector have a learning system that prioritises contin-ual reflection and improvement?

Inputs ► Are key inputs—including drugs and supplies, facility infrastruc-ture, information systems, health workforce, and funds at the facil-ity level—available?

► Are they equitably distributed? ► Are they of sufficiently high quality to meet population health needs?

Population health management ► Are local priorities evidence based and determined in collaboration with local communities and stakeholders?

► Do communities have input to and impact on the way that PHC is financed, governed, and implemented?

► Is a system of empanelment, or rostering, in place to ensure that the entire population is known to the health system and that specific service providers have responsibility for specific panels of patients?

► Does proactive population outreach occur to deliver essential health services to those in need?

facility organisation and management ► Are primary care services organised and delivered by effective pro-vider teams, capable of ensuring comprehensive and coordinated care?

► Are facilities effectively led by managers with the ability to organise operations, motivate staff, and deploy resources?

► Do facilities set performance targets, have staff capacity to capture and use data at the point of care to monitor and improve perfor-mance, and implement quality improvement activities?

► Is supportive supervision routinely conducted?

box 2 relevant lessons learnt on mixed- methods, rubric- based tool design and implementation from a review of existing tools

► Rubric and standards- based tools can be used to assess across multiple, related domains.

► Most assessment tools employ a four or five- level categorisation, which enables nuanced descriptions of performance levels and/or progressive levels of maturity that can more completely cap-ture complex topics and progression over time than simple binary indicators.

► These tools can be completed based on user opinion, qualitative data, quantitative data, document review and/or combinations of any of the above. Assessment can be internally driven by the stake-holders who participate in the activity being assessed or conducted by an objective, external body. Decisions around which methodol-ogy(ies) to use should be driven by the objectives of the assess-ment—for example, for comparison across sites or internal quality improvement purposes.

► The time and resources required to complete assessments are closely linked to the methods used.

► The assessment tool can be a vehicle for bringing together diverse stakeholders whose work relates to the same topic or output, but who may not have the opportunity to regularly interact with and learn from one another.

the best practices around tool design and implementa-tion (box 2). Additional information about these tools is available in online supplementary file 1.

Based on these lessons learnt, PHCPI decided that the new assessment tool should be implemented through a joint internal and external assessment process led by

in- country teams who rigorously document findings to collectively determine scores that are subsequently reviewed by an external team to ensure that results are rooted in evidence and that performance standards are consistently applied across countries. Such a process offered the best means of achieving PHCPI’s two goals for the assessment: (1) to produce a national assess-ment of PHC capacity that is acceptable to and owned by policymakers in LMIC and used to drive improvement efforts and (2) to drive accountability and improvement through the public release of Vital Signs Profiles that are standardised across countries and enable peer- to- peer benchmarking and learning.

develoPmenT of THe ToolPHCPI undertook a structured, iterative, and partici-patory process to develop the PHC Progression Model (figure 2). The first step was a targeted review of relevant global health frameworks, toolkits, and data collection instruments15–20 to identify any indicators or normative standards related to the content to be assessed via the PHC Progression Model. Drawing where possible from these validated tools and supplementing with qualitative indicators to create a set of rubrics outlining four progres-sive performance categories for each topic assessed, we completed a first draft of the PHC Progression Model. To assess the face validity of the tool, we then partnered with the Alliance for Health Policy and Systems Research at the World Health Organization (WHO) to complete mock assessments using the seven Primary Health Care Systems case studies that were available at the time (Bang-ladesh, Ethiopia, Ghana, Kenya, Nigeria, Tanzania and

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

4 Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

BMJ Global Health

Figure 2 Timeline of the development of the PHC Progression Model. LMIC, low- income and middle- income country; PHC, primary healthcare; PHCPI, Primary Health Care Performance Initiative.

Uganda).21 Based on the findings of this exercise, we made adjustments and generated a second draft of the tool.

Next, we identified 75 individuals with expertise in specific areas covered by the PHC Progression Model and conducted key informant interviews and online surveys. Each expert was asked to review the sections of the tool relevant to their area of expertise and reflect on the

following dimensions of each measure: relevance, cali-bration, reliability, and comprehensiveness.

Finally, we convened an expert advisory group made up of international measurement experts as well as policy- makers from seven LMICs to review the content of the PHC Progression Model and assess whether each measure included in the tool captured the right infor-mation, was clear and well calibrated, and was feasible to assess in the represented countries. Based on all expert input received, we updated the PHC Progression Model to a version ready for pilot testing.

THe PHC ProgressIon modelThe resulting PHC Progression Model is a mixed- methods assessment tool for measuring the foundational capacities of PHC.22 The model consists of 32 measures covering the content areas described in box 1. As shown in figure 3, each measure includes a rubric that is used to assign a country to one of four performance catego-ries ranging from level 1 (low) to level 4 (high). The criteria for levels 1–4 vary according to the content of each measure being assessed; individual criteria can be found in the assessment tool.22 Data and evidence for completing the assessment can be drawn from a variety of sources available within countries, including policy docu-ments; routine reports and assessments; data elements from global surveys and/or locally owned and generated data; and key informant interviews with a variety of public and private, governmental and non- governmental organ-isations including at the subnational levels, as appro-priate depending on the measure to ensure inclusion of diverse perspectives.

Scoring of each measure of the PHC Progression Model employs a threshold approach, in which a performance level can only be achieved if all criteria described in the measure meet the performance described in the corre-sponding rubric. The 32 measure scores are summarised into nine subscores, corresponding to each subdomain of the PHCPI conceptual framework being assessed, by taking a simple unweighted average of all the constituent measures. The nine subscores are then summarised into three overall scores—Governance, Inputs, and Popula-tion Health and Facility Management—for display in the Capacity pillar of the Vital Signs Profile, again by taking simple unweighted average of the constituent subscores (figure 3).

PIloTIng of THe PHC ProgressIon modelPHCPI partnered with national Ministries of Health in five LMICs—Argentina, Ghana, Rwanda, Senegal, and the United Republic of Tanzania—to pilot a PHC Progression Model assessment in 2018. Countries were selected based on a formal expression of interest by the minister on behalf of their government, strong pre- existing relationship between one of the PHPCI partners and the ministry, and availability of resources to support the assessment.

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822 5

BMJ Global Health

Figure 3 Structure of the PHC Progression Model and its relationship to the Vital Signs Profile (VSP). Each of the 32 measures of the PHC Progression Model contains a rubric outlining four performance categories (Levels 1–4). Measures are grouped thematically, according to the PHCPI conceptual framework. Raw measure scores are averaged by theme into nine subscores, which are in turn averaged to calculate the three scores that appear in the Capacity pillar of the VSP. Subscores and VSP scores are rounded to the tenths place. PHC, primary healthcare; PHCPI, Primary Health Care Performance Initiative.

Each country formed a core team responsible for imple-menting the assessment in accordance with the methods and standards outlined in a standardised assessment guide. As described in table 1, the size and composition of the core team varied by country depending on factors such as the time and availability of technical staff and preferences for inclusivity across multiple organisations or divisions within the Ministry of Health. Most often, the core team contained technical staff from Ministries of Health with expertise in and oversight of PHC in their country, as well as technical consultants who were often responsible for data collection.

The assessment team began by contextualising the assessment to the country context, a process which entailed agreeing to local definitions for: (1) the package of services considered to make up ‘PHC’, (2) the facili-ties considered to be ‘PHC facilities’ and (3) the human resources for health considered to be ‘PHC human resources’. Next, country teams undertook a detailed review of the types of data that would be required to score each measure, identified potential quantitative and qual-itative sources, and made a plan for efficiently collecting needed data. When key informant interviews at the subnational level were deemed necessary, country teams

also developed a sampling strategy to ensure that subna-tional data sources would yield a representative picture of what was truly occurring in the country.

Countries approached data collection in diverse ways, with some choosing to centralise the process within one or two individuals and others distributing responsibility to multiple members of the assessment team. Teams based decisions around considerations of feasibility, accept-ability and effectiveness, including: how to structure their assessment process based on experience with similar assessment methods, for example, the Joint External Eval-uation12; the time and resources key stakeholders could commit to the process; and an understanding of what steps would be necessary to generate local ownership of the results and be most likely to encourage the use of results to drive improvement efforts. Table 1 summarises the different assessment processes undertaken by each country team.

Data collection plans were approved by stakeholders such as high- level representatives from Ministries of Health and core implementing partners whose accep-tance of the results would be critical for ensuring resulting data would be used to drive improvement efforts. Teams then completed data collection and used templates

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

6 Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

BMJ Global Health

Tab

le 1

P

HC

Pro

gres

sion

Mod

el a

sses

smen

t st

rate

gies

use

d a

cros

s co

untr

ies

Arg

enti

naG

hana

Rw

and

aS

eneg

alTa

nzan

ia

Ext

erna

l (P

HC

PI)

team

Le

ad o

rgan

isat

ion

Wor

ld B

ank

Gro

upR

esul

ts fo

r D

evel

opm

ent

Bill

& M

elin

da

Gat

es F

ound

atio

nW

orld

Ban

k G

roup

Wor

ld B

ank

Gro

up

C

onte

xt o

f re

latio

nshi

ps

in

coun

try

Eng

aged

via

a U

S$3

00 m

illio

n in

vest

men

t fo

cuse

d o

n su

pp

ortin

g ef

fect

ive

Uni

vers

al

Hea

lth C

over

age.

Res

ults

for

Dev

elop

men

t ha

s a

serie

s of

long

- sta

ndin

g p

artn

ersh

ips

with

gov

ernm

enta

l ag

enci

es in

Gha

na, p

rimar

ily

focu

sed

on

heal

th fi

nanc

ing

and

he

alth

sys

tem

s st

reng

then

ing.

Tech

nica

l par

tner

ship

on

pro

ject

s re

late

d t

o te

lehe

alth

and

dro

ne- d

eliv

ered

co

mm

oditi

es.

Eng

agem

ent

with

gov

ernm

ent

thro

ugh

dev

elop

men

t of

Glo

bal

Fi

nanc

ing

Faci

lity

Inve

stm

ent

Cas

e an

d n

ew W

orld

Ban

k In

vest

men

t P

roje

ct fi

nanc

ing

focu

sed

on

imp

rovi

ng m

ater

nal,

child

, and

ad

oles

cent

hea

lth.

Eng

aged

via

a U

S$2

00 m

illio

n P

rogr

am- f

or- R

esul

ts fo

cuse

d

on s

tren

gthe

ning

PH

C s

yste

m

per

form

ance

.

In- c

ount

ry t

eam

M

inis

teria

l en

gage

men

tM

inis

teria

l ap

pro

val t

o in

itiat

e en

gage

men

t.A

ctiv

e m

inis

teria

l inv

olve

men

t an

d e

ngag

emen

t th

roug

hout

d

ata

colle

ctio

n an

d in

tern

al

scor

ing

exer

cise

.M

inis

teria

l eng

agem

ent

to r

evie

w

resu

lts a

nd a

pp

rove

rel

ease

.

Min

iste

rial a

pp

rova

l to

initi

ate

enga

gem

ent.

Ste

erin

g C

omm

ittee

mad

e up

of

lead

ers

from

the

MO

H, G

hana

H

ealth

Ser

vice

, Nat

iona

l Hea

lth

Insu

ranc

e A

utho

rity,

and

loca

l re

pre

sent

ativ

es fr

om t

he W

orld

B

ank

and

WH

O a

pp

rove

d m

etho

ds

and

find

ings

.M

inis

teria

l eng

agem

ent

to r

evie

w

resu

lts a

nd a

pp

rove

rel

ease

.

Min

iste

rial a

pp

rova

l to

initi

ate

enga

gem

ent.

Min

iste

r of

Sta

te e

ngag

ed a

t as

sess

men

t ou

tset

to

app

rove

pro

cess

and

giv

e gr

een

light

to

cond

uct

dat

a co

llect

ion.

Act

ive

min

iste

rial i

nvol

vem

ent

and

en

gage

men

t th

roug

hout

dat

a co

llect

ion

and

inte

rnal

sco

ring

exer

cise

.M

inis

teria

l eng

agem

ent

to r

evie

w r

esul

ts

and

ap

pro

ve r

elea

se.

Min

iste

rial a

pp

rova

l to

initi

ate

enga

gem

ent.

Min

iste

rial e

ngag

emen

t to

re

view

res

ults

and

ap

pro

ve

rele

ase.

Min

iste

rial a

pp

rova

l to

initi

ate

enga

gem

ent.

Per

man

ent

Sec

reta

ry e

ngag

ed

at a

sses

smen

t ou

tset

to

app

rove

p

roce

ss a

nd g

ive

gree

n lig

ht t

o co

nduc

t d

ata

colle

ctio

n.M

inis

teria

l eng

agem

ent

to r

evie

w

resu

lts a

nd a

pp

rove

rel

ease

.

S

enio

r of

ficia

l en

gage

men

tP

ositi

on: U

nder

Sec

reta

ry o

f P

ublic

Hea

lth C

are

Cov

erag

e,

Nat

iona

l Dire

ctio

n of

Qua

lity

in H

ealth

Ser

vice

s an

d H

ealth

R

egul

atio

ns, a

nd O

ffice

of t

he

Gen

eral

Coo

rdin

atio

n U

nit

of t

he

Nat

iona

l MO

HR

ole:

Set

dire

ctio

n of

as

sess

men

t, o

vers

aw t

he

com

pos

ition

of t

echn

ical

tea

m,

enga

ged

as

key

info

rman

ts,

and

pro

vid

ed p

oliti

cal s

upp

ort

thro

ugho

ut t

he p

roce

ss.

Pos

ition

: Dire

ctor

of P

olic

y,

Pla

nnin

g, M

onito

ring

and

E

valu

atio

n w

ithin

Gha

na H

ealth

S

ervi

ceR

ole:

Set

dire

ctio

n of

ass

essm

ent,

ov

ersa

w c

omp

ositi

on o

f tec

hnic

al

team

, eng

aged

as

key

info

rman

t,

and

pro

vid

ed p

oliti

cal s

upp

ort

thro

ugho

ut t

he p

roce

ss.

Pos

ition

: Dire

ctor

Gen

eral

of P

lann

ing,

H

ealth

Fin

anci

ng a

nd In

form

atio

n S

yste

ms

and

Dire

ctor

Gen

eral

of R

wan

da

Bio

med

ical

Cen

ter

Rol

e: A

ctiv

ely

invo

lved

thr

ough

out

dat

a co

llect

ion,

eng

aged

as

key

info

rman

ts,

pro

vid

ed p

oliti

cal s

upp

ort,

and

par

ticip

ated

in

inte

rnal

sco

ring

exer

cise

.

Pos

ition

: Dire

ctor

of P

lann

ing,

R

esea

rch

and

Sta

tistic

sR

ole:

Set

dire

ctio

n of

as

sess

men

t, o

vers

aw

com

pos

ition

of t

echn

ical

tea

m,

enga

ged

as

key

info

rman

t,

and

pro

vid

ed p

oliti

cal s

upp

ort

thro

ugho

ut t

he p

roce

ss.

Pos

ition

: Dire

ctor

of D

epar

tmen

t of

Qua

lity

Ass

uran

ceR

ole:

Set

dire

ctio

n of

as

sess

men

t, o

vers

aw

com

pos

ition

of t

echn

ical

tea

m,

enga

ged

as

key

info

rman

t,

and

pro

vid

ed p

oliti

cal s

upp

ort

thro

ugho

ut t

he p

roce

ss.

Con

tinue

d

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822 7

BMJ Global Health

Arg

enti

naG

hana

Rw

and

aS

eneg

alTa

nzan

ia

Te

chni

cal t

eam

Ext

erna

l, in

- cou

ntry

con

sulta

nt

team

mad

e up

of f

our

rese

arch

ers

from

the

Inst

itute

of

Clin

ical

Effe

ctiv

enes

s an

d H

ealth

P

olic

y.TW

G m

ade

up o

f the

thr

ee

MO

H fo

cal p

oint

s, e

xter

nal

cons

ulta

nts,

and

ad

diti

onal

MO

H

staf

f and

dire

ctor

s (fo

ur o

r m

ore,

d

epen

din

g on

the

top

ic u

nder

d

iscu

ssio

n), w

ith s

upp

ort

from

in

- cou

ntry

Wor

ld B

ank

staf

f.

Tw

o ex

tern

al, i

n- co

untr

y d

octo

ral-

leve

l qua

ntita

tive

and

q

ualit

ativ

e co

nsul

tant

s

Pro

ject

coo

rdin

ator

.

TWG

led

by

Dep

uty

Dire

ctor

of

Pol

icy,

Pla

nnin

g, M

onito

ring

and

Eva

luat

ion

of t

he G

hana

H

ealth

Ser

vice

and

com

pris

ed

rep

rese

ntat

ives

from

:

►

MO

H

►

Uni

vers

ity o

f Gha

na

►

Gha

na H

ealth

Ser

vice

►

G

hana

Sta

tistic

al S

ervi

ce

►

Nat

iona

l Hea

lth In

sura

nce

Aut

horit

y

►

UN

ICE

F G

hana

Tw

o ex

tern

al, i

n- co

untr

y co

nsul

tant

s,

one

PhD

and

one

mas

ters

- lev

el

rese

arch

er.

W

orki

ng g

roup

led

by

Dire

ctor

of

Mon

itorin

g an

d E

valu

atio

n an

d R

wan

da

Bio

med

ical

Cen

ter

and

the

H

ealth

Info

rmat

ion

Sys

tem

s an

d

Rep

ort

Lead

Sp

ecia

list

at t

he M

OH

and

co

mp

rised

rep

rese

ntat

ives

from

:

►

MO

H

►

Rw

and

a B

iom

edic

al C

ente

r

►

Rw

and

a N

ursi

ng C

ounc

il

►

Dis

tric

t ho

spita

ls

►

WH

O R

wan

da

Offi

ce

►

Man

agem

ent

Sci

ence

s fo

r H

ealth

an

d o

ther

dev

elop

men

t p

artn

ers

Tw

o ex

tern

al, i

n- co

untr

y d

octo

ral-

leve

l con

sulta

nts

C

ore

wor

king

gro

up le

d

by

Gen

eral

Dire

ctor

for

Hea

lth S

ervi

ces,

com

pris

ed

rep

rese

ntat

ives

from

:

►

WH

O S

eneg

al C

ount

ry

Offi

ce

►

Wor

ld B

ank

Gro

up

Sen

egal

►

N

atio

nal S

tatis

tics

and

D

emog

rap

hy A

genc

y

►

Priv

ate

Sec

tor

Alli

ance

►

U

nive

rsity

and

Res

earc

h In

stitu

te

E

xter

nal,

in- c

ount

ry d

octo

ral-

le

vel c

onsu

ltant

W

orki

ng G

roup

com

pris

ed:

►

R

epre

sent

ativ

e fr

om

the

MO

H, C

omm

unity

D

evel

opm

ent,

Gen

der

, Eld

erly

an

d C

hild

ren

►

N

atio

nal P

rofe

ssio

nal

Offi

cer

for

Fam

ily a

nd

Rep

rod

uctiv

e H

ealth

of W

HO

Ta

nzan

ia C

ount

ry O

ffice

►

S

enio

r E

cono

mis

t of

the

W

orld

Ban

k G

roup

, Tan

zani

a

Ass

essm

ent

pro

cess

P

rep

arat

ion

PH

CP

I orie

nted

con

sulta

nts

and

p

rovi

ded

ove

rvie

w o

f pro

cess

.TW

G a

nd S

teer

ing

Com

mitt

ee

form

ed.

PH

CP

I orie

nted

TW

G a

nd p

rovi

ded

ov

ervi

ew o

f pro

cess

.

PH

CP

I orie

nted

con

sulta

nts

and

sen

ior

offic

ials

and

pro

vid

ed o

verv

iew

of p

roce

ss.

PH

CP

I orie

nted

con

sulta

nts

and

sen

ior

offic

ials

and

p

rovi

ded

ove

rvie

w o

f pro

cess

.

PH

CP

I orie

nted

and

pro

vid

ed

over

view

of P

HC

Pro

gres

sion

M

odel

ass

essm

ent

pro

cess

to

cons

ulta

nts

and

wor

king

gro

up.

Id

entifi

catio

n of

d

ata

sour

ces

Res

pon

sib

le p

erso

n(s)

: C

onsu

ltant

tea

m.

MO

H fo

cal p

oint

s re

view

ed a

nd

app

rove

d.

Res

pon

sib

le p

erso

n(s)

: TW

G.

Res

pon

sib

le p

erso

n(s)

: Tec

hnic

al t

eam

.W

orki

ng g

roup

rev

iew

ed a

nd a

pp

rove

d.

Res

pon

sib

le p

erso

n(s)

: Cor

e w

orki

ng g

roup

, sen

ior

MO

H

offic

ial a

nd c

onsu

ltant

s.

Res

pon

sib

le p

erso

n(s)

: C

onsu

ltant

and

wor

king

gro

up.

D

ata

colle

ctio

nR

esp

onsi

ble

per

son(

s):

Con

sulta

nts

17 in

terv

iew

s co

mp

lete

d.

22 d

ocum

ents

rev

iew

ed.

Reg

iona

l/loc

al p

ersp

ectiv

e no

t av

aila

ble

due

to

regu

lato

ry

com

plia

nce

and

con

curr

ent

pol

itica

l eve

nts,

so

rele

vant

par

ts

of t

he a

sses

smen

t (P

HFM

) not

co

mp

lete

d a

t th

is t

ime.

Res

pon

sib

le p

erso

n(s)

: Qua

litat

ive

cons

ulta

nt a

nd T

WG

Ap

pro

xim

atel

y 70

inte

rvie

ws

com

ple

ted

in t

otal

by

the

TWG

and

co

nsul

tant

.A

pp

roxi

mat

ely

40 d

ocum

ents

re

view

ed.

Reg

iona

l/loc

al p

ersp

ectiv

e en

sure

d

by

sam

plin

g a

rep

rese

ntat

ive

set

of

five

regi

ons

and

tw

o d

istr

icts

(one

ur

ban

and

one

rur

al) p

er r

egio

n an

d

cond

uctin

g in

terv

iew

s w

ith r

elev

ant

regi

onal

and

dis

tric

t st

aff.

Qua

ntita

tive

dat

a us

ed c

ame

from

re

por

ts a

nd d

ocum

ents

—no

de

novo

ana

lysi

s co

nduc

ted

.

Res

pon

sib

le p

erso

n(s)

: Con

sulta

nts

25 in

terv

iew

s co

mp

lete

d.

105

doc

umen

ts r

evie

wed

and

10

mai

n d

ata

web

site

s co

nsul

ted

.TW

G a

tten

ded

by

25 p

eop

le in

clud

ing

the

MO

H, t

he D

irect

or G

ener

al o

f Rw

and

a B

iom

edic

al C

ente

r, th

e D

irect

or G

ener

al

in c

harg

e of

pla

nnin

g, d

iffer

ent

Min

istr

y’s

dep

artm

ent

rep

rese

ntat

ives

, dis

tric

t ho

spita

ls d

irect

ors,

rep

rese

ntat

ives

of

dev

elop

men

t p

artn

ers,

and

loca

l civ

il so

ciet

y or

gani

satio

ns.

Qua

ntita

tive

dat

a us

ed c

ame

from

rep

orts

an

d d

ocum

ents

—no

de

novo

ana

lysi

s co

nduc

ted

.

Res

pon

sib

le p

erso

n(s)

: C

onsu

ltant

s22

inte

rvie

ws

com

ple

ted

.28

doc

umen

ts r

evie

wed

.R

egio

nal/l

ocal

per

spec

tive

ensu

red

thr

ough

att

end

ance

by

TWG

at

scor

ing

exer

cise

.

Res

pon

sib

le p

erso

n(s)

: C

onsu

ltant

12 in

terv

iew

s co

mp

lete

d b

y co

nsul

tant

.25

doc

umen

ts r

evie

wed

by

cons

ulta

nt.

8 d

atas

ets

min

ed fo

r q

uant

itativ

e d

ata,

incl

udin

g th

e S

TAR

rat

ing

syst

em, a

n ac

cred

itatio

n sy

stem

th

at c

olle

cts

dat

a on

an

annu

al

bas

is fr

om a

ll fa

cilit

ies.

Reg

iona

l/loc

al p

ersp

ectiv

e en

sure

d b

y en

gage

men

t w

ith t

he

Pre

sid

ent’s

Offi

ce fo

r R

egio

nal

and

Loc

al G

over

nanc

e.

D

ata

synt

hesi

sR

esp

onsi

ble

per

son(

s):

Con

sulta

nts

Res

pon

sib

le p

erso

n(s)

: Qua

litat

ive

cons

ulta

nt a

nd p

rogr

amm

e co

ord

inat

or. S

upp

ort

pro

vid

ed b

y R

esul

ts fo

r D

evel

opm

ent

staf

f.

Res

pon

sib

le p

erso

n(s)

: Con

sulta

nts

Res

pon

sib

le p

erso

n(s)

: C

onsu

ltant

sR

esp

onsi

ble

per

son(

s):

Con

sulta

nt

Tab

le 1

C

ontin

ued

Con

tinue

d

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

8 Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

BMJ Global Health

Arg

enti

naG

hana

Rw

and

aS

eneg

alTa

nzan

ia

In

tern

al s

corin

gC

onsu

ltant

s w

ith in

put

by

MO

H’s

fo

cal p

oint

s.TW

G a

nd c

onsu

ltant

s co

mp

lete

d

inte

rnal

sco

ring.

Wor

king

Gro

up a

nd c

onsu

ltant

s co

nven

ed

to c

omp

lete

inte

rnal

sco

ring.

Larg

e TW

G m

ade

up o

f 15

exp

erts

from

acr

oss

Sen

egal

co

nven

ed t

o of

fer

evid

ence

an

d p

ersp

ectiv

es t

hrou

gh t

he

inte

rnal

sco

ring

exer

cise

.

Wor

king

Gro

up a

nd c

onsu

ltant

co

nven

ed t

o co

mp

lete

inte

rnal

sc

orin

g.W

orks

hop

with

bro

ad

rep

rese

ntat

ion

from

rel

evan

t M

OH

dep

artm

ents

and

par

tner

or

gani

satio

ns, i

nclu

din

g al

l who

se

rved

as

key

info

rman

ts, h

eld

to

rev

iew

inte

rnal

sco

ring

and

p

rovi

de

add

ition

al in

put

.

MO

H, M

inis

try

of H

ealth

; PH

C, p

rimar

y he

alth

care

; PH

CP

I, P

rimar

y H

ealth

Car

e P

erfo

rman

ce In

itiat

ive;

PH

FM, P

opul

atio

n H

ealth

and

Fac

ility

Man

agem

ent;

TW

G, T

echn

ical

Wor

king

Gro

up.

Tab

le 1

C

ontin

ued

provided by the PHCPI team to compile and synthesise relevant information from across all data sources for each measure.

These data syntheses were then used as the evidence base for an internal scoring exercise, which consisted of convening a stakeholder group to review all of the assembled evidence for each measure and use the rubric to assign the country’s performance to one of the four performance categories. In the event that assessment teams had been unable to identify sufficient data to score a measure, a score of level 1 was assigned. (In Argentina, the federal administrative nature of the health system and current events made assessment of Population Health and Facility Management unfeasible, and results were instead displayed as ‘N/A’.) Participants in the scoring exercise were selected by the assessment team based on how best to generate buy- in for and acceptance of the results.

Next, the results of the internal scoring exercise, along with all supporting evidence, were shared with the PHCPI country engagement lead and a team from PHCPI partner organisation Ariadne Labs for external validation. The goals of the external validation were to ensure that the available evidence justified the scores given by the country team and that measurement stan-dards were being consistently applied across countries. Often, the external assessment process resulted in the identification of measures where more detailed evidence was needed to justify the internal scores; in these cases, the external and country assessment teams would engage in ongoing dialogue and review of additional evidence until agreement was reached on the appropriate score. Final results were then integrated into the Vital Signs Profile, which was presented to the Minister of Health or equivalent for approval to be released. The typical length of time to complete the entire PHC Progression Model assessment process was approximately 3 months.

fIndIngs And lessons leArnTThe PHC Progression Model is a novel tool for system-atically assessing PHC capacity at a national level and provides a basis for countries to track their progress in creating better conditions for stronger performance over time. Additionally, due to its standardised methodology and structure, the PHC Progression Model and the Vital Signs Profile overall enables countries to engage in cross- country learning and peer- to- peer benchmarking if they choose, though neither tool is intended to be used for direct ranking or comparison purposes.

PHCPI conducted targeted outreach with imple-menters of the first five pilot assessments to collect their insights on the process and lessons learnt. Overall, the results of the five PHC Progression Model assessments (table 2) demonstrate that the process and methodology were feasible and acceptable. The measurement tool was able to be implemented with fidelity and to detect meaningful variation in PHC system capacity across and

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822 9

BMJ Global Health

Tab

le 2

C

omp

aris

on o

f int

erna

l, ex

tern

al a

nd c

onse

nsus

sco

res

acro

ss fi

ve p

ilot

coun

trie

s

VS

P D

om

ain

Sub

do

mai

nM

easu

reA

rgen

tina

Gha

naR

wan

da

Sen

egal

Tanz

ania

Alig

nmen

t o

f in

tern

al, e

xter

nal,

and

co

nsen

sus

sco

res

(n, %

)

Vari

atio

n b

etw

een

inte

rnal

sc

ore

and

co

nsen

sus

sco

re

of

2+ (n

, %)

IE

CI

EC

IE

CI

EC

IE

C

Gov

erna

nce

2.9

2.6

3.0

2.9

2.6

2.8

3.5

3.2

3.5

2.5

2.4

2.4

2.3

2.5

2.9

Gov

erna

nce

and

le

ader

ship

2.4

1.8

2.6

3.2

2.6

3.0

3.6

3.4

3.6

2.4

2.4

2.4

3.2

3.2

3.2

1: P

HC

pol

icie

s (1

/2)

22

24

33

44

42

33

34

32

(40)

0 (0

)

2: P

HC

pol

icie

s (2

/2)

11

23

23

33

32

22

33

33

(60)

0 (0

)

3: Q

ualit

y m

anag

emen

t in

fras

truc

ture

32

34

34

42

32

22

33

32

(40)

0 (0

)

4: S

ocia

l ac

coun

tab

ility

(1/2

)2

22

33

33

44

32

23

33

3 (6

0)0

(0)

5: S

ocia

l ac

coun

tab

ility

(2/2

)4

24

22

24

44

33

34

34

3 (6

0)0

(0)

Ad

just

men

t to

p

opul

atio

n he

alth

ne

eds

3.3

3.3

3.3

2.7

2.7

2.7

3.3

3.0

3.3

2.7

2.3

2.3

2.7

2.7

2.7

6: S

urve

illan

ce4

44

33

33

33

43

33

33

4 (8

0)0

(0)

7: P

riorit

y S

ettin

g3

33

22

23

23

22

22

22

4 (8

0)0

(0)

8: In

nova

tion

and

Le

arni

ng3

33

33

34

44

22

23

33

5 (1

00)

0 (0

) Con

tinue

d

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

10 Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

BMJ Global Health

VS

P D

om

ain

Sub

do

mai

nM

easu

reA

rgen

tina

Gha

naR

wan

da

Sen

egal

Tanz

ania

Alig

nmen

t o

f in

tern

al, e

xter

nal,

and

co

nsen

sus

sco

res

(n, %

)

Vari

atio

n b

etw

een

inte

rnal

sc

ore

and

co

nsen

sus

sco

re

of

2+ (n

, %)

Inp

uts

2.0

1.7

2.7

2.3

2.3

2.4

3.1

2.6

2.6

2.4

2.1

2.1

1.8

1.3

2.2

Dru

gs a

nd s

upp

lies

1.3

1.3

3.0

1.3

1.0

1.3

3.7

3.0

3.0

2.3

2.0

2.0

2.3

1.0

2.3

9: S

tock

- out

of

esse

ntia

l med

icin

es

and

con

sum

able

co

mm

oditi

es

11

41

12

43

31

11

31

21

(20)

1 (2

0)

10: B

asic

eq

uip

men

t2

22

11

24

33

33

32

13

2 (4

0)0

(0)

11: D

iagn

ostic

su

pp

lies

11

32

13

33

33

22

21

21

(20)

1 (2

0)

Faci

lity

infr

astr

uctu

re2.

31.

73.

01.

71.

71.

72.

72.

02.

02.

02.

02.

01.

71.

72.

0

12: F

acili

ty D

ensi

ty3

33

33

32

22

33

32

22

5 (1

00)

0 (0

)

13: F

acili

ty

amen

ities

31

31

11

33

32

22

22

24

(80)

0 (0

)

14: S

tand

ard

saf

ety

pre

caut

ions

and

eq

uip

men

t

11

31

11

31

11

11

11

22

(40)

2 (4

0)

Info

rmat

ion

syst

ems

3.0

3.0

3.3

2.3

2.3

2.3

2.7

2.3

2.3

2.7

1.7

1.7

2.0

1.7

2.0

15: C

ivil

regi

stra

tion

and

vita

l sta

tistic

s3

34

11

11

11

31

11

11

3 (6

0)1

(20)

16: H

ealth

m

anag

emen

t in

form

atio

n sy

stem

s

33

33

33

43

33

33

33

34

(80)

0 (0

)

17: P

erso

nal c

are

reco

rds

33

33

33

33

32

11

21

23

(60)

0 (0

)

Wor

kfor

ce2.

31.

32.

03.

03.

03.

32.

71.

71.

72.

51.

71.

71.

31.

31.

3

18: D

ensi

ty a

nd

dis

trib

utio

n1

11

11

11

11

11

11

11

5 (1

00)

0 (0

)

19: T

rain

ing

31

23

22

42

22

22

22

22

(40)

1 (2

0)

20: C

omm

unity

he

alth

wor

kers

21

23

44

32

23

22

11

11

(20)

0 (0

)

Fund

s1.

01.

02.

33.

33.

33.

34.

04.

04.

02.

73.

03.

03.

33.

33.

3

21: F

acili

ty b

udge

ts1

13

33

34

44

23

33

33

3 (6

0)1

(20)

22: F

inan

cial

m

anag

emen

t in

form

atio

n sy

stem

11

33

33

44

43

33

33

34

(80)

1 (2

0)

23: S

alar

y p

aym

ent

11

14

44

44

43

33

44

45

(100

)0

(0)

Tab

le 2

C

ontin

ued

Con

tinue

d

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822 11

BMJ Global Health

VS

P D

om

ain

Sub

do

mai

nM

easu

reA

rgen

tina

Gha

naR

wan

da

Sen

egal

Tanz

ania

Alig

nmen

t o

f in

tern

al, e

xter

nal,

and

co

nsen

sus

sco

res

(n, %

)

Vari

atio

n b

etw

een

inte

rnal

sc

ore

and

co

nsen

sus

sco

re

of

2+ (n

, %)

Pop

ulat

ion

Hea

lth

and

Fac

ility

M

anag

emen

t

--*

----

2.4

3.0

2.7

3.8

2.4

3.1

2.2

2.3

2.3

2.0

1.0

2.1

Pop

ulat

ion

heal

th

man

agem

ent

----

--2.

03.

02.

53.

52.

53.

32.

32.

32.

32.

31.

52.

0

24: L

ocal

prio

rity

sett

ing

----

--2

33

44

42

22

32

32

(50)

0 (0

)

25: C

omm

unity

en

gage

men

t--

----

34

34

33

22

22

22

2 (5

0)0

(0)

26: E

mp

anel

men

t--

----

11

12

22

22

22

11

2 (5

0)0

(0)

27: P

roac

tive

pop

ulat

ion

outr

each

----

--2

43

41

43

33

21

21

(25)

0 (0

)

Faci

lity

orga

nisa

tion

and

man

agem

ent

----

--2.

83.

02.

84.

02.

23.

02.

22.

42.

42.

02.

02.

2

28: T

eam

- bas

ed

care

org

anis

atio

n--

----

33

34

12

22

21

11

3 (7

5)1

(25)

29: F

acili

ty

man

agem

ent

cap

abili

ty a

nd

lead

ersh

ip

----

--2

22

41

22

22

22

23

(75)

1 (2

5)

30: I

nfor

mat

ion

syst

em u

se--

----

33

34

13

22

22

22

3 (7

5)0

(0)

31: P

erfo

rman

ce

mea

sure

men

t an

d

man

agem

ent

(1/2

)

----

--3

33

44

42

33

34

32

(50)

0 (0

)

32: P

erfo

rman

ce

mea

sure

men

t an

d m

anag

emen

t (2

/2):

Sup

por

tive

Sup

ervi

sion

----

--3

43

44

43

33

21

32

(50)

0 (0

)

Mea

sure

s fo

r w

hich

sco

res

wer

e al

igne

d a

cros

s in

tern

al, e

xter

nal,

and

con

sens

us s

core

s (n

, %)

11 (4

6)22

(69)

17 (5

3)22

(69)

20 (6

3)

Mea

sure

s fo

r w

hich

inte

rnal

sco

re v

arie

d b

y tw

o or

mor

e p

erfo

rman

ce le

vels

from

the

con

sens

us s

core

(n, %

)5

(21)

0 (0

)4

(13)

1 (3

)0

(0)

Col

our

cod

ing:

Red

=sc

ore

of 1

.0–1

.9; o

rang

e=sc

ore

of 2

.0–2

.9; y

ello

w=

scor

e 3.

0–3.

9; g

reen

=sc

ore

of 4

.0.

*Pop

ulat

ion

heal

th a

nd fa

cilit

y m

anag

emen

t w

as n

ot a

sses

sed

in A

rgen

tina.

C, c

onse

nsus

sco

re; E

, ext

erna

l sco

re;I,

inte

rnal

sco

re; P

HC

, prim

ary

heal

th c

are;

VS

P, V

ital S

igns

Pro

file.

Tab

le 2

C

ontin

ued

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

12 Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

BMJ Global Health

box 3 Implementation steps that enabled success across countries

► Positioned within an ongoing strategic effort of the government to improve primary health care so that the assessment is not per-ceived as an ad hoc or standalone effort.

► Obtaining high- level buy- in and leadership from the ministerial level.

► Careful messaging of the assessment’s purpose as the first step in an improvement effort rather than as a punitive or audit tool.

► Fostering a participatory assessment process. ► Customising the assessment strategy—including working group composition, data collection strategies, and approach to the scoring exercise—to meet country expectations and norms.

► Deep engagement by in- country technical teams with ongoing, trusting relationships with government officials.

► Strong relationships between Primary Health Care Performance Initiative and the local assessment teams that established trust, mutual respect, and transparency.

within countries. The tool also had internal validity, with internal and external scores being highly aligned—on average across all countries, 60% of measure scores were fully aligned across internal, external, and consensus scores while only 7% of scores differed by two or more performance levels between internal and final consensus scores. Only one measure had two or more countries differ by at least two performance levels from the internal to final consensus score. Importantly, not all differences between the internal and external scores shown in table 2 were due to PHCPI ‘correcting’ internal scores; in cases with misalignment, external scores were both higher and lower than internal scores, and it was typical that discus-sions about the difference between internal and external scores would surface implicit knowledge being applied by in- country teams and lead to the identification of additional data sources and evidence to justify the initial scores and resolve discrepancies.

Overall, we found that completion of the PHC Progres-sion Model assessment was a process which generated valuable new collaborations and insights for countries. The information needed to score individual measures and complete the assessment was often located within a multitude of documents and key informants. Bringing together all of these different data sources created a unique opportunity to collaboratively and holistically assess and understand an individual country’s PHC capacities in a way that is difficult if not impossible to do otherwise. This process of conducting the assessment and strategy of bringing together diverse stakeholders, each of whom had deep insight into a different piece of PHC, was often as valuable for understanding PHC capacity as the actual assessment results. Stakeholders reported that the process of implementing assessments made the identification of a system’s capacity strengths and weaknesses ‘glaringly obvious’ and that the assess-ment process resulted in learnings that ‘challenged pre- existing expectations’, even for stakeholders who had long been deeply embedded in the system. Since comple-tion of assessments, all five country teams have initiated efforts to use the results of the PHC Progression Model and Vital Signs Profile to identify targeted improvement plans and inform efforts to expand data availability to more routinely measure areas assessed by the tool.

Our results also demonstrate that there are multiple implementation approaches that can be successfully employed to complete a PHC Progression Model assess-ment that is appropriate for the country context. Key core processes shared across countries that enabled success are summarised in box 3 and included contextualising the assessment within ongoing efforts by the government to improve PHC, obtaining both high- level ministerial buy- in and deep technical engagement, and customising the assessment strategy to meet local expectations and norms.

The piloting of the PHC Progression Model iden-tified limitations of the tool and assessment process. Most notably, the assessment was challenging to

implement in a federalised country (Argentina) where the high degree of decentralisation meant that national- level data sources were unable to yield sufficient, timely information about on- the- ground realities across provinces. In federalised countries, subnational rather than national assessments of PHC capacity may be both more feasible to conduct and yield more informative results. Additionally, as noted above, completion of an assessment required invest-ment of resources, including focused time from stake-holders and often necessitated hiring a consultant to support in- country efforts. We anticipate that repeat assessments in a country will be able to build on results and infrastructure established through the initial assessment, and therefore require less time and fewer resources. However, it will be critical for PHCPI to identify ways to streamline the assessment process and increase sustainability of the PHC Progression Model. Finally, the external validation process was designed to ensure that the collected data supported the scores proposed by country teams, however, external teams did not conduct quality reviews of data sources or any independent data collection efforts.

fuTure dIreCTIonsResults of each country’s assessment were released as part of their complete Vital Signs Profile at the Global Conference on PHC in Astana, Kazakhstan in 2018. Following the successful piloting of the PHC Progres-sion Model, PHCPI undertook efforts—including expert consultations and a convening of early implementers—to refine the assessment tool to address any challenges iden-tified during the pilot phase and ensure the measure-ment criteria in the tool reflect guidance and standards released since tool development began in 2017. In April 2019, PHCPI released an updated version of the assess-ment tool.23 Additionally, PHCPI updated the assessment

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822 13

BMJ Global Health

guide to incorporate lessons learnt from the pilot expe-rience; more information about the assessment guide is available on request.22 Moving forward, PHCPI has already initiated partnerships with ten additional coun-tries—with many more planned—to expand the use of this assessment.

ConClusIonThrough a structured, participatory process, PHCPI developed and piloted a novel assessment tool to systematically measure PHC system capacity. Pilot assessments in five countries found the tool to be feasible to implement with fidelity, acceptable to stakeholders, and able to produce valid results. Both the assessment process and results were found to be highly valuable to country stakeholders and are now being used to inform improvement efforts. The PHC Progression Model is a promising new approach for generating comprehensive, standardised, and action-able data on PHC capacity to complement other key performance indicators and develop a holistic under-standing of PHC strengths and weaknesses that can be used to drive improvement efforts.

Author affiliations1Ariadne Labs, Brigham and Women's Hospital & Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA2Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA3Secretaria de Gobierno de Salud, Ministerio de Salud y Desarrollo Social, Buenos Aires, Argentina4Directorate of Planning, Research and Statistics, Ministry of Health and Social Action, Dakar, Senegal5Institute for Clinical Effectiveness and Health Policy, Buenos Aires, Argentina6Ministry of Health, Kigali, Rwanda7Bill & Melinda Gates Foundation, Seattle, Washington, USA8Ministry of Health, Community Development, Gender, Elderly, and Children, Dodoma, United Republic of Tanzania9Results for Development, Washington, District of Columbia, USA10Kwame Nkrumah University of Science and Technology, Kumasi, Ghana11Division of Policy, Planning, Monitoring and Evaluation, Ghana Health Service, Accra, Ghana12Health Section, UNICEF Tanzania, Dar es Salaam, United Republic of Tanzania13Global Health Issues and Solutions, Kigali, Rwanda14Health, Nutrition and Population Global Practice, World Bank Group, Washington, District of Columbia, USA

Acknowledgements The authors gratefully acknowledge the contributions to the development and implementation of the PHC Progression Model made by the entire PHCPI working group and in particular by Beth Tritter and Hong Wang. We also thank Etienne Langlois and Abdul Ghaffar from the Alliance for Health Policy and Systems Research and all those who participated in our April 2018 expert convening for their time and feedback during the development process. Finally, we are immensely grateful to all individuals who supported the implementation of PHC Progression Model assessments in Argentina, Ghana, Rwanda, Senegal, and Tanzania.

Contributors HLR, DS, LRH and AB led the planning of the work described here. All authors contributed to the conduct of this research. HLR drafted the manuscript and all authors provided review and revisions. All authors gave their approval for publication.

funding Funding for this work was provided by the Bill & Melinda Gates Foundation. Foundation staff contributed to the implementation of the work described here and one funder representative is included as a coauthor.

Competing interests None declared.

Patient consent for publication Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

data availability statement No additional data are available.

open access This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https:// creativecommons. org/ licenses/ by/ 4. 0/.

orCId idsHannah L Ratcliffe http:// orcid. org/ 0000- 0003- 4134- 5112Lisa R Hirschhorn http:// orcid. org/ 0000- 0002- 4355- 7437Jocelyn Fifield http:// orcid. org/ 0000- 0001- 9521- 1465Sanam Roder- DeWan http:// orcid. org/ 0000- 0003- 0114- 2505Asaf Bitton http:// orcid. org/ 0000- 0002- 3633- 3806

ReFeRenceS 1. World Health Organization, United Nations Children’s Fund.

Declaration of Astana. World Health Organization and United Nations Children’s Fund, 2018.

2. Das J, Hammer J. Quality of primary care in low- income countries: facts and economics. Annu Rev Econom 2014;6:525–53.

3. Kruk ME, Chukwuma A, Mbaruku G, et al. Variation in quality of primary- care services in Kenya, Malawi, Namibia, Rwanda, Senegal, Uganda and the United Republic of Tanzania. Bull World Health Organ 2017;95:408–18.

4. Kruk M, Gage A, Arsenault C, et al. High quality health systems—time for a revolution: report of the Lancet global health Commission on high quality health systems in the SDG era. Lancet Global Health.

5. Primary Health Care Performance Initiative. Available: http://www. improvingphc. org [Accessed 14 Dec 2018].

6. Bitton A, Ratcliffe HL, Veillard JH, et al. Primary health care as a foundation for strengthening health systems in low- and middle- income countries. J Gen Intern Med 2017;32:566–71.

7. Veillard J, Cowling K, Bitton A, et al. Better measurement for performance improvement in low- and middle- income countries: the primary health care performance initiative (PHCPI) experience of conceptual framework development and indicator selection. Milbank Q 2017;95:836–83.

8. Vital Signs Profiles | PHCPI. Available: https:// improvingphc. org/ vital- signs- profiles [Accessed 5 Jan 2019].

9. Dull D. Personal communication regarding the Bill & Melinda Gates Foundation’s Supply Chain Maturity Model, 2017.

10. Grooten L, Borgermans L, Vrijhoef HJ. An instrument to measure maturity of integrated care: a first validation study. Int J Integr Care 2018;18:10.

11. Patient- Centered medical home assessment (PCMH- A): introduction to the PCMH- A. Safety net medical home initiative. Available: http://www. safetynetmedicalhome. org/ sites/ default/ files/ PCMH- A. pdf

12. Joint external evaluation (JEE) | JEE alliance. Available: https://www. jeealliance. org/ global- health- security- and- ihr- implementation/ joint- external- evaluation- jee/ [Accessed 18 Dec 2018].

13. PEPFAR country operational plan sustainability indices and Dashboards (SIDS). Available: https://www. pepfar. gov/ countries/ cop/ sids/ index. htm [Accessed 18 Dec 2018].

14. Shaw G. Transition readiness assessment tool: user guide v1.0: assessing the sustainability of harm reduction services through and beyond the transition period from global fund support to domestic funding. Eurasian harm reduction network 2016. Available: https:// eecaplatform. org/ en/ resource/ transition- readiness- assessment- tool- trat/ [Accessed 14 Jan 2019].

15. Primary Health Care Performance Initiative. Primary health care performance initiative methodology note, 2015.

16. The DHS Program - Service Provision Assessments (SPA). Available: https:// dhsprogram. com/ What- We- Do/ Survey- Types/ SPA. cfm [Accessed 21 Jan 2018].

17. World Health Organization. WHO | Service Availability and Readiness Assessment (SARA), 2015.

18. Leadership Council of the Sustainable Development Solutions Network. Indicators and a monitoring framework for the sustainable development goals: Launching a data revolution for the SDGs. leadership Council of the sustainable development solutions network, 2015.

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from

14 Ratcliffe HL, et al. BMJ Global Health 2019;4:e001822. doi:10.1136/bmjgh-2019-001822

BMJ Global Health

19. World Health Organization. Global reference list of 100 core health indicators (plus health- related SDGs). World Health Organization, 2018.

20. WHO. Monitoring the building blocks of health systems: a Handbook of indicators and their measurement strategies. World Health Organization, 2010.

21. WHO | Primary Health Care Systems (PRIMASYS) - New case studies available. Available: https://www. who. int/ alliance- hpsr/ projects/ primasys/ en/

22. PHCPI. Primary health care progression model | PHCPI. Available: https:// improvingphc. org/ primary- health- care- progression- model [Accessed 16 May 2019].

23. PHCPI. Primary health care progression model assessment tool. primary health care progression model assessment tool, 2019. Available: https:// improvingphc. org/ sites/ default/ files/ PHC- Progression% 20Model% 202019- 04- 04_ FINAL. pdf

on January 13, 2021 by guest. Protected by copyright.

http://gh.bmj.com

/B

MJ G

lob Health: first published as 10.1136/bm

jgh-2019-001822 on 13 Septem

ber 2019. Dow

nloaded from