CC Lesotho Final Report

CommCare Evaluation

Training and implementation of CommCare in Leribe District’s

TB treatment programs

Ali Flaming [Occidental College]

In collaboration with Dimagi South Africa and ICAP Lesotho

2

Acknowledgements This project would not have been possible without the support and guidance of many people. First, I would like to thank the Occidental College Young Initiative for providing the grant that funded this project. I would also like to thank Professor Khagram, my wonderful academic advisor, for supporting me throughout the entire application and planning process. At Dimagi, I would especially like to thank Nynke Brunner, Jolani De La Porte, Kara Grijalva, Kieran Sharpey-‐Schafer, and all Dimagi staff who provided feedback and support before, during, and after the evaluation in Lesotho. At ICAP, I would like to thank Koen Friederix, Blanche Pitt, Chris Linda, Tlohang Moeketse, Mateboho Mokobocho, Puleng Nthinya, Nkomane Seele, all of the wonderful ICAP drivers, and the entire Maseru and Leribe offices who made me feel so welcome and supported. Last, but not least, I would like to thank my fellow ICAP interns and roommates—Blair Kauffman, Olivia El-‐Sadr Davis, and Elliott Weinstein for being my Lesotho family for four great weeks.

3

Table of Contents Acronyms and Abbreviations Executive Summary I. Introduction to TB Care in Lesotho Background on Tuberculosis in Lesotho ICAP’s Role in Lesotho’s TB Services Key Challenges with TB Care II. CommCare Project Project Goals CommCare TB Applications CommCare HQ Project Timeline III. Evaluation Study Objectives: Research Questions Methodology Limitations User Score Calculation IV. Results Indicators of Successful Adoption Research Question 1: Training Research Question 2: Application Benefits and Challenges V. CommCare Usage Analysis VI. Conclusion: Key Recommendations User-‐Recommended App Modifications Recommendations for ICAP Recommendations for Dimagi Recommendations for Further Research VII. Appendices Appendix A: Translated Survey Appendix B: APS Officer Interview Questions Appendix C: User Score Calculations VIII. Bibliography Primary Sources Secondary Sources

4

Acronyms and Abbreviations APS Adherence and Psycho-‐Social Support ART Anti-‐Retroviral Treatment CCHQ CommCare Headquarters CDS Core DOT Supporter DOT Directly Observed Treatment HIV Human Immunodeficiency Virus HSA Health Service Area ICAP International Center for AIDS Prevention MoH Ministry of Health TB Tuberculosis START Start TB patients on ART and Retain on Treatment

5

This report presents the evaluation of a project that introduced CommCare into the workflow of Core DOT Supporters (Tuberculosis community health workers) and nurses in Leribe District of Lesotho. Both qualitative and quantitative methods were used to gather feedback and usage results for this evaluation. The following sections provide an introduction to TB in Lesotho, the CommCare project in Leribe, the results of the evaluation, and recommendations for ICAP and Dimagi.

I. Introduction to TB Care in Lesotho Background on Tuberculosis in Lesotho The Kingdom of Lesotho is a small mountainous country completely surrounded by South Africa. Among its small population of two million is one of the highest concentrations of detected Tuberculosis (TB) and HIV cases in the world. HIV/AIDS is the current leading cause of death in Lesotho, with the third highest prevalence in the world at 23%. These high rates of HIV have exascerbated the TB epidemic in Lesotho, resulting in the fourth highest TB incidence in the world at 696 patients per 100,000 populations. The burden of these co-‐infectious diseases is amplified by poverty, high rates of migration for mining and other employment in surrounding countries, and lack of access to health services in Lesotho’s rural countryside.1

Lesotho is divided into ten districts: Butha-‐Buthe, Leribe, Mafeteng, Mohale’s Hoek, Mokhotlong, Qacha's Nek, Quthing, Thaba-‐ Tseka, and Maseru. These ten districts are sectioned into Ministry of Health (MoH) Health Service Areas (HSAs) that divide up government health services across the country. Each HSA has a central hospital and several filter clinics, health centers, and community health posts that serve between 6,000 to 10,000 people each.2 Due to the HIV and TB epidemics, there are many international health organizations working in Lesotho to supplement the government health system. ICAP, the Columbia University Mailman School of Public Health’s global health organization, works with the MoH primarily in strengthening TB and HIV services in Lesotho. ICAP’s Role in Lesotho’s TB Services ICAP is the leading partner in care and treatment to Lesotho’s MoH. This partnership provides additional personnel and resources to the existing government health system, primarily in the areas of HIV and TB services. ICAP’s main areas of focus in Lesotho are: TB/HIV service integration, the Nurse Education Partnership Initiative to support nurse training institutes in Lesotho, the Enhanced Prevention in Couples study, mapping of mine 1 ICAP Lesotho Portfolio Description, 2013. 2 ICAP Lesotho Portfolio Description, 2013.

6

workers in Lesotho, the Start TB patients on ART and Retain on Treatment (START) study, and supporting the MoH in reopening the medical college in Maseru.3 One of the main ways that ICAP provides support to Lesotho’s TB programs is through the creation of Core DOT Supporters (CDS). Lesotho uses the Directly Observed Treatment (DOT) program, the globally accepted protocol for TB treatment programs. When a patient is initiated on TB treatment, he/she must identify a DOT supporter—someone who will observe the patient take his/her daily medication, and support the patient through the treatment regimen. ICAP created the Core DOT Supporter position to supervise DOT supporters, and to support clinics with TB registration and case management. CDS work alongside government nurses in the clinic during patient registration and follow-‐up appointments, and conduct monthly home visits to patients’ homes. ICAP supports one CDS at each clinic across Lesotho though financial incentives and supervisor support. The Core DOT Supporter program has increased support for nurses in government clinics, and provides more personalized support to TB patients and their families. Key Challenges with TB Care There are several challenges with TB services contributing to the ongoing TB/HIV epidemic in Lesotho. ICAP’s 2013 Lesotho report lists the following major challenges:

• The true burden of TB is unknown because data collection is minimal and inconsistent, and no TB disease survey has ever been conducted

• The amount of government resources allocated toward TB control does not match the size of the burden

• High patient loss to follow-‐up and death has resulted in treatment success rates far below the 87% global target

• Geographic barriers, especially during rainy and winter seasons, restrict access to critical TB diagnosis and treatment services

• TB medication stock-‐outs are common, and prevent detected patients from accessing consistent treatment

• Planning, collaboration, and communication between treatment programs, laboratory services, and other departments within the MoH is inadequate

Many of these challenges are related to the mismanagement of information and resources that consistently plague TB programs across the globe. CommCare was introduced to Lesotho’s TB programs as a tool to address some of these challenges, and ultimately improve TB care in the country.

3 ICAP Lesotho Portfolio Description, 2013.

7

II. CommCare Project With one of the highest concentrations of detected TB cases in the world, Lesotho has great potential to gain from mobile technologies targeted toward improving the accessibility, quality, and efficiency of TB treatment programs. Many of the current challenges with TB care in Lesotho are preventable or addressable with proper resource allocation and project management. CommCare is a tool designed to improve data collection and monitoring, quality of case-‐management and service delivery, and communication between patients, CHWs, and supervisors. Project Goals

Prior to this project, ICAP had already been using CommCare to optimize case-‐management and health worker performance within the START study. APS manager and key CommCare project leader, Mateboho Mokobocho, supported the introduction of mobile technology to the TB program because she saw mobile technology as an opportunity to improve HIV/TB services: “We saw [CommCare] as an opportunity to bridge the gap in our HIV/TB program. We realized we don’t have very good patient follow-‐up, so we thought maybe this would help this process and identify immediately who has missed appointments.”4 The ultimate goal for ICAP is to increase the retention of patients in TB care. Paper-‐based monitoring was not an effective system for tracking patients and ensuring retention, so ICAP decided to introduce mobile technology into their services. CommCare TB Application

4 Interview with Mateboho

8

The CommCare TB application was designed specifically for ICAP’s TB service workflow. Initially, the application was only going to be used by Core DOT Supporters as a case-‐management tool. Because CDSs are not always in the clinic, sometimes the nurses register or meet with patients without the CDS—so ICAP decided to make a condensed version of the application for nurses. The image above displays the main screen for the two different versions of the application: the nurse version on the left and CDS version on the right. The full CDS version of the application has six modules: patient registration, appointment book, missed appointments, household visit, contact follow-‐up, and education. The workflow of the application is as follows (steps for the CDS, or nurse if the CDS is not present): àWhen a TB patient is registered, the CDS or nurse enters all of the information into the “patient registration” form in CommCare, as well as the paper registration book at the clinic. In this registration form, a two-‐week follow-‐up date is set, automatically creating an appointment in the Appointment Book module. àDuring this initial visit, the CDS plays the “patient education” form in the Education module. If the DOT supporter is present at the initial visit, then this is played too, however sometimes this happens at later clinic visits or during the home visit. à Two weeks after the initial visit, the patient is to return to the clinic for a check-‐up and to get more medication. Two days before this appointment, the patient receives an automated SMS reminder to come to the clinic. This appointment appears in the Appointment Book module for the CDS and nurse to update once the patient comes to the clinic. During this visit, the next appointment is made for one month later. If the appointment is missed, the patient receives an automated SMS two days after the missed appointment date as encouragement to return to the clinic. The following steps are done by the CDS, through the extended version of CommCare: à If the patient does not come for their appointment, an asterisk appears by their name as a late flag the day after the missed appointment date. The CDS then uses the Missed Appointments module to: 1) call the patient, and if fails, 2) assign a CHW to visit the patient’s home, and 3) add the CHW visit outcome, and 4) add the patient’s treatment outcome. à Between the patient’s monthly clinic visits, the CDS conducts home visits using the Household Visit module. When in the patient’s home, the CDS uses this module to complete the DOT checklist and register contacts (people living in close contact with the TB patient).

9

During this time, the CDS also educates contacts and sometimes the DOT supporter using the Education module. à When a contact is registered during the household visit, they may be identified as a TB suspect and referred to a clinic for TB testing. The CDS then completes the Contact Follow-‐up form during the next home visit, one month later. This form asks whether the contact went to the clinic (does not ask TB test result or treatment status), and deletes the patient from the contact list if they did go to the clinic. CommCare HQ All of the data that is captured through the CommCare application described above is sent to the CommCare HQ cloud server. CommCare HQ is the supervisor-‐level of the application that allows case-‐level data, and information about the user’s usage of the application, to be analyzed in real-‐time. Whenever a form is completed by a CDS or nurse, it is submitted and sent using data to CommCare HQ. If the mobile phone does not have data coverage at the time of form completion, all of the information is stored in the mobile phone until a signal is recognized, and then the form is submitted. The image below shows the CommCare HQ home page:

The image above shows the main reports available on CommCare HQ: worker activity, daily form activity, submissions by form, form completion time, case activity, form completion versus submission trends, worker activity times. Apart from these automatically generated reports, supervisors can access more patient-‐specific data through the case list, and save

10

these reports under “My Saved Reports.” These customized reports can provide more specific, programmatic-‐level data. An example of this would be a report showing the number of submissions by form for the “call patient” form in the “Missed Appointments” module; this would show the supervisors how many patients with missed appointments were contacted and reminded to come to the clinic. It is also possible to receive automatic emailed reports at any given time range (ex: weekly, biweekly, monthly, etc.) on any data that is captured through CommCare, so that the supervisor does not even need to login to the CommCare HQ site. The supervisors are able to design these reports based on what indicators are most important to the organization. Project Timeline The first stage of the CommCare project began in early March. The two versions of the application were designed, and then CDS and nurses were trained during the first week of March. Starting March 7th, both users began using the application. The project timeline below was created by Dimagi field managers Jolani De La Porte and Nynke Brunner:5 Solution Pilot sites

Phase 1 : TB Application in limited Sites

- Mobile Application for CDS - Mobile Application for Nurse - SMS Appointment Reminders to

Patient - ICAP Operational Reporting

Leribe: - 1 Hospital - 1 Filter Clinic - 9 Clinics

Phase 2: TB Application at Scale

- Refinement of Nurse and CDS application

- Application for supportive supervision

- Refinement of reporting on programme indicators

- Leribe (26 sites) - Quthing (9 sites) - Qacha (12 sites)

Phase 3: HIV Care Application in limited no. of Sites

5 “ICAP DOTS Project Planning” (Dimagi), Powerpoint by Nynke Brunner and Jolani De La Porte

11

- Mobile Application for VHW - Mobile Application for Nurse - SMS Appointment Reminders to

Patient - ICAP Operational Reporting

- TBD

During the initial planning for the project, the following general timeline was created for Phases one, two, and three:

Phase 1: Implementation and training from February to March, run pilot from March through May

Phase 2: Enhancements based on feedback from May through June, scale-‐up to full ICAP sites from July through October

Phase 3: Integrate HIV application, timeline TBD Phase 1: Training and Implementation Processes During the initial training and implementation phase, several roles were established within ICAP to ensure successful management of the CommCare project. Each of these project roles and training processes are described below for the following roles: technical officer, supervisor, trainer, project coordinator, M&E officer, and users:6 Role #1: The Technical Officer’s role is to manage any technical issues with the mobile phones. They must be 1) able to meet with the CDS on a regular basis, 2) tech-‐savvy, and 3) incentivized to fix the mobile phones for the success of the project. The technical officers for the project are the two Leribe APS officers—Nkomane Seele and Puleng Nthinya —and Leribe M&E Officer. Technical Officer Training Module:

• Application overview • Downloading and installing the application • Deploying CommCare onto the phones • Troubleshooting any technical issues with the phones • Application status report

Role #2: The Supervisor’s role is to supervise and support all CommCare users. The supervisor must be: 1) able to meet with CDS on at least a monthly basis, and 2) have time to monitor worker performance reports on a weekly basis. The supervisors on the project are APS officers Puleng and Nkomane, two Leribe ICAP nurse advisors, and twelve government nurses (one at each site).

6 “HIT Project Roles”, Jolani De La Porte

12

Supervisor Training Module: • Application overview • CommCare HQ overview • Worker monitoring reports • Sending messages • Supervision processes • Phone management plans • Phone contracts • Export data

Role #3: The Trainer’s role is to organize and lead CommCare trainings. The trainers for the project were APS officers Puleng and Nkomane, and APS manager Mateboho Mokobocho. Trainer Training Module:

• Application overview • CommCare HQ • Teaching technology module • Training curriculum

Role #4: The Project Coordinator’s role is to manage and supervise the project using CommCare HQ. The project coordinator for the project was originally ICAP APS manager Mateboho Mokobocho, but she was unavailable, so M&E Manager Tlohang Moeketse was trained. Project Coordinator Training Module:

• CommCare HQ overview • Worker monitoring reports • Basic data management • Mobile user management • Web user management • Saved Reports • Email reports

Role #5: The M&E Officer’s role is to use data from CCHQ to monitor the project. The M&E Officer on this project is ICAP M&E Manger Tlohang Moeketse. M&E Officer Training Module:

• Application overview • CommCare HQ overview • Form exports • Case exports • Custom exports (excel dashboard)

13

Role #6: The User’s role is to use the mobile phone to report data through the CommCare application. The users on this project were the eleven Core DOT Supporters and ten nurses from Leribe (nurses were trained during the Supervisor training). User Training Module

• Application overview Phase 2: Scale-‐Up Phase 2 of the CommCare project requires modifications to the application based on program feedback, and training all CDS (and at least one nurse per clinic) at all twenty-‐six Leribe sites, nine Quthing sites, and twelve Qacha sites. In addition, a CommCare application for supportive supervision will be created for ICAP APS officers, APS manager, and M&E manager. During this scale-‐up phase, the program indicators that CommCare collects data on will be evaluated and refined based on ICAP’s needs.7 These details are currently being discussed between ICAP and Dimagi.

Phase 3: Integration of HIV Application At the initiation of the CommCare project in February and March, plans were discussed to expand Commcare to an integrated TB/HIV application. This is yet to be decided by ICAP and Dimagi.

III. Evaluation Study objectives: Research Questions The CommCare project has been active in Leribe District since March of 2014. The purpose of this evaluation is to investigate the potential impact and barriers of mobile health technology in Lesotho’s Tuberculosis programs. The focus of this evaluation is two-‐fold:

I. What indicators make CommCare adoption successful? Where should the focus be in training and implementation processes?

II. How does CommCare support and challenge TB care delivery in Leribe District’s DOTS program? How can the benefits of the CommCare be amplified?

The evaluation was conducted after CommCare had been active for three and a half months, between Phase I and Phase II of the project. The following report draws from qualitative usage data and quantitative feedback from users and other key project staff to provide recommendations for improvement before the Phase II scale-‐up. Aside from the project

7 “ICAP DOTS Project Planning” (Dimagi), Powerpoint by Nynke Brunner and Jolani De La Porte

14

specific recommendations, this evaluation also provides general recommendations for Dimagi to improve the training and implementation processes used in all CommCare projects. Methodology Both quantitative and qualitative data was gathered for this evaluation. Quantitative usage data from CommCare HQ was analyzed for all twenty-‐two users. Only data on worker activity and form submission trends were utilized (no patient data). While on-‐site, several qualitative methods were used to evaluate the project: 1) individual interviews with twenty-‐one users, 2) observations of mobile proficiency in demo mode for fourteen users, 3) individual surveys with twenty users, and 4) two focus group discussions (one Core DOT supporter and one nurse session). Qualitative Analysis The four qualitative tools were used on-‐site, over the span of three weeks in Lesotho. Observations, individual interviews, and individual surveys were conducted one clinic at a time. Within each of the eleven CommCare sites there are at least two users (a Core DOT supporter and a nurse) using the application. To avoid biased responses, each user at each site was interviewed separately. First, the user was asked basic interview questions regarding the Dimagi training, and the benefits and challenges of CommCare. Second, the user was observed navigating CommCare in demo mode. Third, the user completed the individual survey. While the first user started the survey, they moved to a different location, and the second user started the interview, observation, and survey process. Throughout these three processes, the APS officer was present to translate interviews and provide support with survey questions when necessary. After completing this process at all eleven sites, the CDS and nurses were brought to Motebang Hospital for two separate focus group discussions. A different translator (nurse advisor at Motebang Hospital) was used for the CDS focus group discussion, and the APS officers were not present, in an effort to eliminate the potential response bias from having the user’s supervisor present. After one day of nurse focus groups and one day of CDS focus group, all users and the APS officers returned for a third day where a summary of the qualitative data was presented back to them. The details of each of the four qualitative methods are described below. 1) Individual Interviews: Individual interviews were conducted with twenty-‐one users (eleven CDS and ten nurses) prior to the observations in demo mode and individual surveys (the twenty-‐second user was unavailable for interviews). Interviews with CDSs were translated by the APS officers

15

and most nurse interviews were conducted in English without need for translation. The following interview questions were used with all users:

• Think back to three month ago when you learned how to use CommCare. Was the three-‐day (or two for nurses) training at Malinakana Guesthouse sufficient in length?

• What part of the training process did you find helpful in learning to use CommCare? • What part of the training process did you find confusing or unhelpful? • How do you think the training could be improved?

• How does CommCare benefit your work as a CDS/nurse? • What components of the CommCare application do you find most useful? • How does CommCare create additional challenges in your work? • What components of the CommCare application do you find challenging, or useless? • How could the application be improved? • Do you have any questions for me?

2) Observation Criteria: After the individual interview, each user was observed and scored on their level of proficiency with the CommCare application. A total score out of eight possible points was determined using each of these four categories:

• Does the user know how to login to demo mode? no=0 pts; yes=1 pt • How comfortable is the user with typing patient name and info in the patient registration

form? uncomfortable=1 pt; fairly comfortable=2 pts; very comfortable= 3pts • Does the user know how to update patient status and treatment outcome? no=0 pts; yes= 1 pt • How comfortable is the user, overall, with navigating the CommCare application?

uncomfortable=1 pt; fairly comfortable=2 pts; very comfortable= 3pts Because of time and other logistical limitations, not all users were observed in CommCare demo mode. Observations were conducted with fourteen of the twenty-‐two users (eight CDS and six nurses). 3) Individual Surveys: The individual surveys were conducted with twenty users (ten CDS and ten nurses), as two were unavailable to complete it. The survey was developed prior to arriving on site, and was translated by APS officer Puleng Nthinya. All of the surveys were conducted with an APS officer present to translate if necessary (this potential bias is listed as a limitation in the section below). See Appendix A for survey questions. 4) Focus Group Discussions: Focus group discussions were conducted in two separate groups: one CDS discussion and one nurse discussion. To avoid bias, a nurse advisor from Motebang Hospital translated the

16

CDS discussion (translator not needed for nurses), and the APS officers were not present. Each focus group discussion lasted between one to one-‐and-‐a-‐half hours. All CDS and nurses were reimbursed for their travel expenses and provided a meal after the focus group discussions. The following questions were used as a launching point, although each discussion flowed in its own direction and further questions were asked: Part 1: Training (same questions for CDSs and nurses)

Think back to three months ago when you received training on how to use CommCare. • How did you feel about the training? • Which part of the training did you find most helpful in learning to use CommCare? • Which part of the training did you find unhelpful or confusing? • How do you think training could be improved? • When you have an issue with CommCare, how do you get support?

Part 2: CommCare (CDS questions)

• How does CommCare benefit your work as a Core DOT Supporter? • Which modules on the application do you find most useful? • How does CommCare create challenges in your work as a Core DOT Supporter? • Which module on the application do you find least useful, or most problematic? • When do you use the education module on CommCare? • How do you use CommCare during home visits? • How does CommCare help you when a patient misses an appointment? (do you use CommCare

to call, assign CHW, add CHW visit outcome)? • What would you like to see changed, or added, on CommCare? • If ICAP stopped requiring you to use CommCare, would you still continue to use it? Why or why

not? Part 2: CommCare (nurse questions)

• When working with a CDS, how do you decide who will use CommCare with the patient? • How does CommCare benefit your work as a nurse? • Which modules on the application do you find most useful? • How does CommCare create challenges in your work as a nurse? • Which module on the application do you find least useful, or most problematic? • When do you use the education module on CommCare? • How does CommCare help you when a patient misses an appointment? (do you use CommCare

to call, assign CHW, add CHW visit outcome)? • What would you like to see changed, or added, on CommCare? • If ICAP stopped requiring you to use CommCare, would you still continue to use it? Why or why

not? In addition to the above interviews, observations, surveys, and focus groups discussions, several ICAP and Dimagi employees were interviewed. The following ICAP employees were interviewed for this evaluation: APS officers Puleng and Nkomane (see Appendix B for interview questions), APS manager Mateboho, M&E manager Tlohang, technical director Koen Frederix. Additionally, Dimagi field manager Nynke Brunner and Dimagi project

17

manager Jolani De La Porte were informally interviewed throughout the evaluation process. Limitations Response bias. When gathering feedback from CHWs, there is always danger of a response bias. The CDS, nurses, and even APS officers may have responded to surveys, interviews, and focus group discussions how they thought the interviewer wanted them to respond. Although the interviewer was independent from ICAP and Dimagi, she may have been viewed as such by the respondents, creating a bias in the feedback. Additionally, the APS officers, the respondents’ supervisors, were present during interviews and surveys. This could have been a major bias to answer “correctly” in front of their supervisors. The survey seemed to be most affected by this bias, as many users responded with all “4s” or “5s” to questions evaluating the CommCare project. However, focus group discussions were conducted with an uninvolved nurse advisor at Motebang Hospital to eliminate any response bias. The responses were mostly consistent with the potentially biased interview and survey questions during these discussions, and the feedback seemed genuine and honest. The surveys may have been affected by the response bias because it required written answers—this may have confused, or intimidated respondents who had low levels of literacy. While response bias is always a challenge with qualitative research, it did not appear to limit the data significantly. User-‐reported qualitative data. Although usage data is available on CCHQ, some information provided by users was not possible to cross-‐check. Because all of the benefits and challenges of CommCare were user-‐reported, there is potential for falsely reported information. One example of this limitation is the user-‐reported decrease in missed appointments because of CommCare. The increased adherence to appointments was mentioned by users and APS officers as a benefit of the project, however it was not possible to quantify this change with the logistical restraints of the current evaluation. This is the case with many program indicators, as district-‐ and clinic-‐level data is difficult to access or simply not collected. As a result, this limitation must be kept in consideration when reading the feedback provided in this evaluation. Not all users were observed, interviewed, and surveyed. Due to logistical difficulties, not all of the twenty-‐two CommCare users were observed in demo mode, interviewed, and surveyed. Only fourteen of the twenty-‐two were observed, because of time-‐constraints in the clinic while the observations were taking place. Many of the nurses and CDS had patients or other work, and had to leave before the observation was conducted. This was also the case with interviews, although twenty-‐one of the twenty-‐two users were interviewed. Twenty of the twenty-‐two users conducted the survey. One user was

18

unavailable, and the other did not complete the training and never adopted the technology, so the survey questions were not relevant to her. In retrospect, this inactive CDS should have completed as much of the survey as possible in order to gain feedback from users who are unable to adopt the technology. These gaps in the data are a significant limitation to the study. Small sample size. The largest limitation of this evaluation was the small sample size. Quantitative and qualitative data was analyzed for twenty-‐two CommCare users, which is not large enough of a sample size to draw significant relationships between the data. The feedback collected is extremely valuable for the purpose of the project scale-‐up; however, should not be considered statistically significant or generalizable data on indicators of adoption and usage of CommCare. User Score Calculation

In order to identify indicators for successful CommCare adoption, all Core DOT Supporters and nurses were classified into three user levels: low, medium, and high-‐level users. User levels were calculated using CCHQ “worker activity” percentages from the month of May (percentage of worker’s cases that have been updated within the past month), and APS officer classification as a low, medium, or high-‐level user. Figure 1 below compares five different categories that were considered for the user score calculation:

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Worker Activity

Demo Score Total user Score

Supervisor Score

Self-‐Identioied Comfort

Average Score (out of 1)

Score Criteria

Average User Scores: CDS vs. Nurse CDS Average Nurse Average

n=10 n=12 n=10

n=8

n=x/22

n=12

n=6 n=12 n=10

n=10 n=10

19

Figure 1 compares the average scores for CDS and nurses in the five different categories considered for the user score calculation. Each of the five categories had different samples sizes (see n=x), as not all users were evaluated in all five categories. As explained above, worker activity percentages from CCHQ and supervisor-‐assigned scores were used to calculate the total user score (third bar from the left). The demo scores were assigned during observations in demo mode (see methodology section), and “self-‐identified comfort” was reported by each user in the survey. Worker activity and supervisors scores were used because it was possible to obtain the data for all twenty-‐two users for both of these categories, and they provided a balance of quantitative usage data and qualitative supervisor input. For more details on user level classifications, see Appendix C: User Score Calculations.

IV. Results Indicators of Successful Adoption In the above section, User Score Calculation, the process for calculating individual user proficiency is described. These user level scores were used to deconstruct survey responses and better understand which demographic indicators affect adoption of CommCare. Figure 2 shows the variation across low, medium, and high user levels for the following indicators: average age, education level, average number of years worked at clinic, average number of years owned mobile phone, average level of phone usage for SMS, level of phone usage for calls, and average level of phone usage for internet:

0

1

2

3

4

5

6

7

8

Age (x10) Education Years worked at clinic

Years owned a mobile phone

Usage of phone for SMS

Usage of phone for calls

Usage of phone for internet

Average Value

Variable

Potential Indicators of User Level Low Medium High

n=1 n=7 n=13

20

Figure 2 shows relationships between seven different variables and user level. It is important to note that because the sample size is so small, there is an uneven distribution of sample sizes across the three levels. There was only one low-‐level user, seven medium level users, and thirteen high level users. The survey’s twenty-‐respondent sample size is too small to prove statistically significant correlations between any of the indicators and user levels. There is only data for the low-‐level user group in the age, SMS usage, and internet usage columns because the one low-‐level user did not complete the survey to answer these questions. Despite the small sample size, the trends in Figure 2 present interesting relationships between the various demographic indicators and user level:

1. The first cluster of bars represents the average ages for low, medium, and high-‐level users. According to this graph, age and user level have a slight inverse relationship—as age decreases, user level score increases. There is only one user in the low level group, and the difference in age between medium and high-‐level users is very small. Despite the weak quantitative evidence, age was brought up by the users during focus group discussions as a factor affecting the ease of adoption (elaborated in “Training Length” section below).

2. According to Figure 2, there was no significant relationship between education and user level.

3. The number of years each user group has worked at the clinic decreases as user level increases. This is surprising at first, however age becomes a confounding variable as the outliers (over twenty years of experience) in both medium and high groups are much older than the average age.

4. There was a slight inverse relationship between the number of years of mobile phone ownership and user level—however, the decrease in years of mobile phone ownership from medium to high level users is very minimal.

5. The amount of SMS usage only had an effect on the low-‐level user group, which is only one user and therefore difficult to draw conclusions from.

6. There was a slight inverse relationship between usage of the mobile phone for calls and user level. This decrease in usage for calls from medium to high level users is very minimal (similar to difference in number 1 and 4).

7. There is a slight direct relationship between mobile internet usage and user level (as user level increases, so does mobile internet usage), however this is mostly due to the single low-‐level user. Similar to the slight trends in items 1, 4, and 6, this is likely not significant. Further research is required to confirm or deny these relationships between user level and the factors listed in the graph above.

Figure 2 displays interesting relationships between seven different indicators and user level. Because of the small and unequal distribution of sample size, these relationships can

21

not be understood as statistically significant, however some of the trends were mentioned in the training feedback section below. Further research should investigate these potential indicators of user level with a larger sample size and more consistent methods. The following section introduces the first research question regarding CommCare training processes, and helps explain some of the trends observed above. Research Question I: Training The CommCare Evaluation Survey, individual interviews, and focus groups discussions were designed to address the following research question: What indicators make CommCare adoption successful? Where should the focus be in training and implementation processes? The survey collected feedback from twenty users on the training processes used to implement CommCare. The following three questions were asked to get a general understanding of the training’s value to users:

Figure 3 (Learning) shows that all twenty users at least agreed (79% strongly agree, 21% agree, 0% indifferent, 0% disagree, and 0% strongly disagree) that learning CommCare was easy. Figure 4 (Upon) shows that all twenty users felt comfortable with CommCare after the training (84% strongly agree, 16% agree, 0% indifferent, 0% disagree, and 0% strongly disagree). Figure 5 shows slightly more variation in responses regarding comfort after three months of use: 80% strongly agreed, 15% agreed, 5% were indifferent, 0% disagreed, and 0% strongly disagreed. The above three results demonstrate that most users found CommCare easy to learn, and are comfortable using the application. However, face-‐to-‐face responses from users revealed some variation in ease of adoption among the Core DOT Supporters and nurses. This variation was vocalized by both user groups during focus group discussions. While some users found the mobile application easy to learn, others struggled during the training, and are still adapting to the technology after three months of use. The following section will

79%

21%

Learning to use CommCare was easy

for me:

Strongly Agree Agree

84%

16%

Upon completing the training, I felt comfortable using

CommCare: Strongly Agree Agree 5% 15%

80%

After three months of use, I feel

comfortable using CommCare:

Indifferent

Agree

Strongly Agree

22

explore this variation to understand indicators correlated with the successful adoption of CommCare. Training Length

During individual interviews and focus group discussions, the majority of users said the three-‐day training (or two-‐day for nurses) was sufficient to learn how to use CommCare. According to APS Officer Puleng Nthinya, the users enjoyed learning the new technology during training:

[CommCare was] difficult for everyone the first day, but it was exciting for them to get used to the technology. Everyone was staying at the guesthouse, and at night they were practicing in groups even after the training ended.8

When asked about length, two nurses and three CDSs out of nineteen total respondents said the training was too short, and they would have liked a full week of training to get more practice registering patients with the application.9 One APS officer described a difference in learning curve depending on the trainee’s previous mobile phone usage:

Some of them just owned phones to call and SMS, but not beyond that. It was difficult for them to learn within three days, but for younger CDSs, or those who are using Facebook and Whatsapp, I don’t think it was very difficult to grasp.10

8 Interview with Puleng Nthinya 9 Interview with nurse, Interview with Nurse, Interview with CDS, Interview with CDS, Interview with CDS 10 Interview with Nkomane Seele

5% 5% 5%

27% 58%

The training was suf`icient in length:

Strongly Disagree Disagree Indifferent Agree Strongly Agree

23

During focus group discussions, one nurse reiterated the relationship between the user’s previous mobile internet usage and the speed of their CommCare learning process:

It was long enough for those who are used to this technology, it is easy. But some of us are old, and we are not used to this technology. We need more training and more practice. We don’t use this Whatsapp and we are not used to the phones.11

These user-‐reported relationships between age and previous mobile usage correspond to some of the trends observed in Figure 2 above. While there are some limitations to this user-‐reported qualitative data, these respones hold significant weight and should be investigated in further research. Training Content The individual surveys also asked users about the processes used to train Core DOT Supporters and nurses on CommCare. Figure 6 below displays the average scores (one through five) for the three-‐day training processes across the three user level groups:

Figure 6 shows that medium level users found the trainer demos and group practices sessions equally useful, and high level users found both processes more useful than medium-‐level users did, but equally as useful when compared to each other. Medium level users ranked individual practice sessions as less useful than the first two processes. High-‐level users also rated individual practice as less useful than demos and group practice, and found it less useful than medium level users did. All scores were on a scale from one to five.

11 Nurse Focus Group Discussion

4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8

Demo by trainer Group practice Individiual practice

Average Score

Training Process

3-‐Day Training: Average Process Score by User Level

Medium

High

24

The graph above provides valuable quantitative data on the helpfulness of CommCare training processes. According to the responses, users found demos and group practice more useful than individual practice sessions. However, this feedback was no expressed during focus group discussions. Both CDSs and nurses said that practicing registering patients in demo mode with a partner was the most helpful part of training; some even said that an extra day of training should be added to include more individual practice time.12 This discrepancy may have been caused by a response bias in the survey, or confusion about what “individual” vs. “group” practice meant in the survey question. It is important to keep these limitations and discrepancies in mind when analyzing the data. A similar question asked users to rank the different processes of the entire training package—including supervisor reports and feedback, the take-‐home manual, the supervisor’s phone number for director support, the three day training, and practice in demo mode:

Figure 7 shows the average scores for all five training processes, separated by user level group. The five categories are in ascending order from lowest to highest average score. The order of helpfulness for all five categories was the same among medium and high-‐level users, however the average scores within each category varied by user level. According to survey results, both medium and high-‐level users ranked the five categories of training processes in the following order: supervisor reports and feedback, take-‐home training manual, supervisor phone number for direct support, three-‐day training, and demo mode. High-‐level users found supervisor reports and feedback and the take-‐home manual more helpful than medium-‐level users did, and medium-‐level users found the supervisor phone number, three-‐day training, and demo mode more useful than high-‐level users did. Similar to the above graph, it must be kept in mind that there may have been some confusion about what each category was referring to. For example, the users

12 Nurse and CDS Focus Group Discussions

3.6 3.8 4

4.2 4.4 4.6 4.8 5

Supervisor reports and feedback

Take-‐home manual

Supervisor phone # for direct support

3-‐Day training Demo mode

Average Score

Training Process

Whole Training Package: Average Process Score by User Level

Medium

High

25

identified practicing in demo mode during the three-‐day training as the most helpful way to learn CommCare—this may be why “Demo mode” was the highest ranked process in Figure 7. However, these results are useful in getting a general idea of how useful medium and high-‐level users find the CommCare training processes. Training Recommendations by Users There were three main suggestions proposed by users regarding training: 1) extend the training to five days, 2) focus more on practicing registering “cases” on CommCare in pairs, and 3) organize regular follow-‐up meetings to discuss CommCare challenges and potential solutions. In the training improvement comment box in the survey (question 30), one nurse suggested meeting again after the initial training “to share experiences with other facilities which we as [health center name] may not have recognized.”13 The APS officers had mixed opinions about the length of the training, as some users struggled to learn to use CommCare, while others excelled in the three days. One APS officer suggested telling clinics ahead of time that the training would be technology-‐focused, and then doing an assessment of the users’ mobile literacy before training to adjust the length accordingly.14 During focus group discussions, both CDSs and nurses said a stronger focus on practice rather than lectures and demonstrations would make the training more helpful. Both APS officers agreed that quarterly meetings with CDS and Nurses to discuss challenges and get feedback would help resolve any challenges with adopting the application. Quarterly meetings had been agreed on at the start of the project in March, but none have been organized so far.15 Research Question II: Application Benefits and Challenges The CommCare Evaluation Survey, individual interviews, and focus groups discussions also addressed the second research question: How does CommCare support and challenge TB care delivery in Leribe District’s DOTS program? How can the benefits of the CommCare be amplified? Figure 8 below displays the average scores for the application’s different features. These scores provide a general idea of which components of CommCare are most useful to Core DOT Supporters and nurses.

13 Nurse Survey (#16) 14 Interview with Nkomane Seele 15 Interview with Puleng Nthinya

26

Figure 8 shows the average scores for different components of the CommCare application in ascending order by usefulness: late mark “*,” missed appointments module, contact registration and follow-‐up, appointment list, automatic SMS reminders, education module. While this survey question was originally designed to rank the components compared to each other, this was unclear in the survey and some users scored the components individually. Because of this confusion, the average scores of each components were calculated for this graph. The wording of the components may have also caused confusion, as a lot of users identified the appointment book as most useful during interviews, which includes missed appointments, the appointment list, and SMS reminders. The “missed appointments module” may have been ranked lower in this question because automatic SMS reminders and appointment list were already ranked as most useful. It is interesting that the education module is ranked highest, because DOT supporter and contact education usage remains very low (see page 33-‐36). Respondents may have been referring to patient education, which has high form submission rates and has already surpassed the usage goal per patient (see page 33-‐34). These discrepancies and points of confusion are important to understand so they can be avoided by closer attention to design and translation in future surveys and evaluations. Benefits of CommCare I. Improved Supervision Less Paperwork. The CommCare application digitizes patient registration, the clinic appointment book, TB education, and all materials used during home visits. While the paper TB register is still used in the clinics, all other forms are now completely digitized

4.4 4.6 4.8 5

5.2 5.4 5.6 5.8

Late mark "*" Missed appointments

module

Contact registration and follow-‐up

Appointment list

Automatic SMS

reminders

Education module

Average Score

Application Component

Average Scores of Application Features

27

because of CommCare. This means they are now easily monitored, in real time, by the health workers’ supervisors. Rather than collecting the CDS’s paperwork for each home visit, the APS officers are able to check the weekly CCHQ reports for form submissions: “Now it is much easier to monitor home visits. Earlier I couldn’t keep up with paperwork on the DOT checklist, and it mostly went ignored.”16 More Accountability. According to the APS officers, the increase in monitoring due to digital form submission is creating more accountability. The CommCare HQ cloud infrastructure allows supervisors to monitor all community health worker activity in real time. This means the APS officers can now see exactly when users are—and aren’t—educating, following up with patients, and conducting home visits. According to one APS officer, the ability to monitor patient and contact education has been one of the biggest improvements in accountability: “We could never see that before.”17 The APS officers explained that CommCare allows them to give feedback to the CDSs and nurses on their CommCare usage each time they visit the clinic, which can be one to four times per month depending the clinic’s location. If the supervisor sees anything unusual on the automatic weekly report, they call the user to sort out the issue. This immediate reporting and feedback has increased communication between CDS, nurses, and supervisors on every step of TB care. However, usage of CCHQ by the APS officers has been low since the implementation of the project. Although the officers reported the increase in monitoring and accountability because of CommCare, further input from supervisors and users revealed that CCHQ is highly underused. This is an issue that is discussed in section IV under Challenges (page 35). II. Improved Health Worker Performance Improved case-‐management. Perhaps the most fundamental improvement CommCare provides to the user is the consolidation of all TB case-‐management tools: the patient register, appointment book, TB education, home-‐visit materials, and contact registration. This is especially valuable for Core DOT Supporters, who are in charge of patient care within the clinic, and in the field during home visits:

It puts all of the information in one place. This makes it easier for me, especially when I am in the field, because I serve a very large community. All of the names are in the phone, so it is easy to use. I don’t even carry so much paperwork, so it is easier and saves me time.18

16 Interview with Puleng Nthinya 17 Interview with Puleng Nthinya 18 Interview with CDS

28

Many of the users talked about the appointment book as the most useful case management tool on the application. While clinics were previously supposed to manually send SMS reminders to patients each week, the phone now automatically sends appointment reminders and missed appointment alerts to the patient. One CDS described CommCare as a “very improved linkage between the community and the facility. Now it is not easy to lose patients, and with the appointment list, it is easier to plan my daily schedule.”19 The clinic manager and nurse at one facility explained that CommCare is improving their system for patient follow-‐ups: “With CommCare, we are now able to make follow-‐ups after we register them. CommCare will tell you when, and what to do with the patients, so that is very helpful.”20 One concern during the implementation of CommCare was government nurse buy-‐in. Even though patient registration is digitized, the clinics are still required by the government to document patients in the paper register. Although the agreement was to maintain the paper register and paper appointment book, the APS officers reported that some clinics have stopped using the paper appointment book, as they find CommCare a more useful method of tracking appointments. This is problematic in that it makes cross-‐checking impossible, but it does show the usefulness of the application as an appointment tracking tool. Regarding the paper register, double-‐documentation could be viewed as extra work by the nurses whose clinical responsibilities are not just limited to TB care. However, most nurses said that CommCare makes registration easier and faster:

I find CommCare more reliable, because if I am busy I just register the patient quickly, and then the other activities can be attended. Then the paper registration can come after assessing the patient. It makes work easier.21

During both CDS focus group and nurse focus group discussions there was unanimous agreement that the clinics have seen a decrease in missed appointments: “It reduced the number of missed appointments. The lost ones are found now.”22 Because missed appointments were not well documented before, this effect cannot be measured; however, this is a very significant observation by the users. When asked if they would continue using CommCare if it were no longer required, one nurse responded:

19 Interview with CDS 20 Interview with Nurse 21 Interview with Nurse 22 Nurse Focus Group Discussion

29

Yes. What I have discovered is that the default rate is almost 0% now because we are able to track the patients with CommCare. And following them with the phone is more easy than writing with the register.23 CommCare is like our 3rd hand in improving patients’ adherence.24

Retaining TB patients in care was the primary goal of implementing the CommCare application. Although previous default rates cannot be compared, the effect of the application will be apparent when the six-‐month treatment cycle is completed and the default rate of CommCare patients can be measured. Several respondents reported that TB patients have expressed feeling more supported by the health system with CommCare’s regular appointment reminders. Every time the patient goes in for an appointment—or is visited by a CDS—they are reminded beforehand, and then updated in the system during the visit. According to CDS and nurses, this tracking in CommCare has received positive feedback from patients as they are supported through their TB treatment plan. Improved consistency in TB education. TB education is a critical component of care for the TB patient and family members upon initiation of treatment. Previously, CDSs and nurses did not have any standardized materials for educating patients, and the information received by each patient was often highly variable.25 The education module on CommCare helps fill this gap in TB care: “The education is very useful because sometimes you tend to forget what you need to talk about, and this reminds you,” explained one CDS.26 Both CDS and nurses reported that the education module is improving quality of TB care, because the patients enjoy and retain the information more:

The patients love the education, and they listen attentively whenever I use the education. Even before I play the module, I give the general health talk, but the health talk in the phone reinforces what I am saying.27

Even with patients who were registered a long time ago, I usually use the education. The patient more easily grabs the important information, and then asks questions from there. They learn more.28

23 Nurse Focus Group Discussion 24 Nurse Survey (#16) 25 Interview with Puleng Nthinya 26 Interview with CDS 27 Interview with CDS

30

Although patients were not interviewed for feedback, several respondents (both CDSs and nurses) reported positive feedback from TB patients about the education module. This was especially stressed during focus group discussions, when the users talked amongst each other about how they use the education module. Users reported that patients enjoy the education module, and absorb the information better. While this is valuable to hear from the users, further research should investigate beneficiary perceptions of CommCare to attain more in-‐depth patient feedback. Improved home visits. Core DOT Supporters are responsible for conducting home visits with TB patients and their families once each month, between monthly clinic visits. During these home visits, the CDS has to run through the DOT checklist, register and screen any TB contacts, and educate the patient, and/or DOT supporter, and/or contacts. Previously, all of these activities required paperwork that the CDS carried with them to home visits. With CommCare, CDSs are able complete the DOT checklist, contact registration, and education on their mobile phones, requiring no paperwork. The CDSs reported that replacing all of the paperwork with a single mobile phone has made their jobs much easier, and more efficient.29 The nurse advisor at one clinic reported an improvement in the consistency of home visits in his area: “What I have also noticed is that we are even able to visit the families of the patients which we register in CommCare, which we were not doing well in the past. So the CDSs are going into the community and visiting the patients and their families much more often.”30 According to one APS Officer, CommCare has standardized the contact tracing process, which previously had no formal protocol. She explained that contact tracing is still a weak spot in Lesotho’s system of TB care, but CommCare has helped increase the consistency and tracking of the process.31 Automatic data collection for program monitoring. As is the case in many low-‐income rural countries, there is minimal availability of accurate health system data in Lesotho. While quarterly reports are generated by ICAP, it is difficult to capture accurate, real time data on TB incidence. CommCare is improving this data availability by automatically capturing patient data and health worker activity that can be monitored in real-‐time. According to technical director, Koen Frederix, this was the main draw for implementing mobile technology into ICAP’s workflow. With CommCare, ICAP can now enter the CCHQ cloud system and monitor patient registration, appointment adherence, and community

28 Interview with Nurse 29 CDS Focus Group Discussion 30 Interview with Nurse 31 Interview with Puleng Nthinya

31

health worker activity. Previously, knowing exactly how many patients are in care required flipping through paper registries at each clinic—now the mobile phones are automatically capturing this data for improve program monitoring at each level of the organization. Challenges with CommCare I. Clinic Workflow Issues Only one nurse per clinic is trained on CommCare. Within each Leribe district clinic there is one CDS who supervises all TB cases, and several nurses who are responsible for all areas of medical care—including TB treatment. If the CDS is unavailable or absent when a TB patient comes to the clinic, the nurse is responsible for registering and attending to the patient. When CommCare was implemented, the CDS long with one nurse at each site were trained to use the mobile technology (with the exception of Motebang Hospital, where two CDSs were trained). The supervisors thought that training one nurse to register patients with CommCare would ensure that if the CDS was unavailable, all TB patients would still be entered in the system; however, only training one nurse has created challenges within the clinic. During individual interviews and focus group discussions, the nurses expressed that being the only nurse trained to use CommCare has shifted the balance of work in the clinics. Because they were designated to attend the ‘TB mobile phone training,’ their co-‐workers view them as responsible for all TB patients now, rather than all nurses sharing TB care responsibilities. Several nurses said they have tried training other nurses to register and update patients with the phones—in case they and the CDS are ever both unavailable—and they receive resistance from the nurses who didn’t attend the original Dimagi training.32 Those nurses who have been able to train others to use the mobile phone in their absence feel an unfair burden of responsibility for how and when the airtime is used. According to ICAP protocol, the mobile phones can only be used for TB patients registered in CommCare. Every time airtime is used for these activities, it must be logged and then submitted to the designated APS officer. This is no problem for the CDSs who have their own mobile phones, but for the nurses who ask others in the clinic to update CommCare when they’re absent, they do not always know who has used the phone for what. Several nurses explained that this is too much accountability put on only one nurse per clinic. They suggested that

32 Nurse Focus Group Discussion

32

shorter, on-‐site trainings be conducted in each clinic by the APS officer to distribute the responsibility.33 Treatment stock-‐outs and other interruptions of care. Since the introduction of CommCare into the Leribe district clinics, several existing bottlenecks in the TB workflow are being highlighted. The three most common challenges raised during interviews and focus group discussions were: medicine stock-‐outs, patient migration or transfer, and TB registration number delegation (patient ID number required by the Ministry of Health). The following issues are not caused by CommCare, but rather are highlighted by the efficiency this TB care management tool:

• Stock-‐outs are common in Leribe District, especially in the smaller rural clinics.

What often happens is a TB patient is registered in a rural clinic, and when the patient is scheduled to return for a follow-‐up, the clinic has run out of medication. The patient is then sent to Motebang hospital to pick up the drugs, and told to return immediately to the rural clinic to be updated in the system. This issue often unfolds in three different scenarios: 1) transportation can be costly and difficult, so the medication is not acquired and the patient defaults; 2) the patient is able to get the medication from Motebang hospital, but they don’t return to the rural clinic and are not updated in CommCare; or ideally, 3) the patient gets the medication from Motebang, returns to the rural clinic to be updated in the system, and is continued in CommCare with no treatment interruption. Because CommCare automatically tracks missed appointments, the CDS and nurses are unsure what to enter into the system when a patient misses an appointment because of scenario 1 or 2, rather than simply patient lack of adherence:

There is a district-‐wide problem that the drugs are out of stock. So the client comes, and there are no drugs. That client does not come back, so I have a problem trying to figure out what appointment date to put, because we do not know when we will have drugs next.34

• A similar scenario occurs when patients migrate to South Africa for work, or are transferred to another clinic and/or district. Several CDS and nurses mentioned this as a challenge, as they are unsure what to enter into the mobile appointment book.35 Before mobile tracking, these patients slipped through the cracks and were often labeled as defaulted patients who had “transferred out,” although they should technically still be followed up with. CommCare’s appointment module requires a reason to be entered for each missed appointment, and “transferred out” is

33 Nurse Focus Group Discussion 34 Interview with Nurse 35 CDS Focus Group Discussion

33

currently not a drop-‐down option. According to Koen Frederix, ICAP technical director, “transferred out” should not be a cause of patient default. Patient domestic and cross-‐border migration is often the cause of interrupted treatment in Lesotho, although treatment should just be continued in the destination’s local clinic; however, lack of communication and tracking across districts and borders makes treatment continuation rare.36 With all of the data available through CommCare patient tracking, this issue is being highlighted as a common yet preventable cause of defaulting.

• Upon registration, every patient is assigned a TB registration number by Motebang Hospital. This requires the CDS or nurse at the clinic to call the hospital for a registration number assignment while the patient is being registered. Several users reported that it is often difficult to get ahold of the Motebang TB registration office, which delays the CommCare registration by a day or two. Similar to the above two, this issue has existed for a long time before CommCare, but with the introduction of a more efficient case management tool, it’s obstruction to the quality of TB care is becoming more apparent.

The above three issues were often mentioned during individual interviews and focus group discussions. It is important to understand that they are not caused by CommCare, but rather issues with the system of TB care in Lesotho that are now being highlighted with the introduction of CommCare’s systematic TB care management tool. The APS officers made this very clear: “CommCare is important in that it shows that stock-‐outs and these issues interfere with proper management of TB patients.”37 II. Technical Issues Since the introduction of CommCare into Lesotho’s system of TB care, several technical issues have persisted. This section describes the three most prominent technical issues facing the project currently. It is important to note that all of these issues are fixable, and simply have not been addressed. Issues with network signal near the SA border. Several clinics are located near Lesotho’s border with South Africa, which creates issues with the network signal. All of the CommCare phones have contracts with Lesotho networks, but when the mobile phones are near the border they pick up the signal from SA networks and begin roaming. While roaming, these mobile phones are not able to send forms and sync with the cloud server. The CDS that raised this issue explained that most people in border towns use networks 36 Interview with Koen Frederix 37 Interview with Puleng Nthinya

34

that work on both sides of the border, such as MTN and Vodacom.38 However, according to ICAP M&E manager, Tlohang Moeketse, this issue can simply be fixed by changing the roaming settings on the mobile phones. Whichever solution is necessary, both are easily fixable and just need to be addressed by the APS officers or Moeketse. Forms not sending even when mobile phone has signal. Apart from the phones with roaming issues, several CDSs are having issues syncing with the server even when they have a network signal. This is causing issues in two ways: 1) forms cannot send from the mobile phone to the server, making CCHQ data inaccurate, and 2) CDS and nurse mobile phones do not sync with each other to update patient records, causing confusion within the clinic. As explained above, this is an issue that is fixable, but needs to be communicated and addressed by the correct people. Moeketse was able to fix two mobile phones with this issue by simply removing the SIM card and putting it back in. Others required further action. Education module audio cuts off. During individual interviews, many users reported that the education module audio cuts off toward the end of the form. According to the APS officers, this issue has existed since the launch of CommCare three months ago. The CDS and nurses said they still use the module to educate patients; however, this technical issue may explain low usage. The users may also be closing the education module where the audio cuts off, which means the form is not sent, and CCHQ does not register that the module was used. The APS officer were aware of this issue for the past three months, however Moeketse and the Dimagi project staff were not. Again, this is an issue that can be easily fixed, but has yet to be addressed. All of the technical issues described above are fixable, but have not been communicated up the line to the appropriate people who can address them. To avoid these outstanding issues, any technical issue must be communicated from the user to the APS officer, and from the APS officer to Moeketse. If Moeketse is unable to fix the technical issue, he can ask Dimagi for support. Unfortunately, there has been weak communicated between the APS officers and Moeketse on technical issues, which is why they have persisted for so long. III. Minimal supervision Introducing mobile technology into a CHW system workflow requires the appropriate amount of supervision and support. During the launch of the project, the Dimagi team worked with the ICAP project staff to set up supervision and technical support roles within

38 Interview with Nurse

35

ICAP’s structure (see page 10-‐12 for description of roles). Since the project launch in March, some of the original plans for supervision have not been realized. Unfortunately, APS manager Mateboho Mokobocho had an emergency and has been out of the office since May, leaving a void in the Project Coordinator role. As supervisors to the CDSs and nurses, the APS officers originally decided to conduct monthly meetings with users to discuss challenges and success with CommCare; however, these meetings have not happened yet. The APS officers reported that they check the CCHQ automatic worker activity reports periodically, and provide support to users during their regular visits to Leribe district clinics (one-‐to-‐four times per month).39 With Mateboho gone, and Tlohang’s minimal usage of CCHQ, there is nobody regularly monitoring and responding to the CCHQ worker-‐ and case-‐level reports. From the APS officers, to the APS manager and M&E manager, there is minimal supervision on the CommCare project, and minimal technical support provided to the users. This the primary cause for the outstanding technical issues that have persisted throughout the project, and a likely cause for any under-‐usage of the application. If the users’ technical issues are not being resolved, and their work not being monitored, it is difficult to expect usage to be 100% at this point in the project (see page 36 for CommCare usage levels and analysis). IV. CommCare HQ Introducing CommCare into a CHW system provides two levels of benefits: user-‐level, and supervisor-‐level. CommCare benefits the users—CDSs and nurses in this case—by streamlining case management processes and creating an improved, paperless system for on-‐the-‐go health workers. On the second level, CommCare benefits supervisors and other higher-‐level health officials by providing a tool to monitor CHW activity and improve access to program-‐level outcome data that would otherwise by buried in stacks of paperwork. Both user-‐ and supervisor-‐levels of CommCare must be utilized for the full potential of mobile technology to be leveraged. In this CommCare project, the supervisor-‐level (CCHQ) is not being utilized to its full potential. As Project Coordinator, Mateboho rarely logs in to CCHQ because she does not understand how to navigate the different reports available.40 As M&E manager, Tlohang has a better grasp of the technology, but has many other M&E responsibilities and rarely checks the CCHQ reports for data analysis. The APS officers are not expected to regularly login to CCHQ, but receive weekly emailed reports on worker activity. According to interviews and observations, APS officers periodically check these reports, but not weekly, and almost

39 Interview with Puleng Nthinya, interview with Nkomane Seele 40 Interview with Mateboho Mokobocho

36

never provide feedback to users based on this data. There are three potential causes for the low usage of CCHQ reports at ICAP:

1. The first was stressed by the ICAP supervision staff (APS officers, APS manager, M&E manager): too short of a technical training by Dimagi on CCHQ data management. According to ICAP staff, the one-‐day training on CCHQ was too hurried, and didn’t provide enough hands-‐on practice before the supervisors were expected to manage reports on their own. CCHQ provides a space for ‘saved reports’ where supervisors can create and save custom reports to be accessed easily at any time. However, these reports were not set up during the beginning of the project, and therefore have not been utilized. The APS officers and APS manager suggested that extending the training by one or two days, and including more hands-‐on practice with navigating the site and setting up reports would increase their comfort with CCHQ.

2. The second cause of low CCHQ usage can be traced to overall low levels of supervision on the CommCare project. As described in the section above (“III. Minimal Supervision”), the Project Coordinator/APS manager on the project has been on sick leave for six weeks and counting. This has resulted in minimal CCHQ usage and overall supervision of CDSs by the APS officers. Without someone designated to regularly monitoring CCHQ reports, and checking on the APS officers, there has been very minimal usage of this valuable monitoring and supervision tool. Tlohang has also rarely used the CCHQ reports, although is very capable of navigating the data. As M&E manager, he is in charge of monitoring district and country-‐wide data, and CCHQ only gathers data from 11 sites in one district. While the CCHQ data may not be extremely useful to his large-‐scale duties now, once CommCare expands across Lesotho, all M&E officers will need to start using the CCHQ data exports.

3. The third cause of low usage is linked to the inability of field-‐level supervisors (APS officers) to access patient outcome data on CCHQ. The CCHQ report structure is the same for all projects regardless of the content of the application, so the available reports focus on worker activity and form submission, which are relevant across all CommCare applications. However, because all CommCare applications have different modules and forms, data from specific questions within the application are not easily accessible on these reports. Patient outcomes and other key program monitoring data are often captured in these specific questions within forms. Accessing this question-‐level data requires exporting excel files from CCHQ, which are data-‐heavy. For the APS officers and APS manager (who are not tech savvy), these reports are not easy to decipher.41 ICAP M&E Manager (Tlohang) was trained

41 Interview with Mateboho Mokobocho

37

on accessing these data exports, however has not used them and has since forgotten. The issue at hand is not a lack of data capturing, but rather insufficient training of supervisors to access and act on this data. The Recommendations section below suggests more in-‐depth training to address this issue.

According to Dimagi, low CCHQ usage is a challenge within every project, at varying levels. While some organizations adopt the supervision level of CommCare easily, others struggle to incorporate the data into their existing monitoring and evaluation processes. In order to increase the usefulness of CCHQ, ICAP suggested longer technical training and more accessible outcome data. Dimagi can modify training processes and help the partner organization create custom saved reports, but the successful usage of CCHQ is also dependent on the adopting organization. The current structure within ICAP has not provided enough supervision at each level. Key patient outcome data is available on CCHQ, but it requires someone familiar with excel to regularly monitor the data, and present it in a format that is useful for the lower-‐level supervising staff. Currently, this is not happening. The previous two sections describe the benefits and challenges of CommCare based on feedback collected in surveys, interviews, and focus group discussions. The following section, CommCare Usage Analysis, uses quantitative data on CommCare usage from CCHQ to assess the success of the project in Lesotho.

V. CommCare Usage Analysis Since the launch of the CommCare project in March, CommCare HQ has been collecting data on how and when the CDSs and nurses use the application. This section of the report will analyze the usage data to understand how well CommCare is being utilized in Leribe district TB programs. The most basic way to observe patterns in usage across the three and a half months is through weekly total form submissions. Figure 9 below maps the total number of forms submitted by each of the four groups—Puleng and Nkomane Sites, and CDSs and Nurses— each week:

38

Figure 9 displays the change in total number of forms submitted each week by the four different categories of users: all users at Puleng sites, all users at Nkomane sites, all CDSs, and all nurses. The data comes from CCHQ automatic weekly worker activity reports from March 7 to June 24. When comparing the four groups to each other, it is clear that the CDSs consistently have a higher number of form submissions than nurses, until the last week (June 24th). Similarly, Puleng sites generally have higher total form submissions than Nkomane sites, except for June 3rd and June 17th. Generally, these numbers make sense. CDSs are the primary users of CommCare, and nurses are supposed to register and update patients in CommCare when the CDS is unavailable. Puleng sites should also have a higher number of forms submitted, because there are more Puleng sites than Nkomane sites in Leribe district (six vs. five). However, these trends were not perfectly consistent throughout the three-‐and-‐a-‐half month period. When asked about the change in total number of form submissions each week, the APS officers could not identify specific events or reasons to explain the drastic differences by week. However, total submissions could be affected by the number of TB patients that week, clinic or district-‐wide staff meetings or trainings, public holidays, weather that prohibits home visits, etc. Total number of form submissions gives us an idea of how the four different users’ overall usage compares to each other, but does not provide any detailed information about how CDSs and nurses are using specific components of the application.

0

20

40

60

80

100

120

140

160

180

2-‐Mar 22-‐Mar 11-‐Apr 1-‐May 21-‐May 10-‐Jun 30-‐Jun

total # form

s subm

itted

Date of HQ Report

Weekly Total Form Submission: March 7-‐June 24

Puleng Sites

Nkomane Sites

CDS

Nurses

39

In order to better understand how the CommCare application is being used, usage within the different modules must be broken down. Figure 10 below displays the actual number of submissions by form, as compared to the estimated number that would indicate 100% usage of the application. The 100% usage goals were calculated based off the number of “register patient” forms submitted. It was assumed that for every patient registered there should be two follow-‐ups (“update appointment”), three contacts registered (“contact registration”), and at least one education form submitted (“patient,” “DOT supporter,” and “Contact” education). The 100% usage goal for “contact follow-‐up” was calculated from the number of “contact registration” forms submitted.

Figure 10 compares the total number of forms submitted (as of June 24) to the number of submissions that would indicate 100% usage of the application. All of the 100% usage goals were estimated with the help of Koen Frederix, technical director at ICAP. Patient Registration and Appointment Book Modules Figure 10 is best read from the bottom up. The “register patient” form submission total provides the baseline for the 100% usage goal estimates in the rest of the graph (calculations explained above). The first notable gap is the number of “update appointment” forms submitted compared to the 100% usage goal estimate. The “update appointment” form is submitted every time a patient comes in for a follow-‐up, so this gap indicates a slightly low follow-‐up rate. However, it must be noted that the 100% usage goal was calculated assuming an average of two follow-‐ups per patient, and did not take into account the patients who were registered and then removed. Because both of these factors

0 100 200 300 400 500 600 700 800

Patient registration>Edit patient Patient registration>register patient Patient Registration>Remove patient

Appointment book>make appointment Appointment book>Update appointment

Missed Appointments>Call Patient Missed Appointments>Assign CHW

Missed Appointments>Add CHW Visit Outcome Missed Appointments>Add treatment outcome

Household Visit>Contact registration Household Visit>DOT checklist

contact follow-‐up>contact follow-‐up Education>Patient

Education>DOT supporter Education>contact

Total # Submitted

Form

Type

Total Form Submission: March 7-‐June 24

CDS Submissions Nurse Submissions 100% Usage Goal

0236

40

could be skewing the data, it is important to use the above graph as a general summary of usage, not exact numbers. Within the “Appointment Book” module, form submissions for “make appointment” are very low because the users create appointments through the “register patient” form, and then continue schedule follow-‐up appointments in the “update appointment” form. Missed Appointments Module All form submissions within the “missed appointments” module are extremely low. According to APS officers, and users during focus group discussions, these low numbers are an accurate portrayal of the high appointment retention since the deployment of CommCare. The decrease in missed appointments was mentioned by APS officers and users during interviews, but is difficult to calculate because missed appointments were previously not tracked. Additionally, there could be more missed appointments, but only six were called, three were assigned a CHW, and two of the CHW visits outcomes were updated. It is difficult to tell whether these numbers decreased down the line because patients were brought back to treatment after each step (i.e. three patients were brought back just by calling), or if the CDSs just stopped taking the additional steps to contact defaulted patients. It is difficult to tell if there are truly few missed appointments, or if these numbers only reflect a small portion of the TB patients who have interrupted their treatment. Without consistent use of the paper appointment book (which some clinics have stopped using) it is difficult to track this. Household Visit Module Within the “household visit” module, “contact registration” form submissions are extremely low compared to the 100% usage estimate. Contact registration has always been a weak point in Lesotho’s TB system of care, and these form submission numbers highlight the gap. According to Koen Frederix, there is an average of three contacts per TB patient in Lesotho, and this graph shows that less than one fourth of them are being registered in CommCare. Within the same module, “DOT checklist” form submissions are also low—at less than half of what they should be according to the number of patients registered. Again, these estimates are not exact calculations, but provide a general overview of the gaps in usage. These gaps could be due to the fact that this module requires the CDSs to regularly conduct household visits, which does not always happen. Additionally, one nurse reported that her CDS has trouble completing the long “DOT checklist” form, and often brings it back to the clinic and completes it there, hours later. The length of the form could be causing users to leave it incomplete, or to just ignore it all together. When asked if any of the content could be taken out to make it shorter, the nurses agreed that all of the questions were important, and that time and practice will make the form easier to complete.42

42 Nurse Focus Group Discussions

41

Contact Follow-‐up Form The possible inconsistency of household visits could also be effecting the “contact follow-‐up” form submission numbers, as it requires the CDSs to register the contact during one home visit, and then return the next month and follow-‐up with the contact. According to the form submission data on CCHQ, less than one sixth of the 100% usage goal has been submitted by CDSs in the past three and a half months. Contact follow-‐up numbers could be affected by the CDS’s consistency regarding home visits, and by the contact’s presence (or absence) in the patient’s home during the follow-‐up visit. Ideally, the contact “follow-‐up” form would be completed in the clinic, when the contact comes in for their TB test. This would eliminate the extra step of a second household visit for the form to be completed. However, sometimes the location where the contact goes for his/her test is not the same clinic as the CDS that registered the contact initially. If the contact goes to a different clinic, the contact would still receive the test and treatment if necessary, but the initial contact case registered in CommCare would not be followed-‐up with, making contact tracing very difficult. This is a challenge that ICAP and Dimagi must work through together—the current system is yielding extremely low usage, and some of the simplest alternatives may not work smoothly on the ground. Education Module According to Figure 10, the education module has been underused in the past three and a half months of CommCare deployment. Compared to the 100% usage estimate (one education per patient registered), patient education is at two-‐thirds of full usage, while DOT supporter and contact education are at about one-‐fifth of full usage. These low numbers are concerning, as education is a critical component of the TB treatment protocol. The users and APS officers mentioned during interviews that there was an adoption curve for using the mobile education tool, and education has increased since CommCare was first deployed. Because the values in Figure 10 are cumulative from March 7th to June 24th, this increase in adoption would not be visible above. Figures 11, 12, and 13 below break down the education module usage by month:

42

Figure 11 breaks down the patient education module usage from Figure 10 by month (March, April, May, June). 100% usage goal estimates were calculated based on the number of patient registration forms submitted that month (1 register patient form:1 patient education form). Figure 11, Patient Education Trends: March-‐June, displays a consistent increase in patient education module usage from March to June. Usage in March was extremely low—approximately one-‐seventh of the 100% usage goal was submitted. In April usage doubled, and then doubled again in May. Already by June 24th, more “patient education” forms had been submitted than “register patient” forms for the month of June. This increase in patient education module usage mirrored comments from users and APS officers that adoption has increased since deployment. However; this pattern of drastically increasing adoption did not carry through to the usage of the DOT supporter education and Contact Education modules:

Figure 12 breaks down the DOT supporter education module usage from Figure 10 by month (March, April, May, June). 100% usage goal estimates were calculated based on the number of patient registration forms submitted that month (1 register patient form:1 DOT supporter education form).

-‐20 -‐10 0 10 20 30 40 50 60 70 80

March

April

May

June

# Forms Submitted

Month

Patient Education Trends: March-‐June

Total # Educated

100% Usage Goal Estimation

0 10 20 30 40 50 60 70 80

March

April

May

June

# Forms Submitted

Month

DOT Supporter Education Trends: March-‐June

Total # Educated

100% Usage Goal

43

The monthly breakdown of DOT supporter education module usage also displays an gradual increase, however, usage is still extremely low. DOT supporter education may occur when the DOT supporter accompanies the patient to the clinic, or in the home during a household visit by the CDS. The DOT supporter should receive the education at least once, and the figure above indicates that education is only occuring one-‐third of the time. It makes sense that DOT supporter and contact education usage would be lower than patient education, because the first two require an extra step to occur—either the DOT supporter must come to the clinic, or the CDS must go to the household and educate the DOT supporter or contact. Figure 13 below displays the contact education module usage, which mirrors the DOT supporter education module usage from Figure 10:

Figure 13 breaks down the contact education module usage from Figure 10 by month (March, April, May, June). 100% usage goal estimates were calculated based on the number of patient registration forms submitted that month (1 register patient form:1 contact education form). Similar to DOT supporter education trends, contact education also increased slightly from March to June. The 100% usage goals were calculated based off the number of patient registration forms submitted each month. Knowing that the average TB patient has three contacts, there should be at least one contact education form submitted per patient. This is assuming that all contacts are educated at the same time, during the home visit from the CDS. Considering the extremely low usage of the contact registration form (Figure 10), it is not surprising that contact education is underused by the CDSs. Overall, the frequency of education is increasing with CommCare. While patient education has already surpassed the 100% usage goal, DOT supporter and contact education remain underused. During focus group discussions, users were asked about the low usage of CommCare’s education module. Some users said that if they are in a hurry to register the patient, they kip the education module and give a shortened version of the education verbally. Others said that because the audio cuts of at the end, sometimes they don’t finish the education on the phone. In this case, the form would not send, and CCHQ would not

0 10 20 30 40 50 60 70 80

March

April

May

June

# Forms Submitted

Month

Contact Education Trends: March-‐June

Total # Educated 100% Usage Goal

44

recognize that the patient has been educated—this could be affecting the usage numbers. However, these are minor decreases in the overall usage of the application, and it is clear that DOT supporter and contact education remains underutilized. While this is concerning, CommCare now allows us to identify these gaps in TB care, and act on them. The following section provides recommendations based off the user feedback and CCHQ data analysis outlined in the previous two sections.

VI. Conclusion: Key Recommendations This report evaluates the training and implementation processes used to integrate CommCare into Lesotho’s system of TB care. Feedback from Core DOT Supporters, nurses, APS officers, and other key stakeholders indicate that CommCare is appreciated and valued by its users, and has improved many of the treatment and data collection processes used for TB care. Overall, users described CommCare as an extremely valuable tool: “CommCare is like our 3rd hand in improving patients’ adherence.”43 Surveys, interviews, and focus group discussions with users were used to identify the main benefits and challenges of CommCare in Leribe District TB programs. The main benefits of CommCare are: less paperwork and more accountability for CHW supervisors; improved case management, consistency in TB education, and home visits; and automatic data collection for improved program monitoring. The challenges with CommCare are: clinic workflow issues, technical issues, minimal supervision and support for users, and difficulties with CCHQ data management. These challenges are apparent in the CCHQ usage data, as components of the application are underused. Much of this underutilization is due to the lack of supervision of both the users, and the APS officers. Without accountability, areas like CDS household visits and APS officer supervision and support have been neglected. In order for CommCare to be utilized to its full potential, these key areas must be addressed. The following sections provide recommendations for Dimagi and ICAP to address the current challenges and maximize the benefits of mobile technology in Lesotho’s system of TB care. User-‐Recommended App Modifications A major purpose of this evaluation was to collect user feedback on changes that need to be made to CommCare before it is scale-‐up. The application modifications described below were recommended by users, and either confirmed or rejected by Koen Frederix, technical director at ICAP, in the column to the right. 43 Nurse Survey (#16)

45

Recommended Change User Comments ICAP Decision

1

Registration form: add directions to patient’s home

CDS & Nurses

Information that is in the paper register, not currently on CC registration form

Yes—adopt

2 Registration form: add two options for home address

CDS & Nurses

Same as above. Yes—adopt

3 Registration form: add ART treatment status

CDS & Nurses


4 Registration form: add occupation

CDS & Nurses


5 Registration form: add sputum results

CDS & Nurses


6 Registration form: add drop-‐down options for “remove patient” form

CDS & Nurses

Currently a comment box with no standardized responses. Also causes confusion because have to look in 2 places (update appointment) to find outcome of patient…

Yes—adopt

7 Appointment module: add reminder for nurses 2 days before appointment

Nurses only

Nurses manage more than just TB cases, sometimes forget to check appointment book-‐-‐want alert on phone rather than SMS if possible.

Yes—adopt

8 Appointment module: add “edit appointment date” form

CDS & Nurses

-‐Dangerous if patient changes to later date and is off treatment during that time. -‐CDS/nurses know not to allow this, should have freedom to modify date

Yes—adopt

9 Appointment module: add “followed-‐up at different clinic” option for update appointment form

CDS & Nurses

Should be coming back to clinic after getting meds from hospital

No—leave as is

10 Education module: fix audio recording—cuts out during bus

CDS & Nurses

Yes—adopt

11 Education module: add HIV education

CDS & Nurses

Need to decide when HIV education will be integrated. Possibly when Jolani comes.

Need to discuss further

12 Education module: add cultural myths about TB

CDS & Nurses

See above. Need to discuss further

13 Education module: add warning about alcohol and smoking

CDS & Nurses

See above.

Need to discuss further

14 Contact registration: add CDS & Just need the ability to enter “/” to Yes—adopt

46

“/” as input option for age under 1 year (ie. 4 months old: 4/12)

Nurses indicate age in terms on months for infant

15 Contact registration: give nurses access to contact module

Nurses -‐So that nurses can submit contact follow-‐up form when contact comes in, can put in test outcome -‐Location where contact goes for sputum test is different than clinic where TB patient registers

Maybe—need to discuss better solution

16 Contact registration: add “TB test outcome” and “treatment status” to contact follow-‐up form

CDS & Nurses

Better for tracking outcomes Yes—adopt

17 Contact registration: add automatic SMS reminders for contacts to come to clinic

CDSs said they usually don’t have phones, can put in patient or family member phone #

Yes—adopt

Recommendations for ICAP

Recommended Action Responsible Party

à Implement application modifications • Minor changes to modules as recommended by users

(see “Application Modification” table for specific changes)

• Decide which modifications to make to Education module, and during which phase to implement

• Ali: Recommend modifications to Dimagi

• ICAP: Confirm education module changes

• Dimagi: Implement changes remotely and on-‐site

à Address current technical issues • Change roaming settings for users in border clinics • Fix phones with issues synchronizing to CommCare

HQ and to their CDS/nurse counterpart at the clinic • Fix low volume issue on older Nokia models • Identify users whose SMS reminders are not sending

and if not resolved, report to Dimagi

• APS officers: Identify all current issues

• Tlohang: Support APS officers with ongoing technical issues and report unresolvable issues to Dimagi

• Dimagi: Support Tlohang with unresolvable issues

à Develop a clear protocol for addressing future technical issues

• Create a checklist of the most common technical issues and simple steps to resolve them

• Incorporate Leribe M&E officers for technical assistance

• Report and record all unresolvable technical issues using the “CommCare Technical Issue Report”

• Eliott & Tlohang: Identify most common issues and develop clear checklist for APS officers

• APS officers: Use checklist to resolve minor issues, and report unresolvable technical issues to Tlohang

• Tlohang: Train Leribe M&E

47

• Meet on the last Friday of each month to discuss any ongoing technical challenges, and provide support with understanding CommCare HQ reports

officers to provide technical support

• APS officers & Tlohang: Meet monthly to discuss challenges

à Increase support for Core DOT Supporters and Nurses using CommCare

• Follow-‐up on all technical issues using the above protocol

• Use CommCare HQ reports to provide feedback to CDS and nurses on CommCare usage

• Organize monthly (or quarterly, depending on funding availability) meetings with CDS and nurses to discuss CommCare challenges and share ideas/successes

• APS officers & Tlohang: Consistently follow-‐up on technical issues

• APS officers: Use weekly emailed reports to provide feedback to CDS and nurses, and organize regular meetings with CDS & nurses to provide support

à Increase monitoring of CommCare project outcomes • Consistently check weekly emailed CommCare HQ

reports • Meet monthly to improve APS officer knowledge of

CCHQ • Increase M&E officer monitoring of online CCHQ data • Report monthly CommCare usage and treatment

outcomes • Provide individual clinics with monthly TB reports

• APS officers: Check weekly emailed CCHQ reports

• Tlohang: Increase monitoring of CCHQ data, and meet with APS officers monthly to provide support with CCHQ reports

• Elliott: Develop template for monthly monitoring

à Modify training processes for phase 2 scale-‐up • Expand to 4 days for those who need extra practice • Identify strong current users to assist with scale-‐up

training • Conduct on-‐site trainings for other nurses (with

assistance of current CommCare CDS and nurse at each site)

• APS officers: Identify strong CDS/nurses to assign with scale-‐up training

• APS officers & Mateboho (with assistance from current users): On-‐site trainings for additional nurses

Recommendations for Dimagi

Recommended Action

Project-‐Specific Recommendations

à Implement application modifications • Remote modifications to modules as recommended by users (see “Application Modification”

table for specific changes) • Discuss contact follow-‐up module improvements with ICAP for best solution to increase

contact tracing • Discuss timeline for education module changes with ICAP • Confirm that technical issues have been resolved, as some may require Dimagi assistance

48

(i.e. sms reminders, forms not sending) à Conduct refresher trainings on-‐site with project supervisors

• More training needed for APS officers, APS manager, and M&E on CCHQ—data management and action

• Help APS manager (Mateboho) and M&E manager (Tlohang) set up saved reports and case exports to increase consistent usage

• Encourage ICAP to include district-‐level M&E on monthly supervision—may need to train them as well

à Increase availability of program outcome information • On-‐site assistance with setting up useful CCHQ saved reports and case/form exports that

reflect program outcomes (# on treatment, # missed appointments, #missed appointments brought back to treatment, etc.) rather than usage

• Supportive supervision application for technical support, CCHQ reports, and feedback to users (see Supportive Supervision App Brainstorm doc)

• Impact123—share impact outcomes with Koen if possible

General Implementation Recommendations

à Assess mobile literacy levels of users prior to training, and adjust length appropriately • Ask the partner organization to assess the CHWs’ mobile usage (particularly SMS and

internet) prior to training • If low mobile literacy, ask supervisors to lead the additional training: either 4th day of demo

practice in pairs, or follow-‐up training 2 weeks later (preferred) • Generally, users prefer more time for demo practice in pairs

à Increase focus on capacity building during implementation • Longer training for CCHQ (2 days), more hands-‐on—sit with them as they set up “saved

reports” • Make sure there are clear roles for who will monitor and act on CCHQ data • Stronger focus on technical problem-‐solving. Ex: 1-‐page guide for CHW supervisors on most

common technical issues and simple steps to solve them; tools for technical problem-‐solving, like Jolani’s supportive supervision app, and “CommCare technical issue report template” to improve communication

à Improve data accessibility on CommCare HQ • Clearer reports for outcome tracking—currently available through case/form exports, but

key data is often difficult to find in the large, messy Excel files • Continue expanding Impact123! Will be an extremely valuable tool for all partners

Recommendations for Further Research This study provided valuable feedback on the benefits and challenges of CommCare within Lesotho’s TB system. However, the data gathered in this study had several limitations preventing it from producing generalizable results. Several recommendations for further research address these limitations and would provide a valuable follow-‐up to this evaluation:

49

Larger Study. Another study with a larger sample size exploring the same two research questions would be valuable to better understand patterns in mobile adoption and usage. Such a study could especially provide a deeper understanding of factors that affect adoption, such as user age and previous mobile usage. Factors that contribute to organization-‐wide adoption of mobile technology. This evaluation revealed that CDSs and nurses generally found CommCare useful and easy to use, however several technical and supervision difficulties were limiting 100% usage of the application. In the case of this evaluation, supervision was a challenge to the full adoption of CommCare in Leribe district. Further research should investigate the factors that affect an organizations ability to successfully adopt mobile technology. Some potential research questions could be: What factors contribute to an organization’s successful adoption of mobile health technology—from the users all the way up to program directors? What implementation processes on the part of mobile technology companies facilitate successful adoption?

VII. Appendices Appendix A: Translated Survey

NO ENGLISH SESOTHO This evaluation is being conducted to get

your feedback on the CommCare application and the training you received to use it. Please tell us how to improve so that CommCare can be made as useful as possible.

Lipatlisitso tsena li etsoa ho fumana maikutlo a lona ts’ebelisong ea marangrang a sejoale joale le thupelo eo le e fumaneng ea ts’ebeliso ea ona. Re kopa ore bolelle maikutlo a hao a ka ntlafatsang tsebeliso ea marang rang ana ka mokhoa o phethahetseng

1

2 3 4

5 6

User Background Name Age Highest completed education (circle one): primary; secondary; university CDS or Nurse (circle one) Years worked as CDS/Nurse Health facility:

litaba tsa Mosebelisi lebitso lilemo Thuto e ka holimo-‐limo eo oe fihletseng (khetha ele ‘ngoe) primary; secondary; university O Motsehetsi e moholo kapa Mooki (khetha e le ‘ngoe) Lilemo tseo oli sebelitseng ole Mots’ehetsi e moholo kapa Mooki

50

Setsi sa bophelo

7 8 9 10 11

Mobile Phone Ownership and Usage: Who is the primary owner and user of your mobile phone? I own my own phone, I share with a family member; other: How many years have you owned the mobile phone?

a. Less than 1 year. b. 1-‐3 years c. More than 3 years

How often do you send SMS? a. Daily b. once a week c. Never

How often do you make phone calls?:

a. Daily b. Once a week c. Never

How often do you use the internet on your phone (eg. google, facebook, whatsapp)?:

a. Daily b. Once a week c. Never

Boikarabello mohaleng le tsebelisong ea ona Ke mang motho oa mantlha a sebelisang a bile a na le boikaraelllo mohaleng oa hao? Ke ‘Na Ke arolelena le balelapa Babang : Ke lilemo tse kae o ena le mohala oa thekeng bophelong ba hao?

a. Ka tlase ho selemo b. selemo ho isa ho tse tharo c. Ho feta lilemo tse tharo

O romela melaetsa e ngotsoeng bonyane ha kae?

a. Tsatsi leleng b.Ha ngoe ka beke c. Ha ke so romele molaetsa hohang

Ke ha kae o letsang ka mohala?

a. Tsatsi le leng le le leng b. Hangoe ka beke c. Ha ke so letse ho hang

Ke ha kae o sebelisang marang -‐ rang ka phone joaloka (google, facebook, whatsapp)?:

a. tsatsi le leng le leleng b. Ha ngoe ka beke c. Ha ke so romele molaetsa ka marang-‐

rang

51

12 13 14

CommCare Training: For each of the statements, select the answer that represents what you think about the CommCare training you received three months ago. I received the complete CommCaretraining:

a. Yes: b. No

Learning how to use CommCare was easy for me (1) Strongly Disagree (2 )Disagree (3) Indifferent (4)Agree (5)Strongly Agree Upon completing the training, I felt comfortable using CommCare: (1) Strongly Disagree (2) Disagree (3) Indifferent (4)Agree (5)Strongly Agree

Koetliso ea marang rang Bakeng sa polelo e ngoe le engoe, khetha karabo e bolelang na o ikutloa joang ka koetliso ea marang rang eo o fumaneng likhoeli tse tharo tse fetileng : fumane thupelo e feletseng ea marang rang . a. ee: b. chee: Tsebeliso ea marangrang a Commcare e bile bonolo ho ‘na 1.Ha ke Lumellane le taba eena ho hang 2.Ha ke Lumele 3.Ha ke tsebe hantle. 4 . Ke ea lumela hore koetliso e ts’oana le tsebeliso 5. Ke lumela ka botlalo Kamora ho qeta thupelo , ke utloile ke phuthulohile ho sebelisa commcare: 1.Ha ke Lumellane le taba eena ho hang 2.Ha ke Lumele 3.Ha ke tsebe hantle. 4 . Ke ea lumela hore koetliso e ts’oana le tsebeliso 5. Ke lumela ka botlalo

Training Content: Rate the following sections of the training (circle one number). (1)=Unhelpful (5)=Very helpful

Litaba tsa koetliso: Kala likarolo tse latelang tse amanang le thepelo( etsa lesakana ho ele ‘ngoe e tsamaeang le maikutlo a hao) (1) =Haho thuso eo ke e fumaneng (5)=Ke fumane bohlokoa bo boholo

52

18 19 20

Three day training: 15. Plenary sessions with demos

by the trainer (1) (2) (3) (4) (5)

16. Group practice: (1) (2) (3) (4) (5)

17. Individual practice:

(1) (2) (3) (4) (5) Take-‐home training manual: (1) (2) (3) (4) (5) Practice in demo mode: (1) (2) (3) (4) (5) Feedback and support from supervisors: (1) (2) (3) (4) (5)

Thupelo ea matsatsi a mararo: 15. Boithuto boileng ba etsoa ka

(sets’oantsisi) ke morupelli: (1) (2) (3) (4) (5) 16. Pheta-‐pheto to chorisa ea kutloisiso ka lihlopha : (1) (2) (3) (4) (5) 17. Pheta-‐pheto ha morupeluoa a le mong (1) (2) (3) (4) (5) Ho ea hae ka se thusa thuto (1) (2) (3) (4) (5) Ho pheta hangata ka ‘Demo’ (1) (2) (3) (4) (5) Maikutlo le tsehetso etsoang ho barupeli (1) (2) (3) (4) (5)

21

Training Processes

Rank the different training processes by how helpful they were for you in learning to use CommCare: (1 being least helpful, 5 being most helpful) *Only assign one number (1-‐5) to each training process ____Three day training ____ Take-‐home training manual/handouts ____CommCare demo mode ____Feedback and reports from supervisors ___Supervisor phone number for direct support

Methati ea thupelong

Kala methathi e fapaneng ea thupelo ea marang rang hoea ka bohlokoa boo bi fumaneng ho eona eona(qala ka e bohlokoahali efe 5 hoea ho e bohlokoanyana efe 1) *fana ka palo e le ‘ngoe pakeng tsa (1-‐5) mothating o mong le o momg ____Matsatsi a mararo thupelong ____Honka Lithusa thuto ha o ea hae _____Ho ithuta ka Demo ele he ho malafatsa boithuto _____Ho fana ka maikutlo a thupelo ka barupeli Thuso e o e fumanang ka ho tseba ____nomoro ea mookameli eo o tlamehang ho buoa le eena ha ona le mathata

22

How many times did you contact a supervisor for support using

O kupile thuso ha kae ho mookameli oa marang rang a Commcare bakeng sa tsehetso ea

53

23 24 25 26

CommCare? a. Never b. 1-‐3 times c. More than 3 times

If you selected b or c, how helpful was this? (1)= Unhelpful (5)=Very helpful (1) (2) (3) (4) (5) How many times did you receive feedback without contacting a supervisor first?

a. Never b. 1-‐3 times c. More than 3 times

If you selected b or c, how helpful was this? (1)= Unhelpful (5)=Very helpful (1) (2) (3) (4) (5) Did you have any problems receiving support from a supervisor? Please explain below:

tsebeliso ha o thulana le mathata? a. Ha ke so kope thuso b. Ha ngoe ho isa borarong c. Ho feta ha raro Haeba o khethile karabo ea b kapa c, na oile oa thuseha? (1) =Haho thuso eo ke e fumaneng (5)=Ke fumane bohlokoa bo boholo (1) (2) (3) (4) (5) Ke makhetlo a makae o o fumaneng tharollo o saka oa botsa mookameli oa marangrang a Commcare pele? a. Ha ho so be joalo b. hangoe hoisa borarong c. Ho feta ha raro Haeba o khetha b kapa c , o fumane molemo o feneng? (1) =Haho thuso eo ke e fumaneng (5)=Ke fumane bohlokoa bo boholo (1) (2) (3) (4) (5) Na o kile oa ba le bothata ba ho fumantsoa tsehetso kapa thuso ho tsoa ho mookameli oa marang rang? Fana ka thlaloso ea bothata:

54

27 28 29

The training provided me with information I could not have figured out on my own: (1) Strongly Disagree (2)Disagree (3) Indifferent (4) Agree ( 5) Strongly Agree I will need more help with understanding the application in the future: (1) Strongly Disagree (2)Disagree (3) Indifferent (4) Agree ( 5) Strongly Agree The training was sufficient in length: (1) Strongly Disagree (2)Disagree (3) Indifferent (4) Agree ( 5) Strongly Agree

Thupelo ena e mphile malebela ao keneng nkeke ka a fihlella ka bo ‘na: 1.Ha ke Lumele ho hang 2.Ha ke Lumele 3.Ha ke tsebe hantle. 4 . Ke ea lumela hore koetliso e ts’oana le tsebeliso 5. Ke lumela ka botlalo Ke tla hloka thuso e ‘ngoe hape ho utloisisa tsebeliso ea marang rang nakong e tlang: 1.Ha ke Lumele ho hang 2.Ha ke Lumele 3.Ha ke tsebe hantle. 4 . Ke ea lumela hore koetliso e ts’oana le tsebeliso 5. Ke lumela ka botlalo Bolelele ba thupelo e ne ele bo lekaneng: 1.Ha ke Lumele ho hang 2.Ha ke Lumele 3.Ha ke tsebe hantle. 4 . Ke ea lumela hore koetliso e ts’oana le tsebeliso 5. Ke lumela ka botlalo

30 How do you think the training could be improved?

O nahana hore thupelo ee e ka ntlafatsoa joang?4

31 32

Are you currently using CommCare? a. Yes b. No If not, why?: a. Technical issues with my phone

and/or the application

Na o ntse o sebelisa marang rang ha joale? a.ee b. chee Haeba haho joale, hlalosa hore na hobaneng? a. bothata ba mohala kappa tsebeliso ea ona b. mathata a mang:

55

b. Other:

33 34 35

CommCare Adoption and Experience

After three months of use, I feel comfortable using CommCare: (1) Strongly Disagree (2)Disagree (3) Indifferent (4) Agree ( 5) Strongly Agree Rank the different components of the CommCare application by how useful they are to your job: (1 least useful , 6 most useful) _______ Automated SMSs to patients as

appointment alerts _______ The late mark ‘*’ as reminder to

follow-‐up with this patient _______ Missed appointment module

including workflow of actions; call patient, assign CHW etc.

_______ Register and follow-‐up on contact screening

_______ Educating patients/dot supporter/contacts

_______ Appointment module where next appointments show in order according to appointment date

Overall, CommCare saves me time: (1) Strongly Disagree (2)Disagree (3) Indifferent (4) Agree ( 5) Strongly Agree

Ka mora likhoeli tse tharo tsa tsebeliso, ke ikutloa ke le maemong a matle a ho sebelisa marang rang a commcare 1.Ha ke Lumele ho hang 2.Ha ke Lumele 3.Ha ke tsebe hantle. 4 . Ke ea lumela hore koetliso e ts’oana le tsebeliso 5. Ke lumela ka botlalo kala litsebeletso tse fapaneng tsa marang rang hoea ka bohlokoa ba tsona (6= e bohlokoahali, 1 e bohlokoanyana) ___melaetsa e khophiselitsoeng bakuli ho ba

hlokomelisa ka letsatsi la ho tla setsing ____Lets’oao lena ‘*’ ke sehopotso sa ho sala

mokuli morao ____Buka ea bakuli bas a tlang e kenyeletsa; ho

letsetsa mokuli,tsalo morao ka mosebeletsi oa tsa bophelo motseng le tse ling

____Ngoliso le ts’alo morao ho li patlisisso ho ba phelang le TB

____Thuto ho bakuli/bats’ehetsi/ba phelang le mokuli oa TB

_____Buka ea ho behela bakuli matsatsi a hotla setsing e bonts’a matsatsi ka tatellano

Ka kakaretso, marang rang a baballa nako ebile a potlaka: 1.Ha ke Lumele ho hang 2.Ha ke Lumele 3. Ha ke tsebe hantle. 4. Ke ea lumela hore koetliso e ts’oana le tsebeliso 5. Ke lumela ka botlalo

36. Any additional comments:

Maikutlo a mang ka tsebeliso ea marang rang a commcare:

56

Appendix B: APS Officer Interview Questions CommCare Training -‐ When Commcare was first implemented three months ago, was it difficult for the CDS and nurses to learn to use? -‐ Do you think the training was sufficient, or would the CDS and nurse benefited from more training? -‐ How do you think the training could be improved? -‐ Was it difficult for you to learn how to use the monitoring and reports in CommCare HQ? -‐ What type of training did you receive to learn how to monitor the users in CommCare HQ? -‐ What do you think should be changed, or added to the training? CommCare Adoption -‐ How well do you think the CDS and nurses are using CommCare now? -‐ Do you use the CommCare HQ reports to monitor their progress? -‐ When do go on CommCare HQ, which reports to you look at? What info is useful to you? -‐ Overall, how has CommCare affected your TB care programs? Usage among CDS and nurses -‐ Currently, the CDS are using the app much more than the nurses-‐-‐ Why do you think this is happening? -‐ Is there anything that can be done to improve the application that will increase usage? -‐ What do you think is causing the different levels of usage between each week? Beginning of May usage dropped, and then increased again—do you know what is causing these fluctuations? Supervisor Experience with CommCare HQ -‐ How has CommCare affected the way you monitor your clinics? -‐ How specifically does CommCare benefit your TB programs? -‐ What specific challenges does CommCare create within your TB programs? -‐ How do you think these challenges can be addressed? -‐ What can be changed about CommCare make it more successful for scale-‐up? -‐ Is there anything else you think Dimagi or ICAP should know before scaling up this project? Appendix C: User Score Calculations

Name

Position Worker Activity

WA Rank

Supervisor Rank

Total Score Calculation

Total Score

User Rank

Lelala CDS 86% 3 3 =3*.4+3*.6 3 H

57

Mafere CDS 86% 3 3 =3*.4+3*.6 3 H Makoala CDS 91% 3 2 =3*.4+2*.6 2.4 M Masoetsa CDS 20% 1 1 =1*.4+1*.6 1 L Moholane CDS 45% 1 3 =1*.4+3*.6 2.2 M Mokoteli CDS 100% 3 3 =3*.4+3*.6 3 H Mositi CDS 54% 2 3 =2*.4+3*.6 2.6 H Ramothibi CDS 100% 3 3 =1*.4+3*.6 3 H Rapatala CDS 54% 2 3 =2*.4+3*.6 2.6 H Rasethuntsa CDS 75% 3 3 =3*.4+3*.6 3 H Seema CDS 100% 3 3 =3*.4+3*.6 3 H Tsipo CDS 86% 3 2 =3*.4+2*.6 2.4 M Khatleli Nurse 100% 3 3 =3*.4+3*.6 3 H Lekhela Nurse 86% 3 2 =3*.4+2*.6 2.4 M Mohlaba Nurse 86% 3 3 =3*.4+3*.6 3 H Mpholo Nurse 91% 3 2 =3*.4+2*.6 2.4 M Mphunyane Nurse 75% 3 3 =3*.4+3*.6 3 H Ntene Nurse 20% 1 2 =1*.4+2*.6 1.6 M Ranketoa Nurse 45% 1 3 =1*.4+3*.6 2.2 M Ranthithi Nurse 100% 3 3 =3*.4+3*.6 3 H Semantle Nurse 85% 3 3 =3*.4+3*.6 3 H Tselanyane Nurse 100% 3 3 =3*.4+3*.6 3 H

Activity Rank Low=0-‐50 1 Medium=50-‐74 2 High=75-‐100 3 Calculated total score with 40% of worker activity and 60% of supervisor ranking. I then rounded the scores to their closest ranking: L= 0-‐1.4; M=1.5-‐2.4; H=2.5-‐3.

Bibliography Primary Sources Interview with Mateboho Mokobocho Interview with Puleng Nthinya Interview with Nurse Interview with Nurse Interview with CDS

58

Interview with CDS Interview with CDS Interview with Nkomane Seele Nurse Focus Group Discussion Nurse Survey (#16) Interview with Nkomane Seele CDS Focus Group Discussion Interview with CDS Interview with CDS Interview with Nurse interview with Nurse Interview with CDS Interview with CDS Interview with Nurse Interview with Nurse Interview with Koen Frederix Interview with Nurse Secondary Sources ICAP Lesotho Portfolio Description, 2013. “ICAP DOTS Project Planning” (Dimagi), Powerpoint by Nynke Brunner and Jolani De La Porte “HIT Project Roles”, Jolani De La Porte

CC Lesotho Final Report

Documents

Transcript of CC Lesotho Final Report