Dataset Information

Validity of using mobile phone surveys to evaluate community health worker program in Mali

ABSTRACT:

Background

The monitoring and evaluation of public health programs based on traditional face-to-face interviews in hard-to-reach and unstable regions present many challenges. Mobile phone-based methods are considered to be an effective alternative, but the validity of mobile phone-based data for assessing implementation strength has not been sufficiently studied yet. Nested within an evaluation project for an integrated community case management (iCCM) and family planning program in Mali, this study aimed to assess the validity of a mobile phone-based health provider survey to measure the implementation strength of this program.

Methods

From July to August 2018, a cross-sectional survey was conducted among the community health workers (ASCs) from six rural districts working with the iCCM and family planning program. ASCs were first reached to complete the mobile phone-based survey; within a week, ASCs were visited in their communities to complete the in-person survey. Both surveys used identical implementation strength tools to collect data on program activities related to iCCM and family planning. Sensitivity and specificity were calculated for each implementation strength indicator collected from the phone-based survey, with the in-person survey as the gold standard. A threshold of ≥ 80% for sensitivity and specificity was considered adequate for evaluation purposes.

Results

Of the 157 ASCs interviewed by mobile phone, 115 (73.2%) were reached in person. Most of the training (2/2 indicators), supervision (2/3), treatment/modern contraceptive supply (9/9), and reporting (3/3) indicators reached the 80% threshold for sensitivity, while only one supervision indicator and one supply indicator reached 80% for specificity. In contrast, most of the stock-out indicators (8/9) reached 80% for specificity, while only two indicators reached the threshold for sensitivity.

Conclusions

The validity of mobile phone-based data was adequate for general training, supervision, and supply indicators for iCCM and family planning. With sufficient mobile phone coverage and reliable mobile network connection, mobile phone-based surveys are useful as an alternative for data collection to assess the implementation strength of general activities in hard-to-reach areas.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01317-7.

SUBMITTER: Chen X

PROVIDER: S-EPMC8176601 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundIn Mali, community health workers (CHWs) deliver essential community care (ECC) to rural populations. The dominance of external funding for the program threatens the sustainability of this critical workforce as donor financing decreases. This article summarizes results of analyses aimed at assisting Mali's decision makers and leaders in initiating a transition to a sustainable CHW program supported by domestic funding through strategic and rational investment.MethodsData on ECC implementation norms, workforce, coverage, utilization, cost, and geospatial features were collected between 2016 and 2019. The data informed interlinked CHW financing analyses-situational, services costing, efficiency, and geospatial mapping. Analysis showed distribution of reported expenditures, estimates of required CHW funding, cost-saving options, and spatially visualized discrepancies between spending estimates and normative costs.ResultsThirteen financing sources contributed to CHW program expenditures, 88% of which were from international donors, for a package of 23 curative, preventive, and promotive interventions. In 2015, the CHW program spent US$13.01 million; an estimated US$8.36 million would have been needed to achieve the same service volume under standard care protocols. Medicines and start-up training had US$6.88 million more than needed; supervision, program management, and recurrent training components were underfunded by US$2.2 million. Cost-saving opportunities of US$6.16 million were identified in 41 of 44 districts. Funding reallocation opportunities (after meeting technical efficiency requirements) were identified in 20 of 44 districts (US$2.56 million). Use of geospatial targeting and mapping suggests district- and village-level reallocation options for theoretical funding surpluses.ConclusionCHW costs can be significantly reduced without sacrificing service technical quality. Spending can be geographically targeted to optimize service use by rural populations. Efficiency analyses provide evidence to build stronger engagement, support improved decision making, efficiently prioritize resources, and target investments for sustainable financing of CHW programs.

Project description:BackgroundThe Better Health Outcomes through Mentoring and Assessment (BHOMA) project is a cluster randomized controlled trial aimed at reducing age-standardized mortality rates in three rural districts through involvement of Community Health Workers (CHWs), Traditional Birth Attendants (TBAs), and Neighborhood Health Committees (NHCs). CHWs conduct quarterly surveys on all households using a questionnaire that captures key health events occurring within their catchment population. In order to validate contact with households, we utilize the Lot Quality Assurance Sampling (LQAS) methodology. In this study, we report experiences of applying the LQAS approach to monitor performance of CHWs in Luangwa District.MethodsBetween April 2011 and December 2013, seven health facilities in Luangwa district were enrolled into the BHOMA project. The health facility catchment areas were divided into 33 geographic zones. Quality assurance was performed each quarter by randomly selecting zones representing about 90% of enrolled catchment areas from which 19 households per zone where also randomly identified. The surveys were conducted by CHW supervisors who had been trained on using the LQAS questionnaire. Information collected included household identity number (ID), whether the CHW visited the household, duration of the most recent visit, and what health information was discussed during the CHW visit. The threshold for success was set at 75% household outreach by CHWs in each zone.ResultsThere are 4,616 total households in the 33 zones. This yielded a target of 32,212 household visits by community health workers during the 7 survey rounds. Based on the set cutoff point for passing the surveys (at least 75% households confirmed as visited), only one team of CHWs at Luangwa high school failed to reach the target during round 1 of the surveys; all the teams otherwise registered successful visits in all the surveys.ConclusionsWe have employed the LQAS methodology for assurance that quarterly surveys were successfully done. This methodology proved helpful in identifying poorly performing CHWs and could be useful for evaluating CHW performance in other areas.Trial registrationIdentifier: NCT01942278 . Date of Registration: September 2013.

Project description:Monitoring and evaluating policies and programs in low- and middle-income countries are often difficult because of the lack of routine data. High mobile phone ownership in these countries presents an opportunity for efficient data collection through telephone interviews. This study examined the feasibility of collecting data on medicines through telephone interviews in Kenya. Data on the availability and prices of medicines at 137 health facilities and 639 patients were collected in September 2016 via in-person interviews. Between December 2016 and December 2017, monthly telephone interviews were conducted with health facilities and patients. An unannounced in-person interview was conducted with respondents to validate the telephone interview within 24 h. A bottom-up itemization costing approach was used to estimate the costs of telephone and in-person data collection. In-depth interviews were conducted with data collectors and respondents to explore their perceptions on both modes of data collection. The level of agreement between data on medicines availability collected through phone and in-person interviews was strong at the health facility level [kappa = 0.90; confidence interval (CI) 0.88-0.92] and moderate at the household level (kappa = 0.50, CI 0.39-0.60). Price data from telephone and in-person interviews showed strong intra-class correlation at health facilities [intra-class correlation coefficient (ICC) = 0.96] and moderate intra-class correlation at households (ICC = 0.47). The cost per phone interview at health facilities and households were $19.73 and $16.86, respectively, compared to $186.20 for a baseline in-person interview. Participants considered telephone interviews to be more convenient. In countries with high cell phone penetration, telephone data collection should be considered in monitoring and evaluating public health programs especially at health facilities. Additional strategies may be needed to optimize this mode of data collection at the household level. Variations in cell phone ownership, telecommunication network and data collection costs across different settings may limit the generalizability of the findings from this study.

Project description:Community health workers (CHWs) are essential to primary health care systems and are a cost-effective strategy to achieve the Sustainable Development Goals (SDGs). Nepal is strongly committed to universal health coverage and the SDGs. In 2017, the Nepal Ministry of Health and Population partnered with the nongovernmental organization Nyaya Health Nepal to pilot a program aligned with the 2018 World Health Organization guidelines for CHWs. The program includes CHWs who: (1) receive regular financial compensation; (2) meet a minimum education level; (3) are well supervised; (4) are continuously trained; (5) are integrated into local primary health care systems; (6) use mobile health tools; (7) have consistent supply chain; (8) live in the communities they serve; and (9) provide service without point-of-care user fees. The pilot model has previously demonstrated improved institutional birth rate, antenatal care completion, and postpartum contraception utilization. Here, we performed a retrospective costing analysis from July 16, 2017 to July 15, 2018, in a catchment area population of 60,000. The average per capita annual cost is US$3.05 (range: US$1.94 to US$4.70 across 24 villages) of which 74% is personnel cost. Service delivery and administrative costs and per beneficiary costs for all services are also described. To address the current discourse among Nepali policy makers at the local and federal levels, we also present 3 alternative implementation scenarios that policy makers may consider. Given the Government of Nepal's commitment to increase health care spending (US$51.00 per capita) to 7.0% of the 2030 gross domestic product, paired with recent health care systems decentralization leading to expanded fiscal space in municipalities, this CHW program provides a feasible opportunity to make progress toward achieving universal health coverage and the health-related SDGs. This costing analysis offers insights and practical considerations for policy makers and locally elected officials for deploying a CHW cadre as a mechanism to achieve the SDG targets.

Project description:BackgroundMaternal and neonatal health outcomes in low- and middle-income countries (LMICs) have improved over the last two decades. However, many pregnant women still deliver at home, which increases the health risks for both the mother and the child. Community health worker programs have been broadly employed in LMICs to connect women to antenatal care and delivery locations. More recently, employment of digital tools in maternal health programs have resulted in better care delivery and served as a routine mode of data collection. Despite the availability of rich, patient-level data within these digital tools, there has been limited utilization of this type of data to inform program delivery in LMICs.MethodsWe use program data from 38,787 women enrolled in Safer Deliveries, a community health worker program in Zanzibar, to build a generalizable prediction model that accurately predicts whether a newly enrolled pregnant woman will deliver in a health facility. We use information collected during the enrollment visit, including demographic data, health characteristics and current pregnancy information. We apply four machine learning methods: logistic regression, LASSO regularized logistic regression, random forest and an artificial neural network; and three sampling techniques to address the imbalanced data: undersampling of facility deliveries, oversampling of home deliveries and addition of synthetic home deliveries using SMOTE.ResultsOur models correctly predicted the delivery location for 68%-77% of the women in the test set, with slightly higher accuracy when predicting facility delivery versus home delivery. A random forest model with a balanced training set created using undersampling of existing facility deliveries accurately identified 74.4% of women delivering at home.ConclusionsThis model can provide a "real-time" prediction of the delivery location for new maternal health program enrollees and may enable early provision of extra support for individuals at risk of not delivering in a health facility, which has potential to improve health outcomes for both mothers and their newborns. The framework presented here is applicable in other contexts and the selection of input features can easily be adapted to match data availability and other outcomes, both within and beyond maternal health.

Project description:BackgroundThis is the first study to examine the costs of conducting a mobile phone survey (MPS) through interactive voice response (IVR) to collect information on risk factors for noncommunicable diseases (NCD) in three low- and middle-income countries (LMIC); Bangladesh, Colombia, and Uganda.MethodsThis is a micro-costing study conducted from the perspective of the payer/funder with a 1-year horizon. The study evaluates the fixed costs and variable costs of implementing one nationally representative MPS for NCD risk factors of the adult population. In this costing study, we estimated the sample size of calls required to achieve a population-representative survey and associated incentives. Cost inputs were obtained from direct economic costs incurred by a central study team, from country-specific collaborators, and from platform developers who participated in the deployment of these MPS during 2017. Costs were reported in US dollars (USD). A sensitivity analysis was conducted assessing different scenarios of pricing and incentive strategies. Also, costs were calculated for a survey deployed targeting only adults younger than 45 years.ResultsWe estimated the fixed costs ranging between $47,000 USD and $74,000 USD. Variable costs were found to be between $32,000 USD and $129,000 USD per nationally representative survey. The main cost driver was the number of calls required to meet the sample size, and its variability largely depends on the extent of mobile phone coverage and access in the country. Therefore, a larger number of calls were estimated to survey specific harder-to-reach sub-populations.ConclusionMobile phone surveys have the potential to be a relatively less expensive and timely method of collecting survey information than face-to-face surveys, allowing decision-makers to deploy survey-based monitoring or evaluation programs more frequently than it would be possible having only face-to-face contact. The main driver of variable costs is survey time, and most of the variability across countries is attributable to the sampling differences associated to reaching out to population subgroups with low mobile phone ownership or access.

Project description:INTRODUCTION:This study examined feasibility of a place-based community health worker (CHW) and health advocate (HA) initiative in five public housing developments selected for high chronic disease burden and described early outcomes. METHODS:This intervention was informed by a mixed-method needs assessment performed December 2014-January 2015 (representative telephone survey, n=1,663; six focus groups, n=55). Evaluation design was a non-randomized, controlled quasi-experiment. Intake and 3-month follow-up data were collected February-December 2015 (follow-up response rate, 93%) on 224 intervention and 176 comparison participants, and analyzed in 2016. All participants self-reported diagnoses of hypertension, diabetes, or asthma. The intervention consisted of chronic disease self-management and goal setting through six individual CHW-led health coaching sessions, instrumental support, and facilitated access to insurance/clinical care navigation from community-based HAs. Feasibility measures included CHW service satisfaction and successful goal setting. Preliminary outcomes included clinical measures (blood pressure, BMI); disease management behaviors and self-efficacy; and preventive behaviors (physical activity). RESULTS:At the 3-month follow-up, nearly all intervention participants reported high satisfaction with their CHW (90%) and HA (76%). Intervention participants showed significant improvements in self-reported physical activity (p=0.005) and, among hypertensive participants, self-reported routine blood pressure self-monitoring (p=0.013) compared with comparison participants. No improvements were observed in self-efficacy or clinical measures at the 3-month follow-up. CONCLUSIONS:Housing-based initiatives involving CHW and HA teams are acceptable to public housing residents and can be effectively implemented to achieve rapid improvements in physical activity and chronic disease self-management. At 3-month assessment, additional time and efforts are required to improve clinical outcomes.