How well are aid agencies evaluating programs? An assessment of the quality of global health evaluations.
Ontology highlight
ABSTRACT:
SUBMITTER: Raifman J
PROVIDER: S-EPMC6161748 | biostudies-literature | 2018
REPOSITORIES: biostudies-literature
Publications
How well are aid agencies evaluating programs? An assessment of the quality of global health evaluations.
Raifman Julia J Lam Felix F Keller Janeen Madan JM Radunsky Alexander A Savedoff William W
Journal of development effectiveness 20180321 2
Evaluations are key to learning and accountability. We assessed the methodological quality of 37 randomly selected programme evaluations from 5 major global health funders. Two researchers rated each evaluation for relevance, validity, and reliability and met to resolve discrepancies. Most evaluations asked questions relevant to the health programme, but less than 40 per cent of impact evaluations and less than 10 per cent of performance evaluations used relevant data, followed accepted social s ...[more]
Similar Datasets
Project description:BackgroundThe number of Health Technology Assessment (HTA) agencies increases. One component of HTAs are economic aspects. To incorporate economic aspects commonly economic evaluations are performed. A convergence of recommendations for methods of health economic evaluations between international HTA agencies would facilitate the adaption of results to different settings and avoid unnecessary expense. A first step in this direction is a detailed analysis of existing similarities and differences in recommendations to identify potential for harmonization. The objective is to provide an overview and comparison of the methodological recommendations of international HTA agencies for economic evaluations.MethodsThe webpages of 127 international HTA agencies were searched for guidelines containing recommendations on methods for the preparation of economic evaluations. Additionally, the HTA agencies were requested information on methods for economic evaluations. Recommendations of the included guidelines were extracted in standardized tables according to 13 methodological aspects. All process steps were performed independently by two reviewers.ResultsFinally 25 publications of 14 HTA agencies were included in the analysis. Methods for economic evaluations vary widely. The greatest accordance could be found for the type of analysis and comparator. Cost-utility-analyses or cost-effectiveness-analyses are recommended. The comparator should continuously be usual care. Again the greatest differences were shown in the recommendations on the measurement/sources of effects, discounting and in the analysis of sensitivity. The main difference regarding effects is the focus either on efficacy or effectiveness. Recommended discounting rates range from 1.5%-5% for effects and 3%-5% for costs whereby it is mostly recommended to use the same rate for costs and effects. With respect to the analysis of sensitivity the main difference is that oftentimes the probabilistic or deterministic approach is recommended exclusively. Methods for modeling are only described vaguely and mainly with the rational that the "appropriate model" depends on the decision problem. Considering all other aspects a comparison is challenging as recommendations vary regarding detailedness and addressed issues.ConclusionThere is a considerable unexplainable variance in recommendations. Further effort is needed to harmonize methods for preparing economic evaluations.
| S-EPMC3849629 | biostudies-literature
Project description:QuestionThe aim was to systematically collate and synthesise existing, publicly available patient-reported outcome measure (PROM) information suitable for quality of life (QOL)/well-being measurement in mental health economic evaluations, with specific focus on their applicability in multisectoral, multinational, multiperson economic evaluations and to develop an electronic PROM compendium with meta-data.Study selection and analysisA systematic literature search for non-disease-pecific PROMs and their versions suitable for the measurement of QOL/well-being or recovery was conducted from 2008 to February 2020. Six criteria were applied to judge their suitability in multisectoral, multinational, multiperson economic evaluations: (i) availability of separate adult and child/adolescent versions, (ii) availability of a proxy-completion option, (iii) assessing outcomes beyond health, (iv) availability of translations (≥2 language versions), (v) availability of a preference-based valuation, (vi) availability of value sets in more than one country.FindingsThe final ProgrammE in Costing, resource use measurement and outcome valuation for Use in multisectoral National and International health economic evaluAtions (PECUNIA) PROM-MH Compendium includes 204 unique scales, out of which 88 are individual instruments, while the remaining 116 scales belong to 46 PROM families with more than one distinctive version. Out of the total 134 individual PROMs/PROM families, 72% have at least two language versions, 8% measure broader well-being beyond health-related QOL, 11% have preference-based valuation, with multiple country sets available for 60% of these. None of the identified PROMs met all six proposed criteria.ConclusionsThe PECUNIA PROM-MH Compendium provides a unique overview of the relevant PROMs and their linked meta-data, and should be a helpful tool when choosing a suitable instrument for future mental health economic evaluations.
| S-EPMC9046758 | biostudies-literature
Project description:Increasingly, public and private resources are being dedicated to community-based health improvement programs. But evaluations of these programs typically rely on data about process and a pre-post study design without a comparison community. To better determine the association between the implementation of community-based health improvement programs and county-level health outcomes, we used publicly available data for the period 2002-06 to create a propensity-weighted set of controls for conducting multiple regression analyses. We found that the implementation of community-based health improvement programs was associated with a decrease of less than 0.15 percent in the rate of obesity, an even smaller decrease in the proportion of people reporting being in poor or fair health, and a smaller increase in the rate of smoking. None of these changes was significant. Additionally, program counties tended to have younger residents and higher rates of poverty and unemployment than nonprogram counties. These differences could be driving forces behind program implementation. To better evaluate health improvement programs, funders should provide guidance and expertise in measurement, data collection, and analytic strategies at the beginning of program implementation.
| S-EPMC5845775 | biostudies-literature
Project description:BackgroundTemperature excursions occur during the transport and storage of pharmaceuticals, and often result in considerable losses for public health programs operating in countries with limited resources. After a temperature excursion has been identified, often products are discarded without any additional risk assessments. Consulting the manufacturer is the preferred approach but can be challenging depending on the responsiveness of the manufacturer. However, decisions are often required quickly depending on program needs and available stock in country.MethodsTo provide further guidance, simulations have been conducted based on mean kinetic temperature evaluations using accepted default kinetic parameters to assess loss of shelf-life for scenarios involving various levels of temperature excursions on a model pharmaceutical at different recommended storage conditions, shelf-life, and long-term storage conditions.ResultsAlthough an immediate loss to shelf-life occurred with excursions when the product was stored at the maximum allowed temperature, more extended excursion could be withstood before loss of shelf-life was detected when long-term storage was maintained at temperatures below the maximum storage condition for the product. With the assumption that a shelf-life loss of 2 weeks was negligible when managing program stock, a risk assessment was conducted to outline the various times that excursions at different temperatures could be considered low risk to the program.ConclusionsDepending on the level of the temperature excursion and the guidance provided by the manufacturer, public health programs will have further information with this assessment to guide decisions that impact safety to the end user and resource management due to temperature excursions that can occur.
| S-EPMC8842539 | biostudies-literature
Project description:Provision of reproductive health (RH) services is a minimum standard of health care in humanitarian settings; however access to these services is often limited. This systematic review, one component of a global evaluation of RH in humanitarian settings, sought to explore the evidence regarding RH services provided in humanitarian settings and to determine if programs are being evaluated. In addition, the review explored which RH services receive more attention based on program evaluations and descriptive data. Peer-reviewed papers published between 2004 and 2013 were identified via the Ovid MEDLINE database, followed by a PubMed search. Papers on quantitative evaluations of RH programs, including experimental and non-experimental designs that reported outcome data, implemented in conflict and natural disaster settings, were included. Of 5,669 papers identified in the initial search, 36 papers describing 30 programs met inclusion criteria. Twenty-five papers described programs in sub-Saharan Africa, six in Asia, two in Haiti and three reported data from multiple countries. Some RH technical areas were better represented than others: seven papers reported on maternal and newborn health (including two that also covered family planning), six on family planning, three on sexual violence, 20 on HIV and other sexually transmitted infections and two on general RH topics. In comparison to the program evaluation papers identified, three times as many papers were found that reported RH descriptive or prevalence data in humanitarian settings. While data demonstrating the magnitude of the problem are crucial and were previously lacking, the need for RH services and for evaluations to measure their effectiveness is clear. Program evaluation and implementation science should be incorporated into more programs to determine the best ways to serve the RH needs of people affected by conflict or natural disaster. Standard program design should include rigorous program evaluation, and the results must be shared. The papers demonstrated both that RH programs can be implemented in these challenging settings, and that women and men will use RH services when they are of reasonable quality.
| S-EPMC4328944 | biostudies-literature
Project description: Not available
| S-EPMC11342894 | biostudies-literature
Project description:Conservation actions such as habitat protection, restoration, and translocations are critical actions in preventing further extinctions of threatened species. We used the 152 threatened species on the International Union for the Conservation of Nature's Red List with conservation translocations as a recommended conservation action to access the habitat quality of these species' ranges. We determined where multi-species conservation translocation and forest restoration efforts can be concentrated. To determine the habitat quality of species' ranges, we assessed forest cover, forest restoration potential, protected area status, and invasive species concerns. Forty-four percent (67 species) of species with translocations recommended have part of their range in a protected area, existing forest cover, and currently no invasive species risk. However, the majority (85 species) currently need habitat management (63 species), invasive species control (71 species), or protection (34 species). We also identified key differences between species recommended for reintroductions (115 species) and benign introductions (37 species), such as the percentage of a species' range within a protected area, in which reintroductions (median = 7.4%) had more than benign introductions (median = 0.9%). Mauritius, central Africa, eastern Australia and Himalaya regions each have areas with range overlap of three or more species recommended for translocations and forest restoration potential. For those species with CT programs in place, mean forest cover was 32% and restoration potential was 16%, suggesting potential minimum habitat requirements for initial releases. Results provide a global perspective on reintroduction and translocation needs of threatened species with evidenced-based information on habitat quality, i.e. forest restoration potential, forest cover, protected areas, and invasive species control, to aid conservation translocation scientists and ultimately improve the success of such projects.
| S-EPMC8046235 | biostudies-literature
Project description:BackgroundHealth technology assessment (HTA) organizations play a crucial role in optimizing healthcare resources, but the factors influencing decision making vary by country.ObjectiveHTAs of cancer and hepatitis C drugs were evaluated across developed countries to understand differences in decision processes and criteria.MethodsThe HTA organizations evaluated are from France, Germany, Italy, Spain, the United Kingdom (UK), Australia, Canada and Japan. Economic evaluation types and 28 factors in the following categories were evaluated: clinical uncertainties/issues; disease/population/treatment consideration factors including National Institute for Health and Care Excellence's (NICE) special circumstances factors (e.g. end-of-life and innovation); and International Society for Pharmacoeconomics and Outcomes Research (ISPOR) additional value elements. Qualitative and correspondence analyses were conducted to assess the differences across organizations.ResultsIncremental cost-effectiveness ratio (ICER) using quality-adjusted life-year (QALY) was evaluated in Canada, the UK, Australia and Japan. The highest observed clinical uncertainties were clinical benefits and comparator. For cancer drugs, correspondence analysis showed France, Australia, Canada and the UK to have common attributes observed, such as unmet needs and stakeholder persuasion. In addition, the UK reported end-of-life, issues around current treatment and innovation, whereas Germany reported manageable/insignificant adverse events more frequently. Finally, fear of contagion, equity and scientific spillover value elements were only observed in Australia.ConclusionAlthough clinical factors play a predominant role in the decision to reimburse medicine, HTA organizations consider additional aspects as well. If the methodology of HTA was clearly outlined, there would be more transparency in HTA systems leading to better understanding amongst stakeholders about decision making.
| S-EPMC7895890 | biostudies-literature
Project description:ObjectivesThe aim of this study was to review and compare current health technology assessment (HTA) activities for medical devices across non-European Union HTA agencies.MethodsHTA activities for medical devices were evaluated from three perspectives: organizational structure, processes, and methods. Agencies were primarily selected upon membership of existing HTA networks. The data collection was performed in two stages: stage 1-agency Web-site assessment using a standardized questionnaire, followed by review and validation of the collected data by a representative of the agency; and stage 2-semi-structured telephone interviews with key informants of a sub-sample of agencies.ResultsIn total, thirty-six HTA agencies across twenty non-EU countries assessing medical devices were included. Twenty-seven of thirty-six (75 percent) agencies were judged at stage 1 to have adopted HTA-specific approaches for medical devices (MD-specific agencies) that were largely organizational or procedural. There appeared to be few differences in the organization, process and methods between MD-specific and non-MD-specific agencies. Although the majority (69 percent) of both categories of agency had specific methods guidance or policy for evidence submission, only one MD-specific agency had developed methodological guidelines specific to medical devices. In stage 2, many MD-specific agencies cited insufficient resources (budget, skilled employees), lack of coordination (between regulator and reimbursement bodies), and the inability to generalize findings from evidence synthesis to be key challenges in the HTA of medical devices.ConclusionsThe lack of evidence for differentiation in scientific methods for HTA of devices raises the question of whether HTA needs to develop new methods for medical devices but rather adapt existing methodological approaches. In contrast, organizational and/or procedural adaptation of existing HTA agency frameworks to accommodate medical devices appear relatively commonplace.
| S-EPMC4535322 | biostudies-literature
Project description:BackgroundThe Government of Indonesia implemented health technology assessment (HTA) to ensure quality and cost control in the National Health Insurance Program (Jaminan Kesehatan Nasional/JKN). The current aim of the study was to improve the usefulness of future economic evaluation for resource allocation by appraising current methodology, reporting, and source of evidence quality of studies.MethodsThe inclusion and exclusion criteria were applied to search for relevant studies using a systematic review. The methodology and reporting adherence were appraised according to Indonesia's HTA Guideline issued in 2017. The differences in adherence before and after the guideline dissemination were compared using Chi-square and Fisher's exact tests for methodology adherence wherever appropriate, and the Mann-Whitney test for reporting adherence. The source of evidence quality was assessed using evidence hierarchy. Two scenarios of the study start date and the guideline dissemination period were tested using sensitivity analyses.FindingsEighty-four studies were obtained from PubMed, Embase, Ovid, and two local journals. Only two articles cited the guideline. No statistically significant difference (P > 0.05) was found between the pre-dissemination and post-dissemination period with respect to methodology adherence, except for outcome choice. Studies during the post-dissemination period showed a higher score for reporting which was statistically significant (P = 0.01). However, the sensitivity analyses revealed no statistically significant difference (P > 0.05) in methodology (except for modelling type, P = 0.03) and reporting adherence between the two periods.InterpretationThe guideline did not impact the methodology and reporting standard used in the included studies. Recommendations were provided to improve the usefulness of economic evaluations for Indonesia.FundingThe Access and Delivery Partnership (ADP) hosted by the United Nations Development Programme (UNDP) and the Health Systems Research Institute (HSRI).
| S-EPMC10306002 | biostudies-literature