A comparison of methods for health policy evaluation with controlled pre-post designs.
Ontology highlight
ABSTRACT:
SUBMITTER: O'Neill S
PROVIDER: S-EPMC7080394 | biostudies-literature | 2020 Apr
REPOSITORIES: biostudies-literature
Publications
A comparison of methods for health policy evaluation with controlled pre-post designs.
O'Neill Stephen S Kreif Noemi N Sutton Matt M Grieve Richard R
Health services research 20200212 2
<h4>Objective</h4>To compare interactive fixed effects (IFE) and generalized synthetic control (GSC) methods to methods prevalent in health policy evaluation and re-evaluate the impact of the hip fracture best practice tariffs introduced for hospitals in England in 2010.<h4>Data sources</h4>Simulations and Hospital Episode Statistics.<h4>Study design</h4>Best practice tariffs aimed to incentivize providers to deliver care in line with guidelines. Under the scheme, 62 providers received an additi ...[more]
Similar Datasets
Project description:We discuss tradeoffs and errors associated with approaches to modeling health economic decisions. Through an application in pharmacogenomic (PGx) testing to guide drug selection for individuals with a genetic variant, we assessed model accuracy, optimal decisions, and computation time for an identical decision scenario modeled 4 ways: using 1) coupled-time differential equations (DEQ), 2) a cohort-based discrete-time state transition model (MARKOV), 3) an individual discrete-time state transition microsimulation model (MICROSIM), and 4) discrete event simulation (DES). Relative to DEQ, the net monetary benefit for PGx testing (v. a reference strategy of no testing) based on MARKOV with rate-to-probability conversions using commonly used formulas resulted in different optimal decisions. MARKOV was nearly identical to DEQ when transition probabilities were embedded using a transition intensity matrix. Among stochastic models, DES model outputs converged to DEQ with substantially fewer simulated patients (1 million) v. MICROSIM (1 billion). Overall, properly embedded Markov models provided the most favorable mix of accuracy and runtime but introduced additional complexity for calculating cost and quality-adjusted life year outcomes due to the inclusion of "jumpover" states after proper embedding of transition probabilities. Among stochastic models, DES offered the most favorable mix of accuracy, reliability, and speed.
| S-EPMC9181506 | biostudies-literature
Project description:Often repeated measures data are summarized into pre-post-treatment measurements. Various methods exist in the literature for estimating and testing treatment effect, including ANOVA, analysis of covariance (ANCOVA), and linear mixed modeling (LMM). Under the first two methods, outcomes can either be modeled as the post treatment measurement (ANOVA-POST or ANCOVA-POST), or a change score between pre and post measurements (ANOVA-CHANGE, ANCOVA-CHANGE). In LMM, the outcome is modeled as a vector of responses with or without Kenward-Rogers adjustment. We consider five methods common in the literature, and discuss them in terms of supporting simulations and theoretical derivations of variance. Consistent with existing literature, our results demonstrate that each method leads to unbiased treatment effect estimates, and based on precision of estimates, 95% coverage probability, and power, ANCOVA modeling of either change scores or post-treatment score as the outcome, prove to be the most effective. We further demonstrate each method in terms of a real data example to exemplify comparisons in real clinical context.
| S-EPMC6290914 | biostudies-literature
Project description:BackgroundRandomized pre-post designs, with outcomes measured at baseline and after treatment, have been commonly used to compare the clinical effectiveness of two competing treatments. There are vast, but often conflicting, amount of information in current literature about the best analytic methods for pre-post designs. It is challenging for applied researchers to make an informed choice.MethodsWe discuss six methods commonly used in literature: one way analysis of variance ("ANOVA"), analysis of covariance main effect and interaction models on the post-treatment score ("ANCOVAI" and "ANCOVAII"), ANOVA on the change score between the baseline and post-treatment scores ("ANOVA-Change"), repeated measures ("RM") and constrained repeated measures ("cRM") models on the baseline and post-treatment scores as joint outcomes. We review a number of study endpoints in randomized pre-post designs and identify the mean difference in the post-treatment score as the common treatment effect that all six methods target. We delineate the underlying differences and connections between these competing methods in homogeneous and heterogeneous study populations.ResultsANCOVA and cRM outperform other alternative methods because their treatment effect estimators have the smallest variances. cRM has comparable performance to ANCOVAI in the homogeneous scenario and to ANCOVAII in the heterogeneous scenario. In spite of that, ANCOVA has several advantages over cRM: i) the baseline score is adjusted as covariate because it is not an outcome by definition; ii) it is very convenient to incorporate other baseline variables and easy to handle complex heteroscedasticity patterns in a linear regression framework.ConclusionsANCOVA is a simple and the most efficient approach for analyzing pre-post randomized designs.
| S-EPMC8305561 | biostudies-literature
Project description:BackgroundTransdisciplinary approaches can streamline processes and build workforce capacity by blurring traditional responsibilities and integrating aspects of care. Emerging evidence shows transdisciplinary approaches can improve time-efficiency, quality of care and cost-effectiveness across various healthcare settings, however no empirical study is based on an acute stroke unit.MethodsThe SPIRIT checklist was used to guide the content of the research protocol. The study is a pragmatic pre-/post- mixed methods four-phase study with a 3-month follow up, based at the Mater Hospital Brisbane. Participants experiencing stroke symptoms will be recruited as they are admitted to the acute stroke unit. Patients presenting with mild stroke symptoms or Transient Ischaemic Attack will be allocated to Phase 1 (baseline) or Phase 2 (implementation), while patients presenting with moderate to severe stroke symptoms will be allocated to Phase 3 (baseline) or Phase 4 (implementation). Participants in baseline Phases 1 and 3 will receive standard allied health assessment, while participants in implementation Phases 2 and 4 will receive the novel transdisciplinary assessment. For the primary aim, allied health professionals will time their assessments to evaluate time taken to administer a novel transdisciplinary assessment, compared to usual discipline-specific assessments. Non-inferiority of the novel transdisciplinary assessment will also be explored in terms of patient safety, compliance to national standards, use of the assessment, and stakeholder perceptions. A retrospective medical record audit, staff focus group, patient/staff surveys, and patient phone interviews at 3-months will be completed. Quantitative results will be estimated using general linear and logistic regression models in Stata 15.1. Qualitative results will be analysed using frequency counts and NVivo software. An economic evaluation will be performed using three scopes including the allied health assessment, hospital admission, and patient outcomes at 3-months.DiscussionWhen designing the study, pragmatic factors related to staff willingness to be involved, patient safety, and existing clinical pathways/processes were considered. To address those factors, a co-design approach was taken, resulting in staff buy-in, clinically relevant outcome measures, and the pre-/post- four-phase study design.Trial registrationAustralian New Zealand Clinical Trials Registry (ANZCTR), ACTRN12621000380897. Registered 06 April 2021 - retrospectively registered, https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=381339&isReview=true.
| S-EPMC9789550 | biostudies-literature
Project description:BackgroundEpidemiological evidence on the social determinants of health inequity is well-advanced, but considerably less attention has been given to evaluating the impact of public policies addressing those social determinants. Methodological challenges to produce evidence on policy outcomes present a significant barrier to mobilising policy actions for health equities. This review aims to examine methodological approaches to policy evaluation of health equity outcomes and identify promising approaches for future research.MethodsWe conducted a systematic narrative review of literature critically evaluating policy impact on health equity, synthesizing information on the methodological approaches used. We searched and screened records from five electronic databases, using pre-defined protocols resulting in a total of 50 studies included for review. We coded the studies according to (1) type of policy analysed; (2) research design; (3) analytical techniques; (4) health outcomes; and (5) equity dimensions evaluated.ResultsWe found a growing number of a wide range of policies being evaluated for health equity outcomes using a variety of research designs. The majority of studies employed an observational research design, most of which were cross-sectional, however, other approaches included experimental designs, simulation modelling, and meta-analysis. Regression techniques dominated the analytical approaches, although a number of novel techniques were used which may offer advantages over traditional regression analysis for the study of distributional impacts of policy. Few studies made intra-national or cross-national comparisons or collected primary data. Despite longstanding challenges of attribution in policy outcome evaluation, the majority of the studies attributed change in physical or mental health outcomes to the policy being evaluated.ConclusionOur review provides an overview of methodological approaches to health equity policy outcome evaluation, demonstrating what is most commonplace and opportunities from novel approaches. We found the number of studies evaluating the impacts of public policies on health equity are on the rise, but this area of policy evaluation still requires more attention given growing inequities.
| S-EPMC6037500 | biostudies-literature
Project description:Existing literature comparing statistical properties of nested case-control and case-cohort methods have become insufficient for present day epidemiologists. The literature has not reconciled conflicting conclusions about the standard methods. Moreover, a comparison including newly developed methods, such as inverse probability weighting methods, is needed. Two analytical methods for nested case-control studies and six methods for case-cohort studies using proportional hazards regression model were summarized and their statistical properties were compared. The answer to which design and method is more powerful was more nuanced than what was previously reported. For both nested case-control and case-cohort designs, inverse probability weighting methods were more powerful than the standard methods. However, the difference became negligible when the proportion of failure events was very low (<1%) in the full cohort. The comparison between two designs depended on the censoring types and incidence proportion: with random censoring, nested case-control designs coupled with the inverse probability weighting method yielded the highest statistical power among all methods for both designs. With fixed censoring times, there was little difference in efficiency between two designs when inverse probability weighting methods were used; however, the standard case-cohort methods were more powerful than the conditional logistic method for nested case-control designs. As the proportion of failure events in the full cohort became smaller (<10%), nested case-control methods outperformed all case-cohort methods and the choice of analytic methods within each design became less important. When the predictor of interest was binary, the standard case-cohort methods were often more powerful than the conditional logistic method for nested case-control designs.
| S-EPMC4374014 | biostudies-literature
Project description:BackgroundInterprofessional practice (IPP) has been shown to contribute to quality service provision and improved health outcomes. This knowledge has led to the integration of interprofessional education into course curricula for many health-care disciplines. Offering interprofessional education in rural areas to students undertaking work integrated learning placements is challenging particularly because of the diversity of students and placement dates combined with the student focus on the assessable placement curriculum. This research investigated and evaluated the utility of an escape room as an educational modality that facilitates learning whilst providing a supportive and motivating learning environment. Our project focused on the acquisition of interprofessional practice knowledge and experience by a health professional student cohort.MethodsThis study used the novel intervention of an escape room combined with an interactive teaching session to test student engagement and learning about interprofessional practice and teamwork. The research used a mixed methods single group pre-post design.ResultsFifty students (78% female) from seventeen universities and seven professions participated in teams of three to six members. Most participants (66%) had not previously completed an escape room. The results showed that the intervention provided effective and engaging learning and was intrinsically appealing to students despite its non-assessable nature. Individual student reflection on their participation showed developing insight into the critical importance of clear communication and intentional team member collaboration in the provision of effective interprofessional practice.ConclusionsThe escape room intervention added value to the placement curriculum and proved flexible for a heterogeneous student cohort.
| S-EPMC8056636 | biostudies-literature
Project description:BackgroundCurrently, thanks to the growing number of public database resources, most evidence on planning and management, healthcare institutions, policies and practices is becoming available to everyone. However, one of the limitations for the advancement of data and literature-driven research has been the lack of flexibility of the methodological resources used in qualitative research. There is a need to incorporate friendly, cheaper and faster tools for the systematic, unbiased analysis of large data corpora, in particular regarding the qualitative aspects of the information (often overlooked).MethodsThis article proposes a series of novel techniques, exemplified by the case of the role of Institutional Committees of Bioethics to (1) massively identify the documents relevant to a given issue, (2) extract the fundamental content, focusing on qualitative analysis, (3) synthesize the findings in the published literature, (4) categorize and visualize the evidence, and (5) analyse and report the results.ResultsA critical study of the institutional role of public health policies and practices in Institutional Committees of Bioethics was used as an example application of the method. Interactive strategies were helpful to define and conceptualise variables, propose research questions and refine research interpretation. These methods are additional aids to systematic reviews, pre-coding schemes and construction of a priori diagrams to survey and analyse social science literature.ConclusionsThese novel methods have proven to facilitate the formulation and testing of hypotheses on the subjects to be studied. Such tools may allow important advances going from descriptive approaches to decision-making and even institutional assessment and policy redesign, by pragmatic reason of time and costs.
| S-EPMC6332644 | biostudies-literature
Project description:Mathematical simulation models are commonly used to inform health policy decisions. These health policy models represent the social and biological mechanisms that determine health and economic outcomes, combine multiple sources of evidence about how policy alternatives will impact those outcomes, and synthesize outcomes into summary measures salient for the policy decision. Calibrating these health policy models to fit empirical data can provide face validity and improve the quality of model predictions. Bayesian methods provide powerful tools for model calibration. These methods summarize information relevant to a particular policy decision into (1) prior distributions for model parameters, (2) structural assumptions of the model, and (3) a likelihood function created from the calibration data, combining these different sources of evidence via Bayes' theorem. This article provides a tutorial on Bayesian approaches for model calibration, describing the theoretical basis for Bayesian calibration approaches as well as pragmatic considerations that arise in the tasks of creating calibration targets, estimating the posterior distribution, and obtaining results to inform the policy decision. These considerations, as well as the specific steps for implementing the calibration, are described in the context of an extended worked example about the policy choice to provide (or not provide) treatment for a hypothetical infectious disease. Given the many simplifications and subjective decisions required to create prior distributions, model structure, and likelihood, calibration should be considered an exercise in creating a reasonable model that produces valid evidence for policy, rather than as a technique for identifying a unique theoretically optimal summary of the evidence.
| S-EPMC5448142 | biostudies-literature
Project description:IntroductionThe impact of policies on COVID-19 outcomes is one of the most important questions of our time. Unfortunately, there are substantial concerns about the strength and quality of the literature examining policy impacts. This study systematically assessed the currently published COVID-19 policy impact literature for a checklist of study design elements and methodological issues.MethodsWe included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26 or earlier and screening, all studies were reviewed by three reviewers independently and in consensus. The review tool was based on review guidance for assessing COVID-19 health policy impact evaluation analyses, including first identifying the assumptions behind the methods used, followed by assessing graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating.ResultsAfter 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside due to inappropriate study design (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, we found that only four (or by a stricter standard, only one) of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.DiscussionThe current literature directly evaluating the impact of COVID-19 policies largely fails to meet key design criteria for useful inference. This may be partially due to the circumstances for evaluation being particularly difficult, as well as a context with desire for rapid publication, the importance of the topic, and weak peer review processes. Importantly, weak evidence is non-informative and does not indicate how effective these policies were on COVID-19 outcomes.
| S-EPMC7836129 | biostudies-literature