The Role of Expert Judgment in Statistical Inference and Evidence-Based Decision-Making.
Ontology highlight
ABSTRACT:
SUBMITTER: Brownstein NC
PROVIDER: S-EPMC6474725 | biostudies-literature | 2019
REPOSITORIES: biostudies-literature
Publications
The Role of Expert Judgment in Statistical Inference and Evidence-Based Decision-Making.
Brownstein Naomi C NC Louis Thomas A TA O'Hagan Anthony A Pendergast Jane J
The American statistician 20190320 1
This article resulted from our participation in the session on the "role of expert opinion and judgment in statistical inference" at the October 2017 ASA Symposium on Statistical Inference. We present a strong, unified statement on roles of expert judgment in statistics with processes for obtaining input, whether from a Bayesian or frequentist perspective. Topics include the role of subjectivity in the cycle of scientific inference and decisions, followed by a clinical trial and a greenhouse gas ...[more]
Similar Datasets
Project description:Online decision-making problem requires us to make a sequence of decisions based on incremental information. Common solutions often need to learn a reward model of different actions given the contextual information and then maximize the long-term reward. It is meaningful to know if the posited model is reasonable and how the model performs in the asymptotic sense. We study this problem under the setup of the contextual bandit framework with a linear reward model. The ε-greedy policy is adopted to address the classic exploration-and-exploitation dilemma. Using the martingale central limit theorem, we show that the online ordinary least squares estimator of model parameters is asymptotically normal. When the linear model is misspecified, we propose the online weighted least squares estimator using the inverse propensity score weighting and also establish its asymptotic normality. Based on the properties of the parameter estimators, we further show that the in-sample inverse propensity weighted value estimator is asymptotically normal. We illustrate our results using simulations and an application to a news article recommendation dataset from Yahoo!.
| S-EPMC7962379 | biostudies-literature
Project description:People often use expert advice when making decisions in our society, but how we are influenced by this advice has yet to be understood. To address this, using functional magnetic resonance imaging, we provided expert and novice advice to participants during an estimation task. Participants reported that they valued expert advice more than novice advice, and activity in the ventral striatum correlated with this valuation, even before decisions with the advice were made. When using advice, participants compared their initial opinion to their advisor's opinion. This comparison, termed the "opinion difference", influenced advice utilization and was represented in reward-sensitive brain regions. Finally, the left lateral orbitofrontal cortex integrated both the size of the opinion difference and the advisor's level of expertise, and average activity in this area correlated with mean advice utilization across participants. Taken together, these findings provide neural evidence for how advice engenders behavioral change during the decision-making process.
| S-EPMC3504100 | biostudies-literature
Project description:Inferring parameters of computational models that capture experimental data are a central task in cognitive neuroscience. Bayesian statistical inference methods usually require the ability to evaluate the likelihood of the model-however, for many models of interest in cognitive neuroscience, the associated likelihoods cannot be computed efficiently. Simulation-based inference (SBI) offers a solution to this problem by only requiring access to simulations produced by the model. Previously, Fengler et al. introduced likelihood approximation networks (LANs, Fengler et al., 2021) which make it possible to apply SBI to models of decision-making, but require billions of simulations for training. Here, we provide a new SBI method that is substantially more simulation efficient. Our approach, mixed neural likelihood estimation (MNLE), trains neural density estimators on model simulations to emulate the simulator, and is designed to capture both the continuous (e.g., reaction times) and discrete (choices) data of decision-making models. The likelihoods of the emulator can then be used to perform Bayesian parameter inference on experimental data using standard approximate inference methods like Markov Chain Monte Carlo sampling. We demonstrate MNLE on two variants of the drift-diffusion model and show that it is substantially more efficient than LANs: MNLE achieves similar likelihood accuracy with six orders of magnitude fewer training simulations, and is significantly more accurate than LANs when both are trained with the same budget. Our approach enables researchers to perform SBI on custom-tailored models of decision-making, leading to fast iteration of model design for scientific discovery.
| S-EPMC9374439 | biostudies-literature
Project description:Evidence-based practice is the foundation of rehabilitation for maximizing client outcomes. However, an unacceptably high number of ineffective or outdated interventions are still implemented, leading to sub-optimal outcomes for clients. This paper proposes the Rehabilitation Evidence bAsed Decision-Making (READ) Model, a decision-making algorithm for evidence-based decision-making in rehabilitation settings. The READ Model outlines a step-by-step layered process for healthcare professionals to collaboratively set goals, and to select appropriate interventions. The READ Model acknowledges the important multi-layered contributions of client's preferences and values, family supports available, and external environmental factors such as funding, availability of services and access. Healthcare professionals can apply the READ Model to choose interventions that are evidence-based, with an appropriate mode, dose, and with regular review, in order to achieve client's goals. Two case studies are used to demonstrate application of the READ Model: cerebral palsy and autism spectrum disorder. The READ Model applies the four central principles of evidence-based practice and can be applied across multiple rehabilitation settings.
| S-EPMC9397823 | biostudies-literature
Project description:The increase of multidrug resistance and resistance to last-line antibiotics is a major global public health threat. Although surveillance programs provide useful current and historical information on the scale of the problem, the future emergence and spread of antibiotic resistance is uncertain, and quantifying this uncertainty is crucial for guiding decisions about investment in antibiotics and resistance control strategies. Mathematical and statistical models capable of projecting future rates are challenged by the paucity of data and the complexity of the emergence and spread of resistance, but experts have relevant knowledge. We use the Classical Model of structured expert judgment to elicit projections with uncertainty bounds of resistance rates through 2026 for nine pathogen-antibiotic pairs in four European countries and empirically validate the assessments against data on a set of calibration questions. The performance-weighted combination of experts in France, Spain, and the United Kingdom projected that resistance for five pairs on the World Health Organization's priority pathogens list (E. coli and K. pneumoniae resistant to third-generation cephalosporins and carbapenems and MRSA) would remain below 50% in 2026. In Italy, although upper bounds of 90% credible ranges exceed 50% resistance for some pairs, the medians suggest Italy will sustain or improve its current rates. We compare these expert projections to statistical forecasts based on historical data from the European Antimicrobial Resistance Surveillance Network (EARS-Net). Results from the statistical models differ from each other and from the judgmental forecasts in many cases. The judgmental forecasts include information from the experts about the impact of current and future shifts in infection control, antibiotic usage, and other factors that cannot be easily captured in statistical forecasts, demonstrating the potential of structured expert judgment as a tool for better understanding the uncertainty about future antibiotic resistance.
| S-EPMC6611586 | biostudies-literature
Project description: Not available
| S-EPMC8820069 | biostudies-literature
Project description:OBJECTIVES:The South Korean government required the submission of economic evidence when it implemented the Positive-List System in December 2006. This study investigates the key factors that influenced actual public insurance reimbursement decisions, including the role of economic evidence, after 10 years of decision practice under compulsory health technology assessment (HTA) for new drugs. METHOD:Logistic regression analysis was used to estimate the impact of the variables involved, including cost-effectiveness ratio as a key variable, on reimbursement decisions. The latter were defined as "yes" or "no" at a submitted price and indication. Only cases (n = 91) that present a cost-effectiveness ratio, and that have been reviewed based on this ratio from January 2007 to December 2016, were included in the analysis. RESULTS:Cases with higher cost-effectiveness ratios were less likely to be accepted. In addition, drugs that were used to treat severe diseases and drugs with no substitute were more likely to be recommended. The probability of acceptance declined along with the level of uncertainty in the submitted evidence. The acceptance rate for severe-disease drugs has increased since 2013, when the government introduced several policies that lowered the existing barriers to positive reimbursement. However, such an increase was not statistically significant. CONCLUSIONS:Cost-effectiveness is one of the most influential factors in drug-reimbursement decisions. However, inclusion of other explanatory variables, in addition to the cost-effectiveness ratio, predicted the results of decisions more accurately.
| S-EPMC6200251 | biostudies-other
Project description:In many everyday situations, humans must make precise decisions in the presence of uncertain sensory information. For example, when asked to combine information from multiple sources we often assign greater weight to the more reliable information. It has been proposed that statistical-optimality often observed in human perception and decision-making requires that humans have access to the uncertainty of both their senses and their decisions. However, the mechanisms underlying the processes of uncertainty estimation remain largely unexplored. In this paper we introduce a novel visual tracking experiment that requires subjects to continuously report their evolving perception of the mean and uncertainty of noisy visual cues over time. We show that subjects accumulate sensory information over the course of a trial to form a continuous estimate of the mean, hindered only by natural kinematic constraints (sensorimotor latency etc.). Furthermore, subjects have access to a measure of their continuous objective uncertainty, rapidly acquired from sensory information available within a trial, but limited by natural kinematic constraints and a conservative margin for error. Our results provide the first direct evidence of the continuous mean and uncertainty estimation mechanisms in humans that may underlie optimal decision making.
| S-EPMC3382620 | biostudies-literature
Project description:Achieving accurate judgment ('judgmental achievement') is of utmost importance in daily life across multiple domains. The lens model and the lens model equation provide useful frameworks for modeling components of judgmental achievement and for creating tools to help decision makers (e.g., physicians, teachers) reach better judgments (e.g., a correct diagnosis, an accurate estimation of intelligence). Previous meta-analyses of judgment and decision-making studies have attempted to evaluate overall judgmental achievement and have provided the basis for evaluating the success of bootstrapping (i.e., replacing judges by linear models that guide decision making). However, previous meta-analyses have failed to appropriately correct for a number of study design artifacts (e.g., measurement error, dichotomization), which may have potentially biased estimations (e.g., of the variability between studies) and led to erroneous interpretations (e.g., with regards to moderator variables). In the current study we therefore conduct the first psychometric meta-analysis of judgmental achievement studies that corrects for a number of study design artifacts. We identified 31 lens model studies (N?=?1,151, k?=?49) that met our inclusion criteria. We evaluated overall judgmental achievement as well as whether judgmental achievement depended on decision domain (e.g., medicine, education) and/or the level of expertise (expert vs. novice). We also evaluated whether using corrected estimates affected conclusions with regards to the success of bootstrapping with psychometrically-corrected models. Further, we introduce a new psychometric trim-and-fill method to estimate the effect sizes of potentially missing studies correct psychometric meta-analyses for effects of publication bias. Comparison of the results of the psychometric meta-analysis with the results of a traditional meta-analysis (which only corrected for sampling error) indicated that artifact correction leads to a) an increase in values of the lens model components, b) reduced heterogeneity between studies, and c) increases the success of bootstrapping. We argue that psychometric meta-analysis is useful for accurately evaluating human judgment and show the success of bootstrapping.
| S-EPMC3877076 | biostudies-literature
Project description:BackgroundThe evidence used to inform health care decision making (HCDM) is typically uncertain. In these situations, the experience of experts is essential to help decision makers reach a decision. Structured expert elicitation (referred to as elicitation) is a quantitative process to capture experts' beliefs. There is heterogeneity in the existing elicitation methodology used in HCDM, and it is not clear if existing guidelines are appropriate for use in this context. In this article, we seek to establish reference case methods for elicitation to inform HCDM.MethodsWe collated the methods available for elicitation using reviews and critique. In addition, we conducted controlled experiments to test the accuracy of alternative methods. We determined the suitability of the methods choices for use in HCDM according to a predefined set of principles for elicitation in HCDM, which we have also generated. We determined reference case methods for elicitation in HCDM for health technology assessment (HTA).ResultsIn almost all methods choices available for elicitation, we found a lack of empirical evidence supporting recommendations. Despite this, it is possible to define reference case methods for HTA. The reference methods include a focus on gathering experts with substantive knowledge of the quantities being elicited as opposed to those trained in probability and statistics, eliciting quantities that the expert might observe directly, and individual elicitation of beliefs, rather than solely consensus methods. It is likely that there are additional considerations for decision makers in health care outside of HTA.ConclusionsThe reference case developed here allows the use of different methods, depending on the decision-making setting. Further applied examples of elicitation methods would be useful. Experimental evidence comparing methods should be generated.
| S-EPMC8777312 | biostudies-literature