Dataset Information

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations.

ABSTRACT: The purpose of the current study was to apply and evaluate a procedure to elicit expert judgments about correlations, and to update this information with empirical data. The result is a face-to-face group elicitation procedure with as its central element a trial roulette question that elicits experts' judgments expressed as distributions. During the elicitation procedure, a concordance probability question was used to provide feedback to the experts on their judgments. We evaluated the elicitation procedure in terms of validity and reliability by means of an application with a small sample of experts. Validity means that the elicited distributions accurately represent the experts' judgments. Reliability concerns the consistency of the elicited judgments over time. Four behavioral scientists provided their judgments with respect to the correlation between cognitive potential and academic performance for two separate populations enrolled at a specific school in the Netherlands that provides special education to youth with severe behavioral problems: youth with autism spectrum disorder (ASD), and youth with diagnoses other than ASD. Measures of face-validity, feasibility, convergent validity, coherence, and intra-rater reliability showed promising results. Furthermore, the current study illustrates the use of the elicitation procedure and elicited distributions in a social science application. The elicited distributions were used as a prior for the correlation, and updated with data for both populations collected at the school of interest. The current study shows that the newly developed elicitation procedure combining the trial roulette method with the elicitation of correlations is a promising tool, and that the results of the procedure are useful as prior information in a Bayesian analysis.

SUBMITTER: Zondervan-Zwijnenburg M

PROVIDER: S-EPMC5282462 | biostudies-literature | 2017

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations.

Zondervan-Zwijnenburg Mariëlle M van de Schoot-Hubeek Wenneke W Lek Kimberley K Hoijtink Herbert H van de Schoot Rens R

Frontiers in psychology 20170131

The purpose of the current study was to apply and evaluate a procedure to elicit expert judgments about correlations, and to update this information with empirical data. The result is a face-to-face group elicitation procedure with as its central element a trial roulette question that elicits experts' judgments expressed as distributions. During the elicitation procedure, a concordance probability question was used to provide feedback to the experts on their judgments. We evaluated the elicitati ...[more]

PMID: 28197115

Similar Datasets

Project description:For many societally important science-based decisions, data are inadequate, unreliable or non-existent, and expert advice is sought. In such cases, procedures for eliciting structured expert judgments (SEJ) are increasingly used. This raises questions regarding validity and reproducibility. This paper presents new findings from a large-scale international SEJ study intended to estimate the global burden of foodborne disease on behalf of WHO. The study involved 72 experts distributed over 134 expert panels, with panels comprising thirteen experts on average. Elicitations were conducted in five languages. Performance-based weighted solutions for target questions of interest were formed for each panel. These weights were based on individual expert's statistical accuracy and informativeness, determined using between ten and fifteen calibration variables from the experts' field with known values. Equal weights combinations were also calculated. The main conclusions on expert performance are: (1) SEJ does provide a science-based method for attribution of the global burden of foodborne diseases; (2) equal weighting of experts per panel increased statistical accuracy to acceptable levels, but at the cost of informativeness; (3) performance-based weighting increased informativeness, while retaining accuracy; (4) due to study constraints individual experts' accuracies were generally lower than in other SEJ studies, and (5) there was a negative correlation between experts' informativeness and statistical accuracy which attenuated as accuracy improved, revealing that the least accurate experts drive the negative correlation. It is shown, however, that performance-based weighting has the ability to yield statistically accurate and informative combinations of experts' judgments, thereby offsetting this contrary influence. The present findings suggest that application of SEJ on a large scale is feasible, and motivate the development of enhanced training and tools for remote elicitation of multiple, internationally-dispersed panels.

Project description:Background. Blood cannot be artificially manufactured, and there is currently no substitute for human blood. The supply of blood in transfusion facilities requires constant and timely collection of blood from donors. Modeling and forecasting trends in blood collections are critical for determining both the current and future capacity requirements and appropriate models of adequate blood provision. Objectives. The objective of this study is to determine blood collection or donation patterns and develop time-series models that can be updated and refined in predicting future blood donations in Zimbabwe when given the historical data. Materials and Methods. Monthly blood donation data for the period 2009 to 2019 were collected retrospectively from the National Blood Service Zimbabwe database. Time-series models (i.e., the Seasonal Autoregressive Integrated Moving Average [SARIMA] and Error, Trend and Seasonal [ETS]) models were applied and compared. The models were chosen because of their ability to handle the seasonality and other time-series components evident in the blood donation data. Expert opinions and experience were used in selecting the models and in making inferences in the analysis. Results. Time-series plots of blood donations showed seasonal patterns, with significant drops in blood donations in months associated with Zimbabwe's school holidays (April, August, and December) and public holidays. During these holidays, there is a reduced number of school donors, while at about the same time, there is increasing blood demand as a result of road accidents. Model identification procedures established the SARIMA(1,1,2)(0,1,1)12 model as the appropriate model for forecasting total blood donation in Zimbabwe. The results and forecasts show an upward trend in blood donations. According to the accuracy measures used, the SARIMA model outperforms the ETS model. Conclusions. Expert knowledge in the blood donation process, coupled with statistical models, can help explain trends exhibited in blood donation data in Zimbabwe. These findings help the blood authorities plan for blood donor campaign drives. The findings are key indicators of where to allocate more resources toward blood donation and when to collect more blood units. The increasing blood donation projections ensure a stable blood bank inventory in the near future.HighlightsA SARIMA model can be used to predict the flow of blood donations in Zimbabwe.The seasonal blood donation pattern peaks in the months of March, June/July, and September.The donations troughs are in the months of April, August, December, and January. These are the months coinciding with school holidays in Zimbabwe.Both the SARIMA and ETS models provided similar forecasts, but measures of fit and expert knowledge gave a slight preference to the SARIMA(1,1,2)(0,1,1)12 model in predicting the flow of blood donations in Zimbabwe.These model results are useful for guiding allocation of blood donation resources and blood donation drive timing.

Dataset Information

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations.

Publications

Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets