Dataset Information

Mathematically aggregating experts' predictions of possible futures.

ABSTRACT: Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the "best" final prediction. When experts' performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts' estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.

SUBMITTER: Hanea AM

PROVIDER: S-EPMC8412308 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BackgroundPrognostic accuracy in palliative care is valued by patients, carers, and healthcare professionals. Previous reviews suggest clinicians are inaccurate at survival estimates, but have only reported the accuracy of estimates on patients with a cancer diagnosis.ObjectivesTo examine the accuracy of clinicians' estimates of survival and to determine if any clinical profession is better at doing so than another.Data sourcesMEDLINE, Embase, CINAHL, and the Cochrane Database of Systematic Reviews and Trials. All databases were searched from the start of the database up to June 2015. Reference lists of eligible articles were also checked.Eligibility criteriaInclusion criteriapatients over 18, palliative population and setting, quantifiable estimate based on real patients, full publication written in English.Exclusion criteriaif the estimate was following an intervention, such as surgery, or the patient was artificially ventilated or in intensive care.Study appraisal and synthesis methodsA quality assessment was completed with the QUIPS tool. Data on the reported accuracy of estimates and information about the clinicians were extracted. Studies were grouped by type of estimate: categorical (the clinician had a predetermined list of outcomes to choose from), continuous (open-ended estimate), or probabilistic (likelihood of surviving a particular time frame).Results4,642 records were identified; 42 studies fully met the review criteria. Wide variation was shown with categorical estimates (range 23% to 78%) and continuous estimates ranged between an underestimate of 86 days to an overestimate of 93 days. The four papers which used probabilistic estimates tended to show greater accuracy (c-statistics of 0.74-0.78). Information available about the clinicians providing the estimates was limited. Overall, there was no clear "expert" subgroup of clinicians identified.LimitationsHigh heterogeneity limited the analyses possible and prevented an overall accuracy being reported. Data were extracted using a standardised tool, by one reviewer, which could have introduced bias. Devising search terms for prognostic studies is challenging. Every attempt was made to devise search terms that were sufficiently sensitive to detect all prognostic studies; however, it remains possible that some studies were not identified.ConclusionStudies of prognostic accuracy in palliative care are heterogeneous, but the evidence suggests that clinicians' predictions are frequently inaccurate. No sub-group of clinicians was consistently shown to be more accurate than any other.Implications of key findingsFurther research is needed to understand how clinical predictions are formulated and how their accuracy can be improved.

Dataset Information

Mathematically aggregating experts' predictions of possible futures.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets