Dataset Information

A comparative evaluation of PDQ-Evidence.

ABSTRACT: A strategy for minimising the time and obstacles to accessing systematic reviews of health system evidence is to collect them in a freely available database and make them easy to find through a simple 'Google-style' search interface. PDQ-Evidence was developed in this way. The objective of this study was to compare PDQ-Evidence to six other databases, namely Cochrane Library, EVIPNet VHL, Google Scholar, Health Systems Evidence, PubMed and Trip.We recruited healthcare policy-makers, managers and health researchers in low-, middle- and high-income countries. Participants selected one of six pre-determined questions. They searched for a systematic review that addressed the chosen question and one question of their own in PDQ-Evidence and in two of the other six databases which they would normally have searched. We randomly allocated participants to search PDQ-Evidence first or to search the two other databases first. The primary outcomes were whether a systematic review was found and the time taken to find it. Secondary outcomes were perceived ease of use and perceived time spent searching. We asked open-ended questions about PDQ-Evidence, including likes, dislikes, challenges and suggestions for improvements.A total of 89 people from 21 countries completed the study; 83 were included in the primary analyses and 6 were excluded because of data errors that could not be corrected. Most participants chose PubMed and Cochrane Library as the other two databases. Participants were more likely to find a systematic review using PDQ-Evidence than using Cochrane Library or PubMed for the pre-defined questions. For their own questions, this difference was not found. Overall, it took slightly less time to find a systematic review using PDQ-Evidence. Participants perceived that it took less time, and most participants perceived PDQ-Evidence to be slightly easier to use than the two other databases. However, there were conflicting views about the design of PDQ-Evidence.PDQ-Evidence is at least as efficient as other databases for finding health system evidence. However, using PDQ-Evidence is not intuitive for some people.The trial was prospectively registered in the ISRCTN registry 17 April 2015. Registration number: ISRCTN12742235 .

SUBMITTER: Johansen M

PROVIDER: S-EPMC5856385 | biostudies-literature | 2018 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A comparative evaluation of PDQ-Evidence.

Johansen Marit M Rada Gabriel G Rosenbaum Sarah S Paulsen Elizabeth E Motaze Nkengafac Villyen NV Opiyo Newton N Wiysonge Charles S CS Ding Yunpeng Y Mukinda Fidele K FK Oxman Andrew D AD

Health research policy and systems 20180315 1

<h4>Background</h4>A strategy for minimising the time and obstacles to accessing systematic reviews of health system evidence is to collect them in a freely available database and make them easy to find through a simple 'Google-style' search interface. PDQ-Evidence was developed in this way. The objective of this study was to compare PDQ-Evidence to six other databases, namely Cochrane Library, EVIPNet VHL, Google Scholar, Health Systems Evidence, PubMed and Trip.<h4>Methods</h4>We recruited hea ...[more]

PMID: 29544510

Similar Datasets

Project description:BACKGROUND:A utility value is a health-related quality of life metric (HRQoL) metric used in a cost-effectiveness analysis. While utilities as outcomes in the treatment of advanced Parkinson's disease (PD) with deep brain stimulation (DBS) are available, they do not currently exist for PD with early motor complications. The objectives of this study were to predict utilities from observed disease-specific HRQoL data using two mapping algorithms, and investigate their performance in terms of longitudinal changes within and between treatment groups, and distribution by PD severity. METHODS:This is a post hoc analysis of data from the EARLYSTIM trial of DBS compared with best medical therapy (BMT) in PD patients with early motor complications We used two published algorithms comprising ordinal and multinomial regression models to map EQ-5D-3L utilities from observed PD-specific 39 item Questionnaire (PDQ-39) scores in EARLYSTIM. Utilities were calculated using the predicted functioning levels of EQ-5D-3L dimensions and the established EQ-5D-3L UK tariffs. Statistical analyses (analysis of variance, two-tailed Student's t-test) were used to test the change from baseline within groups and difference in change from baseline between groups in utilities. Boxplots were developed to investigate the distribution of predicted utilities by PD severity, measured using the Hoehn and Yahr scale. RESULTS:The change from baseline in predicted mean utilities was statistically significant at all visits up to 24 months for the DBS group (p < 0.001) with both algorithms, and statistically significant at 12 months only (p = 0.04) for the BMT group with one algorithm. With both algorithms, the between-groups difference in change from baseline in predicted mean utilities favored DBS at all follow-up visits (p < 0.001). Based on the Hoehn and Yahr scale, predicted utilities deteriorated with increasing disease severity. CONCLUSIONS:Among PD patients with early motor complications, utilities predicted by both mapping algorithms using PDQ-39 data demonstrated a statistically and clinically meaningful improvement with DBS compared with BMT. It was not possible to conclude if one algorithm was more responsive than other. In the absence of utilities collected directly from patients, mapping is an acceptable option permitting economic evaluations to be undertaken.

Project description:BackgroundAs artificial intelligence (AI) tools become widely accessible, more patients and medical professionals will turn to them for medical information. Large language models (LLMs), a subset of AI, excel in natural language processing tasks and hold considerable promise for clinical use. Fields such as oncology, in which clinical decisions are highly dependent on a continuous influx of new clinical trial data and evolving guidelines, stand to gain immensely from such advancements. It is therefore of critical importance to benchmark these models and describe their performance characteristics to guide their safe application to clinical oncology. Accordingly, the primary objectives of this work were to conduct comprehensive evaluations of LLMs in the field of oncology and to identify and characterize strategies that medical professionals can use to bolster their confidence in a model's response.MethodsThis study tested five publicly available LLMs (LLaMA 1, PaLM 2, Claude-v1, generative pretrained transformer 3.5 [GPT-3.5], and GPT-4) on a comprehensive battery of 2044 oncology questions, including topics from medical oncology, surgical oncology, radiation oncology, medical statistics, medical physics, and cancer biology. Model prompts were presented independently of each other, and each prompt was repeated three times to assess output consistency. For each response, models were instructed to provide a self-appraised confidence score (from 1 to 4). Model performance was also evaluated against a novel validation set comprising 50 oncology questions curated to eliminate any risk of overlap with the data used to train the LLMs.ResultsThere was significant heterogeneity in performance between models (analysis of variance, P<0.001). Relative to a human benchmark (2013 and 2014 examination results), GPT-4 was the only model to perform above the 50th percentile. Overall, model performance varied as a function of subject area across all models, with worse performance observed in clinical oncology subcategories compared with foundational topics (medical statistics, medical physics, and cancer biology). Within the clinical oncology subdomain, worse performance was observed in female-predominant malignancies. A combination of model selection, prompt repetition, and confidence self-appraisal allowed for the identification of high-performing subgroups of questions with observed accuracies of 81.7 and 81.1% in the Claude-v1 and GPT-4 models, respectively. Evaluation of the novel validation question set produced similar trends in model performance while also highlighting improved performance in newer, centrally hosted models (GPT-4 Turbo and Gemini 1.0 Ultra) and local models (Mixtral 8×7B and LLaMA 2).ConclusionsOf the models tested on a standardized set of oncology questions, GPT-4 was observed to have the highest performance. Although this performance is impressive, all LLMs continue to have clinically significant error rates, including examples of overconfidence and consistent inaccuracies. Given the enthusiasm to integrate these new implementations of AI into clinical practice, continued standardized evaluations of the strengths and limitations of these products will be critical to guide both patients and medical professionals. (Funded by the National Institutes of Health Clinical Center for Research and the Intramural Research Program of the National Institutes of Health; Z99 CA999999.).

Project description:BackgroundMultiple data-analytic methods have been proposed for evaluating gene-expression levels in specific biological pathways, assessing differential expression associated with a binary phenotype. Following Goeman and Bühlmann's recent review, we compared statistical performance of three methods, namely Global Test, ANCOVA Global Test, and SAM-GS, that test "self-contained null hypotheses" Via. subject sampling. The three methods were compared based on a simulation experiment and analyses of three real-world microarray datasets.ResultsIn the simulation experiment, we found that the use of the asymptotic distribution in the two Global Tests leads to a statistical test with an incorrect size. Specifically, p-values calculated by the scaled chi2 distribution of Global Test and the asymptotic distribution of ANCOVA Global Test are too liberal, while the asymptotic distribution with a quadratic form of the Global Test results in p-values that are too conservative. The two Global Tests with permutation-based inference, however, gave a correct size. While the three methods showed similar power using permutation inference after a proper standardization of gene expression data, SAM-GS showed slightly higher power than the Global Tests. In the analysis of a real-world microarray dataset, the two Global Tests gave markedly different results, compared to SAM-GS, in identifying pathways whose gene expressions are associated with p53 mutation in cancer cell lines. A proper standardization of gene expression variances is necessary for the two Global Tests in order to produce biologically sensible results. After the standardization, the three methods gave very similar biologically-sensible results, with slightly higher statistical significance given by SAM-GS. The three methods gave similar patterns of results in the analysis of the other two microarray datasets.ConclusionAn appropriate standardization makes the performance of all three methods similar, given the use of permutation-based inference. SAM-GS tends to have slightly higher power in the lower alpha-level region (i.e. gene sets that are of the greatest interest). Global Test and ANCOVA Global Test have the important advantage of being able to analyze continuous and survival phenotypes and to adjust for covariates. A free Microsoft Excel Add-In to perform SAM-GS is available from http://www.ualberta.ca/~yyasui/homepage.html.

Dataset Information

A comparative evaluation of PDQ-Evidence.

Publications

A comparative evaluation of PDQ-Evidence.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets