Dataset Information

Quality science from quality measurement: The role of measurement type with respect to replication and effect size magnitude in psychological research.

ABSTRACT: The quality of psychological studies is currently a major concern. The Many Labs Project (MLP) and the Open-Science-Collaboration (OSC) have collected key data on replicability and statistical effect sizes. We build on this work by investigating the role played by three measurement types: ratings, proportions and unbounded (measures without conceptual upper limits, e.g. time). Both replicability and effect sizes are dependent on the amount of variability due to extraneous factors. We predicted that the role of such extraneous factors might depend on measurement type, and would be greatest for ratings, intermediate for proportions and least for unbounded. Our results support this conjecture. OSC replication rates for unbounded, 43% and proportion 40% combined are reliably higher than those for ratings at 20% (effect size, w = .20). MLP replication rates for the original studies are: proportion = .74, ratings = .40 (effect size w = .33). Original effect sizes (Cohen's d) are highest for: unbounded OSC cognitive = 1.45, OSC social = .90); next for proportions (OSC cognitive = 1.01, OSC social = .84, MLP = .82); and lowest for ratings (OSC social = .64, MLP = .31). These findings are of key importance to scientific methodology and design, even if the reasons for their occurrence are still at the level of conjecture.

SUBMITTER: Kornbrot DE

PROVIDER: S-EPMC5809054 | biostudies-literature | 2018

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Quality science from quality measurement: The role of measurement type with respect to replication and effect size magnitude in psychological research.

Kornbrot Diana E DE Wiseman Richard R Georgiou George J GJ

PloS one 20180212 2

The quality of psychological studies is currently a major concern. The Many Labs Project (MLP) and the Open-Science-Collaboration (OSC) have collected key data on replicability and statistical effect sizes. We build on this work by investigating the role played by three measurement types: ratings, proportions and unbounded (measures without conceptual upper limits, e.g. time). Both replicability and effect sizes are dependent on the amount of variability due to extraneous factors. We predicted t ...[more]

PMID: 29432479

Similar Datasets

Project description:Rationale, aims, and objectivesIt is generally believed that evidence from low quality of evidence generate inaccurate estimates about treatment effects more often than evidence from high (certainty) quality evidence (CoE). As a result, we would expect that (a) estimates of effects of health interventions initially based on high CoE change less frequently than the effects estimated by lower CoE (b) the estimates of magnitude of effect size differ between high and low CoE. Empirical assessment of these foundational principles of evidence-based medicine has been lacking.MethodsWe reviewed the Cochrane Database of Systematic Reviews from January 2016 through May 2021 for pairs of original and updated reviews for change in CoE assessments based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method. We assessed the difference in effect sizes between the original versus updated reviews as a function of change in CoE, which we report as a ratio of odds ratio (ROR). We compared ROR generated in the studies in which CoE changed from very low/low (VL/L) to moderate/high (M/H) versus M/H to VL/L. Heterogeneity and inconsistency were assessed using the tau and I2 statistic. We also assessed the change in precision of effect estimates (by calculating the ratio of standard errors) (seR), and the absolute deviation in estimates of treatment effects (aROR).ResultsFour hundred and nineteen pairs of reviews were included of which 414 (207 × 2) informed the CoE appraisal and 384 (192 × 2) the assessment of effect size. We found that CoE originally appraised as VL/L had 2.1 [95% confidence interval (CI): 1.19-4.12; p = 0.0091] times higher odds to be changed in the future studies than M/H CoE. However, the effect size was not different (p = 1) when CoE changed from VL/L → M/H [ROR = 1.02 (95% CI: 0.74-1.39)] compared with M/H → VL/L (ROR = 1.02 [95% CI: 0.44-2.37]). Similar overlap in aROR between the VL/L → M/H versus M/H → VL/L subgroups was observed [median (IQR): 1.12 (1.07-1.57) vs. 1.21 (1.12-2.43)]. We observed large inconsistency across ROR estimates (I2 = 99%). There was larger imprecision in treatment effects when CoE changed from VL/L → M/H (seR = 1.46) than when it changed from M/H → VL/L (seR = 0.72).ConclusionsWe found that low-quality evidence changes more often than high CoE. However, the effect size did not systematically differ between the studies with low versus high CoE. The finding that the effect size did not differ between low and high CoE indicate urgent need to refine current EBM critical appraisal methods.

Project description:BackgroundEvaluation of changes in tumor size from images acquired by ultrasound (US), computed tomography (CT) or magnetic resonance imaging (MRI) is a common measure of cancer chemotherapy efficacy. Tumor size measurement based on either the World Health Organization (WHO) criteria or the Response Evaluation Criteria in Solid Tumors (RECIST) is the only imaging biomarker for anti-cancer drug testing presently approved by the United States Food and Drug Administration (FDA). The aim of this paper was to design and test a quality assurance phantom with the capability of monitoring tumor size changes with multiple preclinical imaging scanners (US, CT and MRI) in order to facilitate preclinical anti-cancer drug testing.MethodsThree phantoms (Gammex/UTHSCSA Mark 1, Gammex/UTHSCSA Mark 2 and UTHSCSA multimodality tumor measurement phantom) containing tumor-simulating test objects were designed and constructed. All three phantoms were scanned in US, CT and MRI devices. The size of test objects in the phantoms was measured from the US, CT and MRI images. RECIST, WHO and volume analyses were performed.ResultsThe smaller phantom size, simplified design and better test object CT contrast of the UTHSCSA multimodality tumor measurement phantom allowed scanning of the phantom in preclinical US, CT and MRI scanners compared with only limited preclinical scanning capability of Mark 1 and Mark 2 phantoms. For all imaging modalities, RECIST and WHO errors were reduced for UTHSCSA multimodality tumor measurement phantom (≤1.69 ± 0.33%) compared with both Mark 1 (≤ -7.56 ± 6.52%) and Mark 2 (≤ 5.66 ± 1.41%) phantoms. For the UTHSCSA multimodality tumor measurement phantom, measured tumor volumes were highly correlated with NIST traceable design volumes for US (R2 = 1.000, p < 0.0001), CT (R2 = 0.9999, p < 0.0001) and MRI (R2 = 0.9998, p < 0.0001).ConclusionsThe UTHSCSA multimodality tumor measurement phantom described in this study can potentially be a useful quality assurance tool for verifying radiologic assessment of tumor size change during preclinical anti-cancer therapy testing with multiple imaging modalities.

Dataset Information

Quality science from quality measurement: The role of measurement type with respect to replication and effect size magnitude in psychological research.

Publications

Quality science from quality measurement: The role of measurement type with respect to replication and effect size magnitude in psychological research.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets