Dataset Information

Comparison between 2000 and 2018 on the reporting of statistical significance and clinical relevance in physiotherapy clinical trials in six major physiotherapy journals: a meta-research design.

ABSTRACT:

Design

Meta-research.

Objective

To compare the prevalence of reporting p values, effect estimates and clinical relevance in physiotherapy randomised controlled trials (RCTs) published in the years 2000 and 2018.

Methods

We performed a meta-research study of physiotherapy RCTs obtained from six major physiotherapy peer-reviewed journals that were published in the years 2000 and 2018. We searched the databases Embase, Medline and PubMed in May 2019, and extracted data on the study characteristics and whether articles reported on statistical significance, effect estimates and confidence intervals for baseline, between-group, and within-group differences, and clinical relevance. Data were presented using descriptive statistics and inferences were made based on proportions. A 20% difference between 2000 and 2018 was regarded as a meaningful difference.

Results

We found 140 RCTs: 39 were published in 2000 and 101 in 2018. Overall, there was a high prevalence (>90%) of reporting p values for the main (between-group) analysis, with no difference between years. Statistical significance testing was frequently used for evaluating baseline differences, increasing from 28% in 2000 to 61.4% in 2018. The prevalence of reporting effect estimates, CIs and the mention of clinical relevance increased from 2000 to 2018 by 26.6%, 34% and 32.8% respectively. Despite an increase in use in 2018, over 40% of RCTs failed to report effect estimates, CIs and clinical relevance of results.

Conclusion

The prevalence of using p values remains high in physiotherapy research. Although the proportion of reporting effect estimates, CIs and clinical relevance is higher in 2018 compared to 2000, many publications still fail to report and interpret study findings in this way.

SUBMITTER: Verhagen A

PROVIDER: S-EPMC8724707 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:ImportanceSince the 1990s, reporting guidelines have developed that uniformly require authors to report a measure of precision (confidence intervals [CIs]) in addition to effect size.ObjectiveTo investigate the time trend of statistical inference and statistical reporting style in abstracts of major cancer journals.Design, setting, and participantsThis cross-sectional study reviewed all abstracts published between January 1, 1990, and December 31, 2020, in 10 high-ranking cancer journals (Lancet Oncology, Journal of Clinical Oncology, Cancer Discovery, Cancer Cell, JAMA Oncology, Annals of Oncology, Molecular Cancer, Journal of Thoracic Oncology, Journal of the National Cancer Institute, and Trends in Cancer) using a previously validated computerized algorithm to search the PubMed database. For the time trend analyses, 2 journals with only a few years of existence (JAMA Oncology and Trends in Cancer) were excluded.ExposuresCalendar year, journal, and type of abstract (randomized clinical trial or other).Main outcomes and measuresProportions of abstracts containing CIs, P values without CIs, and qualitative expressions of statistical significance only were compared over time among journals.ResultsOverall, 24 034 of 42 509 abstracts (56.5%) contained statistical inference. Reporting of CIs increased over time in 5 of 8 journals. From 2016 to 2020, the most prevailing statistical reporting style was the presentation of CIs (3070 of 4895 [62.7%]). However, the proportion of abstracts reporting statistical inference based solely on the terms significant or nonsignificant was still 1195 of 4895 (24.4%) during this period and was most prevalent among basic science-oriented cancer journals (eg, 63 of 66 [95.5%] in Cancer Cell). A higher prevalence of CI reporting was associated with reporting of results from randomized clinical trials and the requirement to report according to guidelines (eg, 522 of 574 [90.9%] in Lancet Oncology).Conclusions and relevanceThese findings suggest that the reporting style of statistical inference in abstracts of major cancer journals has improved over time. A requirement in journals' instructions for authors to present statistical inference in accordance with reporting guidelines and the implementation of these guidelines in submitted manuscripts on the part of journal editors may improve reporting.

Project description:BackgroundAppropriately defining and using the minimal important change (MIC) and the minimal clinically important difference (MCID) are crucial for determining whether the results are clinically significant. The aim of this study is to survey the status of randomized controlled trials (RCTs) for insomnia interventions to assess the inclusion and interpretation of MIC/MCID values.MethodsWe conducted a cross-sectional study to survey the status of RCTs for insomnia interventions to assess the inclusion and appropriate interpretation of MIC/MCID values. A literature search was conducted by searching the main sleep medicine journals indexed in PubMed, the Excerpta Medica Database (EMBASE), and the Cochrane Central Register of Controlled Trials (CENTRAL) to identify a broad range of search terms. We included RCTs with no restriction on the intervention. The included studies used the Insomnia Severity Index (ISI) or the Pittsburgh Sleep Quality Index (PSQI) questionnaire as the outcome measures.Results81 eligible studies were identified, and more than one-third of the included studies used MIC/MCID (n = 31, 38.3%). Among them, 21 studies with ISI as the outcome used MIC defined as a relative decrease ranging from 3 to 8 points. The most frequently used MIC value was a 6-point decrease (n = 7), followed by 8-point (n = 6) and 7-point decrease (n = 4), a 4 to 5-points decrease (n = 3), and a 30% reduction from baseline; 6 studies used MCID values, ranging from 2.8 to 4 points. The most frequently used MCID value was a 4-point decrease in the ISI (n = 4). 4 studies with PSQI as the outcome used a 3-point change as the MIC (n = 2) and a 2.5 to 2.7-point difference as MCID (n = 2). 4 non-inferiority design studies considered interval estimation when drawing clinically significant conclusions in their MCID usage.ConclusionsThe lack of consistent MIC/MCID interpretation and usage in outcome measures for insomnia highlights the urgent need for further efforts to address this issue and improve reporting practices.

Project description:Multivariable logistic regression (MLR) has been increasingly used in Chinese clinical medical research during the past few years. However, few evaluations of the quality of the reporting strategies in these studies are available.To evaluate the reporting quality and model accuracy of MLR used in published work, and related advice for authors, readers, reviewers, and editors.A total of 316 articles published in 5 leading Chinese clinical medical journals with high impact factor from January 2010 to July 2015 were selected for evaluation. Articles were evaluated according 12 established criteria for proper use and reporting of MLR models.Among the articles, the highest quality score was 9, the lowest 1, and the median 5 (4-5). A total of 85.1% of the articles scored below 6. No significant differences were found among these journals with respect to quality score (χ = 6.706, P = .15). More than 50% of the articles met the following 5 criteria: complete identification of the statistical software application that was used (97.2%), calculation of the odds ratio and its confidence interval (86.4%), description of sufficient events (>10) per variable, selection of variables, and fitting procedure (78.2%, 69.3%, and 58.5%, respectively). Less than 35% of the articles reported the coding of variables (18.7%). The remaining 5 criteria were not satisfied by a sufficient number of articles: goodness-of-fit (10.1%), interactions (3.8%), checking for outliers (3.2%), collinearity (1.9%), and participation of statisticians and epidemiologists (0.3%). The criterion of conformity with linear gradients was applicable to 186 articles; however, only 7 (3.8%) mentioned or tested it.The reporting quality and model accuracy of MLR in selected articles were not satisfactory. In fact, severe deficiencies were noted. Only 1 article scored 9. We recommend authors, readers, reviewers, and editors to consider MLR models more carefully and cooperate more closely with statisticians and epidemiologists. Journals should develop statistical reporting guidelines concerning MLR.

Project description:BackgroundWhen profiling hospital performance, quality inicators are commonly evaluated through hospital-specific adjusted means with confidence intervals. When identifying deviations from a norm, large hospitals can have statistically significant results even for clinically irrelevant deviations while important deviations in small hospitals can remain undiscovered. We have used data from the Swedish Stroke Register (Riksstroke) to illustrate the properties of a benchmarking method that integrates considerations of both clinical relevance and level of statistical significance.MethodsThe performance measure used was case-mix adjusted risk of death or dependency in activities of daily living within 3 months after stroke. A hospital was labeled as having outlying performance if its case-mix adjusted risk exceeded a benchmark value with a specified statistical confidence level. The benchmark was expressed relative to the population risk and should reflect the clinically relevant deviation that is to be detected. A simulation study based on Riksstroke patient data from 2008-2009 was performed to investigate the effect of the choice of the statistical confidence level and benchmark value on the diagnostic properties of the method.ResultsSimulations were based on 18,309 patients in 76 hospitals. The widely used setting, comparing 95% confidence intervals to the national average, resulted in low sensitivity (0.252) and high specificity (0.991). There were large variations in sensitivity and specificity for different requirements of statistical confidence. Lowering statistical confidence improved sensitivity with a relatively smaller loss of specificity. Variations due to different benchmark values were smaller, especially for sensitivity. This allows the choice of a clinically relevant benchmark to be driven by clinical factors without major concerns about sufficiently reliable evidence.ConclusionsThe study emphasizes the importance of combining clinical relevance and level of statistical confidence when profiling hospital performance. To guide the decision process a web-based tool that gives ROC-curves for different scenarios is provided.