Dataset Information

Double sampling with multiple imputation to answer large sample meta-research questions: introduction and illustration by evaluating adherence to two simple CONSORT guidelines.

ABSTRACT:

Background

Meta-research can involve manual retrieval and evaluation of research, which is resource intensive. Creation of high throughput methods (e.g., search heuristics, crowdsourcing) has improved feasibility of large meta-research questions, but possibly at the cost of accuracy.

Objective

To evaluate the use of double sampling combined with multiple imputation (DS?+?MI) to address meta-research questions, using as an example adherence of PubMed entries to two simple consolidated standards of reporting trials guidelines for titles and abstracts.

Methods

For the DS large sample, we retrieved all PubMed entries satisfying the filters: RCT, human, abstract available, and English language (n?=?322, 107). For the DS subsample, we randomly sampled 500 entries from the large sample. The large sample was evaluated with a lower rigor, higher throughput (RLOTHI) method using search heuristics, while the subsample was evaluated using a higher rigor, lower throughput (RHITLO) human rating method. Multiple imputation of the missing-completely at-random RHITLO data for the large sample was informed by: RHITLO data from the subsample; RLOTHI data from the large sample; whether a study was an RCT; and country and year of publication.

Results

The RHITLO and RLOTHI methods in the subsample largely agreed (phi coefficients: title?=?1.00, abstract?=?0.92). Compliance with abstract and title criteria has increased over time, with non-US countries improving more rapidly. DS?+?MI logistic regression estimates were more precise than subsample estimates (e.g., 95% CI for change in title and abstract compliance by year: subsample RHITLO 1.050-1.174 vs. DS?+?MI 1.082-1.151). As evidence of improved accuracy, DS?+?MI coefficient estimates were closer to RHITLO than the large sample RLOTHI.

Conclusion

Our results support our hypothesis that DS?+?MI would result in improved precision and accuracy. This method is flexible and may provide a practical way to examine large corpora of literature.

SUBMITTER: Capers PL

PROVIDER: S-EPMC4428480 | biostudies-literature | 2015

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Double sampling with multiple imputation to answer large sample meta-research questions: introduction and illustration by evaluating adherence to two simple CONSORT guidelines.

Capers Patrice L PL Brown Andrew W AW Dawson John A JA Allison David B DB

Frontiers in nutrition 20150309

<h4>Background</h4>Meta-research can involve manual retrieval and evaluation of research, which is resource intensive. Creation of high throughput methods (e.g., search heuristics, crowdsourcing) has improved feasibility of large meta-research questions, but possibly at the cost of accuracy.<h4>Objective</h4>To evaluate the use of double sampling combined with multiple imputation (DS + MI) to address meta-research questions, using as an example adherence of PubMed entries to two simple consolida ...[more]

PMID: 25988135

Similar Datasets

Project description:To systematically assess adherence of randomised trials in surgery to Consolidated Standards of Reporting Trials (CONSORT) guidelines for non-pharmacological treatments (NPT). Surgical trials are considered more difficult to design and execute than pharmacological trials. Furthermore, the original CONSORT statement does not address some aspects that are vital to the transparent reporting of surgical trials. The CONSORT-NPT extension was designed to address these issues but adherence in medical and surgical journals has not been assessed.Cross-sectional study.We identified eight general medical and eight surgical journals, indexed in PubMed and published in 2011, with the highest impact factors in their respective categories.Adherence to CONSORT statement and CONSORT-NPT extension items.We identified 54 surgical trials (22 published in medical journals and 32 in surgical journals). There were eight items for which there was less than 30% overall compliance (seven were specific to the CONSORT-NPT extension). These seven items are related to: a full description of the care providers, centres and blinding status in the abstract (n=7/54, 13%), eligibility criteria for centres performing the interventions (n=13/54, 24%), how adherence of care providers with the protocol was assessed or enhanced (n=7/54, 13%), how clustering by care providers or centres was addressed as it relates to sample size (n=3/54, 6%), how care providers were allocated to each group (n=9/54, 17%), how clustering by care providers or centres was addressed as it relates to statistical methods (n=2/54, 4%), a description of care providers (case volume, qualification, expertise, etc) and centres (volume) in each group (n=0/54, 0%).Adherence of surgical trials to CONSORT-NPT extension items is much poorer than to the standard CONSORT statement. Adherence also appears to be superior in general medical journals compared with surgical journals. Raising awareness and conducting qualitative research to identify areas for specific intervention will be important going forward.

Project description:BackgroundThe huge amount of biomedical-molecular data increasingly produced is providing scientists with potentially valuable information. Yet, such data quantity makes difficult to find and extract those data that are most reliable and most related to the biomedical questions to be answered, which are increasingly complex and often involve many different biomedical-molecular aspects. Such questions can be addressed only by comprehensively searching and exploring different types of data, which frequently are ordered and provided by different data sources. Search Computing has been proposed for the management and integration of ranked results from heterogeneous search services. Here, we present its novel application to the explorative search of distributed biomedical-molecular data and the integration of the search results to answer complex biomedical questions.ResultsA set of available bioinformatics search services has been modelled and registered in the Search Computing framework, and a Bioinformatics Search Computing application (Bio-SeCo) using such services has been created and made publicly available at http://www.bioinformatics.deib.polimi.it/bio-seco/seco/. It offers an integrated environment which eases search, exploration and ranking-aware combination of heterogeneous data provided by the available registered services, and supplies global results that can support answering complex multi-topic biomedical questions.ConclusionsBy using Bio-SeCo, scientists can explore the very large and very heterogeneous biomedical-molecular data available. They can easily make different explorative search attempts, inspect obtained results, select the most appropriate, expand or refine them and move forward and backward in the construction of a global complex biomedical query on multiple distributed sources that could eventually find the most relevant results. Thus, it provides an extremely useful automated support for exploratory integrated bio search, which is fundamental for Life Science data driven knowledge discovery.

Project description:ObjectiveTo determine abstracts' adherence to the Consolidated Standards of Reporting Trials for Abstracts (CONSORT-A) statement and to explore the factors associated with reporting quality.DesignAn observational study.SettingAbstracts of randomised controlled trials published between 2010 and 2019, found searching the MEDLINE database.ParticipantsA total of 451 abstracts of the clinical trials on Helicobacter pylori infections were included.Primary and secondary outcome measuresAbstracts' reporting quality was determined by assessing their adherence to 17-item CONSORT-A checklist, with overall score being calculated as the sum of items that were adequately reported for each abstract. Additional factors that might influence the reporting quality of the abstracts were analysed, with univariate and multivariate linear regression used to determine how those factors influenced the overall reporting quality.ResultsIncluded abstracts had an overall median quality score of 8/17 (IQR 7-9). Large proportions of abstracts adequately reported interventions, participants, objectives, numbers randomised and conclusions (97.1, 99.3, 89.1. 94.7 and 98.4% of abstracts, respectively). Trial design, randomisation, blinding and funding were severely under-reported with only 8.0, 2.7, 11.0 and 2.0% of abstracts reporting each item. Overall quality scores for H. pylori abstracts were higher in association with CONSORT-A endorsement (B=5.698; 95% CI 1.781 to 9.615), pharmacological interventions (B=4.063; 95% CI 0.224 to 7.902), multicentre settings (B=5.057; 95% CI 2.370 to 7.743), higher numbers of participants (B=3.607; 95% CI 1.272 to 5.942), hospital settings (B=4.827; 95% CI 1.753 to 7.901) and longer abstracts (B=3.878; 95% CI 0.787 to 6.969 for abstracts with 251-300 words and B=7.404; 95% CI 3.930 to 10.878 for abstracts with more than 300 words).ConclusionsThe overall reporting quality of abstracts was inadequate. The endorsement of CONSORT-A guidelines by more journals might improve the standards of reporting.

Project description:Multiple choice questions (MCQs) offer high reliability and easy machine-marking, but allow for cueing and stimulate recognition-based learning. Very short answer questions (VSAQs), which are open-ended questions requiring a very short answer, may circumvent these limitations. Although VSAQ use in medical assessment increases, almost all research on reliability and validity of VSAQs in medical education has been performed by a single research group with extensive experience in the development of VSAQs. Therefore, we aimed to validate previous findings about VSAQ reliability, discrimination, and acceptability in undergraduate medical students and teachers with limited experience in VSAQs development. To validate the results presented in previous studies, we partially replicated a previous study and extended results on student experiences. Dutch undergraduate medical students (n = 375) were randomized to VSAQs first and MCQs second or vice versa in a formative exam in two courses, to determine reliability, discrimination, and cueing. Acceptability for teachers (i.e., VSAQ review time) was determined in the summative exam. Reliability (Cronbach's α) was 0.74 for VSAQs and 0.57 for MCQs in one course. In the other course, Cronbach's α was 0.87 for VSAQs and 0.83 for MCQs. Discrimination (average Rir) was 0.27 vs. 0.17 and 0.43 vs. 0.39 for VSAQs vs. MCQs, respectively. Reviewing time of one VSAQ for the entire student cohort was ±2 minutes on average. Positive cueing occurred more in MCQs than in VSAQs (20% vs. 4% and 20.8% vs. 8.3% of questions per person in both courses). This study validates the positive results regarding VSAQs reliability, discrimination, and acceptability in undergraduate medical students. Furthermore, we demonstrate that VSAQ use is reliable among teachers with limited experience in writing and marking VSAQs. The short learning curve for teachers, favourable marking time and applicability regardless of the topic suggest that VSAQs might also be valuable beyond medical assessment.

Dataset Information

Double sampling with multiple imputation to answer large sample meta-research questions: introduction and illustration by evaluating adherence to two simple CONSORT guidelines.

Background

Objective

Methods

Results

Conclusion

Publications

Double sampling with multiple imputation to answer large sample meta-research questions: introduction and illustration by evaluating adherence to two simple CONSORT guidelines.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets